Imported from aaron-peloquin/ax-analytics (.agents/skills/ax-analytics-telemetry/SKILL.md). Install upstream with npx skills add aaron-peloquin/ax-analytics --skill ax-analytics-telemetry. Copyright stays with the author.

AX Analytics Telemetry Integration Skill

System Overview

AX Analytics is a high-throughput, language-agnostic AI agent and web telemetry platform. It provides real-time visualization of multi-turn AI agent trajectory flows (Sankey diagrams), side-by-side tool performance hubs (Sunburst charts), sticky A/B experiment resolution, and OpenTelemetry / Opik trace correlation.

Key Operational Capabilities

• 3-Tier Identity Trajectory Flows: Connects user sessions (sessionId), orchestrators (multiagentIdentity), and worker agents (agentIdentity / entityId) into Directed Acyclic Graph (DAG) Sankey trajectory flows (previousToolName -> invokedToolName).
• Outcome Metrics: Measures Seconds to Resolution (executionTimeMs / 1000) and Cost Per Outcome ($ expenditure per successful run).
• OTEL GenAI Attribute Tracking: Records LLM vendor (provider), model (model), prompt tokens (inputTokens), and completion tokens (outputTokens).
• Sticky A/B Experimentation: Deterministic 0-100% variant hashing based on hash(entity_id + experiment_id) % 100.
• Parameter Share Analysis: Evaluates parameter values with an automatic ≥ 8% share threshold grouping rule.
• OpenTelemetry Correlation: Direct trace/span mapping via W3C traceparent headers (otelTraceId, otelSpanId).

3-Tier Identity Taxonomy & Metadata Groupings

To ensure accurate aggregation across analytics dashboards, trajectory flows, and cost models, telemetry callers MUST categorize metadata across three distinct hierarchical scopes:

[!IMPORTANT] Implementation Guidance for multiagentIdentity: multiagentIdentity is NOT restricted to root spans or primary orchestrators. It is an overarching context string stamped on every single event emitted by every participating agent throughout the entire execution chain. It binds all constituent agent steps (entityId) into a unified multi-agent system view for whole-chain cost tracking, full-pipeline latency aggregation, and cross-agent trajectory flow mapping.

Prescriptive Agent Entity Naming Standard (entityId)

To ensure accurate aggregation across Sankey flow diagrams and Sunburst hubs, callers MUST adhere to prescriptive entity naming conventions for entityId:

1. Agent Personas: Use clean, descriptive hyphenated or colon-scoped names representing the specific AI persona or worker role (e.g. retrieval-agent, sql-generator, sales-assistant).
2. System & Route Endpoints: When telemetry originates from internal service routes or MCP servers, set entityId to the endpoint path (e.g. /api/v1/chat, /api/mcp).
3. Stage-Aware Execution Contexts: Append execution stages for fine-grained trajectory tracking when an agent transitions through phases (e.g. "rag-agent: warming" during index load vs "rag-agent" during live synthesis).
4. Deterministic Fallback Cascade: When entityId is omitted, telemetry integrations MUST evaluate fallbacks in order of specificity: entityId -> agentIdentity -> service route path -> "agent-service".

Telemetry Semantics & OpenTelemetry Boundary Guidelines

To ensure clean Sankey trajectory flows and accurate Sunburst performance hubs, callers MUST adhere to the following telemetry boundaries:

1. Agent Tool Calls vs. Internal Code Execution (Temporal Activities, DB, STT/TTS)

• Strictly Agentic: eventType: "tool_call" (and invokedToolName) is strictly reserved for Agentic Tool Invocations—specifically when an AI agent reasons and autonomously decides to call an external tool (e.g., an MCP server tool like get_campaign_plan or search_lore).
• No Synthetic Tool Calls for Background Code: Standard code-level tasks or background workflow activities (e.g., Temporal activities, PostgreSQL queries, speech-to-text transcriptions, text-to-speech audio synthesis, or vector indexing) MUST NOT emit synthetic tool_call or custom temporal_activity events to AX Analytics. Doing so pollutes the Sankey trajectory flow diagrams and Sunburst hubs.

2. Entity Type (entityType) Semantics

• entityType: "agent": MUST only be attached when an AI persona or agent entity explicitly takes an autonomous action (e.g. dm-persona, bookkeeper-agent).
• entityType: "human": Used for real human user interactions (e.g. UI clicks, form submissions, frontend page views).
• Deterministic System Workers: If deterministic backend code or a background system worker is running a routine function, entityType MUST be left undefined / null (omitted) rather than defaulting to "agent".

3. Telemetry Destination Decision Matrix

Automated Session Trajectory Flow Tracking

To eliminate manual caller overhead when recording sequential tool invocations, telemetry integrations SHOULD maintain an in-memory session registry (sessionLastToolMap):

• Automatic Preceding Tool Lookup: When an event with eventType: "tool_call" is logged without an explicit previousToolName, retrieve the preceding tool from sessionLastToolMap.get(sessionId).
• Bootstrap Initialization: If no preceding tool exists for sessionId, default previousToolName to "init_session".
• Registry Update: Immediately update the session registry with sessionLastToolMap.set(sessionId, invokedToolName).

Dynamic Endpoint & Environment Configuration

Telemetry integrations SHOULD dynamically resolve host and app key configurations based on runtime environment context:

• Browser Environments: Proxy ingestion requests through /api/ax-analytics (via Next.js rewrite or equivalent web server proxy) to avoid CORS issues and expose a unified endpoint.
• Server Environments: Read environment variables AX_ANALYTICS_HOST or NEXT_PUBLIC_AX_ANALYTICS_HOST, falling back to http://localhost:4400.
• App Key Resolution: Read AX_ANALYTICS_APP_KEY or NEXT_PUBLIC_AX_ANALYTICS_APP_KEY, specifying the target application/tenant scope (e.g. customer_support_prod, doc_assistant_dev).

Ingestion Payload Field Contract (_v1)

Sequential Integration Workflow

1. Initialize Session Context: Extract W3C traceparent (00-<traceId>-<spanId>-01) or provide a unique sessionId (or let server auto-generate ax_sess_*).
2. Resolve Experiment Variant (Optional): Query POST /v1/experiments/variant with experimentKey and prescriptive entityId.
3. Execute Tool / LLM Inference: Record turn start time and execute the requested agent tool or inference.
4. Emit Telemetry Event: Call POST /v1/telemetry/event (or helper function trackAgentToolCall) with execution time, token cost, input/output tokens, provider, model, parameters, results, and trace IDs.
5. Submit User Feedback: Send thumbs up/down votes and comments via POST /v1/feedback at session completion.

API Endpoints & cURL Specifications

[!TIP] Before making new network requests, search for an reuse any existing helpers that previous engineers may have already setup for your project.

1. Ingest Telemetry Event / Tool Call (POST /v1/telemetry/event)

curl -X POST http://localhost:4400/v1/telemetry/event \
  -H "Content-Type: application/json" \
  -d '{
    "appKey": "customer_support_prod",
    "sessionId": "sess_usr_98124_chat",
    "multiagentIdentity": "customer-triage-system",
    "entityId": "retrieval-agent",
    "entityType": "agent",
    "eventType": "tool_call",
    "invokedToolName": "search_knowledge_base",
    "previousToolName": "init_session",
    "provider": "openai",
    "model": "gpt-4o",
    "inputTokens": 850,
    "outputTokens": 120,
    "params": { "query": "order refund status", "topK": 3 },
    "results": { "status": "active", "docsFound": 3 },
    "statusCode": "SUCCESS",
    "executionTimeMs": 240,
    "tokenCost": 0.0018,
    "otelTraceId": "4bf92f3577b34da6a3ce929d0e0e4736",
    "otelSpanId": "00f067aa0ba902b7"
  }'

2. Ingest Non-Tool LLM Inference (POST /v1/telemetry/event)

curl -X POST http://localhost:4400/v1/telemetry/event \
  -H "Content-Type: application/json" \
  -d '{
    "appKey": "customer_support_prod",
    "sessionId": "sess_usr_98124_chat",
    "multiagentIdentity": "customer-triage-system",
    "entityId": "summarization-worker",
    "entityType": "agent",
    "eventType": "llm_inference",
    "provider": "anthropic",
    "model": "claude-3-5-sonnet",
    "inputTokens": 1200,
    "outputTokens": 350,
    "params": {
      "prompt": "Summarize customer refund conversation.",
      "temperature": 0.3
    },
    "results": {
      "response": "Customer requested a refund due to delayed shipment. Resolution issued under policy #402."
    },
    "statusCode": "SUCCESS",
    "executionTimeMs": 450,
    "tokenCost": 0.0032,
    "otelTraceId": "4bf92f3577b34da6a3ce929d0e0e4736",
    "otelSpanId": "00f067aa0ba902b7"
  }'

3. Ingest Human Web Pageview OTLP Trace (POST /v1/telemetry/otlp/v1/traces)

[!NOTE] user.type: "human" denotes real human visitors navigating the web frontend. Human users generate documentLoad pageview events (URL path, title, referrer, visibility state, device desktop/mobile). This is distinct from AI agents (user.type: "agent"), which execute backend tools/prompts.

curl -X POST http://localhost:4400/v1/telemetry/otlp/v1/traces \
  -H "Content-Type: application/json" \
  -d '{
    "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
    "spanId": "00f067aa0ba902b7",
    "parentSpanId": "5e1074e531853683",
    "name": "documentLoad",
    "startTime": [1700000000, 100000000],
    "endTime": [1700000000, 850000000],
    "attributes": {
      "url.full": "https://example.com/products/headphones",
      "url.scheme": "https",
      "url.path": "/products/headphones",
      "user_agent.original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/122.0.0.0 Safari/537.36",
      "document.title": "Wireless Headphones - Shop",
      "document.referrer": "https://google.com",
      "document.visibilityState": "visible",
      "app.key": "app_live_8832109",
      "app.event_type": "page_view",
      "session.id": "web_sess_99201",
      "user.id": "user_4821",
      "user.type": "human"
    },
    "resource": {
      "attributes": {
        "service.name": "web-frontend",
        "browser.platform": "macOS",
        "browser.mobile": false,
        "browser.brands": ["Google Chrome 122", "Chromium 122"]
      }
    }
  }'

3. Resolve Sticky A/B Experiment (POST /v1/experiments/variant)

curl -X POST http://localhost:4400/v1/experiments/variant \
  -H "Content-Type: application/json" \
  -d '{
    "appKey": "customer_support_prod",
    "experimentKey": "proactive_rag_retrieval_v2",
    "entityId": "retrieval-agent"
  }'

4. Reset Sticky User/Agent Assignments (POST /v1/experiments/reset-assignments)

curl -X POST http://localhost:4400/v1/experiments/reset-assignments \
  -H "Content-Type: application/json" \
  -d '{
    "experimentKey": "proactive_rag_retrieval_v2"
  }'

5. Submit Session Feedback (POST /v1/feedback)

curl -X POST http://localhost:4400/v1/feedback \
  -H "Content-Type: application/json" \
  -d '{
    "appKey": "customer_support_prod",
    "sessionId": "sess_usr_98124_chat",
    "entityId": "retrieval-agent",
    "vote": 1,
    "comment": "Resolved user query accurately with RAG context."
  }'

Structural Scaffolding & Formatting