<!-- OpenSmartRoute: Tracing and observability. Source https://github.com/isathish/OpenSmartRoute/blob/main/docs/OBSERVABILITY.md; HTML https://opensmartroute.ai/docs/OBSERVABILITY -->
# Tracing and observability

Every stage of a request - signal extraction, policy filtering, ranking, planning, execution, learning,
health, cache, guard, shadow rollout, fair share and the autopilot - emits **spans** and **events**
into a `Tracer`. Sinks decide what happens to them: keep the last N in memory for `/trace`, aggregate
them into Prometheus counters, log them as JSON lines, append them to a file or forward them to
OpenTelemetry. Instrumentation is always on and free when no sink is attached (a shared no-op span,
no allocation), so the SDK stays zero-dependency and the cost of capturing everything is opt-in.

```python
from opensmartroute import Router, configure_tracing
from opensmartroute.observability import MemorySink, MetricsSink

tracer = configure_tracing(MemorySink(), MetricsSink())      # process-wide; or configure_tracing() -> settings
router = Router(reg)                                          # picks up the global tracer
d = router.route("Prove that sqrt(2) is irrational")
for e in tracer.find(MemorySink).trace(d.request_id):        # spans + events of that request, in time order
    print(e["kind"], e["name"], e.get("duration_ms"), e["attributes"])
print(tracer.find(MetricsSink).prometheus())                 # osr_* exposition text
```

```
span  route            2.31ms {'text_sha256': '9f2c…', 'text_len': 35, 'target': 'large', 'confidence': 0.82, 'elapsed_ms': 2.1, ...}
event route.signals          {'duration_ms': 0.4, 'complexity': 0.71, 'domains': ['math'], 'reasoning_need': 0.9, 'contains_pii': False, ...}
event route.policy           {'pool': 2, 'admissible': 2, 'rejections': {}}
event route.rank             {'role': None, 'candidates': 2, 'strategies': {'capability': 0.05, ...}, 'top': [{'id': 'large', 'utility': 0.82, ...}]}
```

`osr route "text" --events` prints the same trace after the decision; `osr serve` exposes it over HTTP and
the hosted platform shows it per workspace in the dashboard (activity drawer, `/platform/dashboard/events`,
playground *Trace* tab - see the platform endpoints below).

## What is captured

| Span | Opened by | Attributes (end of span) |
|---|---|---|
| `request` | `EnterpriseRouter.route` | `middleware` (class names), `tenant`, `target`, `confidence`, `audited` |
| `http.request` | `osr serve` middleware | `method`, `path`, `app`, `status_code`, `target`, inbound `traceparent` |
| `route` | `Router.route` | `text_sha256`, `text_len`, `tenant`, `objective`, `kinds`, `candidates`, `exclude`, `plan`, `task_id`, `session_id`, `target`, `kind`, `confidence`, `elapsed_ms`, `alternatives`, `abstain`, `plan_slots` |
| `plan` | `Router._build_plan` (inside `route`) | `primary`, `kind`, `slots` |
| `execute` | `execution.execute` / `aexecute` | `target`, `kind`, `plan_slots`, `task_id`, `runner`, `ok`, `latency_ms`, `cost_usd`, `tokens`, `outcomes`, `system_prompt_len`, `pii_redacted`, `pii_restored` |
| `autopilot.cycle` | `Autopilot.run_once` | `reason`, `online`, `accepted`, `outcome`, `cycle` |

| Event | Emitted when |
|---|---|
| `route.signals` | signals extracted: `duration_ms`, complexity, domains, actions, language, token estimate, PII, jailbreak risk, tools, task type, reasoning need |
| `route.policy` | hard constraints applied: `pool`, `admissible`, `rejections` (target -> reason) |
| `route.no_route` | no admissible target (before `NoRouteError` is raised) |
| `route.pinned` | a rule pinned a target (`kept` tells whether it survived the policy) |
| `route.narrowed` | the retriever shortlisted a large catalogue |
| `route.shortlist` | a strategy shortlisted candidates |
| `route.rank` | strategies scored (`strategies` with per-strategy ms, `top` 3, `excluded_by_floor`) |
| `route.escalate` | the LLM judge was consulted because confidence was low |
| `route.explore` | the bandit chose to explore (`served` tells whether the exploration was taken) |
| `route.abstain` | the calibrated confidence was below the abstention threshold |
| `route.fallback` | a fallback target replaced an unavailable one |
| `plan.slot` | a plan slot was filled (or not: `filled=False`, `reason`) |
| `execute.step` | one execution step (persona, skill, primary) finished |
| `learn.outcome` | `Router.learn` / `learn_correction` recorded an outcome (`corrected`, `learner`) |
| `learn.task_credit` | task-level credit assigned to the plan's targets |
| `learn.calibrate` | the confidence calibrator was refit (temperature) |
| `learn.improve` | a `SelfImprover` cycle finished (`rows`, `champion_accuracy`, `challenger_accuracy`, `accepted`, `reason`; inside an `autopilot.cycle` span when the autopilot ran it) |
| `learn.promote` | a challenger SLM replaced the serving champion (`rows`, `trained_at`, `targets`, `saved`) |
| `health.breaker` | a circuit breaker changed state (`previous`, `state`; level `warning` when it opens) |
| `cache.hit` / `cache.miss` | `CacheMiddleware` lookup (`target`, `age_s` on a hit) |
| `tenant.rejected` | `TenantMiddleware` refused a request (`reason`, `tenant`) |
| `route.slow` | `TimeoutMiddleware` saw a decision exceed its budget |
| `guard.blocked` / `guard.flagged` / `guard.redacted` | `GuardMiddleware` verdicts (`reasons`, `gadget`, `entities`) |
| `shadow.compare` / `shadow.verdict` | `ShadowMiddleware` compared production and candidate; the A/B test reached a verdict |
| `fairshare.throttled` / `fairshare.capped` | `FairShareMiddleware` delayed or refused a tenant |
| `autopilot.drift` | the autopilot detected quality drift and scheduled a cycle |

`opensmartroute.observability.SPAN_NAMES` and `EVENT_NAMES` list the vocabulary; a test asserts every
emitted name belongs to it.

**Privacy.** Request text never enters a span or event: the `route` span carries `text_sha256` (16 hex
characters) and `text_len` (`text_digest()`); attribute strings are truncated to
`OSR_OBSERVABILITY_ATTRIBUTE_MAX_LEN` (500) and nested values are JSON-cleaned. Sinks receive ids,
numbers, target ids, reasons and scores.

## Spans, events, trace ids

An `Event` is a flat record: `name`, `kind` (`event` or `span`), `ts`, `trace_id`, `span_id`,
`parent_id`, `request_id`, `level`, `status` and `duration_ms` (spans only) and `attributes`. A span's
closing record is the same shape as an event so every sink handles one type.

The trace id of a request is derived once, at the root span: an inbound W3C `traceparent` wins,
otherwise a UUID request id is reused, otherwise the request id is hashed (so a `request_id` supplied
by the caller yields a stable trace id). Nested spans inherit trace id, request id and the sampling
decision through a `contextvars` context, so threads and `asyncio` tasks spawned by the router carry
the right parent. `Tracer.current()` returns the innermost open span; `current_tracer()` resolves the
tracer to report to (the open span's, else one bound with `use_tracer`, else the global one), which is
how middleware, the health registry and the executor emit without holding a reference.

## Sinks

| Sink | Purpose |
|---|---|
| `MemorySink(max_events=1000)` | ring buffer; `events(request_id=, trace_id=, name=, kind=, level=, limit=)` (`name` may end with `*`), `trace(request_id)`; backs `/events`, `/trace/{id}` and `osr route --events` |
| `MetricsSink()` | counters and p50/p95/p99 per span name; `snapshot()` for `/stats`, `prometheus()` for `/metrics` (`osr_events_total`, `osr_spans_total`, `osr_span_duration_ms`, `osr_span_decisions_total`, `osr_executions_total`, `osr_execution_cost_usd_total`, `osr_learn_outcomes_total`, `osr_breaker_transitions_total`, `osr_cache_events_total`, `osr_span_errors_total`) |
| `LoggingSink(log=None)` | one JSON line per event on the `opensmartroute.events` logger, level follows the event |
| `FileSink(path)` | append-only JSONL, one event per line (ship with any log forwarder) |
| `OpenTelemetrySink()` (`adapters.optional`, `otel` extra) | live bridge: each span becomes an OTel span (parented through the OTel context, joined to `traceparent`), each event a span event and an `osr.events` counter, durations an `osr.span_duration_ms` histogram; the host configures the SDK, exporter and resource |

Write your own by subclassing `EventSink` and implementing `emit(event)` (optionally `span_start`,
`span_end`, `snapshot`). A sink that raises is counted in `Tracer.sink_errors` and logged (first ten
occurrences); it never breaks routing.

## Configuration

`configure_tracing(*sinks, sample_rate=None)` installs sinks on the process-wide tracer that every
`Router` uses by default. With no sinks it reads `ObservabilitySettings` (`OSR_OBSERVABILITY_*`):

| Setting | Env | Default | Effect |
|---|---|---|---|
| `metrics` | `OSR_OBSERVABILITY_METRICS` | `true` | attach a `MetricsSink` |
| `memory_events` | `OSR_OBSERVABILITY_MEMORY_EVENTS` | `1000` | attach a `MemorySink` of that size (`0` disables) |
| `log_events` | `OSR_OBSERVABILITY_LOG_EVENTS` | `false` | attach a `LoggingSink` |
| `events_file` | `OSR_OBSERVABILITY_EVENTS_FILE` | `""` | attach a `FileSink` at that path |
| `otel` | `OSR_OBSERVABILITY_OTEL` | `false` | attach an `OpenTelemetrySink` (needs the `otel` extra) |
| `sample_rate` | `OSR_OBSERVABILITY_SAMPLE_RATE` | `1.0` | share of root spans that are recorded |
| `attribute_max_len` | `OSR_OBSERVABILITY_ATTRIBUTE_MAX_LEN` | `500` | longest attribute string kept |

`Router(reg, tracer=Tracer([...]))` gives one router a private tracer;
`RouterBuilder(...).with_tracing(*sinks_or_tracer, sample_rate=)` does the same for an
`EnterpriseRouter` (no arguments: build from settings). `osr`, `osr serve`, the Docker image and the
hosted platform call `configure_tracing()` / `Tracer.from_settings()` at start-up, so the environment
variables above are all that is needed in a container.

## HTTP endpoints (`osr serve`)

| Endpoint | Returns |
|---|---|
| `GET /metrics` | `MetricsTelemetry` exposition followed by the tracer's `MetricsSink` counters (names are disjoint) |
| `GET /events?request_id=&trace_id=&name=route.*&kind=&level=&limit=200` | recent events from the memory buffer (404 when `OSR_OBSERVABILITY_MEMORY_EVENTS=0`) |
| `GET /trace/{request_id}` | every span and event of one request, in time order, with its `trace_id` |
| `GET /stats` | adds an `observability` block: sinks, sample rate, sink errors and each sink's snapshot |

Every traced response carries `X-OSR-Trace-Id`; a client that sends `traceparent` gets its own trace id
back, so the routing spans line up with the caller's trace in Jaeger, Tempo, Azure Monitor or any OTel
backend. Probes (`/healthz`, `/readyz`), `/metrics`, `/events`, `/trace/*`, `/whoami` and the OpenAPI
pages are never traced. The endpoints follow the server's access control: with `OSR_SERVER_AUTH_TOKENS`
set, `/events`, `/trace/*` and `/stats` need a token (`/metrics` stays open for scrapers).

## HTTP endpoints (hosted platform)

The platform API (`platform/api`) exposes the same tracer per workspace, so a tenant only ever sees its
own requests - and it is the platform's own observability: it persists every span and event to its
database (`osr_platform.telemetry`, `OSR_PLATFORM_TELEMETRY_STORE`, on by default, off the request path
through a queued writer; `OSR_PLATFORM_TELEMETRY_RETENTION_DAYS`), keeps per-minute HTTP counters for a
day, computes time series from its usage records and evaluates alert rules in-process
(`osr_platform.alerts`). Nothing has to be exported to a metrics, tracing or alerting system.

| Endpoint | Returns |
|---|---|
| `GET /api/v1/trace/{request_id}` | the activity row of the request plus its spans and events (from the buffer, or from the store once evicted - `source` says which), the feedback reported for it (`outcomes`) and its audit records (`audit`); `404` for another workspace's request |
| `GET /api/v1/events?name=&kind=&level=&request_id=&since=&until=&limit=` | events joined to the workspace through its request ids (deployment-wide events on plans with `stats`); `since` / `until` select a time range from the store (default the last 6 hours) |
| `GET /api/v1/telemetry/series?window=` | the workspace's requests, failures, p50 / p95 latency, cost and tokens per bucket (`1h` .. `30d`) with per-target and per-endpoint totals; `http` (plans with `stats`) adds the deployment's requests, 4xx / 5xx and latency percentiles per bucket |
| `GET /api/v1/alerts` | active conditions - budgets, failing requests, and on plans with `stats` readiness, 5xx rate, latency SLO, breakers, drift, autopilot errors, a stale SLM, tracing off - each with the dashboard page that shows or fixes it |
| `GET /api/v1/status` | public readiness with the tracing state, for status pages (`503` while a check fails) |
| `GET /api/v1/activity` | request log with `traced: true` on rows whose spans are kept and `outcome` summarising the feedback per request |

`POST /api/v1/route` returns `trace_id` and `X-OSR-Trace-Id`. The dashboard renders the history at
`/platform/dashboard/events` (1 hour to 7 days), a span waterfall from every traced activity row and a *Trace* tab
in the playground; the overview shows the 24-hour series and the alerts, `/platform/dashboard/health` the
deployment series, the *Systems* card (`/api/v1/status`) and the same alerts.

## OpenTelemetry end to end

```python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))

from opensmartroute import configure_tracing
from opensmartroute.adapters.optional import OpenTelemetrySink

configure_tracing(OpenTelemetrySink())      # or OSR_OBSERVABILITY_OTEL=true
```

OTel spans are named `osr.route`, `osr.plan`, `osr.execute`, ...; attributes carry the `osr.` prefix
(`osr.target`, `osr.confidence`, `osr.request_id`, `osr.trace_id`). `OpenTelemetryTelemetry` (the
older `Telemetry` port) still records the `osr.route_latency_ms` histogram per decision; the sink is the
richer, per-stage view and the two can run side by side.

## Relationship to telemetry and audit

`Telemetry` sinks (`LoggingTelemetry`, `MetricsTelemetry`, `OpenTelemetryTelemetry`) observe *decisions
and outcomes* on an `EnterpriseRouter`; `AuditSink` keeps a hash-chained, tamper-evident record of
*who got what*. The tracer observes *how* the decision was made - every stage, with timings, on plain
and enterprise routers alike. Use audit for compliance, telemetry for dashboards you already have and
the tracer for debugging, SLOs per stage and distributed tracing. `RouteTrace.explain()` remains the
human-readable explanation attached to a `RouteDecision`.

`MetricsTelemetry` is the dashboard store: `snapshot()` returns `decisions_total`, `decisions_by_target`,
`policy_rejections_total`, `errors_by_type`, `route_latency_ms` (`p50`, `p95`, `p99`, `mean`, `count`),
`route_latency_histogram_ms`, `mean_confidence`, `outcomes_by_target` (count, success rate, cost, quality,
latency) and `uptime_s`; `prometheus(namespace="osr", extra={...})` renders the same as Prometheus text
(`osr_route_decisions_total{target="..."}`, `osr_route_latency_ms_bucket{le="..."}`,
`osr_outcome_cost_usd_total`, `osr_decision_cache_hits_total`, extras as gauges). `osr serve` and the
hosted platform expose it at `GET /metrics`; `RouterBuilder.with_cache(ttl_s, max_size)` adds a decision
cache whose `CacheMiddleware.stats()` feed the `osr_decision_cache_*` series, and
`with_audit(sink, outcomes=True)` chains outcomes into the audit trail alongside decisions.