<!-- OpenSmartRoute: Architecture. https://opensmartroute.ai/docs/ARCHITECTURE -->
# Architecture

OpenSmartRoute is a **decision layer**, not a gateway. Given a request and a catalogue of targets it
returns *which target should handle this, how confident we are, and why* — and it learns from what
happened next. Executing the target is optional and pluggable.

```mermaid
flowchart LR
    subgraph In
        REQ[RouteRequest]
    end
    subgraph Middleware
        G[Guard] --> T[Tenant] --> C[Cache]
    end
    subgraph Core
        S[Signals] --> P[Policy] --> ST[Strategies] --> E[Ensemble] --> D[Decision + Trace]
    end
    subgraph Learn
        O[Outcome] --> AL[AutoLearner] --> ST
        O --> H[Health] --> P
    end
    REQ --> G
    C --> S
    D --> X[Executor / handler]
    X --> O
    CAT[(TargetRegistry)] --> P
```

## Layers

| Layer | Package | Responsibility | Depends on |
|---|---|---|---|
| **Core model** | `core` | `RouteTarget`, `RouteRequest`, `RouteDecision`, `RouteTrace`, `Outcome`, `TargetRegistry` | stdlib |
| **Signals** | `signals` | Deterministic features of a request: complexity, domain/action, language, modality, PII, jailbreak, token estimate; opt-in verbalised difficulty, draft-response and response-uncertainty features (`signals.uncertainty`) | core |
| **Policy** | `policy` | Hard constraints → admissible set + rejection reasons. Never traded off. | core, signals |
| **Strategies** | `strategies` | Soft scorers ∈ [0,1] with rationale: rules, capability fit, similarity, Thompson bandit, LLM judge, cascade executor; opt-in hidden-state probe, token budgets, edge/cloud tiers, auctions; post-answer controls (self-escalation, mixture-of-agents, protocol selection) | core, signals |
| **Learning** | `learning`, `math` | Strategies that update from outcomes (IRT, Bradley–Terry, LinUCB, Markov/MDP, multi-turn history embeddings, per-user shrinkage) and the pure-math they rest on (bandits, calibration, Dirichlet probe, energy model, mixture-cure hand-off); `AutoLearner` fan-out, drift, atomic persistence; the routing SLM (`RouterSLM`, `SLMStrategy`, `distill_router`) and its champion/challenger `SelfImprover` | strategies |
| **Discovery** | `discovery`, `retrieval` | Retrieve-then-rank narrowing, schema-aware tool matching, cache-preserving tool order, skill graphs and submodular skill sets for large catalogues | core, strategies |
| **Router** | `router` | Orchestrates signals → policy → strategies → confidence-weighted ensemble → decision; optional judge escalation; optional plan (persona → skill → model) | all above |
| **Real-time** | `realtime` | Circuit breaker, token bucket, rolling budget, `HealthRegistry`; `HealthPolicy` (hard) and `HealthStrategy` (soft) | policy, strategies, math |
| **Enterprise** | `enterprise` | `RouterBuilder`, `EnterpriseRouter`, middleware chain, telemetry/state/audit ports and in-process adapters | router, realtime, learning |
| **Security** | `security` | `InputGuard` (size, normalisation, confounder gadgets), `Redactor`, `GuardMiddleware`, prompt sanitiser, secret loading | enterprise |
| **Adapters** | `adapters` | OpenAI-compatible HTTP client + glue (judge, embedder, handler); agent harnesses; MCP tools and server recommendation, A2A cards, SKILL.md and persona loaders, semantic-router import; live model catalogue (OpenRouter prices, Hugging Face cards) and web-search knowledge (https-only, byte-capped, risk-scored); optional sentence-transformers and OpenTelemetry | strategies, enterprise |
| **Config** | `config` | JSON/YAML loaders for targets and rules with validation errors | core, strategies |
| **Surface** | `aio`, `server`, `cli`, `eval` | Async façade, FastAPI app, `osr` CLI, RouterBench-style evaluation, dataset collection from the Hugging Face Hub (`eval.collect`) | router, enterprise |

Dependency direction is strictly downward in this table. `math` depends on nothing but stdlib and
is usable on its own.

## The decision pipeline, precisely

1. **Signals** — `extract_signals(request)`; ~0.1 ms; no I/O.
2. **Candidate pool** — `registry.all(primary_only=True)`, optionally filtered by `kinds`.
3. **Policy** — `Policy.filter(pool)` → `(admissible, rejections)`. `HealthPolicy` adds breaker/rate/budget
   checks (non-consuming). Empty admissible set → `NoRouteError` with the reasons.
4. **Pinned rules** — a matching `Rule(pin=True)` narrows the admissible set (Arch-Router semantics).
5. **Scoring** — every strategy returns `{target_id: StrategyScore(score, rationale, confidence)}`.
6. **Ensemble** — quality estimate $\hat q = \sum_k w_k\kappa_k s_k / \sum_k w_k\kappa_k$ where $\kappa$ is
   the strategy's self-confidence (learners report $\kappa=\min(1,n/n_0)$, so they are silent until they
   have evidence). Utility $U=\,w_q\hat q - w_c\,\tilde c - w_\ell\,\tilde\ell$ with log-min-max normalised
   cost/latency and a hard quality floor.
7. **Confidence** — softmax margin over utilities ($\tau=0.1$). If below `escalate_llm_judge_below` and a
   judge is configured, re-score once with the judge included.
8. **Decision** — top target, ranked alternatives, full trace. Optionally a `RoutePlan` where each slot
   (persona, skill, llm) is its own sub-routing over that kind; a slot is left empty when its best
   candidate's quality estimate is below `slot_quality_floor`, so irrelevant skills are never attached.
9. **Execute** (optional, `execution.py`) — `Router.run()` walks the plan: the `skill` slot's handler runs as a
   pre-processor (may rewrite the request or attach `skill_output`), persona/skill/primary `instructions` are
   composed into `context["system"]`, then the primary handler is called (`chat_handler`, a skill function, or an
   agent harness via `CallableHarness`/`HTTPHarness`/`SubprocessHarness`). One `Outcome` per participant
   (`role=None|"persona"|"skill"`, shared `task_id`) is recorded — success, latency, cost from tokens × unit cost.
10. **Learn** — `Outcome` → every strategy's `update()`, drift detectors, health registry, feedback store.

## Extension points

| To add… | Implement | Register via |
|---|---|---|
| a target | `RouteTarget(...)` or a YAML entry | `registry.add()` / `load_targets()` |
| a signal | `SignalExtractor.extract()` | `Router(extractors=[...])` |
| a hard rule | `Policy.check()` → reason or `None` | `RouterBuilder.with_policy()` |
| a scorer | `Strategy.score()` (+ `update()` if it learns) | `with_strategy(s, weight)` |
| pre/post processing | `Middleware.__call__(request, next_)` | `with_middleware()` |
| metrics/tracing | `Telemetry.on_decision/on_outcome/on_error` | `with_telemetry()` |
| span/event capture | `EventSink.emit(event)` (+ `span_start/span_end` for live bridges) | `with_tracing(sink)` / `configure_tracing(sink)` |
| persistence | `StateStore.get/put/delete` | pass to learners / caches |
| a provider | `RouteTarget.handler = chat_handler(client, model)` | `adapters.OpenAICompatClient` |

## Concurrency model

- `Router.route()` is pure with respect to router state (no writes) and safe to call from many threads.
- Learners, health registry, caches and stores take their own locks on write.
- `AsyncRouter` runs routing in the default executor and awaits coroutine handlers.
- Multi-replica: learner updates are commutative sums (Beta counts, IRT gradients are small and
  order-insensitive in practice, BT strengths, LinUCB `A`/`b`, Markov counts), so replicas can be folded
  together with `merge()` / `learning.merge_learners()`, or readers can adopt a writer's snapshot through a
  `StateStore` with `AutoLearner.refresh()`. Corrupt snapshots are quarantined, never fatal
  (`AutoLearner.quarantined`).

## Performance envelope

Measured with `python scripts/bench.py --n 200 --scale` (CPython 3.11, x86-64 Windows laptop, pure
Python, no C extensions). Numbers are wall-clock per `route()` including signal extraction, policy,
every strategy and the ensemble; re-run the script on your own hardware before sizing.

**16-target example catalogue** (`examples/targets.yaml`, `examples/eval_dataset.jsonl`):

| Configuration | p50 | p95 | p99 |
|---|---|---|---|
| rules + capability + similarity + bandit | 5.8 ms | 6.2 ms | 6.5 ms |
| + IRT + preference + LinUCB + Markov + health | 8.6 ms | 12.9 ms | 17.8 ms |

**Scaling with catalogue size** (synthetic LLM catalogue, defaults + auto-learning). Cost is linear in
admissible targets x strategies until retrieval narrowing (BM25 + hashed dense, reciprocal-rank fusion)
caps the candidate pool; the default `Settings.routing` only narrows above 500 admissible targets (to 50),
`RouterBuilder.with_retrieval(narrow_above=32, narrow_to=24)` turns it on earlier:

| Targets | Narrowing | p50 | p95 | p99 |
|---|---|---|---|---|
| 16 | default (none) | 9.2 ms | 9.5 ms | 10.0 ms |
| 16 | top-24 above 32 (none triggered) | 9.2 ms | 9.6 ms | 10.5 ms |
| 64 | default (none) | 35.7 ms | 36.6 ms | 37.1 ms |
| 64 | top-24 above 32 | 14.5 ms | 15.1 ms | 15.6 ms |
| 256 | default (none) | 142 ms | 145 ms | 158 ms |
| 256 | top-24 above 32 | 16.7 ms | 17.6 ms | 18.5 ms |
| 1024 | default (auto top-50) | 38.7 ms | 43.5 ms | 49.1 ms |
| 1024 | top-24 above 32 | 24.1 ms | 28.2 ms | 32.9 ms |

Rule of thumb: keep the scored pool at or below ~50 targets and routing stays under 25 ms p99 in pure
Python regardless of catalogue size; the retrieval stage itself is ~10 ms at 1k targets and grows
roughly linearly in the catalogue. The LLM judge, when triggered, adds one provider round-trip; keep
`escalate_llm_judge_below` low (<= 0.5) so it fires only on genuinely ambiguous requests.

## Deployment shapes

1. **Library** — import in-process. Lowest latency. State on local disk or in-memory.
2. **Sidecar** — `osr serve` per application (container image from `deploy/Dockerfile`); shared
   `StateStore` for global learning.
3. **Control plane** — one HA deployment (Helm chart in `deploy/helm/opensmartroute`); a gateway
   (LiteLLM, Envoy, aisix) calls `/route`, executes the chosen target, and posts `/feedback`.

Multi-replica learning: run one *writer* replica that learns and saves through a `StateStore`, and have
readers call `AutoLearner.refresh()` on a timer; or let every replica learn independently and fold them
together periodically with `learning.merge_learners()`. See [deploy/README.md](https://opensmartroute.ai/docs/deploy.md).

## Non-goals

Model serving, protocol translation between providers, prompt versioning, chat UI. See
[ROADMAP.md](https://opensmartroute.ai/docs/ROADMAP.md#out-of-scope).