Architecture

OpenSmartRoute is a decision layer, not a gateway. Given a request and a catalogue of targets it returns which target should handle this, how confident we are, and why — and it learns from what happened next. Executing the target is optional and pluggable.

flowchart LR
    subgraph In
        REQ[RouteRequest]
    end
    subgraph Middleware
        G[Guard] --> T[Tenant] --> C[Cache]
    end
    subgraph Core
        S[Signals] --> P[Policy] --> ST[Strategies] --> E[Ensemble] --> D[Decision + Trace]
    end
    subgraph Learn
        O[Outcome] --> AL[AutoLearner] --> ST
        O --> H[Health] --> P
    end
    REQ --> G
    C --> S
    D --> X[Executor / handler]
    X --> O
    CAT[(TargetRegistry)] --> P

Layers

Dependency direction is strictly downward in this table. math depends on nothing but stdlib and is usable on its own.

The decision pipeline, precisely

1. Signals — extract_signals(request); ~0.1 ms; no I/O.
2. Candidate pool — registry.all(primary_only=True), optionally filtered by kinds.
3. Policy — Policy.filter(pool) → (admissible, rejections). HealthPolicy adds breaker/rate/budget checks (non-consuming). Empty admissible set → NoRouteError with the reasons.
4. Pinned rules — a matching Rule(pin=True) narrows the admissible set (Arch-Router semantics).
5. Scoring — every strategy returns {target_id: StrategyScore(score, rationale, confidence)}.
6. Ensemble — quality estimate $\hat q = \sum_k w_k\kappa_k s_k / \sum_k w_k\kappa_k$ where $\kappa$ is the strategy's self-confidence (learners report $\kappa=\min(1,n/n_0)$, so they are silent until they have evidence). Utility $U=\,w_q\hat q - w_c\,\tilde c - w_\ell\,\tilde\ell$ with log-min-max normalised cost/latency and a hard quality floor.
7. Confidence — softmax margin over utilities ($\tau=0.1$). If below escalate_llm_judge_below and a judge is configured, re-score once with the judge included.
8. Decision — top target, ranked alternatives, full trace. Optionally a RoutePlan where each slot (persona, skill, llm, tool) is its own sub-routing over that kind; a slot is left empty when its best candidate's quality estimate is below slot_quality_floor (tool_quality_floor for tools), so irrelevant skills and tools are never attached. A model or agent primary gets up to tool_slots tool slots - the functions it may call.
9. Execute (optional, execution.py) — Router.run() walks the plan: the skill slot's handler runs as a pre-processor (may rewrite the request or attach skill_output), persona/skill/primary instructions are composed into context["system"], tool slots become function-calling specs in context["openai"]["tools"], the prompt is compacted (compress.py: whitespace, repeated lines, history budget; code fences untouched), then the primary handler is called (chat_handler, a skill function, or an agent harness via CallableHarness/HTTPHarness/SubprocessHarness). Tool calls the model makes to offered tools are run and fed back for up to tool_rounds rounds. One Outcome per participant (role=None|"persona"|"skill"|"tool", shared task_id) is recorded — success, latency, cost from tokens × unit cost.
10. Learn — Outcome → every strategy's update(), drift detectors, health registry, feedback store.

Extension points

Concurrency model

• Router.route() is pure with respect to router state (no writes) and safe to call from many threads.
• Learners, health registry, caches and stores take their own locks on write.
• AsyncRouter runs routing in the default executor and awaits coroutine handlers.
• Multi-replica: learner updates are commutative sums (Beta counts, IRT gradients are small and order-insensitive in practice, BT strengths, LinUCB A/b, Markov counts), so replicas can be folded together with merge() / learning.merge_learners(), or readers can adopt a writer's snapshot through a StateStore with AutoLearner.refresh(). Corrupt snapshots are quarantined, never fatal (AutoLearner.quarantined).

Performance envelope

Measured with the reproducible routing benchmark, 300 requests per configuration (CPython 3.11, x86-64 Windows laptop, pure Python, no C extensions; 2026-10-09). Numbers are wall-clock per route() including signal extraction, policy, every strategy and the ensemble; re-run the benchmark on your own hardware before sizing. The continuous build runs the same benchmark and fails when the full configuration's p95 passes 50 ms, so the overhead cannot drift up unnoticed.

27-target example catalogue with its rules (examples/targets.yaml, examples/rules.yaml, examples/eval_dataset.jsonl):

Scaling with catalogue size (synthetic LLM catalogue, defaults + auto-learning). Cost is linear in admissible targets x strategies until retrieval narrowing (BM25 + hashed dense, reciprocal-rank fusion) caps the candidate pool; the default Settings.routing only narrows above 500 admissible targets (to 50), RouterBuilder.with_retrieval(narrow_above=32, narrow_to=24) turns it on earlier:

Rule of thumb: keep the scored pool at or below ~50 targets and routing stays under 25 ms p99 in pure Python regardless of catalogue size; the retrieval stage itself is ~10 ms at 1k targets and grows roughly linearly in the catalogue. The LLM judge, when triggered, adds one provider round-trip; keep escalate_llm_judge_below low (<= 0.5) so it fires only on genuinely ambiguous requests.

Deployment shapes

1. Library — import in-process. Lowest latency. State on local disk or in-memory.
2. Sidecar — osr serve per application (container image from deploy/Dockerfile); shared StateStore for global learning.
3. Control plane — one HA deployment (the published Helm chart); a gateway (LiteLLM, Envoy, aisix) calls /route, executes the chosen target, and posts /feedback.
4. Hosted platform — the same router behind accounts, API keys, plans and quotas, BYOK providers, tenants, SSO, billing and the dashboard (Platform guide), reached over REST, the /v1 dialects, MCP, the Python CLI (osr login) or the TypeScript client (@opensmartroute/sdk, which also targets an osr serve).

Multi-replica learning: run one writer replica that learns and saves through a StateStore, and have readers call AutoLearner.refresh() on a timer; or let every replica learn independently and fold them together periodically with learning.merge_learners(). See Deploy with Docker and Helm.

Non-goals

Model serving, prompt versioning, chat UI, and provider-to-provider protocol translation as a product: the responses, anthropic and media modules exist so that clients speaking the OpenAI or Anthropic dialects can reach the router with one base-URL change - each becomes one routed chat call, not a general translation layer.