Architecture

OpenSmartRoute is a decision layer, not a gateway. Given a request and a catalogue of targets it returns which target should handle this, how confident we are, and why — and it learns from what happened next. Executing the target is optional and pluggable.

flowchart LR
    subgraph In
        REQ[RouteRequest]
    end
    subgraph Middleware
        G[Guard] --> T[Tenant] --> C[Cache]
    end
    subgraph Core
        S[Signals] --> P[Policy] --> ST[Strategies] --> E[Ensemble] --> D[Decision + Trace]
    end
    subgraph Learn
        O[Outcome] --> AL[AutoLearner] --> ST
        O --> H[Health] --> P
    end
    REQ --> G
    C --> S
    D --> X[Executor / handler]
    X --> O
    CAT[(TargetRegistry)] --> P

Layers#

Layer	Package	Responsibility	Depends on
Core model	core	RouteTarget, RouteRequest, RouteDecision, RouteTrace, Outcome, TargetRegistry	stdlib
Signals	signals	Deterministic features of a request: complexity, domain/action, language, modality, PII, jailbreak, token estimate; opt-in verbalised difficulty, draft-response and response-uncertainty features (signals.uncertainty)	core
Policy	policy	Hard constraints → admissible set + rejection reasons. Never traded off.	core, signals
Strategies	strategies	Soft scorers ∈ [0,1] with rationale: rules, capability fit, similarity, Thompson bandit, LLM judge, cascade executor; opt-in hidden-state probe, token budgets, edge/cloud tiers, auctions; post-answer controls (self-escalation, mixture-of-agents, protocol selection)	core, signals
Learning	learning, math	Strategies that update from outcomes (IRT, Bradley–Terry, LinUCB, Markov/MDP, multi-turn history embeddings, per-user shrinkage) and the pure-math they rest on (bandits, calibration, Dirichlet probe, energy model, mixture-cure hand-off); AutoLearner fan-out, drift, atomic persistence; the routing SLM (RouterSLM, SLMStrategy, distill_router) and its champion/challenger SelfImprover	strategies
Discovery	discovery, retrieval	Retrieve-then-rank narrowing, schema-aware tool matching, cache-preserving tool order, skill graphs and submodular skill sets for large catalogues	core, strategies
Router	router	Orchestrates signals → policy → strategies → confidence-weighted ensemble → decision; optional judge escalation; optional plan (persona → skill → model)	all above
Real-time	realtime	Circuit breaker, token bucket, rolling budget, HealthRegistry; HealthPolicy (hard) and HealthStrategy (soft)	policy, strategies, math
Enterprise	enterprise	RouterBuilder, EnterpriseRouter, middleware chain, telemetry/state/audit ports and in-process adapters	router, realtime, learning
Security	security	InputGuard (size, normalisation, confounder gadgets), Redactor, GuardMiddleware, prompt sanitiser, secret loading	enterprise
Adapters	adapters	OpenAI-compatible HTTP client + glue (judge, embedder, handler); agent harnesses; MCP tools and server recommendation, A2A cards, SKILL.md and persona loaders, semantic-router import; live model catalogue (OpenRouter prices, Hugging Face cards) and web-search knowledge (https-only, byte-capped, risk-scored); optional sentence-transformers and OpenTelemetry	strategies, enterprise
Config	config	JSON/YAML loaders for targets and rules with validation errors	core, strategies
Surface	aio, server, cli, eval	Async façade, FastAPI app, osr CLI, RouterBench-style evaluation, dataset collection from the Hugging Face Hub (eval.collect)	router, enterprise

Dependency direction is strictly downward in this table. math depends on nothing but stdlib and is usable on its own.

The decision pipeline, precisely#

1. Signals — extract_signals(request); ~0.1 ms; no I/O.
2. Candidate pool — registry.all(primary_only=True), optionally filtered by kinds.
3. Policy — Policy.filter(pool) → (admissible, rejections). HealthPolicy adds breaker/rate/budget checks (non-consuming). Empty admissible set → NoRouteError with the reasons.
4. Pinned rules — a matching Rule(pin=True) narrows the admissible set (Arch-Router semantics).
5. Scoring — every strategy returns {target_id: StrategyScore(score, rationale, confidence)}.
6. Ensemble — quality estimate $\hat q = \sum_k w_k\kappa_k s_k / \sum_k w_k\kappa_k$ where $\kappa$ is the strategy's self-confidence (learners report $\kappa=\min(1,n/n_0)$, so they are silent until they have evidence). Utility $U=,w_q\hat q - w_c,\tilde c - w_\ell,\tilde\ell$ with log-min-max normalised cost/latency and a hard quality floor.
7. Confidence — softmax margin over utilities ($\tau=0.1$). If below escalate_llm_judge_below and a judge is configured, re-score once with the judge included.
8. Decision — top target, ranked alternatives, full trace. Optionally a RoutePlan where each slot (persona, skill, llm) is its own sub-routing over that kind; a slot is left empty when its best candidate's quality estimate is below slot_quality_floor, so irrelevant skills are never attached.
9. Execute (optional, execution.py) — Router.run() walks the plan: the skill slot's handler runs as a pre-processor (may rewrite the request or attach skill_output), persona/skill/primary instructions are composed into context["system"], then the primary handler is called (chat_handler, a skill function, or an agent harness via CallableHarness/HTTPHarness/SubprocessHarness). One Outcome per participant (role=None|"persona"|"skill", shared task_id) is recorded — success, latency, cost from tokens × unit cost.
10. Learn — Outcome → every strategy's update(), drift detectors, health registry, feedback store.

Extension points#

To add…	Implement	Register via
a target	RouteTarget(...) or a YAML entry	registry.add() / load_targets()
a signal	SignalExtractor.extract()	Router(extractors=[...])
a hard rule	Policy.check() → reason or None	RouterBuilder.with_policy()
a scorer	Strategy.score() (+ update() if it learns)	with_strategy(s, weight)
pre/post processing	Middleware.__call__(request, next_)	with_middleware()
metrics/tracing	Telemetry.on_decision/on_outcome/on_error	with_telemetry()
span/event capture	EventSink.emit(event) (+ span_start/span_end for live bridges)	with_tracing(sink) / configure_tracing(sink)
persistence	StateStore.get/put/delete	pass to learners / caches
a provider	RouteTarget.handler = chat_handler(client, model)	adapters.OpenAICompatClient

Concurrency model#

• Router.route() is pure with respect to router state (no writes) and safe to call from many threads.
• Learners, health registry, caches and stores take their own locks on write.
• AsyncRouter runs routing in the default executor and awaits coroutine handlers.
• Multi-replica: learner updates are commutative sums (Beta counts, IRT gradients are small and order-insensitive in practice, BT strengths, LinUCB A/b, Markov counts), so replicas can be folded together with merge() / learning.merge_learners(), or readers can adopt a writer's snapshot through a StateStore with AutoLearner.refresh(). Corrupt snapshots are quarantined, never fatal (AutoLearner.quarantined).

Performance envelope#

Measured with python scripts/bench.py --n 200 --scale (CPython 3.11, x86-64 Windows laptop, pure Python, no C extensions). Numbers are wall-clock per route() including signal extraction, policy, every strategy and the ensemble; re-run the script on your own hardware before sizing.

16-target example catalogue (examples/targets.yaml, examples/eval_dataset.jsonl):

Configuration	p50	p95	p99
rules + capability + similarity + bandit	5.8 ms	6.2 ms	6.5 ms
+ IRT + preference + LinUCB + Markov + health	8.6 ms	12.9 ms	17.8 ms

Scaling with catalogue size (synthetic LLM catalogue, defaults + auto-learning). Cost is linear in admissible targets x strategies until retrieval narrowing (BM25 + hashed dense, reciprocal-rank fusion) caps the candidate pool; the default Settings.routing only narrows above 500 admissible targets (to 50), RouterBuilder.with_retrieval(narrow_above=32, narrow_to=24) turns it on earlier:

Targets	Narrowing	p50	p95	p99
16	default (none)	9.2 ms	9.5 ms	10.0 ms
16	top-24 above 32 (none triggered)	9.2 ms	9.6 ms	10.5 ms
64	default (none)	35.7 ms	36.6 ms	37.1 ms
64	top-24 above 32	14.5 ms	15.1 ms	15.6 ms
256	default (none)	142 ms	145 ms	158 ms
256	top-24 above 32	16.7 ms	17.6 ms	18.5 ms
1024	default (auto top-50)	38.7 ms	43.5 ms	49.1 ms
1024	top-24 above 32	24.1 ms	28.2 ms	32.9 ms

Rule of thumb: keep the scored pool at or below ~50 targets and routing stays under 25 ms p99 in pure Python regardless of catalogue size; the retrieval stage itself is ~10 ms at 1k targets and grows roughly linearly in the catalogue. The LLM judge, when triggered, adds one provider round-trip; keep escalate_llm_judge_below low (<= 0.5) so it fires only on genuinely ambiguous requests.

Deployment shapes#

1. Library — import in-process. Lowest latency. State on local disk or in-memory.
2. Sidecar — osr serve per application (container image from deploy/Dockerfile); shared StateStore for global learning.
3. Control plane — one HA deployment (Helm chart in deploy/helm/opensmartroute); a gateway (LiteLLM, Envoy, aisix) calls /route, executes the chosen target, and posts /feedback.

Multi-replica learning: run one writer replica that learns and saves through a StateStore, and have readers call AutoLearner.refresh() on a timer; or let every replica learn independently and fold them together periodically with learning.merge_learners(). See deploy/README.md.

Non-goals#

Model serving, protocol translation between providers, prompt versioning, chat UI. See ROADMAP.md.