# User guide Everything the README leaves out: the full quick-start sequence, running plans end to end, wiring real providers, the enterprise builder, the research-track modules, routing latency and the repository layout. API reference material lives in [SDK.md](https://opensmartroute.ai/docs/SDK.md) (decorator SDK, components, settings) and [REFERENCE.md](https://opensmartroute.ai/docs/REFERENCE.md) (generated; every module and exported name); design in [ARCHITECTURE.md](https://opensmartroute.ai/docs/ARCHITECTURE.md). Contents 1. [Targets](#1-targets) 2. [Routing](#2-routing) 3. [Executing a plan](#3-executing-a-plan) 4. [Real providers](#4-real-providers) 5. [Learning from outcomes](#5-learning-from-outcomes) 6. [Declaring components with decorators](#6-declaring-components-with-decorators) 7. [Enterprise builder](#7-enterprise-builder) 8. [Research-track modules](#8-research-track-modules) 9. [Command line](#9-command-line) 10. [Routing latency](#10-routing-latency) 11. [Repository layout](#11-repository-layout) ## 1. Targets Every routable thing is a `RouteTarget` with the same contract: an `id`, a `kind`, declared `capabilities`, `cost`, `latency_ms`, a `quality_prior`, optional `examples`, `instructions` and a `handler`. Adding a target never requires retraining; it is routable from its declaration and learns from its outcomes. | Kind | What it is | Typical `capabilities` | Wins the primary route? | Executed by | |---|---|---|---|---| | `llm` | A model endpoint (optionally per reasoning effort) | `domains`, `min/max_complexity`, `languages`, `context_window` | yes | `chat_handler` / any callable | | `skill` | A deterministic or narrow capability (SQL, translate, summarise, a SKILL.md package) | `actions`, `domains`, token footprint as `cost` | yes, or fills the `skill` plan slot | your function; `instructions` are disclosed to the model | | `persona` | System-prompt / behaviour layer (`primary: false`) | `domains`, `actions` | no; fills the `persona` slot on top of a model or agent | `instructions` become the system prompt in `run()` | | `agent` | An agent harness: coding, research or support runtime with tools, memory and its own loop | `supports_tools`, `tags: [autonomous]`, `actions` | yes | `CallableHarness` / `HTTPHarness` / `SubprocessHarness` | | `tool` | A single MCP / function tool | `actions`, `input_schema` in metadata | yes | tool call | | `workflow` | A fixed multi-step pipeline | `actions`, `domains` | yes | workflow engine | | `human` | Queue, team or expert (Erlang-C capacity maths) | `actions: [escalate]`, `tags: [safety]` | yes; also the abstention target | ticketing / hand-off | ```yaml # examples/targets.yaml (excerpt) - id: coding-agent # agent harness: repo access, tests, review loop kind: agent capabilities: { domains: [coding], actions: [code_generation, code_review, action], min_complexity: 0.3, supports_tools: true, tags: [autonomous] } cost: { usd_per_1k_tokens: 0.008 } latency_ms: 6000 examples: ["Refactor this class to remove the circular import and run the test suite."] - id: skill-sql # deterministic skill; `instructions` is its SKILL.md-style body kind: skill capabilities: { domains: [data_analysis, coding], actions: [code_generation, extract] } cost: { usd_per_1k_tokens: 0.001 } instructions: "Always qualify table names with the schema; never SELECT *." - id: persona-legal-counsel # never primary; layered on whichever model/agent wins kind: persona primary: false capabilities: { domains: [legal], actions: [reasoning, summarize, qa] } instructions: "You are cautious in-house counsel. Cite the clause, flag jurisdiction risk." ``` Catalogues load from JSON or YAML (`config.load_targets`), from MCP `tools/list` payloads (`adapters.tools_from_mcp`), A2A agent cards (`adapters.agent_from_card`), SKILL.md directories (`adapters.load_skills`), persona folders (`adapters.load_personas`) and vLLM semantic-router configurations (`adapters.load_semantic_router_config`). ### Agent skills for coding assistants .claude/skills/ (`.claude/skills`) ships nine Agent-Skills packages (`SKILL.md`, agentskills.io format) that teach a coding agent how to use and extend OpenSmartRoute: `osr-routing-catalogue`, `osr-decorator-sdk`, `osr-enterprise-builder`, `osr-evaluation`, `osr-security-hardening`, `osr-integrations`, `osr-deploy-serve`, `osr-platform` and `osr-contributing`. Claude Code discovers them automatically in this repository; other agents can load the directory, and each package is published on the documentation site under `/docs/skills/`. They are also valid routing targets: the same loader the SDK uses (`load_skills`) validates them in CI and the router fills the plan's skill slot with the right one. ```bash osr skills # validate + list (default root .claude/skills) osr -t examples/targets.yaml --skills .claude/skills route "Create a JSONL eval dataset and gate CI on accuracy" --plan # plan: llm:llm-mid -> skill:osr-evaluation ``` ## 2. Routing ```python from opensmartroute import Router, RouteRequest, RequestConstraints, Objective router = Router(registry) # plain text d = router.route("Prove that sqrt(2) is irrational, step by step.") print(d.target.id, f"{d.confidence:.2f}") print(d.trace.explain()) # per-strategy scores and rationales # hard constraints and a per-request objective req = RouteRequest( text="Summarize this patient intake note.", constraints=RequestConstraints(region="eu", data_boundary="private", max_cost_per_1k=0.005), objective=Objective(quality=1.0, cost=0.5, latency=0.1, quality_floor=0.6), ) d = router.route(req) print(d.trace.policy_rejections) # {'llm-frontier': 'cost 0.015 > budget 0.005', ...} # a composed plan: persona -> skill -> model d = router.route("Review this NDA clause for GDPR liability.", plan=True) for slot in d.plan.slots: print(slot.role, slot.target.id, f"{slot.confidence:.2f}") ``` Policy runs before scoring and is never traded off: enable flags, allow / deny lists, kind, region, data boundary, PII, tenant, cost / latency / token SLOs, modality, language and jailbreak risk. Strategies then score the survivors and a confidence-weighted ensemble produces a quality estimate; utility is `w_q * quality - w_c * norm(cost) - w_l * norm(latency)` with a hard `quality_floor`. With a `ConformalCalibrator`, `RouteDecision.candidate_set` is the smallest set with the requested coverage and `abstain` is set when no target is safe enough. A generative judge can be consulted only when the ensemble is unsure: ```python from opensmartroute import LLMJudgeStrategy router = Router(registry, llm_judge=LLMJudgeStrategy(llm=lambda prompt: my_llm(prompt)), escalate_llm_judge_below=0.6) ``` ## 3. Executing a plan `route()` decides; `run()` decides and executes the whole plan: skill pre-processors run first, persona and skill `instructions` are composed into the system prompt, the primary target is called, and an `Outcome` is recorded for every participant so each one learns from the result. ```python from opensmartroute.adapters import (CallableHarness, HTTPHarness, SubprocessHarness, chat_handler, harness_handler, load_skills) registry.get("llm-frontier").handler = chat_handler(openai, model="gpt-4o") registry.get("coding-agent").handler = harness_handler( SubprocessHarness(["my-coding-agent", "--repo", "."], system_flag="--system")) registry.get("research-agent").handler = harness_handler( HTTPHarness("https://agents.internal/research/run", api_key_env="AGENT_TOKEN")) registry.get("skill-sql").handler = lambda req: lookup_schema(req.text) # runs BEFORE the model for skill in load_skills("examples/skills"): # SKILL.md folders registry.add(skill) res = router.run("Is this indemnification clause enforceable in California?", task_id="ticket-812") res.text # the primary target's answer res.steps # [persona:persona-legal-counsel, llm:llm-frontier] with per-step latency res.system_prompt # composed persona + skill instructions res.outcomes # one Outcome per participant, role="persona"/"skill"/None, shared task_id ``` A skill handler may return a `RouteRequest` (rewrite the request), a `str` (attached as `context["skill_output"]` and shown to the model) or `None` (instructions only). Async handlers work through `AsyncRouter.run()`; `EnterpriseRouter.run()` adds health, budgets and telemetry. See examples/end_to_end.py (`examples/end_to_end.py`) and [SDK.md](https://opensmartroute.ai/docs/SDK.md#executing-end-to-end). ## 4. Real providers The bundled client is stdlib only and speaks the OpenAI chat and embeddings API, which covers OpenAI, Azure OpenAI, vLLM, Ollama, LiteLLM, OpenRouter, Groq and Mistral. ```python from opensmartroute import Router, LLMJudgeStrategy from opensmartroute.adapters import OpenAICompatClient, chat_handler, embedder, judge_fn from opensmartroute.strategies import CapabilityStrategy, SimilarityStrategy openai = OpenAICompatClient() # OPENAI_API_KEY from env ollama = OpenAICompatClient("http://localhost:11434/v1", api_key="ollama") registry.get("llm-frontier").handler = chat_handler(openai, model="gpt-4o") registry.get("llm-small").handler = chat_handler(ollama, model="llama3.1") router = Router(registry, strategies=[CapabilityStrategy(), SimilarityStrategy(embedder=embedder(openai, model="text-embedding-3-small"))], llm_judge=LLMJudgeStrategy(judge_fn(openai, model="gpt-4o-mini")), escalate_llm_judge_below=0.5) d = router.route(req) result = router.execute(d, req) # ChatResult(text, prompt_tokens, completion_tokens, latency_ms) res = router.run(req) # route + execute the full plan ``` `adapters.sentence_transformers_embedder()` and `adapters.OpenTelemetryTelemetry` are lazy imports behind the `embeddings` and `otel` extras. `osr serve` exposes a FastAPI app with an OpenAI-compatible `/v1/chat/completions` proxy that routes, executes and learns. ## 5. Learning from outcomes ```python from opensmartroute import Outcome router.learn(Outcome(request_id=d.request_id, target_id=d.target.id, success=True, quality=0.9, cost_usd=0.002, latency_ms=1800, domains=d.trace.signals.domains)) ``` One `Outcome` fans out to every learner: Thompson bandits per domain, IRT ability and difficulty, Bradley-Terry preferences, LinUCB on the signal vector, the Markov / MDP lookahead, the task table and the personalisation posteriors. `Outcome.task_id` and `step` join delayed, task-level rewards to the routing decisions that produced them (`learning.TaskCredit`, `learning.TaskPins`). Page-Hinkley drift detection flags targets whose success rate degrades; every learner forgets with a configurable `decay` and can be merged across replicas (`learning.merge_learners`). Each `learn()` also records a `learn.outcome` event on the request's trace and, with `with_audit(sink, outcomes=True)`, an `outcome` record in the audit chain, so a decision and its result stay joined by `request_id` ([OBSERVABILITY.md](https://opensmartroute.ai/docs/OBSERVABILITY.md); on the hosted platform `POST /api/v1/feedback` does the same). Offline warm starts: `learning.warm_start_from_matrix()` replays a `(prompt, {target: reward})` matrix, `learning.warm_start()` copies posteriors from the nearest declared neighbours and `osr train --from rows.jsonl --out models.json` fits the hashed signal models. ### The routing SLM and the self-improvement loop `learning.RouterSLM` is a small routing model in one JSON file: a hashed dual encoder (`SLMSettings.dim` x `feature_dim`), a temperature calibrator and a snapshot of every target's price, latency and quality prior. It has no dependencies, predicts in microseconds and can be trained from any `EvalRow` source - labelled datasets, your own outcomes, synthetic ontology prompts or the full router's decisions (`distill_router`, which compresses rules + capability fit + learners + judge into one model). ```python from opensmartroute import RouterSLM, SLMStrategy, Router, RouteRequest from opensmartroute.eval import DatasetCollector, synthetic_rows from opensmartroute.learning import distill_router slm = RouterSLM(reg.all(), seed=0) slm.fit(synthetic_rows(reg.all())) # no data yet: seed prompts scored by capability fit slm.fit(DatasetCollector("data").corpus()) # collected battle datasets (RouteLLM, LMArena) student, report = distill_router(Router(reg), texts) # or copy the whole ensemble print(report.summary()) # accuracy, loss, ECE, Brier, cost/1k slm.save("slm.json") router = Router(reg, strategies=[SLMStrategy(RouterSLM.from_file("slm.json"))]) d = router.route(RouteRequest(text="prove that sqrt(2) is irrational")) print(slm.predict(d.request.text, objective=Objective(cost=2.0))[:3]) # calibrated p, quality, $/1k, utility ``` `SLMStrategy` sits in the ensemble under weight `OSR_WEIGHTS_SLM` and, like the other learners, updates the model online from every `router.learn(Outcome)`. Data comes from three collectors that all degrade gracefully when offline: - `adapters.ModelCatalogue` merges OpenRouter prices and Hugging Face model cards (downloads, likes, `model-index` benchmarks) into `ModelCard`s, turns them into `RouteTarget`s (`targets(min_quality=, max_usd_per_1k=, measured=True)`), prints a cost/quality benchmark and the Pareto frontier. `refresh(leaderboard=True)` adds Open LLM Leaderboard accuracies (IFEval, BBH, MATH, GPQA, MuSR, MMLU-Pro) as measured quality; `apply_quality()` takes the Bradley-Terry win rates that `eval.model_quality` derives from collected battles. Third-party descriptions are risk-scored and replaced when they look like injections. - `eval.DatasetCollector` pulls routing datasets from the Hugging Face datasets-server into a JSONL cache. Ten public sources are on by default: the pairwise battles `routellm-battles` (GPT-4-judged), `arena-55k`, `arena-100k`, `arena-140k` (LMArena human votes, with the code / language / math / hard-prompt tags), `ppe-human`, `webdev-arena`, `mt-bench-human` and `reward-bench`, the per-model judge scores of `ultrafeedback`, and `routellm-gpt4` (RouteLLM's "was Mixtral good enough for this prompt" labels). RouterBench / RouterEval / the raw arena conversations are known but gated: download them with your own Hub token and load the file with `osr collect --file routerbench=routerbench.jsonl --preset routerbench` (`.jsonl` / `.json` / `.csv` in the preset's shape, `--tier` folds model columns onto your targets), and any dataset works through `DatasetSource`. Battles name a moving population of models, so `--tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small` folds every side onto your tiers as it is parsed (`tier_of()`: the `ARENA_TIERS` table, name rules for the 2024-26 generations, then the parameter count in the name) - the RouteLLM question "was the cheaper tier good enough?". Collection is polite and resumable: pages are paced, 429s are retried with backoff, and a run the server cuts short keeps its rows and continues from the recorded offset on the next `osr collect` (`.meta.json`). `rows_from_feedback` turns your `FeedbackStore` into labelled rows. - `adapters.WebKnowledge` searches Hugging Face (text-generation repos, per-keyword fallback) and DuckDuckGo (Brave with `BRAVE_API_KEY`) for new models and datasets; fetches are https-only, byte-capped and page text is risk-scored. `learning.SelfImprover` closes the loop as a champion/challenger cycle: refresh the catalogue, discover, gather evidence, split, train a challenger, calibrate it on the holdout and promote it only when holdout accuracy improves by at least `OSR_SLM_MIN_GAIN` (or loss drops at equal accuracy). Every cycle is appended to a JSONL history. ```bash osr catalogue --cache data/catalogue.json --hf "llama instruct" --leaderboard --markdown # live cost / quality osr collect --cache-dir data --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small osr -t targets.yaml slm train --out slm.json --cache-dir data --report # every *-tiers cache, holdout metrics osr slm predict slm.json "write a python function to parse csv" --cost-weight 1.0 osr -t targets.yaml improve --slm slm.json --cache-dir data --catalogue data/catalogue.json \ --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small \ --search "qwen2.5 instruct" --interval 3600 # daemon; --offline for air-gapped ``` Serve the result: `osr -t targets.yaml --slm slm.json serve`, or in the container `OSR_SLM=/config/slm.json` (Helm `config.slm`). With `OSR_LLM_BASE_URL` (and `OSR_LLM_API_KEY` / `OSR_LLM_MODEL`) pointing at any OpenAI-compatible endpoint - OpenAI, Azure OpenAI, vLLM, Ollama, OpenRouter - every LLM target gets a chat handler from `metadata.model`, so `/v1/chat/completions` with `model: "auto"` routes *and* answers. #### Speed, and what the numbers mean Training is plain SGD over hashed features; with numpy installed (extra `fast`, included in the serve image) the same steps run on dense arrays - 18 000 rows train in under a minute, ten times the pure-Python speed - and prediction projects through a dense mirror of the weights. `OSR_SLM_BACKEND=auto|numpy|python` chooses; both paths seed identically and write the same model file, so a zero-dependency deployment reads a numpy-trained model unchanged. The model carries a learned per-target bias (the base rate a unit-norm dot product cannot express) and trains with inverse-time learning-rate decay (`OSR_SLM_LR_DECAY`), four epochs and `l2` 1e-3 by default - on the 18 000-row mixed corpus the old constant-rate, eight-epoch defaults overfit *below* the constant baseline. Read `--report` against that baseline. Human preference between tiers is noisy: on the ten-source corpus "always the mid tier" is acceptable for 64 % of holdout prompts and the SLM reaches 64-66 % with a holdout cross-entropy of 0.98-1.00 (the constant predictor: 1.03), well calibrated (ECE 0.05-0.08). It is much stronger where the label really depends on the prompt - 94 % on `routellm-gpt4` (baseline 86 %), +12 points on `arena-100k` - and no better than the prior on sources where the prompt carries no signal. That is what a learned routing prior is; the ensemble adds rules, capability, similarity and the bandit on top. A corpus of model battles only knows the models it compared. Scored rows therefore train a softmax over the *scored population* (every target any row scores, plus `OSR_SLM_NULL_TARGETS` learned "none of these" logits that keep it from saturating), so a tool, skill or agent the data never mentions keeps its catalogue embedding instead of being pushed to zero, and `SLMStrategy` abstains on such targets - its confidence is scaled by the share of the request's candidates it has evidence about. Rows that name only an `expected` target (your own feedback, `examples/eval_dataset.jsonl`) contrast against the whole catalogue, which is how the model learns the non-LLM targets once outcomes arrive. #### Where a transformer helps The hashed encoder is a bag of words, bigrams and character grams: it cannot tell "explain this SQL" from "SQL that explains itself". Two opt-in encoders add that context, both persisted in the same JSON file and restored by `RouterSLM.from_file`: - `OSR_SLM_ENCODER=attention` adds `learning.AttentionEncoder`, a pure-Python transformer block (hashed token embeddings, sinusoidal positions, `OSR_SLM_ATTENTION_HEADS` heads of `OSR_SLM_ATTENTION_HEAD_DIM`, residual, attention pooling) to the query encoder: $q = W\,\phi(x) + \operatorname{Attn}(x)$. No dependencies, deterministic, hand-derived gradients, but quadratic in `OSR_SLM_ATTENTION_MAX_TOKENS` - a few milliseconds per prompt and roughly four times the training time - so measure on your holdout (`osr slm train --report`) before switching. On the public arena corpora (six thousand pairwise rows, seventeen tiered targets) it gained two points at 1 500 rows but *lost* five at full size with the default budget: the hashed encoder stays the default, and attention earns its keep on order-sensitive traffic, not on generic chat. - `OSR_SLM_EMBEDDER=sentence-transformers/all-MiniLM-L6-v2` (extra `embeddings`) or `RouterSLM(embedder=callable)` appends a *frozen* pretrained embedding as dense features (`learning.EmbeddingFeaturizer`): `W` then learns a linear head on top of a real transformer while the hashed features keep the surface cues (code fences, length, exact vocabulary). The file records the embedder's name; a custom callable is passed again to `from_file(embedder=...)`. #### Self-operation `osr serve --autopilot` runs the `SelfImprover` inside the server (`learning.Autopilot`): a cycle every `OSR_SLM_AUTOPILOT_INTERVAL_S` seconds, an early one when `learning.DriftMonitor` (Page-Hinkley over the success and quality of every outcome the router learns from, via `Router.observers`) sees the served quality drop, never closer than `OSR_SLM_AUTOPILOT_MIN_GAP_S`. Feedback is joined back to prompts through the router's request memory (`OSR_SLM_AUTOPILOT_REMEMBER` texts), an accepted challenger is hot-swapped into the `SLMStrategy` and written back to disk, a failing cycle is logged and the server keeps serving. `GET /stats` reports `autopilot` (cycles, promotions, drift, last report, next run) and `POST /autopilot/cycle` schedules one now. In the container: `OSR_AUTOPILOT=1` with `OSR_STATE` mounted (Helm `autopilot.enabled`); `--offline` keeps it off the network. Each cycle is traced as an `autopilot.cycle` span with a `learn.improve` event (and `learn.promote` when the challenger wins), so `GET /events?name=learn.*` shows the loop at work. The hosted platform runs the same loop from `OSR_PLATFORM_SLM` / `OSR_PLATFORM_AUTOPILOT` and shows it at `/platform/dashboard/learning` ([PLATFORM.md](https://opensmartroute.ai/docs/PLATFORM.md), section 10). ```bash osr -t targets.yaml --slm slm.json --state state serve --autopilot --autopilot-interval 1800 \ --catalogue data/catalogue.json --source arena-55k --tier frontier=llm-frontier --tier small=llm-small curl -s localhost:8000/stats | jq .autopilot ``` ## 6. Declaring components with decorators Targets, strategies, signal extractors, policy rules, middleware and telemetry can be declared where they live and assembled by a `ComponentRegistry` (details in [SDK.md](https://opensmartroute.ai/docs/SDK.md)): ```python import opensmartroute as osr @osr.tool("weather", domains=["travel"], actions=["lookup"], cost=0.0, examples=["weather in paris"]) def weather(request, **kw): """Current weather for a city.""" # docstring -> target.description return lookup(request.text) @osr.strategy(weight=0.8) # fn(request, signals, candidates) -> {id: score} def geo_affinity(request, signals, candidates): region = request.constraints.region return {t.id: 1.0 if region in t.constraints.regions else 0.4 for t in candidates} @osr.policy_rule # fn(target, request, signals) -> reason | None def business_hours_only(target, request, signals): if target.kind is osr.TargetKind.HUMAN and request.context.get("after_hours"): return "human queues are closed" router = osr.components.router(registry) app = osr.components.builder(registry).with_auto_learning().build() ``` Every tunable (ensemble weights, thresholds, scales) lives in one typed `Settings` object and can be overridden per process (`osr.configure(...)`) or through `OSR__` environment variables; `osr settings` prints the effective values. ## 7. Enterprise builder ```python from opensmartroute.enterprise import (RouterBuilder, CacheMiddleware, TenantMiddleware, LoggingTelemetry, MetricsTelemetry, FileAuditSink) from opensmartroute.security import GuardMiddleware metrics = MetricsTelemetry() app = ( RouterBuilder(registry) .with_rules(rules) # declarative preferences .with_defaults() # capability + similarity + Thompson bandit .with_auto_learning(state_dir=".osr-state") # IRT + Bradley-Terry + LinUCB + Markov lookahead .with_health(latency_slo_ms=3000) # circuit breakers, rate limits, budgets .with_middleware(GuardMiddleware(redact=True), # gadget defence + PII redaction TenantMiddleware({"acme": {"max_cost_per_1k": 0.005, "data_boundary": "private"}}), CacheMiddleware(ttl_s=30)) .with_telemetry(LoggingTelemetry(), metrics) .with_audit(FileAuditSink("audit.jsonl")) .build() ) d = app.route(RouteRequest("Summarize this contract", constraints=RequestConstraints(tenant="acme"))) app.learn(Outcome(d.request_id, d.target.id, success=True, quality=0.9, cost_usd=0.002, latency_ms=1200, domains=d.trace.signals.domains, complexity=d.trace.signals.complexity)) print(metrics.snapshot()["route_latency_ms"], app.health_snapshot()) ``` `with_retrieval(narrow_above=, narrow_to=)` switches to retrieve-then-rank for large catalogues, `with_queue_awareness()` replaces static latency with a queue-aware estimate, `with_shadow(candidate, mode="shadow" | "ab")` compares policies with SPRT, `with_calibration()` adds temperature scaling and conformal candidate sets, `with_fair_share()` enforces tenant weights and `with_state_store()` accepts Redis, SQL, encrypted, versioned or batched stores. Ports (state, telemetry, audit) are ABCs; see [ENTERPRISE.md](https://opensmartroute.ai/docs/ENTERPRISE.md). ## 8. Research-track modules Each module below implements one roadmap item with tests; papers are cited in [RESEARCH.md](https://opensmartroute.ai/docs/RESEARCH.md) and formulas in [MATH.md](https://opensmartroute.ai/docs/MATH.md). ### Signals from a draft or a small model ```python from opensmartroute.signals import (DEFAULT_EXTRACTORS, VerbalisedDifficultySignal, DraftResponseSignal, extract_signals) extractors = [*DEFAULT_EXTRACTORS, VerbalisedDifficultySignal(source=lambda req: small_model_difficulty(req.text)), DraftResponseSignal(drafter=lambda req: cheap_model(req.text))] router = Router(registry, extractors=extractors) ``` `VerbalisedDifficultySignal` accepts numbers, `7/10` or words (`trivial` ... `expert`) from `context["difficulty"]` or a callable and blends them into `complexity`. `DraftResponseSignal` measures hedging, self-corrections, query overlap and length ratio of a cheap draft. ### Response-side uncertainty ```python from opensmartroute.signals import UncertaintyGate, EventTrigger, TriggerRule, response_uncertainty from opensmartroute.strategies import Cascade gate = UncertaintyGate(sampler=lambda req, n: [cheap_model(req.text) for _ in range(n)], n_samples=4) cascade = Cascade(quality_gate=gate, threshold=0.6) # escalate when samples disagree result = cascade.run(req, router.route(req).alternatives) feats = response_uncertainty(samples, p_true=judge_p_true) trigger = EventTrigger([TriggerRule("semantic_entropy", 0.5, "reroute"), TriggerRule("agreement", 0.5, "ask_human", below=True)]) trigger.check(feats) # ["reroute", ...] ``` ### Hidden-state Dirichlet probe ```python from opensmartroute.strategies import HiddenStateStrategy probe = HiddenStateStrategy(state_fn=lambda req: host_model.hidden_state(req.text), targets=["small", "large"], dim=4096) probe.fit(logged_states, logged_winners, epochs=20) # warm start router = Router(registry, strategies=[*default_strategies(), probe]) ``` The probe's confidence is `1 - K/S` (epistemic uncertainty), so unfamiliar states defer to the other strategies. `math.DirichletProbe` is usable on its own with any feature vector. ### Elastic models and token budgets ```python from opensmartroute.strategies import BudgetVariant, TokenBudgetStrategy, expand_elastic for sib in expand_elastic(registry.get("reasoner"), [BudgetVariant(1024, quality=0.7, cost_scale=0.4), BudgetVariant(8192, quality=0.9)]): registry.add(sib) # reasoner@1024, reasoner@8192 router = Router(registry, strategies=[*default_strategies(), TokenBudgetStrategy()]) ``` ### Edge and cloud tiers Tag targets with `metadata: {tier: edge}` or `{tier: cloud}` and add `EdgeCloudStrategy()`. Edge success is a Thompson posterior per complexity bucket, penalised when decode time eats into the latency SLO; the cloud tier is penalised by upload time and excluded when the request's `data_boundary` is `private` or `on_prem` and the target is `public`. ### Auctions `AuctionStrategy(bid_fn=None, value=0.01)` lets each target bid a claimed success probability and a price (`default_bid` derives both from the catalogue). Claims are corrected by each bidder's observed bias; the highest corrected surplus wins and pays the second price. Supply a `bid_fn` for multi-vendor catalogues where providers self-report. ### Protocol selection, aggregation, self-escalation, hand-off ```python from opensmartroute.strategies import ProtocolPolicy, MixtureOfAgents, SelfEscalation, wrap_stream from opensmartroute.learning import HandoffPolicy, MixtureCureModel policy = ProtocolPolicy() # risk / budget -> protocol choice = policy.choose(req, decision, decision.trace.signals, uncertainty=gate(req, draft)) if choice.protocol == "aggregate": res = MixtureOfAgents(k=3, budget_usd=0.05).run(decision, req) # top-k answers, majority vote for o in res.outcomes(req): router.learn(o) monitor = SelfEscalation.for_target(decision.target) # streaming competence posterior streamed = wrap_stream(provider.stream(req.text), monitor) if streamed.escalated: ... # restart on a stronger target handoff = HandoffPolicy(MixtureCureModel(), fallback_target="senior-agent", pins=task_pins) if handoff.step(task_id, teacher_risk): # permanent hand-off target = handoff.target_for(task_id, decision.target.id) handoff.finish(task_id, failed=False) # censored observation ``` ### Personalisation `learning.UserAdaptiveStrategy` reads `context["user_id"]` (or `profile["user_id"]`) and keeps a Beta posterior per user and target, shrunk toward similar users and the global posterior; `observe_user()` warm-starts from logged interactions. `learning.SkillAffinity.relevance(profile, skills)` feeds `retrieval.select_skill_set(relevance=)` so skill sets depend on who is asking. ### Multi-turn conversations ```python from opensmartroute.learning import HistoryTargetStrategy router = Router(registry, strategies=[*default_strategies(), HistoryTargetStrategy(turns=6)]) d = router.route(RouteRequest("can you explain more?", history=chat_history, context={"last_target": "llm-small", "last_failed": True})) ``` The same follow-up message routes differently depending on what the conversation has been about: the strategy learns a logistic model over the joint embedding of the recent history and each target's catalogue embedding, so a target that keeps failing in legal threads but succeeds in coding threads is scored accordingly, and an unseen target is still scored through its description and examples. `last_target` gives the incumbent a small continuity bonus that turns into a penalty when the previous turn failed. ### Discovery beyond text similarity ```python from opensmartroute.discovery import SchemaAwareStrategy, CachePreservingSelector, SkillGraph router = Router(registry, strategies=[*default_strategies(), SchemaAwareStrategy()]) selector = CachePreservingSelector(prefix_size=8, evict_after=5) tools = selector.select(session_id, candidates, needed=[d.target.id]) # prefix-stable tool order graph = SkillGraph(load_skills("skills")) # osr-requires / osr-conflicts ordered, dropped = graph.compose(["report-builder"]) ``` ### MCP server recommendation and semantic-router import ```python from opensmartroute.adapters import recommend_servers, load_semantic_router_config for rec in recommend_servers("open a pull request", server_cards, k=2, constraints=req.constraints, allowed_auth=["none", "token"]): print(rec.server.name, rec.score, rec.rationale) imported = load_semantic_router_config("config.yaml") # vLLM semantic-router router = Router(imported.registry, strategies=[RulesStrategy(imported.rules), *default_strategies()]) ``` ### Energy and carbon `math.EnergyModel` fits Wh per target from measured samples; `math.hardware_profile("h100-sxm")` gives priors when no meter exists. Both convert to gCO2 with a grid factor and feed `wh_per_1k_tokens` / `gco2_per_1k_tokens` on the target's `cost`, which `Objective(energy=, carbon=)` trades against quality. ## 9. Command line Install `osr` with the platform installer (`curl -LsSf https://opensmartroute.ai/install.sh | sh` on Linux and macOS, `irm https://opensmartroute.ai/install.ps1 | iex` in PowerShell) or with `uv tool install 'opensmartroute[yaml,server]'` (add `fast` for numpy-accelerated SLM training); `osr --version` confirms the install. `osr` takes the catalogue with `-t targets.yaml`, optional rules with `-r rules.yaml` and an optional routing SLM with `--slm slm.json`; `--help` on any subcommand lists its options. Grouped by task: ```bash # Sign in and manage access tokens osr login # browser sign-in to the hosted platform osr login --url https://osr.example.com --profile work # another deployment, kept as a profile osr login --url http://router:8000 --token osr_local_... # self-hosted osr serve token echo "$TOKEN" | osr login --with-token # non-interactive (CI) osr whoami # workspace, plan, edition, token source osr token generate # mint an osr_local_ token for osr serve osr token create --name ci # new platform API key (shown once) osr token list # keys of the signed-in workspace osr token revoke osr logout # forget the profile (--all for every one) # Route, quote and inspect osr -t examples/targets.yaml -r examples/rules.yaml route "I want a refund for order #123" --plan osr -t examples/targets.yaml route "Summarise this HR complaint" \ --constraint data_boundary=on_prem --constraint region=eu # hard constraints; --kinds agent,tool osr -t examples/targets.yaml -r examples/rules.yaml route "" --event order.shipped --plan # textless event osr -t examples/targets.yaml estimate "Translate this contract" --monthly 100000 # cost per candidate + projection osr -t examples/targets.yaml targets # the catalogue as the router sees it osr -t examples/targets.yaml stats # feedback statistics per target osr settings --json # effective tunables # Evaluate and audit osr -t examples/targets.yaml -r examples/rules.yaml eval examples/eval_dataset.jsonl --frontier --calibration osr -t examples/targets.yaml audit traffic.jsonl --baseline llm-frontier --monthly 3000000 --markdown # Routing Audit: savings + violations osr -t examples/targets.yaml ope decisions.jsonl # off-policy estimate (IPS/SNIPS/DR) of logged decisions osr -t examples/targets.yaml safety --learned-guard # red-team suite # Learn osr train --from rows.jsonl --out models.json # fit signal models osr catalogue --cache data/catalogue.json --hf "llama instruct" # live model prices + quality osr collect --cache-dir data --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small # Hub datasets osr collect --cache-dir data --file routerbench=routerbench.jsonl --preset routerbench # a gated file you hold osr -t examples/targets.yaml slm train --out slm.json --report # routing SLM (see section 5) osr slm info slm.json # rows, sources, encoder, calibration, fit history osr -t examples/targets.yaml --slm slm.json route "parse this csv" # SLM inside the ensemble osr slm eval slm.json rows.jsonl # SLM accuracy on labelled rows osr slm predict slm.json "parse this csv" --top 3 # what the SLM alone would rank osr slm info slm.json # SLM metadata, targets and size osr -t examples/targets.yaml improve --slm slm.json --cache-dir data # self-improvement cycle # Serve and integrate osr -t examples/targets.yaml -r examples/rules.yaml serve # http://127.0.0.1:8000/docs (open, local dev) osr -t examples/targets.yaml serve --generate-token # prints an osr_local_ token; all routes need it osr -t examples/targets.yaml serve --require-auth --token "$TOKEN" # production: refuse to start without a token osr -t examples/targets.yaml mcp # MCP server over stdio for an IDE osr mcp --remote # stdio bridge to the platform you signed in to osr mcp --url https:///mcp --api-key osr_... # ...or an explicit endpoint and key osr mcp-manifest tools.json --key-env OSR_MCP_KEY --out signed.json # sign an MCP tools/list (--verify to check) # Manifests, stacks and skills osr validate examples/ocm # Open Capability Manifests osr -t examples/targets.yaml export-ocm --out manifests/ # catalogue -> OCM manifests osr stack init --out stack.yaml # starter declarative stack osr stack validate examples/stack.yaml # check (imports resolved) osr stack plan examples/stack.yaml --against deployed.yaml # diff against what runs today osr stack apply examples/stack.yaml --route "refund order 42?" # build the router, route once osr stack apply registry://support-desk@1.0.0 --registry $OSR # marketplace template osr skills # validate SKILL.md packages ``` `osr login` without `--token` runs the OAuth 2.0 device authorization grant against the hosted platform (community or enterprise edition alike): the CLI prints a short code, opens `/platform/cli/authorize` in the browser, and after you approve it there the platform mints a workspace API key (`osr_live_...`) for that machine. Credentials are stored per profile in `~/.config/opensmartroute/credentials.json` (`%APPDATA%\opensmartroute` on Windows, `OSR_CONFIG_DIR` overrides); flags, then `OSR_API_URL` / `OSR_API_KEY`, then the saved profile decide which one a command uses. A self-hosted `osr serve` has no accounts: `osr token generate` (or `serve --generate-token`) mints an `osr_local_...` token, the server accepts it as `Authorization: Bearer` or `X-API-Key` on every route except the health, metrics, `/whoami` and OpenAPI paths, and `OSR_SERVER_AUTH_TOKENS` / `OSR_SERVER_REQUIRE_AUTH` configure the same thing from the environment (see :class:`ServerSettings` in [SDK.md](https://opensmartroute.ai/docs/SDK.md#settings)). The Routing Audit replays logged traffic (`{"text": ..., "model": ..., "cost_usd": ...}` per line) through the catalogue and reports savings against a baseline target, policy violations the logged route would have committed, and the route mix; `--min-savings` makes it a CI gate. In production the same numbers come from `RouterBuilder.with_savings(baseline="max")` and `ep.savings.report().to_markdown()`. ## 10. Routing latency Routing is pure Python on the request path. Measured with `python scripts/bench.py --n 200 --scale` on the 16-target example catalogue, CPython 3.11, x86-64 laptop: | Configuration | p50 | p95 | p99 | |---|---|---|---| | signals + policy + rules + capability + similarity + bandit | 5.8 ms | 6.2 ms | 6.5 ms | | + IRT + preference + LinUCB + Markov + health | 8.6 ms | 12.9 ms | 17.8 ms | | + LLM judge (only when `confidence < escalate_llm_judge_below`) | + one provider round-trip | n/a | n/a | Catalogue size: 64 targets route in 14.5 ms p50 and 256 in 16.7 ms with `RouterBuilder.with_retrieval(narrow_above=32, narrow_to=24)` (versus 36 ms and 142 ms scoring every target); 1024 targets route in 24 ms. Cost is linear in `scored targets x strategies`; signal extraction is under 1 ms and independent of catalogue size. `RouteDecision.trace.elapsed_ms` records the routing time per request, `MetricsTelemetry` exposes `route_latency_ms` percentiles and the OpenTelemetry adapter emits the `osr.route_latency_ms` histogram. Full tables in [ARCHITECTURE.md](https://opensmartroute.ai/docs/ARCHITECTURE.md#performance-envelope). ## 11. Repository layout ``` src/opensmartroute/ core/ types (RouteTarget, RouteRequest, RouteDecision, Outcome, ...), registry signals/ lexicon extractors, task ontology, learned hashed models, verbalised difficulty, draft-response and response-uncertainty features policy/ hard-constraint filter strategies/ rules, capability, similarity, bandit, llm_judge, cascade, task_table, defer, progress, probe (Dirichlet), elastic, edge, auction, protocol, escalation, aggregate learning/ IRT, preference, LinUCB, Markov, AutoLearner, cold start, task credit, personal (user adaptation, skill affinity), handoff (mixture cure), multiturn (history-target joint embeddings) math/ bandits, irt, preference, markov, calibration, estimators, decision, dirichlet, energy realtime/ circuit breaker, rate limit, budget, health enterprise/ RouterBuilder, EnterpriseRouter, middleware, telemetry, audit, stores, ops security/ InputGuard, learned gadget detector, injection risk, redaction, limits, provenance adapters/ OpenAI-compatible client, harnesses, MCP (tools + server recommendation), A2A, frameworks, personas, SKILL.md, semantic-router import, embeddings, OpenTelemetry eval/ harness, baselines, robustness, frontiers, off-policy evaluation, dataset presets discovery.py schema-aware matching, cache-preserving selection, skill graph retrieval.py BM25 + dense retrieval, reciprocal-rank fusion, skill-set selection, meta-tools execution.py plan executor sdk.py decorator SDK and plugin discovery settings.py typed, environment-overridable tunables config.py load_targets / load_rules (JSON, YAML) feedback/ outcome store and preference export router.py orchestrator and plan builder aio.py AsyncRouter server.py FastAPI app and OpenAI-compatible proxy cli.py osr command deploy/ Dockerfile, entrypoint, Helm chart examples/ targets.yaml, rules.yaml, skills/, eval_dataset.jsonl, demos .claude/skills Agent-Skills packages for coding agents; also routable skill targets tests/ unit and integration tests, public API snapshot, docs coverage gate (test_docs.py) scripts/ bench.py, release.py, brand_build.py, api_reference.py, exit_criteria.py docs/ GUIDE, REFERENCE (generated), RESEARCH, ARCHITECTURE, MATH, ENTERPRISE, SECURITY, SDK, ROADMAP, BRAND ```