User guide

Everything the README leaves out: the full quick-start sequence, running plans end to end, wiring real providers, the enterprise builder, the research-track modules, routing latency and the repository layout. API reference material lives in SDK guide (decorator SDK, components, settings) and Python API reference (generated; every module and exported name); design in Architecture.

Contents

1. Targets
2. Routing
3. Executing a plan
4. Real providers
5. Learning from outcomes
6. Declaring components with decorators
7. Enterprise builder
8. Research-track modules
9. Command line
10. Routing latency
11. Package layout

1. Targets

Every routable thing is a RouteTarget with the same contract: an id, a kind, declared capabilities, cost, latency_ms, a quality_prior, optional examples, instructions and a handler. Adding a target never requires retraining; it is routable from its declaration and learns from its outcomes.

# examples/targets.yaml (excerpt)
- id: coding-agent            # agent harness: repo access, tests, review loop
  kind: agent
  capabilities: { domains: [coding], actions: [code_generation, code_review, action],
                  min_complexity: 0.3, supports_tools: true, tags: [autonomous] }
  cost: { usd_per_1k_tokens: 0.008 }
  latency_ms: 6000
  examples: ["Refactor this class to remove the circular import and run the test suite."]

- id: skill-sql               # deterministic skill; `instructions` is its SKILL.md-style body
  kind: skill
  capabilities: { domains: [data_analysis, coding], actions: [code_generation, extract] }
  cost: { usd_per_1k_tokens: 0.001 }
  instructions: "Always qualify table names with the schema; never SELECT *."

- id: persona-legal-counsel   # never primary; layered on whichever model/agent wins
  kind: persona
  primary: false
  capabilities: { domains: [legal], actions: [reasoning, summarize, qa] }
  instructions: "You are cautious in-house counsel. Cite the clause, flag jurisdiction risk."

Catalogues load from JSON or YAML (config.load_targets), from MCP tools/list payloads (adapters.tools_from_mcp), A2A agent cards (adapters.agent_from_card), SKILL.md directories (adapters.load_skills), persona folders (adapters.load_personas) and vLLM semantic-router configurations (adapters.load_semantic_router_config).

latency_ms is the time to an answer whatever its length; a target that answers token by token at a known pace - a self-hosted model on CPU at five tokens a second, say - also declares latency_per_token_ms (200 there), and the router then expects latency_ms + latency_per_token_ms x expected output tokens for each request (RouteTarget.estimated_latency_ms): the slow box keeps the one-line classification and loses the long draft, the hard max_latency_ms constraint, the health strategy's SLO fit and the pre-call estimate all read the same figure. The hosted platform fills both from what it measures (see docs/PLATFORM.md, Self-hosted models).

Agent skills for coding assistants

OpenSmartRoute ships Agent-Skills packages (SKILL.md, agentskills.io format) that teach a coding agent how to use and extend OpenSmartRoute. The product-facing ones - osr-routing-catalogue, osr-decorator-sdk, osr-enterprise-builder, osr-evaluation, osr-security-hardening, osr-integrations and osr-deploy-serve - are published on the documentation site under /docs/skills/<name> and on the marketplace; a coding agent loads a folder of such packages, and they are also valid routing targets: the same loader the SDK uses (load_skills) validates them and the router fills the plan's skill slot with the right one.

osr skills <folder>                                  # validate + list the SKILL.md packages in a folder
osr -t examples/targets.yaml --skills <folder> route "Create a JSONL eval dataset and gate CI on accuracy" --plan
# plan: llm:llm-mid -> skill:osr-evaluation

2. Routing

from opensmartroute import Router, RouteRequest, RequestConstraints, Objective

router = Router(registry)

# plain text
d = router.route("Prove that sqrt(2) is irrational, step by step.")
print(d.target.id, f"{d.confidence:.2f}")
print(d.trace.explain())  # per-strategy scores and rationales

# hard constraints and a per-request objective
req = RouteRequest(
    text="Summarize this patient intake note.",
    constraints=RequestConstraints(region="eu", data_boundary="private", max_cost_per_1k=0.005),
    objective=Objective(quality=1.0, cost=0.5, latency=0.1, quality_floor=0.6),
)
d = router.route(req)
print(d.trace.policy_rejections)  # {'llm-frontier': 'cost 0.015 > budget 0.005', ...}

# a composed plan: persona -> skill -> model (+ the tools the model may call)
d = router.route("Review this NDA clause for GDPR liability.", plan=True)
for slot in d.plan.slots:
    print(slot.role, slot.target.id, f"{slot.confidence:.2f}")

Policy runs before scoring and is never traded off: enable flags, allow / deny lists, kind, region, data boundary, PII, tenant, cost / latency / token SLOs, modality, language and jailbreak risk. Strategies then score the survivors and a confidence-weighted ensemble produces a quality estimate; utility is w_q * quality - w_c * norm(cost) - w_l * norm(latency) with a hard quality_floor. With a ConformalCalibrator, RouteDecision.candidate_set is the smallest set with the requested coverage and abstain is set when no target is safe enough.

What the signals read out of a request is an ontology of twenty-five task types in seven families (signals.TASK_TYPES; osr route prints the detected domains and actions on its signals: line): the transformations (summarisation, translation, rewriting - which includes the editing help the extension and the desktop bar send), analysis (extraction, reading a scanned or photographed document - document_intelligence, classification, factual questions, web_search for a lookup on the live web, embeddings), reasoning (explanation, maths, planning, observability - the root cause of an incident from logs, traces, metrics and alerts), coding, generation (creative writing, images, speech), interaction (chit-chat, support, escalation) and action (tool_action, browsing - open the page, log in, fill the form, click through, download - and desktop_action - rename, move, paste, type into the window in front). Each type maps to the actions targets declare (search, ocr, browse, action ... - signals.ACTION_LEXICON) and requests carry up to three domains (signals.DOMAIN_LEXICON: coding, finance, legal, medical, observability, science, education ...). A human target is scored as a last resort: unless the request asks to escalate its capability fit is halved (capability.human_fallback_penalty, OSR_CAPABILITY_HUMAN_FALLBACK_PENALTY), so a queue with a high quality prior never wins a plain question on that prior alone.

A generative judge can be consulted only when the ensemble is unsure:

from opensmartroute import LLMJudgeStrategy

router = Router(registry, llm_judge=LLMJudgeStrategy(llm=lambda prompt: my_llm(prompt)), escalate_llm_judge_below=0.6)

3. Executing a plan

route() decides; run() decides and executes the whole plan: skill pre-processors run first, persona and skill instructions are composed into the system prompt, the primary target is called, and an Outcome is recorded for every participant so each one learns from the result.

from opensmartroute.adapters import (
    CallableHarness,
    HTTPHarness,
    SubprocessHarness,
    chat_handler,
    harness_handler,
    load_skills,
)

registry.get("llm-frontier").handler = chat_handler(openai, model="gpt-4o")
registry.get("coding-agent").handler = harness_handler(
    SubprocessHarness(["my-coding-agent", "--repo", "."], system_flag="--system")
)
registry.get("research-agent").handler = harness_handler(
    HTTPHarness("https://agents.internal/research/run", api_key_env="AGENT_TOKEN")
)
registry.get("skill-sql").handler = lambda req: lookup_schema(req.text)  # runs BEFORE the model
for skill in load_skills("examples/skills"):  # SKILL.md folders
    registry.add(skill)

res = router.run("Is this indemnification clause enforceable in California?", task_id="ticket-812")
res.text  # the primary target's answer
res.steps  # [persona:persona-legal-counsel, llm:llm-frontier] with per-step latency
res.system_prompt  # composed persona + skill instructions
res.outcomes  # one Outcome per participant, role="persona"/"skill"/None, shared task_id

A skill handler may return a RouteRequest (rewrite the request), a str (attached as context["skill_output"] and shown to the model) or None (instructions only). Async handlers work through AsyncRouter.run(); EnterpriseRouter.run() adds health, budgets and telemetry. See examples/end_to_end.py and SDK guide.

Tools the model may call

A model or agent plan also carries tool slots: the tool targets (MCP tools, @tool handlers) whose ensemble quality for this request clears RoutingSettings.tool_quality_floor (default 0.5; measured on a live catalogue, a tool whose examples match the request scores 0.53-0.75 and an unrelated one stays below 0.47 - the model still decides whether to call it, so an offered-but-unused tool only costs its spec tokens), at most tool_slots of them (default 2; Router(reg, tool_slots=0) or OSR_ROUTING_TOOL_SLOTS=0 turns the slot off). run() offers them to the model as OpenAI function-calling specs (execution.tool_spec: the tool's MCP input_schema, or a single input string for handler-only tools) through context["openai"]["tools"]. When the model calls one, the tool target's handler runs (context["tool_arguments"] = the parsed arguments, or the input argument as the request text), the result goes back as a tool message and the model is asked again - up to RoutingSettings.tool_rounds rounds - so the caller receives the finished answer. A request that brings its own tools keeps its own tool protocol (nothing is added), and a call to a tool the router did not offer is handed back untouched. res.steps shows tool … offered / called entries and every tool that was called is credited with a role="tool" outcome, so the learners find out which tools actually help.

A skill or tool slot is only filled when the request names the companion, whatever its quality score: an action it declares (a translation for the translate skill), one of its domains, a need for fresh facts for a tool tagged web, or a text close to its own examples (RoutingSettings.companion_similarity_floor, default 0.7 on the similarity strategy's score). A greeting or a bare word names nothing and gets a model and, at most, a persona - a companion is a request of its own, and its record on other requests is no reason to run it here. The trace shows a slot left empty for this reason as plan.slot … "nothing in the request names one".

res = router.run("How many orders shipped last week?")
[(s.role, s.target_id, s.note) for s in res.steps]
# [('tool', 'warehouse:sql_query', 'offered'), ('tool', 'warehouse:sql_query', 'called'), ('llm', 'llm-frontier', 'primary')]

4. Real providers

The bundled client is stdlib only and speaks the OpenAI chat and embeddings API, which covers OpenAI, Azure OpenAI, vLLM, Ollama, LiteLLM, OpenRouter, Groq and Mistral.

from opensmartroute import Router, LLMJudgeStrategy
from opensmartroute.adapters import OpenAICompatClient, chat_handler, embedder, judge_fn
from opensmartroute.strategies import CapabilityStrategy, SimilarityStrategy

openai = OpenAICompatClient()  # OPENAI_API_KEY from env
ollama = OpenAICompatClient("http://localhost:11434/v1", api_key="ollama")

registry.get("llm-frontier").handler = chat_handler(openai, model="gpt-4o")
registry.get("llm-small").handler = chat_handler(ollama, model="llama3.1")

router = Router(
    registry,
    strategies=[CapabilityStrategy(), SimilarityStrategy(embedder=embedder(openai, model="text-embedding-3-small"))],
    llm_judge=LLMJudgeStrategy(judge_fn(openai, model="gpt-4o-mini")),
    escalate_llm_judge_below=0.5,
)

d = router.route(req)
result = router.execute(d, req)  # ChatResult(text, prompt_tokens, completion_tokens, latency_ms)
res = router.run(req)  # route + execute the full plan

adapters.sentence_transformers_embedder() and adapters.OpenTelemetryTelemetry are lazy imports behind the embeddings and otel extras. osr serve exposes a FastAPI app with an OpenAI-compatible /v1/chat/completions proxy that routes, executes and learns.

adapters.AnthropicClient speaks the Messages API natively behind the same chat / stream_chat surface and places prompt-cache breakpoints on its own when the caller sent none: on the tool list, on a system prompt long enough to be cached and on the conversation prefix (the user turn before the current one), so an agent loop that resends the same tools and history pays Anthropic's cache-read rate for them instead of the input rate; prompt_cache=False turns it off, markers the caller placed are respected, and the cached and written token counts come back in usage (prompt_tokens_details.cached_tokens, cache_creation_input_tokens). adapters.MockClient is the provider for tests and demos: deterministic answers without a network, streaming, embeddings and failure on cue (OSR_LLM_BASE_URL=mock:// gives every LLM target of osr serve one).

The OpenAI-compatible proxy

osr serve (opensmartroute.server.create_app) is a drop-in base URL for the official OpenAI SDKs: point a client at http://router:8000/v1 and keep the rest of the code.

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="osr_local_...")  # the token from `osr token generate`
client.chat.completions.create(model="osr/auto", messages=[{"role": "user", "content": "..."}])

• Model names. osr/auto (or auto) lets the router choose, osr/auto:cheap|fast|quality|private|agentic|llm presets the objective, constraints or kinds, and any target id pins that target. models: ["llm-a", "llm-b"] names the candidates: the router picks among them and, when the chosen handler fails, falls back down the list (osr.fallbacks: false turns that off; OSR_SERVER_MAX_FALLBACKS caps the re-routes). Every failed attempt is a failed outcome for the learners and health breakers; the response's opensmartroute.fallback_from lists them.
• Streaming. stream: true answers with chat.completion.chunk server-sent events: a role chunk, one content delta per provider token (relayed while the upstream is still generating), tool-call deltas, then a final chunk with finish_reason, usage and the opensmartroute metadata, then data: [DONE]. Errors before the first token keep their HTTP status (404 unknown model, 422 no admissible target, 503 provider down); the response headers X-OSR-Target and X-OSR-Request-Id name the decision. A handler that does not stream is chunked after it returns (OSR_SERVER_STREAM_CHUNK_CHARS characters per delta). The response is held for the first token for at most OSR_SERVER_STREAM_FIRST_TOKEN_WAIT_S (15 s); a model still silent after that - a cold load, a long prompt on a CPU, a queue - gets its stream opened anyway and an SSE comment (: keepalive) every OSR_SERVER_STREAM_KEEPALIVE_S (10 s) until it speaks, so an ingress idle timeout never cuts a slow answer and the client sees the request is alive; a failure after that point arrives in-band as an error event with the status it would have had.
• Tools and passthrough fields. tools, tool_choice, response_format, stop, n, seed, logprobs, top_logprobs, parallel_tool_calls and user reach the target's handler through request.context["openai"]; chat_handler forwards them to the upstream untouched. Assistant tool_calls and tool messages are kept in the conversation, and the response carries message.tool_calls, message.refusal and the real finish_reason (tool_calls when the model called tools). A request that sends tools only routes to LLM targets that declare capabilities.supports_tools: true.
• Prompt compaction. Before any model or agent handler runs, execution compacts the prompt (opensmartroute.compress, OSR_COMPRESSION_*, on by default): trailing whitespace, zero-width characters and runs of blank lines go, consecutive duplicate lines of 24+ characters (pasted logs, repeated boilerplate) collapse to one, and the same is applied to the history and the composed system prompt. Fenced code blocks are kept byte-for-byte and no line is rewritten, so the model reads the same words with fewer tokens. OSR_COMPRESSION_HISTORY_BUDGET_TOKENS adds a conversation budget: the oldest turns are dropped until the history fits, never splitting an assistant tool_calls from its tool results and never touching the newest OSR_COMPRESSION_KEEP_LAST_TURNS turns. The estimated saving is reported as tokens_saved in the response metadata, ExecutionResult.tokens_saved and a compress step (-N tokens); OSR_COMPRESSION_ENABLED=0 turns it off. Tool, workflow and human targets receive the request verbatim. Token estimates come from estimate.estimate_tokens, calibrated against billed gpt-4.1 prompt tokens on prose, chat, logs, code, JSON, Markdown, URLs, numbers, German and Japanese (mean ratio 1.02, every type within about 20 %). examples/precall_check.py re-measures the whole story against a live endpoint - routing, the tool slot, the compaction - and prints real tokens, cost and correctness next to an always-frontier baseline.
• Multi-part content. Text parts of the current turn join into the routed text; image_url, input_audio and file parts land in context["attachments"] (and images / audio / files for the modality signal, so an image only routes to a target whose capabilities.modalities include image). chat_handler sends them on as multi-part content; earlier turns keep their text.
• Embeddings. POST /v1/embeddings (input: str | list[str]) routes osr/auto among LLM targets tagged embeddings (capabilities.tags) or with metadata.embeddings: true; a target id pins one. The target's handler receives context["embeddings_input"] and returns {"embeddings": [[...]], "model": ..., "prompt_tokens": n} - adapters.openai_compat.embed_handler(client, model) does exactly that for any OpenAI-compatible endpoint, and OSR_LLM_BASE_URL attaches it automatically to embeddings-tagged targets (the example catalogue ships one, embed-small). Embeddings-only targets are never chosen for a chat completion.
• Other dialects. POST /v1/responses (the OpenAI Responses API, opensmartroute.responses) and POST /v1/messages (the Anthropic Messages API, opensmartroute.anthropic) are translated onto the same chat path, so the OpenAI Agents SDK, the Anthropic SDKs and Claude Code use the router as their base URL unchanged: content blocks, tools, tool results, stop sequences and the streaming events of each dialect are mapped both ways, the opensmartroute block rides along, and POST /v1/messages/count_tokens answers from the router's token estimate without calling a provider.
• GET /v1/models lists osr/auto, its variants and every executable primary target: those with a handler, plus instructions-only skills and personas when a model with a handler can run them.
• Async handlers. Handlers may be async def; the server runs routing and execution in worker threads and drives coroutine handlers to completion, so sync and async targets mix freely. Handler keywords such as on_delta are only passed to handlers whose signature accepts them (on_delta is also in request.context).
• Token scopes. An entry in --token / OSR_SERVER_AUTH_TOKENS may be <token> (full access), <token>:ro (read-only: every GET, /route, /v1/* and /mcp, but 403 on /feedback and /autopilot/cycle) and carry an expiry :exp=<unix seconds> (401 once passed). GET /whoami reports the scope of the presented token.

The same building blocks are public in the SDK: execution.execute_with_fallback (route -> execute -> re-route on failure), execution.ChatStream / completion_chunks (OpenAI chunk events from any handler) and server.proxy_request / proxy_response / embeddings_response (the body conversions).

This is tested against the real clients, not a look-alike: the test suite starts an osr serve process on the example catalogue and drives it with the official openai SDK (models, chat, streaming, a tool call, embeddings, a rejected token) and the official mcp SDK (POST /mcp over Streamable HTTP, and osr mcp over stdio).

5. Learning from outcomes

from opensmartroute import Outcome

router.learn(
    Outcome(
        request_id=d.request_id,
        target_id=d.target.id,
        success=True,
        quality=0.9,
        cost_usd=0.002,
        latency_ms=1800,
        domains=d.trace.signals.domains,
    )
)

One Outcome fans out to every learner: Thompson bandits per domain, IRT ability and difficulty, Bradley-Terry preferences, LinUCB on the signal vector, the Markov / MDP lookahead, the task table and the personalisation posteriors. Outcome.task_id and step join delayed, task-level rewards to the routing decisions that produced them (learning.TaskCredit, learning.TaskPins). Page-Hinkley drift detection flags targets whose success rate degrades; every learner forgets with a configurable decay and can be merged across replicas (learning.merge_learners). Each learn() also records a learn.outcome event on the request's trace and, with with_audit(sink, outcomes=True), an outcome record in the audit chain, so a decision and its result stay joined by request_id (Tracing and observability; on the hosted platform POST /api/v1/feedback does the same).

Offline warm starts: learning.warm_start_from_matrix() replays a (prompt, {target: reward}) matrix, learning.warm_start() copies posteriors from the nearest declared neighbours and osr train --from rows.jsonl --out models.json fits the hashed signal models.

Implicit feedback: learning from the request and the response

Almost no traffic calls /feedback, so the router also reads the evidence it already has (learning.implicit, settings group implicit):

• The answer itself. Whenever execute() (and with it osr serve's /v1/chat/completions, /route
  • /execute, the MCP tool) runs an LLM, agent, skill or persona target whose handler reports no quality of its own, learning.assess_response judges the response against the request: a refusal (message.refusal, finish_reason: content_filter, "I cannot ..."), an empty answer, a truncated one (finish_reason: length), hedging and self-correction phrases, a coding request answered without any code, and a call slower than OSR_IMPLICIT_SLOW_FACTOR times the target's typical latency all lower the outcome's quality (OSR_IMPLICIT_REFUSAL_QUALITY, OSR_IMPLICIT_TRUNCATED_QUALITY, ...). An answer with nothing wrong gets OSR_IMPLICIT_NEUTRAL_QUALITY (0.7) - good enough to learn from, not the 1.0 that a bare success=True used to mean, which had every learner reinforcing whatever the router picked simply because the call returned. Outcome.source records the verdict (response:refusal, response:ok, ...).
• The next turn. With context["session_id"] (the proxy's session_id / metadata.session_id), learning.ReactionLearner reads each new turn of a session as feedback on the previous one: a complaint ("that's wrong", "try again", "no, I meant ..."), a near-identical re-ask (token Jaccard >= OSR_IMPLICIT_RETRY_SIMILARITY, a retry) or thanks become an outcome for the previous request's target (reaction:negative 0.2, reaction:retry 0.3, reaction:positive 0.95) within OSR_IMPLICIT_REACTION_WINDOW_S. The router feeds it through learn_correction(), so it reaches every learner - including the SLM's online step, which now accepts a later correction of a request it already saw.
• Into the self-improvement loop. eval.rows_from_feedback reconciles several outcomes for one request by trust (reaction > explicit feedback > response assessment) and drops response:ok outcomes - "it did not fail" says nothing about which target was best, and training on it would only teach the SLM its own policy. A row that scores a single target badly (learning.negative_only) trains as "not this one" - a push towards every other target - in both objectives; it used to become a full positive for the target it complained about, because a softmax over one score is 1.0.

OSR_IMPLICIT_ENABLED=false returns to plain success/failure outcomes; OSR_IMPLICIT_REACTIONS=false turns off the next-turn reading. A handler that returns its own quality (a judge, an exact-match check, an agent harness) is always believed over the assessment.

The routing SLM and the self-improvement loop

learning.RouterSLM is a small routing model in one JSON file: a hashed dual encoder (SLMSettings.dim x feature_dim), a two-stage calibrator and a snapshot of every target's price, latency and quality prior. It has no dependencies, predicts in microseconds and can be trained from any EvalRow source - labelled datasets, your own outcomes, synthetic ontology prompts or the full router's decisions (distill_router, which compresses rules + capability fit + learners + judge into one model). Calibration fits the softmax temperature on the validation rows and, from 40 rows up, also a monotone (isotonic) map from the tempered top probability to the observed hit rate, shrunk toward the tempered value while the calibration set is small (n / (n + 100)): a single temperature cannot fix a model that is right at 0.6 as often as at 0.9, and the ensemble weighs the SLM by the probability it reports. Measured out of sample on the example catalogue (calibrate on half the holdout, score the other half, ~120 rows): ECE 0.07-0.14 -> 0.03-0.09 and Brier down a third for both encoders, at unchanged accuracy; the ranking never changes (the winner stays the winner).

from opensmartroute import RouterSLM, SLMStrategy, Router, RouteRequest
from opensmartroute.eval import DatasetCollector, synthetic_rows
from opensmartroute.learning import distill_router

slm = RouterSLM(reg.all(), seed=0)
slm.fit(synthetic_rows(reg.all()))  # no data yet: seed prompts scored by capability fit
slm.fit(DatasetCollector("data").corpus())  # collected battle datasets (RouteLLM, LMArena)
student, report = distill_router(Router(reg), texts)  # or copy the whole ensemble
print(report.summary())  # accuracy, loss, ECE, Brier, cost/1k
slm.save("slm.json")

router = Router(reg, strategies=[SLMStrategy(RouterSLM.from_file("slm.json"))])
d = router.route(RouteRequest(text="prove that sqrt(2) is irrational"))
print(slm.predict(d.request.text, objective=Objective(cost=2.0))[:3])  # calibrated p, quality, $/1k, utility

SLMStrategy sits in the ensemble under weight OSR_WEIGHTS_SLM and, like the other learners, updates the model online from every router.learn(Outcome). Its scores and confidence are read from the SLM's calibrated probabilities (the fitted temperature), the same numbers osr slm predict prints.

Slot models: which persona, skill or tool suits the request

The primary encoder answers one question - which target answers. A plan asks more: which persona, which skill, which tools. Those decisions get their own small encoders, the slot models (RouterSLM.slots, one per role in OSR_SLM_SLOT_ROLES - persona, skill, tool), trained on the outcomes credited to that slot (Outcome.role; eval.rows_from_feedback(..., role="skill")) and saved in the same file, so a promotion or a rollback moves them with the primary. SLMStrategy reads the slot the router is filling (signals.extra["role"]) and votes with that role's model - a marketplace skill with a record is learned exactly like a catalogue one - and abstains for a role it has no model for; slot outcomes never enter the primary corpus (a persona is not an answer), and the online update of a planned request keeps the primary's and each slot's decision apart. The improve cycle trains a slot once it has OSR_SLM_SLOT_MIN_ROWS (20) rows and reports each role under slots (rows, accuracy against the constant predictor, skill); RouterSLM.info()["slots"] and osr slm info show what is served.

rows = rows_from_feedback(feedback, texts, role="persona")  # the persona slot's outcomes, prompt by prompt
report = slm.fit_slot("persona", rows)  # its own encoder and calibration
slm.slots["persona"].predict_proba("review this contract clause", slm.slot_candidates("persona"))

Conversation context

A follow-up says nothing about the task on its own: "why?", "shorter please", "and what about the edge cases?" after a proof, a translation or a code review each belong with the target that served the turn before. The SLM reads such a message together with the last OSR_SLM_CONTEXT_TURNS (2) user turns of RouteRequest.history (learning.routing_text: "<message> || <previous user turn> ...", at most OSR_SLM_CONTEXT_CHARS (400) of context, newest first) - SLMStrategy does it for every request, RouterSLM.predict*(text, history=...) for a direct call, EvalRow.history for training and evaluation rows. Only a follow-up gets context (learning.is_followup): a message of at most OSR_SLM_CONTEXT_MAX_WORDS (3) words, or one that opens with a continuation (learning.slm.FOLLOWUP_RE: "and", "now", "instead", "try again", "can you elaborate" ...) and stays under OSR_SLM_CONTEXT_MAX_TEXT_CHARS (240). "Translate this paragraph into French" after a coding chat names its own task and routes on its own words. Context words hash under their own ctx: namespace (signals.models.CONTEXT_MARK, weight CONTEXT_WEIGHT 0.7; words and bigrams only, in the attention block too), so "python" said earlier and "python" said now are different features and a model trained without conversations reads unchanged. eval.synthetic_followup_rows builds follow-up conversations (and, with switches=, topic switches) from any labelled rows for training; explain() reports the text the encoder actually read and names context features ctx:w:....

The online update is reinforcement learning

An outcome is a reward for the routed target, not a label: the served model learns from it with a REINFORCE step on its own softmax (OSR_SLM_ONLINE_POLICY=reinforce, the default) - the reward's advantage over the target's running mean reward (OSR_SLM_REWARD_BASELINE_RATE, the baseline starts at 0.5) raises or lowers the target's log-probability, so the first good answer of a new target is a surprise worth learning from and the hundredth is not, and a bad outcome moves probability to the other candidates in proportion to what the model already gives them. Because the router picked the target, not the SLM, the step is importance-weighted by p_SLM(target) / propensity (capped at OSR_SLM_IPS_CLIP, 5): Router.learn fills Outcome.propensity from its request memory when the caller did not, the same logged propensity eval.ope uses, so traffic routed by the whole ensemble trains the SLM without bias. OSR_SLM_ONLINE_POLICY=supervised is the plain label update (a good outcome is a positive, a bad one spreads the label over the others). SLMStrategy.status() reports online_policy and last_advantage; the same REINFORCE with a learned baseline also drives the stand-alone learning.PolicyGradientStrategy over hashed features.

The bundled model

The package ships a trained model, so an on-prem or air-gapped install routes with a learned SLM before it has seen a request: RouterSLM.bundled() (or the path "bundled" anywhere a model file is accepted - --slm bundled, osr slm info bundled, RouterBuilder.with_slm(), OSR_SLM=bundled in the container, OSR_PLATFORM_SLM=bundled on the platform). It is an attention-encoder model trained on the reference catalogue (examples/targets.yaml, 27 targets) and labelled by the reference ensemble - rules, capability fit, cost and quality priors together - rather than by the capability fit alone, whose near-ties between generalists had sent a quarter of everyday prompts to the on-prem model. Its corpus is the ontology's seed templates over three seeds plus the everyday phrasings of every task type (signals.corpus.NATURAL_PROMPTS: terse chat, office English, pasted context, typos, a second language), the multicloud benchmark prompts routed by the ensemble, one follow-up conversation per row, the 70 labelled example rows with their follow-ups - weighted ten times so ground truth outvotes near-tied synthetic labels - and 64 bare follow-ups with a flat label; a 128-token attention window, learning rate 0.08. The labelled rows cover what the browser extension's writing help, the desktop bar and the newsroom's writer actually send (grammar fixes, shorter drafts, e-mail replies, subject lines, voice memos, read-aloud, embeddings, release notes) and what the new agents and tools are for (search the web for..., OCR this receipt, root-cause this alert, open the portal and download the statement, rename the files in this folder). It is measured two ways. The probe holdout - a fourth seed's templates and a fifth of the everyday phrasings the probe never trained on - reads 0.77 (follow-ups 0.76, topic switches 0.77; a template-only holdout had read 0.97 of a model that routed unseen wording at 0.49). The held-out set (examples/eval_heldout.jsonl, 117 everyday asks naming every target, in none of the training data) reads 0.73 for the model alone, and the ensemble with it 0.82 against 0.84 without - which is why OSR_WEIGHTS_SLM defaults to 0.5 and the strategy reports its calibrated top probability as its confidence, scaled by that held-out skill (meta.heldout). On the labelled set the model alone routes all 70 rows acceptably and the ensemble with it 69 of 70; ~1.8 MB as an int8 file. The learning rate matters more than the seed: at the attention encoder's default 0.2 the holdout swung between 0.71 and 0.88 from one epoch to the next and the file depended on where early stopping landed, at 0.08 every epoch improves on the last, so the recipe is reproducible - the repository's training script rebuilds the shipped file bit for bit. The file also carries the plan's slot models: a persona slot and a skill slot (RouterSLM.slots, trained by fit_slot on the same seed prompts scored for every persona and skill of the catalogue plus the labelled rows that name one), so plan=True ranks which persona speaks and which skill instructs with a learned model from the first request - 0.99 holdout accuracy on both slots. Its targets are the reference ids. Its calibration - temperature plus the isotonic stage - is the one fitted on rows the model never trained on (the script's probe fit, everyday phrasings included), not on its own training rows, so the probability it reports is honest; the final fit runs for the number of epochs the probe's early stopping chose. Your own targets are added when it is loaded; the ensemble hears the model only on targets it has compared, so with a different catalogue it starts silent and osr improve / the autopilot

• which now start warm from it whenever no model file exists yet - train a challenger that knows your targets on the first cycle (the file records origin: bundled through every promotion). Retrain the shipped file with the script; never edit it by hand.

from opensmartroute import RouterSLM
from opensmartroute.enterprise import RouterBuilder

app = RouterBuilder(reg).with_defaults().with_slm().build()  # the packaged model, your targets added
print(RouterSLM.bundled().info()["holdout"])  # what it was measured at

The package ships the hand-labelled set the bundled model was measured on next to it (opensmartroute/data/labelled.jsonl, a copy of examples/eval_dataset.jsonl), and RouterSLM.labelled_check() scores any model on it - on its own, on the rows whose targets the model can choose between. That is the labelled floor of the self-improvement loop (SLMSettings.labelled_floor, on): a holdout is self-agreement with the router's own decisions on your traffic, and a deployment whose traffic is one kind of job can promote itself into a model that is right on that job and wrong on everything else while every holdout reads 0.95. A challenger that routes fewer of the applicable labelled rows acceptably than the champion - beyond labelled_tolerance (5 %) of them - is refused whatever its holdout says (ImprovementReport.labelled carries both scores), and SelfImprover.floor_check(RouterSLM.bundled()) holds a served champion to the same floor against the packaged model, adopting it when the champion fell below. Fewer than labelled_min_rows (20) applicable rows - a catalogue of your own - switch the floor off. The floor is also rehearsed: the applicable labelled rows train every challenger next to your traffic, replicated to about labelled_rehearsal (25 %) of its training rows, so a challenger that learns one kind of traffic keeps the reference behaviour instead of being refused every cycle (the holdout stays your own rows; ImprovementReport.sources["labelled"] counts them).

Data comes from three collectors that all degrade gracefully when offline:

• adapters.ModelCatalogue merges OpenRouter prices and Hugging Face model cards (downloads, likes, model-index benchmarks) into ModelCards, turns them into RouteTargets (targets(min_quality=, max_usd_per_1k=, measured=True)), prints a cost/quality benchmark and the Pareto frontier. refresh(leaderboard=True) adds Open LLM Leaderboard accuracies (IFEval, BBH, MATH, GPQA, MuSR, MMLU-Pro) as measured quality; apply_quality() takes the Bradley-Terry win rates that eval.model_quality derives from collected battles. Third-party descriptions are risk-scored and replaced when they look like injections.
• eval.DatasetCollector pulls routing datasets from the Hugging Face datasets-server into a JSONL cache. Fifteen public sources are on by default: the pairwise battles routellm-battles (GPT-4-judged), arena-55k, arena-100k, arena-140k (LMArena human votes, with the code / language / math / hard-prompt tags), ppe-human, webdev-arena, mt-bench-human and reward-bench, the 2025 releases arena-expert-5k (domain experts judging 2025 models; the conversation column is a Python repr, parsed as such), search-arena-7k (search-augmented models on live questions - the names carry their own tier rules), repochat-arena (answers over a repository) and llama3-70b-battles, the per-model judge scores of ultrafeedback, and RouteLLM's routellm-gpt4 ("was Mixtral good enough for this prompt") and routellm-mmlu battles. RouterBench / RouterEval / the raw arena conversations are known but gated: download them with your own Hub token and load the file with osr collect --file routerbench=routerbench.jsonl --preset routerbench (.jsonl / .json / .csv in the preset's shape, --tier folds model columns onto your targets), and any dataset works through DatasetSource. Battles name a moving population of models, so --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small folds every side onto your tiers as it is parsed (tier_of(): the ARENA_TIERS table, name rules for the 2024-26 generations, then the parameter count in the name) - the RouteLLM question "was the cheaper tier good enough?". Collection is polite and resumable: pages are paced, 429s are retried with backoff, and a run the server cuts short keeps its rows and continues from the recorded offset on the next osr collect (<name>.meta.json). rows_from_feedback turns your FeedbackStore into labelled rows.
• adapters.WebKnowledge searches Hugging Face (text-generation repos, per-keyword fallback), Wikipedia and DuckDuckGo (a self-hosted SearXNG with SEARXNG_URL, Brave with BRAVE_API_KEY) for new models, datasets and live context; fetches are https-only, byte-capped and page text is risk-scored. read_feeds(urls) reads RSS / Atom article feeds - vendor blogs, release notes, arXiv listings (adapters.parse_feed, stdlib XML, newest OSR_SLM_FEED_MAX_ENTRIES entries within OSR_SLM_FEED_MAX_AGE_DAYS) - so the loop hears about a model launch the day it is announced, not when a benchmark gets around to it.

learning.SelfImprover closes the loop as a champion/challenger cycle: refresh the catalogue, discover, gather evidence, split, train a challenger, calibrate it on the holdout and promote it only when it really wins there. Articles feed the discover step (feeds= / --feed URL / OSR_SLM_FEEDS): each cycle counts which catalogue models the articles mention (ImprovementReport.mentions), looks up Hugging Face cards for model names the catalogue has never seen (gemini-2.5-pro announced in a post becomes a card lookup and a discovered_models entry) and lets priced cards the articles talk about join the candidate targets (trending) even before a leaderboard has measured them - the evidence rows still decide whether the challenger learns to name them, and the promotion gates below still apply, so an article can make a model eligible, never preferred. Three gates stand between a challenger and production: the holdout must hold at least OSR_SLM_PROMOTE_MIN_HOLDOUT rows (a four-row holdout cannot judge anything); holdout accuracy must improve by at least OSR_SLM_MIN_GAIN (or loss must drop at equal accuracy); and the rows the challenger newly gets right must outnumber the rows it newly gets wrong with a one-sided exact binomial p-value of at most OSR_SLM_PROMOTE_ALPHA (learning.paired_test, McNemar's discordant pairs - two flipped rows out of a hundred are noise, not a win; 1.0 turns the test off). The replaced champion is kept next to the model as <model>.prev.json (OSR_SLM_KEEP_PREVIOUS), so osr slm rollback slm.json - or SelfImprover.rollback()

• undoes a promotion that looked good on the holdout and bad in traffic; the files swap, so a second call redoes it. Every cycle is appended to a JSONL history with the champion and challenger reports, the discordant counts and the p-value, and the model file carries a version that counts the training runs behind it.

osr catalogue --cache data/catalogue.json --hf "llama instruct" --leaderboard --markdown   # live cost / quality
osr collect --cache-dir data --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small
osr -t targets.yaml slm train --out slm.json --cache-dir data --report          # every *-tiers cache, holdout metrics
osr -t targets.yaml slm train --out slm.json --from rows.jsonl --encoder attention --warm bundled   # fine-tune the packaged model
osr slm predict slm.json "write a python function to parse csv" --cost-weight 1.0 --explain
osr -t targets.yaml improve --slm slm.json --cache-dir data --catalogue data/catalogue.json \
    --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small \
    --search "qwen2.5 instruct" --feed https://openai.com/news/rss.xml \
    --feed https://huggingface.co/blog/feed.xml --interval 3600         # daemon; --offline for air-gapped
osr slm rollback slm.json                                                    # undo the last promotion

--explain (and RouterSLM.explain) prints the hashed features that decided the pick - each word, bigram or character gram contributes v_j (W_j . e_t) / tau to a target's logit, so the list is the exact decomposition of the logit margin between the pick and its runner-up, plus the learned per-target bias. The hosted SLM service returns the same block from POST /predict with "explain": true.

Serve the result: osr -t targets.yaml --slm slm.json serve, or in the container OSR_SLM=/config/slm.json (Helm config.slm). With OSR_LLM_BASE_URL (and OSR_LLM_API_KEY / OSR_LLM_MODEL) pointing at any OpenAI-compatible endpoint - OpenAI, Azure OpenAI, vLLM, Ollama, OpenRouter - every LLM target gets a chat handler from metadata.model, so /v1/chat/completions with model: "auto" routes and answers.

Speed, and what the numbers mean

Training is plain SGD over hashed features; with numpy installed (extra fast, included in the serve image) the same steps run on dense arrays - 18 000 rows train in under a minute, ten times the pure-Python speed - and prediction projects through a dense mirror of the weights. OSR_SLM_BACKEND=auto|numpy|python chooses; both paths seed identically and write the same model file, so a zero-dependency deployment reads a numpy-trained model unchanged. The model carries a learned per-target bias (the base rate a unit-norm dot product cannot express) and trains with inverse-time learning-rate decay (OSR_SLM_LR_DECAY) and l2 1e-3.

The model file: quantised weights

The file is JSON, but its weights are not text: every row of the projection W, of the target embeddings E and of the attention encoder's token table is stored as symmetric per-row integers - q = round(x / s) with one scale s = max|x| / 127 per row - packed into a base64 block (opensmartroute.learning.quant). At the default OSR_SLM_QUANTIZE_BITS=8 that is a fifth of the bytes of the four-decimal floats it replaces with a relative RMS weight error near 0.5 %; 4 packs two codes per byte (a tenth of the bytes, ~5 % error - measure before shipping); 0 writes floats. Inference dequantises on load and stays in floating point, so a quantised model ranks like the float one: the packaged model went from 2.15 MB to 0.58 MB with identical top-1 picks on the example dataset. Files written by earlier versions (format 1, float rows) load unchanged; a file written at 8 bits carries format 2 and is refused with a clear error by a package that predates it.

osr slm quantize slm.json --bits 8 --check rows.jsonl      # rewrite in place; size, error and top-1 agreement
osr slm quantize slm.json --bits 4 --out slm-4bit.json      # smaller still, for edge deployments
osr slm quantize slm-4bit.json --bits 0 --out slm-float.json
osr -t targets.yaml slm train --out slm.json --quantize 0   # train and keep floats in the file

osr slm info reports quantization.loaded_bits (what the file carried) and quantization.bits (what the next save writes).

How long to train is decided by the data, not a constant: from OSR_SLM_VALIDATION_MIN_ROWS rows up, fit splits a deterministic OSR_SLM_HOLDOUT share off as validation, scores it after every epoch, stops once the validation loss has not improved for OSR_SLM_PATIENCE epochs (at most OSR_SLM_EPOCHS) and keeps the best epoch's weights; the calibrator and the report are fitted on those validation rows, so --report is honest even without a separate holdout. Every row also trains through OSR_SLM_AUGMENT_VIEWS (2) word-dropout views - copies with each word dropped at OSR_SLM_AUGMENT_DROPOUT (0.15), drawn once per fit so both arithmetic backends see the same corpus - which teaches the hashed features that a prompt survives losing a word, exactly how the validation prompts differ from the training ones. OSR_SLM_LABEL_SMOOTHING (0, off) spreads that share of every label over the row's candidates; it changed nothing measurable here and is left for corpora with noisier labels. On the example catalogue (24 targets, 391 synthetic training rows, 97 holdout, three seeds) the old fixed four epochs reached 0.66-0.70 holdout accuracy at loss 2.22-2.30; early stopping alone 0.68-0.72 at loss 2.17-2.24; early stopping with the two views 0.80-0.87 at loss 1.81-1.88 (ECE 0.05-0.11), training in 1.5 s (numpy) or 11 s (pure Python) and predicting in 0.3 ms. Views are not rows: trained_rows, the report and the confidence ramp count the originals, and rare-feature pruning counts a feature's rows without them. Every history entry records epochs, best_epoch and validation. The file also records the feature-space signature it was trained under (hashed/v1:dim=...); a model built on a different tokenisation or hash width is refused at load instead of routing on scrambled weights.

The encoder reads at most OSR_SLM_MAX_TOKENS (512) whitespace tokens of a prompt, for scoring, familiarity and explain alike (0 = every token); prompts below the cap are untouched, and the cap is a runtime setting, so an older model file honours it too. A longer prompt is read through a reading window (learning.window, ReadingWindow) in two views:

• the window: OSR_SLM_WINDOW_HEAD (0.4) and OSR_SLM_WINDOW_TAIL (0.4) of the budget keep the opening and the closing tokens - where instructions usually live - and the rest goes to the middle: it is cut into sentences, each is read where it holds the most instruction cues (the words a request opens with, taken from the first word of every task template of the ontology - write, translate, summarise, extract, fix, what, why ... - plus a question mark) and scored for salience (cues per square root of the length), and the most salient sentences are kept and placed first, before the head, because the encoders read an opening more closely than anything after it. Budget left over goes to OSR_SLM_WINDOW_RUN-token (8) runs spaced evenly through the middle, so bigrams and character grams stay meaningful and the body's own vocabulary (legal, medical, code) still reaches the model;
• the ask: the sentences that ask for something - among the opening and closing spans and the salient middle - read on their own, at most OSR_SLM_WINDOW_ASK_TOKENS tokens (0 = two sentence spans), and added to the query as a unit vector weighted OSR_SLM_WINDOW_ASK (2.0) against the window's: twelve words of request among five hundred of document are two per cent of a bag of features, and the body decided; now the request is read the way it would be on its own (0 = window only). A fit and an online step read a long row through the same two views, on both arithmetic backends, so training and serving agree. OSR_SLM_WINDOW_NOVELTY (1.0, in 0..1) is the share of a sentence's salience that depends on its words being rare in the rest of the prompt, so a question the document itself repeats on every page (an FAQ, a form) is never mistaken for the request.

Measured with the bundled model on the 70 labelled rows, each buried at the head, the middle or the tail of a thousand-word document (a finance report without punctuation, legal clauses, FAQ questions, a code dump): head-and-tail reading routed 7-44 of 70 acceptably, the window alone 29-44 (9 on code, 7 on the FAQ), window and ask view 63-69 of 70 on every body; the same rows on their own route 70/70 before and after. explain reports the text the window view read and the ask next to it. The attention block has its own, tighter window (below) and cuts through the same policy.

A served encoder also remembers the features and query vector of the last OSR_SLM_ENCODE_CACHE (512) prompts it scored (0 = none), dropped on every weight change (an online step, a fit, a reload): the familiarity gate reads the features the score just computed instead of hashing the prompt a second time, and a repeated prompt - a retry, an evaluation set, the labelled floor check - is encoded once. Measured on the bundled model with numpy: SLMStrategy.score 735 µs -> 600 µs on a prompt never seen, 130 µs on a repeat; predict_proba 400 µs -> 35 µs on a repeat.

Read --report against the constant baseline. Human preference between tiers is noisy: on the ten-source corpus "always the mid tier" is acceptable for 64 % of holdout prompts and the SLM reaches 64-66 % with a holdout cross-entropy of 0.98-1.00 (the constant predictor: 1.03), well calibrated (ECE 0.05-0.08). It is much stronger where the label really depends on the prompt - 94 % on routellm-gpt4 (baseline 86 %), +12 points on arena-100k - and no better than the prior on sources where the prompt carries no signal. That is what a learned routing prior is; the ensemble adds rules, capability, similarity and the bandit on top. The corollary: an SLM trained only on synthetic seed rows is a compressed capability scorer, and in a rule-driven ensemble it can cost accuracy on labelled traffic (on the 33 example rows: 0.88 without it, 0.82 with it at weight 1.0). Every report therefore also states the baseline - the accuracy of always predicting the most common acceptable target of those rows - and the skill = (accuracy - baseline) / (1 - baseline), the share of the constant predictor's errors the model actually fixes; SLMStrategy speaks as loudly as that skill (OSR_SLM_TRUST_BY_SKILL: confidence x (0.5 + 0.5 x skill) of the last validated fit, so a model no better than the constant predictor is heard at half volume, one that fixes every error at full). Warm-start with synthetic rows, but let osr eval on your own labelled rows set OSR_WEIGHTS_SLM until real outcomes have trained it. OSR_SLM_ABSTAIN_MARGIN makes SLMStrategy return no scores at all when its top two probabilities are closer than the margin (top-1 accuracy on such near-ties was 22 % against 72 % elsewhere); it is off by default because a near-tie still carries the model's opinion about every other target, and the ensemble breaks the tie anyway - measure before turning it on. The familiarity gate is on: the share of a prompt's features the model has a trained row for scales its confidence from nothing at OSR_SLM_FAMILIARITY_NONE (0.7) to full at OSR_SLM_FAMILIARITY_FULL (0.95; 0 = no gate) - a model is calibrated on the prompts it saw, and a prompt made of features it never saw is where it is most confidently wrong. Measured with the production model trained on synthetic rows against the 33 labelled example rows: 0.879 for the ensemble alone, 0.758 with the SLM ungated, 0.879 gated; on prompts from its own training distribution nine in ten score above 0.94 familiarity and pass at full volume. The trace shows the gate (familiarity 0.51 -> x0.00), and the same number feeds the input-drift monitor below.

Online learning is bounded so one prompt or one noisy stretch cannot drag the served model: a request's outcomes apply at most OSR_SLM_ONLINE_MAX_PER_REQUEST (3) SGD steps, and a served model takes at most OSR_SLM_ONLINE_BUDGET (5000) steps in total before updates pause until the next fit or promotion re-anchors it on the full corpus (SLMStrategy.online_steps / online_skipped). fit also drops the projection rows of features seen in fewer than OSR_SLM_MIN_FEATURE_COUNT (2) training rows once the corpus has 50 rows or more - a feature seen once carries its random seed row and nothing else, so an unseen feature contributes nothing instead of noise: measured -18 % file size at identical accuracy.

One SLMStrategy can serve one model per tenant: SLMStrategy(slm, resolver=lambda req: models.get(req.context.get("account_id"))) scores a request with the tenant's own RouterSLM when the resolver returns one (the shared model otherwise), trains that same model from the request's outcomes and reports confidence from that model's skill. The hosted platform uses it for per-workspace models (Platform guide, "Learning").

A corpus of model battles only knows the models it compared. Scored rows therefore train a softmax over the scored population (every target any row scores, plus OSR_SLM_NULL_TARGETS learned "none of these" logits that keep it from saturating), so a tool, skill or agent the data never mentions keeps its catalogue embedding instead of being pushed to zero, and SLMStrategy abstains on such targets - its confidence is scaled by the share of the request's candidates it has evidence about. Rows that name only an expected target (your own feedback, examples/eval_dataset.jsonl) contrast against the whole catalogue, which is how the model learns the non-LLM targets once outcomes arrive.

Where a transformer helps

The hashed encoder is a bag of words, bigrams and character grams: it cannot tell "explain this SQL" from "SQL that explains itself". Two opt-in encoders add that context, both persisted in the same JSON file and restored by RouterSLM.from_file:

• OSR_SLM_ENCODER=attention adds learning.AttentionEncoder, a transformer block (hashed token embeddings, sinusoidal positions, OSR_SLM_ATTENTION_HEADS heads of OSR_SLM_ATTENTION_HEAD_DIM, residual, attention pooling) to the query encoder: $q = W\,\phi(x) + \operatorname{Attn}(x)$. No dependencies, deterministic, hand-derived gradients. It reads a prompt up to OSR_SLM_ATTENTION_MAX_TOKENS (128; word and punctuation tokens) through the same reading window as the hashed features - head, the salient middle first, tail - and is quadratic in that window - at 128 tokens prediction on a 150-word prompt still takes about a millisecond. With numpy (extra fast, in every image) the block runs as dense matrix products over a mirror of its weights: a fit that took four and a half minutes in pure Python takes 12-30 s, prediction 0.7 ms instead of 6 ms, and both paths draw the same random numbers and end within float rounding of each other (the model file is identical). The block learns slower than the linear projection, so it trains on its own budget: OSR_SLM_ATTENTION_EPOCHS (40), OSR_SLM_ATTENTION_LR (0.2) and OSR_SLM_ATTENTION_PATIENCE (4) replace epochs, lr and patience when the encoder is on (0 = the shared value). With four heads of sixteen dimensions and that budget it reaches 0.95-0.97 holdout accuracy on the example catalogue's synthetic rows where the hashed encoder with views stops at 0.80-0.87 (the views add nothing measurable to the attention encoder, which sees word order already, and double its fit time); with the old budget (two heads of eight, twelve epochs at 0.05) it lost three points, and on the public arena corpora it lost five at full size - measure on your holdout (osr slm train --report) before switching. The bundled model and the hosted platform's SLM service use this encoder (OSR_SLM_ENCODER on the trainer). The default is auto: a loaded model keeps the encoder it was trained with - the improver clones a bundled or promoted attention champion and fine-tunes it on the attention budget - and a fresh model is hashed. Naming an encoder explicitly after a champion was trained does not clone it - SelfImprover builds a fresh model with the configured architecture (ImprovementReport.encoder vs champion_encoder) and the promotion gate decides, so the switch takes effect only where it wins.
• OSR_SLM_EMBEDDER=openai:<model>[@<base_url>] appends a frozen pretrained embedding as dense features (learning.EmbeddingFeaturizer) from any OpenAI-compatible /embeddings endpoint over the stdlib client - no extra, no download: the embedding model a deployment already serves (openai:qwen3-embedding:4b@http://osr-ollama:11434/v1), Azure OpenAI (openai:text-embedding-3-small@https://<resource>.openai.azure.com/openai/v1) or OpenAI itself (openai:text-embedding-3-small). The key comes from the variable OSR_SLM_EMBEDDER_API_KEY_ENV names (OSR_LLM_API_KEY; an in-cluster model server needs none), each call waits OSR_SLM_EMBEDDER_TIMEOUT_S (20) and texts are cut at their first 6,000 characters. Every vector is cut to its first OSR_SLM_EMBEDDER_DIMS (256) components and renormalised - what the providers' dimensions parameter does for a Matryoshka-trained model (text-embedding-3, Qwen3-Embedding, nomic, jina v3 keep almost all of their quality there) and what keeps a training corpus of tens of thousands of rows in memory; 0 keeps the full width. The dense block enters at OSR_SLM_EMBEDDER_SCALE (4) times the weight of one hashed feature - at 1 the hundreds of sparse cues drowned it. W then learns a linear head on top of a real transformer while the hashed features keep the surface cues (code fences, length, exact vocabulary); a fit embeds its corpus in batches of 64 and a text is embedded once per process, so a training run costs one pass over its rows. Measured on the reference catalogue's 117 held-out phrasings with Azure text-embedding-3-small: the model alone routes 0.79 of them acceptably against 0.74 for the hashed encoder, and the ensemble with it 0.83. The file records the embedder's name and the cut, so a reload rebuilds the same client from settings alone; when the endpoint fails at request time SLMStrategy sits that request out (SLMStrategy.status()["embedder_errors"]) and the other strategies decide, so an embedding outage never fails a route. OSR_SLM_EMBEDDER=sentence-transformers/all-MiniLM-L6-v2 (extra embeddings) runs the transformer in process instead, and RouterSLM(embedder=callable) takes any callable (passed again to from_file(embedder=...)). The packaged model ships without an embedder - it must load with zero dependencies and no endpoint - so a deployment that wants the dense features trains its own: the hosted platform's SLM service and osr slm train read OSR_SLM_EMBEDDER, and SelfImprover builds a fresh challenger with the configured embedder when the champion was trained without one, promoted only when it wins the gate.
• OSR_SLM_EMBEDDER=keras:/path/head (extra tensorflow) names a saved learning.NeuralRoutingHead - a Keras classifier over the same hashed features, trained on the same rows - whose representation layer becomes the dense embedder, so the SLM's linear head sees a non-linear view of the prompt next to the raw hashed cues. The head reads conversation context the same way the SLM does (context=True: EvalRow.history at training time, predict_proba(text, history=...) at prediction time) and carries an experience-replay memory: observe(text, target, reward, history=) keeps every outcome in a bounded learning.ReplayMemory, replay_fit() fine-tunes on it (good outcomes as positives, bad ones as reward-weighted mass on the other targets), and the memory is saved next to the model, so a restart keeps what live traffic taught. The same extra offers learning.TokenCountModel, which learns billed token counts from (text, tokens) pairs and, once installed (estimate.set_token_counter), replaces the heuristic behind estimate_tokens (SDK guide).

Live web context

A router picks the model; it cannot make a model know what happened this morning. signals.freshness_need scores how much a request depends on facts that change - news, prices, weather, scores, "latest", "today", an explicit "look it up", this year's date - into Signals.freshness_need; the capability strategy reads it as a nudge toward targets tagged web / search / browsing (OSR_CAPABILITY_WEB_BONUS), and enterprise.WebGroundingMiddleware acts on it: at or above its threshold (0.5) it runs the request's first sentence through adapters.WebKnowledge - keyless Wikipedia (adapters.wikipedia_search: encyclopaedic, hours to days behind the news, but real text for "who / what is X"), DuckDuckGo instant answers and Hugging Face by default; a self-hosted SearXNG instance for the whole live web when SEARXNG_URL is set (adapters.searxng_search), Brave with BRAVE_API_KEY, or any provider you register - screens every hit with the injection lexicon (security.injection_risk; hits at or above OSR_SLM_PAGE_RISK_THRESHOLD are dropped) and attaches the rest to the request - context["web_context"] for the trace and context["system"] as a Live web context block the OpenAI-compatible chat handler puts in front of the model, so the answer is grounded in what the web says today and can cite it. context["web"] = True forces it for one request, False disables it; identical queries within cache_ttl_s come from memory; a failing or slow provider never blocks routing (web.grounded / web.empty / web.failed events). Turn it on with RouterBuilder.with_web_grounding(), harden(router, web_grounding=True) or osr serve --web-grounding (OSR_SERVER_WEB_GROUNDING=true).

Knowledge graph and context engineering

Outcomes say which target did well on which task; conversations say what was said; the knowledge graph (opensmartroute.knowledge) keeps facts - typed, sourced, weighted, expiring triples - about the people who ask, the targets that answer, the topics that come up and the pages that were consulted, and turns them into the context a model should see for one request:

• KnowledgeGraph stores Triple(subject, predicate, object) facts with a confidence, an evidence weight (a fact stated twice grows instead of duplicating), a source (catalogue, conversation:<id>, web:<provider>, outcome, user), timestamps and an optional expiry; it answers pattern queries (query), neighbourhood walks (neighbors), word search (search), forgets on request (forget), decays what nobody confirmed (decay, OSR_KNOWLEDGE_DECAY_PER_DAY), bounds itself (OSR_KNOWLEDGE_MAX_TRIPLES) and persists to one JSONL file. Interoperability with the open-source graph ecosystem is by format, not by dependency: to_jsonld (JSON-LD 1.1), to_ntriples (RDF for rdflib, Jena, GraphDB, Blazegraph), to_graphml (Gephi, yEd, igraph), to_cypher (Neo4j / Memgraph MERGE statements) and to_networkx (when networkx is installed); from_jsonld reads back. The vocabulary is knowledge.PREDICATES; the person-subject ones (PERSONAL_PREDICATES) are never exported to a shared graph unless asked.
• Ingestion is rules, not a model: extract_facts reads what a person states about themselves (my name is, I work at, I'm a data engineer based in Toronto, we use Python and dbt, I prefer, answer in French, I'm working on, remember that), mentions the entities a message names; ingest_turn, ingest_targets (the catalogue as facts), ingest_decision (task type and domains routed_to the chosen target, the person asked_about them), ingest_outcome (target performed_on task with the running mean quality and the count) and ingest_web_context (a live search's hits as expiring cites facts) write into the graph and return what they added.
• ContextEngineer builds the block: About the user (the subject's facts, identity first, then relevance), Relevant knowledge (shared facts whose words overlap the request) and Live web context (the hits a WebGroundingMiddleware attached, numbered so the model can cite them); every candidate line has a score and lines are admitted in score order until OSR_KNOWLEDGE_BUDGET_TOKENS (600) is spent, at most OSR_KNOWLEDGE_MAX_FACTS per section and OSR_KNOWLEDGE_MAX_WEB_HITS hits. EngineeredContext records what was placed and what was dropped and why (budget, limit, duplicate); apply puts the block in front of context["system"] and the accounting under context["knowledge"], so a trace shows the model's context the way it shows the routing decision.
• ContextMiddleware runs the engineer in front of every routed request (context["user_id"] / ["account_id"] / ["session_id"] or profile["id"] names the person) and teaches the graph from the traffic: the message's facts before routing, the decision and any web hits after; it emits context.engineered. KnowledgeStrategy (knowledge in the ensemble, OSR_WEIGHTS_KNOWLEDGE) scores candidates by what the graph knows worked on the request's task type and domains - it abstains for a target with fewer than OSR_KNOWLEDGE_STRATEGY_MIN_EVIDENCE outcomes on the topic and trusts itself fully at OSR_KNOWLEDGE_STRATEGY_FULL_EVIDENCE; outcomes reaching it become new performed_on evidence.
• A learned link predictor closes the loop (optional extra tensorflow): learning.KnowledgeEmbedding trains a TransE embedding of every entity and predicate in Keras (fit(graph); personal facts left out) and then answers with numpy alone - predict_tail("target:llm-small", "has_domain") (which domain is it probably good at that nobody declared), predict_head("performed_on", "task:coding"), similar, score; saved as .npz + .json or to_dict() for a database row. Attached as graph.embedding it backs KnowledgeGraph.suggest, which otherwise counts what peers of the same type carry.

from opensmartroute.enterprise import RouterBuilder

router = (
    RouterBuilder(registry)
    .with_defaults()
    .with_web_grounding()
    .with_knowledge("state/knowledge.jsonl", budget_tokens=600)  # ingests the catalogue; learns from traffic
    .build()
)
d = router.route(RouteRequest(text="I work at Acme; how do I test a dbt model?", context={"user_id": "u1"}))
print(d.trace.signals.task_type, router.knowledge.query(subject="person:u1"))
print(router.knowledge.to_cypher())

harden(router, knowledge=True) (or a KnowledgeGraph / a path) does the same for a router that already exists. The hosted platform runs one graph per workspace as its knowledge service (Platform guide).

Self-operation

osr serve --autopilot runs the SelfImprover inside the server (learning.Autopilot): a cycle every OSR_SLM_AUTOPILOT_INTERVAL_S seconds, an early one when learning.DriftMonitor (Page-Hinkley over the success and quality of every outcome the router learns from, via Router.observers) sees the served quality drop or when learning.InputDriftMonitor (Page-Hinkley over the SLM's top probability and over the share of each prompt's features it has a trained row for - both fall when traffic moves to prompts it was not trained on, long before outcomes show it; OSR_SLM_INPUT_DRIFT_THRESHOLD, OSR_SLM_INPUT_DRIFT_MIN_N) sees unfamiliar traffic, never closer than OSR_SLM_AUTOPILOT_MIN_GAP_S. Feedback is joined back to prompts through the router's request memory (OSR_SLM_AUTOPILOT_REMEMBER texts; RequestMemory(path=...) keeps it in a JSONL file across restarts), which also records the decision the router took on every prompt: prompts without an outcome become distillation rows (the ensemble's propensities as soft labels, newest OSR_SLM_DECISION_ROWS_MAX per cycle), so the SLM learns the router's behaviour on real traffic first and the outcomes correct it. While real rows are scarce (fewer than the cycle's synthetic_below, 200) synthetic rows over the catalogue fill in, OSR_SLM_SYNTHETIC_PER_TEMPLATE per prompt template (2; the hosted stacks use 8). An accepted challenger is hot-swapped into the SLMStrategy and written back to disk, a failing cycle is logged and the server keeps serving. GET /stats reports slm whenever the ensemble serves one (SLMStrategy.status(): rows, version, encoder, origin, calibration, trust, online steps, abstentions, input drift) and autopilot (cycles, promotions, drift, input drift, last report, next run) and POST /autopilot/cycle schedules one now. In the container: OSR_AUTOPILOT=1 with OSR_STATE mounted (Helm autopilot.enabled); --offline keeps it off the network. --slm is a warm start, not a prerequisite: without it the loop begins with the bundled model over the catalogue (or the one a previous run promoted into <cache-dir>/slm.json) and trains a challenger on the first cycle. Each cycle is traced as an autopilot.cycle span with a learn.improve event (and learn.promote when the challenger wins), so GET /events?name=learn.* shows the loop at work. The hosted platform runs the same loop from OSR_PLATFORM_SLM / OSR_PLATFORM_AUTOPILOT and shows it at /platform/dashboard/learning (Platform guide, section 10).

osr -t targets.yaml --slm slm.json --state state serve --autopilot --autopilot-interval 1800 \
    --catalogue data/catalogue.json --source arena-55k --tier frontier=llm-frontier --tier small=llm-small \
    --feed https://openai.com/news/rss.xml --marketplace-refresh 86400
curl -s localhost:8000/stats | jq '.autopilot, .marketplace'

The same server follows the marketplace (--marketplace-refresh / OSR_MARKETPLACE_REFRESH_S, Marketplace): new and re-versioned skills and tools land in the served catalogue on the next tick, and the autopilot's next cycle trains with them as candidates.

6. Declaring components with decorators

Targets, strategies, signal extractors, policy rules, middleware and telemetry can be declared where they live and assembled by a ComponentRegistry (details in SDK guide):

import opensmartroute as osr


@osr.tool("weather", domains=["travel"], actions=["lookup"], cost=0.0, examples=["weather in paris"])
def weather(request, **kw):
    """Current weather for a city."""  # docstring -> target.description
    return lookup(request.text)


@osr.strategy(weight=0.8)  # fn(request, signals, candidates) -> {id: score}
def geo_affinity(request, signals, candidates):
    region = request.constraints.region
    return {t.id: 1.0 if region in t.constraints.regions else 0.4 for t in candidates}


@osr.policy_rule  # fn(target, request, signals) -> reason | None
def business_hours_only(target, request, signals):
    if target.kind is osr.TargetKind.HUMAN and request.context.get("after_hours"):
        return "human queues are closed"


router = osr.components.router(registry)
app = osr.components.builder(registry).with_auto_learning().build()

The short forms take the rest of the boilerplate away - a rule without rules.yaml, code for a catalogue's targets, a guard and hooks without subclasses, and routing without building a router:

osr.components.load("targets.yaml", "rules.yaml")  # the catalogue and its rules as components


@osr.handler("llm-small", "llm-frontier")  # one function behind several catalogue targets
def chat(request, **kw):
    return provider.complete(request.text)


osr.rule("pii-stays-home", prefer=["llm-onprem"], pin=True, contains_pii=True)  # a typo in a condition raises


@osr.guard  # None passes, a str refuses (SecurityError), a RouteRequest rewrites
def no_empty(request):
    return "empty request" if not request.text.strip() else None


@osr.on_outcome  # also @on_decision(request, decision), @on_error(request, error)
def count(outcome):
    metrics.inc(outcome.target_id, outcome.success)


d = osr.route("Refund order #123, the customer is upset")  # router built from the registry on first use
osr.feedback(d, success=True, quality=0.9)
print(osr.run("Summarise this ticket: ...").text)  # route + execute + learn

examples/decorator_sdk.py runs the whole loop offline.

Every tunable (ensemble weights, thresholds, scales) lives in one typed Settings object and can be overridden per process (osr.configure(...)) or through OSR_<GROUP>_<FIELD> environment variables; osr settings prints the effective values.

7. Enterprise builder

from opensmartroute.enterprise import (
    RouterBuilder,
    CacheMiddleware,
    TenantMiddleware,
    LoggingTelemetry,
    MetricsTelemetry,
    FileAuditSink,
)
from opensmartroute.security import GuardMiddleware

metrics = MetricsTelemetry()
app = (
    RouterBuilder(registry)
    .with_rules(rules)  # declarative preferences
    .with_defaults()  # capability + similarity + Thompson bandit
    .with_auto_learning(state_dir=".osr-state")  # IRT + Bradley-Terry + LinUCB + Markov lookahead
    .with_health(latency_slo_ms=3000)  # circuit breakers, rate limits, budgets
    .with_middleware(
        GuardMiddleware(redact=True),  # gadget defence + PII redaction
        TenantMiddleware({"acme": {"max_cost_per_1k": 0.005, "data_boundary": "private"}}),
        CacheMiddleware(ttl_s=30),
    )
    .with_telemetry(LoggingTelemetry(), metrics)
    .with_audit(FileAuditSink("audit.jsonl"))
    .build()
)

d = app.route(RouteRequest("Summarize this contract", constraints=RequestConstraints(tenant="acme")))
app.learn(
    Outcome(
        d.request_id,
        d.target.id,
        success=True,
        quality=0.9,
        cost_usd=0.002,
        latency_ms=1200,
        domains=d.trace.signals.domains,
        complexity=d.trace.signals.complexity,
    )
)
print(metrics.snapshot()["route_latency_ms"], app.health_snapshot())

with_retrieval(narrow_above=, narrow_to=) switches to retrieve-then-rank for large catalogues, with_queue_awareness() replaces static latency with a queue-aware estimate, with_shadow(candidate, mode="shadow" | "ab") compares policies with SPRT, with_calibration() adds temperature scaling and conformal candidate sets, with_fair_share() enforces tenant weights and with_state_store() accepts Redis, SQL, encrypted, versioned or batched stores. Ports (state, telemetry, audit) are ABCs; see Enterprise builder.

8. Research-track modules

Each module below implements one roadmap item with tests; papers are cited in Research and formulas in Mathematics.

Signals from a draft or a small model

from opensmartroute.signals import DEFAULT_EXTRACTORS, VerbalisedDifficultySignal, DraftResponseSignal, extract_signals

extractors = [
    *DEFAULT_EXTRACTORS,
    VerbalisedDifficultySignal(source=lambda req: small_model_difficulty(req.text)),
    DraftResponseSignal(drafter=lambda req: cheap_model(req.text)),
]
router = Router(registry, extractors=extractors)

VerbalisedDifficultySignal accepts numbers, 7/10 or words (trivial ... expert) from context["difficulty"] or a callable and blends them into complexity. DraftResponseSignal measures hedging, self-corrections, query overlap and length ratio of a cheap draft.

Response-side uncertainty

from opensmartroute.signals import UncertaintyGate, EventTrigger, TriggerRule, response_uncertainty
from opensmartroute.strategies import Cascade

gate = UncertaintyGate(sampler=lambda req, n: [cheap_model(req.text) for _ in range(n)], n_samples=4)
cascade = Cascade(quality_gate=gate, threshold=0.6)  # escalate when samples disagree
result = cascade.run(req, router.route(req).alternatives)

feats = response_uncertainty(samples, p_true=judge_p_true)
trigger = EventTrigger(
    [TriggerRule("semantic_entropy", 0.5, "reroute"), TriggerRule("agreement", 0.5, "ask_human", below=True)]
)
trigger.check(feats)  # ["reroute", ...]

Hidden-state Dirichlet probe

from opensmartroute.strategies import HiddenStateStrategy

probe = HiddenStateStrategy(
    state_fn=lambda req: host_model.hidden_state(req.text), targets=["small", "large"], dim=4096
)
probe.fit(logged_states, logged_winners, epochs=20)  # warm start
router = Router(registry, strategies=[*default_strategies(), probe])

The probe's confidence is 1 - K/S (epistemic uncertainty), so unfamiliar states defer to the other strategies. math.DirichletProbe is usable on its own with any feature vector.

Elastic models and token budgets

from opensmartroute.strategies import BudgetVariant, TokenBudgetStrategy, expand_elastic

for sib in expand_elastic(
    registry.get("reasoner"), [BudgetVariant(1024, quality=0.7, cost_scale=0.4), BudgetVariant(8192, quality=0.9)]
):
    registry.add(sib)  # reasoner@1024, reasoner@8192
router = Router(registry, strategies=[*default_strategies(), TokenBudgetStrategy()])

Edge and cloud tiers

Tag targets with metadata: {tier: edge} or {tier: cloud} and add EdgeCloudStrategy(). Edge success is a Thompson posterior per complexity bucket, penalised when decode time eats into the latency SLO; the cloud tier is penalised by upload time and excluded when the request's data_boundary is private or on_prem and the target is public.

Auctions

AuctionStrategy(bid_fn=None, value=0.01) lets each target bid a claimed success probability and a price (default_bid derives both from the catalogue). Claims are corrected by each bidder's observed bias; the highest corrected surplus wins and pays the second price. Supply a bid_fn for multi-vendor catalogues where providers self-report.

Protocol selection, aggregation, self-escalation, hand-off

from opensmartroute.strategies import ProtocolPolicy, MixtureOfAgents, SelfEscalation, wrap_stream
from opensmartroute.learning import HandoffPolicy, MixtureCureModel

policy = ProtocolPolicy()  # risk / budget -> protocol
choice = policy.choose(req, decision, decision.trace.signals, uncertainty=gate(req, draft))
if choice.protocol == "aggregate":
    res = MixtureOfAgents(k=3, budget_usd=0.05).run(decision, req)  # top-k answers, majority vote
    for o in res.outcomes(req):
        router.learn(o)

monitor = SelfEscalation.for_target(decision.target)  # streaming competence posterior
streamed = wrap_stream(provider.stream(req.text), monitor)
if streamed.escalated:
    ...  # restart on a stronger target

handoff = HandoffPolicy(MixtureCureModel(), fallback_target="senior-agent", pins=task_pins)
if handoff.step(task_id, teacher_risk):  # permanent hand-off
    target = handoff.target_for(task_id, decision.target.id)
handoff.finish(task_id, failed=False)  # censored observation

Personalisation

learning.UserAdaptiveStrategy reads context["user_id"] (or profile["user_id"]) and keeps a Beta posterior per user and target, shrunk toward similar users and the global posterior; observe_user() warm-starts from logged interactions. learning.SkillAffinity.relevance(profile, skills) feeds retrieval.select_skill_set(relevance=) so skill sets depend on who is asking.

Multi-turn conversations

from opensmartroute.learning import HistoryTargetStrategy

router = Router(registry, strategies=[*default_strategies(), HistoryTargetStrategy(turns=6)])
d = router.route(
    RouteRequest(
        "can you explain more?", history=chat_history, context={"last_target": "llm-small", "last_failed": True}
    )
)

The same follow-up message routes differently depending on what the conversation has been about: the strategy learns a logistic model over the joint embedding of the recent history and each target's catalogue embedding, so a target that keeps failing in legal threads but succeeds in coding threads is scored accordingly, and an unseen target is still scored through its description and examples. last_target gives the incumbent a small continuity bonus that turns into a penalty when the previous turn failed.

Discovery beyond text similarity

from opensmartroute.discovery import SchemaAwareStrategy, CachePreservingSelector, SkillGraph

router = Router(registry, strategies=[*default_strategies(), SchemaAwareStrategy()])

selector = CachePreservingSelector(prefix_size=8, evict_after=5)
tools = selector.select(session_id, candidates, needed=[d.target.id])  # prefix-stable tool order

graph = SkillGraph(load_skills("skills"))  # osr-requires / osr-conflicts
ordered, dropped = graph.compose(["report-builder"])

MCP server recommendation and semantic-router import

from opensmartroute.adapters import recommend_servers, load_semantic_router_config

for rec in recommend_servers(
    "open a pull request", server_cards, k=2, constraints=req.constraints, allowed_auth=["none", "token"]
):
    print(rec.server.name, rec.score, rec.rationale)

imported = load_semantic_router_config("config.yaml")  # vLLM semantic-router
router = Router(imported.registry, strategies=[RulesStrategy(imported.rules), *default_strategies()])

Energy and carbon

math.EnergyModel fits Wh per target from measured samples; math.hardware_profile("h100-sxm") gives priors when no meter exists. Both convert to gCO2 with a grid factor and feed wh_per_1k_tokens / gco2_per_1k_tokens on the target's cost, which Objective(energy=, carbon=) trades against quality.

9. Command line

Install osr with the platform installer (curl -LsSf https://opensmartroute.ai/install.sh | sh on Linux and macOS, irm https://opensmartroute.ai/install.ps1 | iex in PowerShell) or with uv tool install 'opensmartroute[yaml,server]' (add fast for numpy-accelerated SLM training); osr --version confirms the install.

osr takes the catalogue with -t targets.yaml, optional rules with -r rules.yaml and an optional routing SLM with --slm slm.json; --help on any subcommand lists its options. python -m opensmartroute ... is the same command for environments where the script is not on PATH (IDE MCP configurations, containers, uv run). Grouped by task:

# Sign in and manage access tokens
osr login                                                            # browser sign-in to the hosted platform
osr login --url https://osr.example.com --profile work               # another deployment, kept as a profile
osr login --url http://router:8000 --token osr_local_...             # self-hosted osr serve token
echo "$TOKEN" | osr login --with-token                               # non-interactive (CI)
osr whoami                                                           # workspace, plan, edition, token source
osr trace <request-id>                                               # span tree of one request (server or platform)
osr events --name guard.* --limit 20                                 # recent tracer events there
osr token generate                                                   # mint an osr_local_ token for osr serve
osr token create --name ci                                           # new platform API key (shown once)
osr token list                                                       # keys of the signed-in workspace
osr token revoke <key-id>
osr logout                                                           # forget the profile (--all for every one)
# Route, quote and inspect
osr -t examples/targets.yaml -r examples/rules.yaml route "I want a refund for order #123" --plan
osr -t examples/targets.yaml route "Summarise this HR complaint" \
    --constraint data_boundary=on_prem --constraint region=eu     # hard constraints; --kinds agent,tool
osr -t examples/targets.yaml -r examples/rules.yaml route "" --event order.shipped --plan   # textless event
osr -t examples/targets.yaml estimate "Translate this contract" --monthly 100000   # cost per candidate + projection
osr -t examples/targets.yaml targets                                 # the catalogue as the router sees it
osr -t examples/targets.yaml stats                                   # feedback statistics per target
osr settings --json                                                  # effective tunables

# Evaluate and audit
osr -t examples/targets.yaml -r examples/rules.yaml eval examples/eval_dataset.jsonl --frontier --calibration
osr -t examples/targets.yaml audit traffic.jsonl --baseline llm-frontier --monthly 3000000 --markdown
                                                                     # Routing Audit: savings + violations
osr -t examples/targets.yaml ope decisions.jsonl                     # off-policy estimate (IPS/SNIPS/DR) of logged decisions
osr -t examples/targets.yaml safety --learned-guard                  # red-team suite

# Learn
osr train --from rows.jsonl --out models.json                        # fit signal models
osr catalogue --cache data/catalogue.json --hf "llama instruct"      # live model prices + quality
osr collect --cache-dir data --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small   # Hub datasets
osr collect --cache-dir data --file routerbench=routerbench.jsonl --preset routerbench             # a gated file you hold
osr -t examples/targets.yaml slm train --out slm.json --report       # routing SLM (see section 5)
osr slm info slm.json                                                # rows, sources, encoder, calibration, fit history
osr -t examples/targets.yaml --slm slm.json route "parse this csv"   # SLM inside the ensemble
osr slm eval slm.json rows.jsonl                                     # SLM accuracy on labelled rows
osr slm predict slm.json "parse this csv" --top 3                    # what the SLM alone would rank
osr slm info slm.json                                                # SLM metadata, targets and size
osr -t examples/targets.yaml improve --slm slm.json --cache-dir data # self-improvement cycle

# Serve and integrate
osr -t examples/targets.yaml -r examples/rules.yaml serve            # http://127.0.0.1:8000/docs (open, local dev)
osr -t examples/targets.yaml serve --generate-token                  # prints an osr_local_ token; all routes need it
osr -t examples/targets.yaml serve --require-auth --token "$TOKEN"   # production: refuse to start without a token
osr -t examples/targets.yaml mcp                                     # MCP server over stdio for an IDE
python -m opensmartroute -t examples/targets.yaml mcp                # the same, when `osr` is not on PATH
osr mcp --remote                                                     # stdio bridge to the platform you signed in to
osr mcp --url https://<platform>/mcp --api-key osr_...               # ...or an explicit endpoint and key
osr mcp-manifest tools.json --key-env OSR_MCP_KEY --out signed.json  # sign an MCP tools/list (--verify to check)
OSR_LLM_BASE_URL=mock:// osr -t examples/targets.yaml serve           # every LLM answers in-process: demos and CI, no key

# Launch a coding agent with the router as its model endpoint (the saved sign-in becomes its key)
osr run --list                                                       # the tools and how each one is wired
osr run claude-code                                                  # ANTHROPIC_BASE_URL + key -> the router, model auto
osr run codex --model osr/auto:cheap -- "fix the failing test"      # anything after -- goes to the tool untouched
osr run gemini-cli --print-env                                       # the variables, for your own shell

# Import another gateway's configuration as a catalogue
osr migrate litellm config.yaml --out ./router                       # targets + providers.json + aliases policy, and what is left to do by hand

# Size self-hosted model servers (CPU / RAM)
osr capacity show qwen3.5:4b glm-4.7-flash --memory 28Gi --cores 4  # footprint, fits?, tokens/s, warm vs cold latency
osr capacity plan qwen3.5:4b qwen2.5-coder:7b deepseek-r1:8b --memory 8Gi --hosts 3   # pack onto hosts + OLLAMA_* env
osr capacity plan glm-4.7-flash --memory 14Gi --shrink --strict      # coarser KV / shorter context to fit; exit 2 if not
osr capacity show --url http://ollama:11434 --memory 28Gi            # exact shapes from the server's /api/show
osr capacity ps --url http://ollama:11434                            # what the server holds in memory right now

# Manifests, stacks and skills
osr validate examples/ocm                                            # Open Capability Manifests
osr -t examples/targets.yaml export-ocm --out manifests/             # catalogue -> OCM manifests
osr stack init --out stack.yaml                                      # starter declarative stack
osr stack validate examples/stack.yaml                               # check (imports resolved)
osr stack plan examples/stack.yaml --against deployed.yaml           # diff against what runs today
osr stack apply examples/stack.yaml --route "refund order 42?"       # build the router, route once
osr stack apply registry://support-desk@1.0.0 --registry $OSR        # marketplace template
osr skills                                                           # validate SKILL.md packages

osr login without --token runs the OAuth 2.0 device authorization grant against the hosted platform (community or enterprise edition alike): the CLI prints a short code, opens /platform/cli/authorize in the browser, and after you approve it there the platform mints a workspace API key (osr_live_...) for that machine. Credentials are stored per profile in ~/.config/opensmartroute/credentials.json (%APPDATA%\opensmartroute on Windows, OSR_CONFIG_DIR overrides); flags, then OSR_API_URL / OSR_API_KEY, then the saved profile decide which one a command uses. A self-hosted osr serve has no accounts: osr token generate (or serve --generate-token) mints an osr_local_... token, the server accepts it as Authorization: Bearer or X-API-Key on every route except the health, metrics, /whoami and OpenAPI paths, and OSR_SERVER_AUTH_TOKENS / OSR_SERVER_REQUIRE_AUTH configure the same thing from the environment (see :class:ServerSettings in SDK guide).

The Routing Audit replays logged traffic ({"text": ..., "model": ..., "cost_usd": ...} per line) through the catalogue and reports savings against a baseline target, policy violations the logged route would have committed, and the route mix; --min-savings makes it a CI gate. In production the same numbers come from RouterBuilder.with_savings(baseline="max") and ep.savings.report().to_markdown().

osr run <agent> is the one-line way to put a coding agent or AI client on the router: Claude Code, Codex, the Gemini CLI, aider, OpenCode (launched), and Cursor, Continue, Zed, LibreChat, Open WebUI (configured in their own settings - --print-env prints what to paste). The tool's provider variables (ANTHROPIC_BASE_URL, OPENAI_BASE_URL, GOOGLE_GEMINI_BASE_URL and their keys) are set from the saved sign-in or --url / --token, --model names what the tool asks for (auto by default: the router decides per request), and the process is replaced by the tool's, so Ctrl-C and exit codes are the tool's own. Everything after the agent name - or after -- - goes to the tool untouched.

osr migrate litellm <config.yaml> reads a LiteLLM proxy configuration - its model_list (deployment names, provider models, API bases, credential variables, prices, context windows, embedding mode, rate limits, budgets) and the fallback chains - and writes, under --out, the catalogue (targets.yaml; one target per deployment name with cost, context window, rate_limit / budget metadata and the embeddings tag), providers.json (providers with base_url / api_key_env per distinct endpoint, models binding each target to provider/model; a literal key in the config is never copied, the variable it should come from is named) and, when a deployment name is not a valid target id, a policy file with the aliases that keep callers' model names working on the hosted platform. The report lists what was skipped and why, and what a proxy keeps in its database rather than its file - virtual keys, teams, users, budgets - as the steps left to do by hand.

10. Routing latency

Routing is pure Python on the request path. Measured with the reproducible routing benchmark (300 requests per configuration, 2026-10-09) on the 27-target example catalogue with its rules, CPython 3.11, x86-64 Windows laptop:

Catalogue size: 64 targets route in 14.5 ms p50 and 256 in 16.7 ms with RouterBuilder.with_retrieval(narrow_above=32, narrow_to=24) (versus 36 ms and 142 ms scoring every target); 1024 targets route in 24 ms. Cost is linear in scored targets x strategies; signal extraction is under 1 ms and independent of catalogue size. RouteDecision.trace.elapsed_ms records the routing time per request, MetricsTelemetry exposes route_latency_ms percentiles and the OpenTelemetry adapter emits the osr.route_latency_ms histogram. Full tables in Architecture.

11. Package layout

opensmartroute/
  core/        types (RouteTarget, RouteRequest, RouteDecision, Outcome, ...), registry
  signals/     lexicon extractors, task ontology, learned hashed models, verbalised difficulty,
               draft-response and response-uncertainty features
  policy/      hard-constraint filter
  strategies/  rules, capability, similarity, bandit, llm_judge, cascade, task_table, defer, progress,
               probe (Dirichlet), elastic, edge, auction, protocol, escalation, aggregate
  learning/    IRT, preference, LinUCB, Markov, AutoLearner, cold start, task credit,
               personal (user adaptation, skill affinity), handoff (mixture cure),
               multiturn (history-target joint embeddings)
  math/        bandits, irt, preference, markov, calibration, estimators, decision, dirichlet, energy
  realtime/    circuit breaker, rate limit, budget, health
  enterprise/  RouterBuilder, EnterpriseRouter, middleware, telemetry, audit, stores, ops
  security/    InputGuard, learned gadget detector, injection risk, redaction, limits, provenance
  adapters/    OpenAI-compatible client, harnesses, MCP (tools + server recommendation), A2A,
               frameworks, personas, SKILL.md, semantic-router import, embeddings, OpenTelemetry
  eval/        harness, baselines, robustness, frontiers, off-policy evaluation, dataset presets
  discovery.py schema-aware matching, cache-preserving selection, skill graph
  retrieval.py BM25 + dense retrieval, reciprocal-rank fusion, skill-set selection, meta-tools
  execution.py plan executor
  sdk.py       decorator SDK and plugin discovery
  settings.py  typed, environment-overridable tunables
  config.py    load_targets / load_rules (JSON, YAML)
  feedback/    outcome store and preference export
  router.py    orchestrator and plan builder
  aio.py       AsyncRouter
  server.py    FastAPI app and OpenAI-compatible proxy
  cli.py       osr command