Skip to content
OpenSmartRoute

User guide

Targets, rules, plans, providers, execution, learning, the CLI and SKILL.md packages.

docs/GUIDE.md

Everything the README leaves out: the full quick-start sequence, running plans end to end, wiring real providers, the enterprise builder, the research-track modules, routing latency and the repository layout. API reference material lives in SDK.md (decorator SDK, components, settings) and REFERENCE.md (generated; every module and exported name); design in ARCHITECTURE.md.

Contents

  1. Targets
  2. Routing
  3. Executing a plan
  4. Real providers
  5. Learning from outcomes
  6. Declaring components with decorators
  7. Enterprise builder
  8. Research-track modules
  9. Command line
  10. Routing latency
  11. Repository layout

1. Targets#

Every routable thing is a RouteTarget with the same contract: an id, a kind, declared capabilities, cost, latency_ms, a quality_prior, optional examples, instructions and a handler. Adding a target never requires retraining; it is routable from its declaration and learns from its outcomes.

KindWhat it isTypical capabilitiesWins the primary route?Executed by
llmA model endpoint (optionally per reasoning effort)domains, min/max_complexity, languages, context_windowyeschat_handler / any callable
skillA deterministic or narrow capability (SQL, translate, summarise, a SKILL.md package)actions, domains, token footprint as costyes, or fills the skill plan slotyour function; instructions are disclosed to the model
personaSystem-prompt / behaviour layer (primary: false)domains, actionsno; fills the persona slot on top of a model or agentinstructions become the system prompt in run()
agentAn agent harness: coding, research or support runtime with tools, memory and its own loopsupports_tools, tags: [autonomous], actionsyesCallableHarness / HTTPHarness / SubprocessHarness
toolA single MCP / function toolactions, input_schema in metadatayestool call
workflowA fixed multi-step pipelineactions, domainsyesworkflow engine
humanQueue, team or expert (Erlang-C capacity maths)actions: [escalate], tags: [safety]yes; also the abstention targetticketing / hand-off
# examples/targets.yaml (excerpt)
- id: coding-agent            # agent harness: repo access, tests, review loop
  kind: agent
  capabilities: { domains: [coding], actions: [code_generation, code_review, action],
                  min_complexity: 0.3, supports_tools: true, tags: [autonomous] }
  cost: { usd_per_1k_tokens: 0.008 }
  latency_ms: 6000
  examples: ["Refactor this class to remove the circular import and run the test suite."]

- id: skill-sql               # deterministic skill; `instructions` is its SKILL.md-style body
  kind: skill
  capabilities: { domains: [data_analysis, coding], actions: [code_generation, extract] }
  cost: { usd_per_1k_tokens: 0.001 }
  instructions: "Always qualify table names with the schema; never SELECT *."

- id: persona-legal-counsel   # never primary; layered on whichever model/agent wins
  kind: persona
  primary: false
  capabilities: { domains: [legal], actions: [reasoning, summarize, qa] }
  instructions: "You are cautious in-house counsel. Cite the clause, flag jurisdiction risk."

Catalogues load from JSON or YAML (config.load_targets), from MCP tools/list payloads (adapters.tools_from_mcp), A2A agent cards (adapters.agent_from_card), SKILL.md directories (adapters.load_skills), persona folders (adapters.load_personas) and vLLM semantic-router configurations (adapters.load_semantic_router_config).

Agent skills for coding assistants#

.claude/skills/ ships nine Agent-Skills packages (SKILL.md, agentskills.io format) that teach a coding agent how to use and extend OpenSmartRoute: osr-routing-catalogue, osr-decorator-sdk, osr-enterprise-builder, osr-evaluation, osr-security-hardening, osr-integrations, osr-deploy-serve, osr-platform and osr-contributing. Claude Code discovers them automatically in this repository; other agents can load the directory, and each package is published on the documentation site under /docs/skills/<name>. They are also valid routing targets: the same loader the SDK uses (load_skills) validates them in CI and the router fills the plan's skill slot with the right one.

osr skills                                           # validate + list (default root .claude/skills)
osr -t examples/targets.yaml --skills .claude/skills route "Create a JSONL eval dataset and gate CI on accuracy" --plan
# plan: llm:llm-mid -> skill:osr-evaluation

2. Routing#

from opensmartroute import Router, RouteRequest, RequestConstraints, Objective

router = Router(registry)

# plain text
d = router.route("Prove that sqrt(2) is irrational, step by step.")
print(d.target.id, f"{d.confidence:.2f}")
print(d.trace.explain())                   # per-strategy scores and rationales

# hard constraints and a per-request objective
req = RouteRequest(
    text="Summarize this patient intake note.",
    constraints=RequestConstraints(region="eu", data_boundary="private", max_cost_per_1k=0.005),
    objective=Objective(quality=1.0, cost=0.5, latency=0.1, quality_floor=0.6),
)
d = router.route(req)
print(d.trace.policy_rejections)           # {'llm-frontier': 'cost 0.015 > budget 0.005', ...}

# a composed plan: persona -> skill -> model
d = router.route("Review this NDA clause for GDPR liability.", plan=True)
for slot in d.plan.slots:
    print(slot.role, slot.target.id, f"{slot.confidence:.2f}")

Policy runs before scoring and is never traded off: enable flags, allow / deny lists, kind, region, data boundary, PII, tenant, cost / latency / token SLOs, modality, language and jailbreak risk. Strategies then score the survivors and a confidence-weighted ensemble produces a quality estimate; utility is w_q * quality - w_c * norm(cost) - w_l * norm(latency) with a hard quality_floor. With a ConformalCalibrator, RouteDecision.candidate_set is the smallest set with the requested coverage and abstain is set when no target is safe enough.

A generative judge can be consulted only when the ensemble is unsure:

from opensmartroute import LLMJudgeStrategy
router = Router(registry,
                llm_judge=LLMJudgeStrategy(llm=lambda prompt: my_llm(prompt)),
                escalate_llm_judge_below=0.6)

3. Executing a plan#

route() decides; run() decides and executes the whole plan: skill pre-processors run first, persona and skill instructions are composed into the system prompt, the primary target is called, and an Outcome is recorded for every participant so each one learns from the result.

from opensmartroute.adapters import (CallableHarness, HTTPHarness, SubprocessHarness,
                                     chat_handler, harness_handler, load_skills)

registry.get("llm-frontier").handler = chat_handler(openai, model="gpt-4o")
registry.get("coding-agent").handler = harness_handler(
    SubprocessHarness(["my-coding-agent", "--repo", "."], system_flag="--system"))
registry.get("research-agent").handler = harness_handler(
    HTTPHarness("https://agents.internal/research/run", api_key_env="AGENT_TOKEN"))
registry.get("skill-sql").handler = lambda req: lookup_schema(req.text)   # runs BEFORE the model
for skill in load_skills("examples/skills"):                              # SKILL.md folders
    registry.add(skill)

res = router.run("Is this indemnification clause enforceable in California?", task_id="ticket-812")
res.text                 # the primary target's answer
res.steps                # [persona:persona-legal-counsel, llm:llm-frontier] with per-step latency
res.system_prompt        # composed persona + skill instructions
res.outcomes             # one Outcome per participant, role="persona"/"skill"/None, shared task_id

A skill handler may return a RouteRequest (rewrite the request), a str (attached as context["skill_output"] and shown to the model) or None (instructions only). Async handlers work through AsyncRouter.run(); EnterpriseRouter.run() adds health, budgets and telemetry. See examples/end_to_end.py and SDK.md.

4. Real providers#

The bundled client is stdlib only and speaks the OpenAI chat and embeddings API, which covers OpenAI, Azure OpenAI, vLLM, Ollama, LiteLLM, OpenRouter, Groq and Mistral.

from opensmartroute import Router, LLMJudgeStrategy
from opensmartroute.adapters import OpenAICompatClient, chat_handler, embedder, judge_fn
from opensmartroute.strategies import CapabilityStrategy, SimilarityStrategy

openai = OpenAICompatClient()                                             # OPENAI_API_KEY from env
ollama = OpenAICompatClient("http://localhost:11434/v1", api_key="ollama")

registry.get("llm-frontier").handler = chat_handler(openai, model="gpt-4o")
registry.get("llm-small").handler   = chat_handler(ollama, model="llama3.1")

router = Router(registry,
                strategies=[CapabilityStrategy(),
                            SimilarityStrategy(embedder=embedder(openai, model="text-embedding-3-small"))],
                llm_judge=LLMJudgeStrategy(judge_fn(openai, model="gpt-4o-mini")),
                escalate_llm_judge_below=0.5)

d = router.route(req)
result = router.execute(d, req)          # ChatResult(text, prompt_tokens, completion_tokens, latency_ms)
res = router.run(req)                    # route + execute the full plan

adapters.sentence_transformers_embedder() and adapters.OpenTelemetryTelemetry are lazy imports behind the embeddings and otel extras. osr serve exposes a FastAPI app with an OpenAI-compatible /v1/chat/completions proxy that routes, executes and learns.

5. Learning from outcomes#

from opensmartroute import Outcome
router.learn(Outcome(request_id=d.request_id, target_id=d.target.id,
                     success=True, quality=0.9, cost_usd=0.002, latency_ms=1800,
                     domains=d.trace.signals.domains))

One Outcome fans out to every learner: Thompson bandits per domain, IRT ability and difficulty, Bradley-Terry preferences, LinUCB on the signal vector, the Markov / MDP lookahead, the task table and the personalisation posteriors. Outcome.task_id and step join delayed, task-level rewards to the routing decisions that produced them (learning.TaskCredit, learning.TaskPins). Page-Hinkley drift detection flags targets whose success rate degrades; every learner forgets with a configurable decay and can be merged across replicas (learning.merge_learners). Each learn() also records a learn.outcome event on the request's trace and, with with_audit(sink, outcomes=True), an outcome record in the audit chain, so a decision and its result stay joined by request_id (OBSERVABILITY.md; on the hosted platform POST /api/v1/feedback does the same).

Offline warm starts: learning.warm_start_from_matrix() replays a (prompt, {target: reward}) matrix, learning.warm_start() copies posteriors from the nearest declared neighbours and osr train --from rows.jsonl --out models.json fits the hashed signal models.

The routing SLM and the self-improvement loop#

learning.RouterSLM is a small routing model in one JSON file: a hashed dual encoder (SLMSettings.dim x feature_dim), a temperature calibrator and a snapshot of every target's price, latency and quality prior. It has no dependencies, predicts in microseconds and can be trained from any EvalRow source - labelled datasets, your own outcomes, synthetic ontology prompts or the full router's decisions (distill_router, which compresses rules + capability fit + learners + judge into one model).

from opensmartroute import RouterSLM, SLMStrategy, Router, RouteRequest
from opensmartroute.eval import DatasetCollector, synthetic_rows
from opensmartroute.learning import distill_router

slm = RouterSLM(reg.all(), seed=0)
slm.fit(synthetic_rows(reg.all()))                         # no data yet: seed prompts scored by capability fit
slm.fit(DatasetCollector("data").corpus())                 # collected battle datasets (RouteLLM, LMArena)
student, report = distill_router(Router(reg), texts)       # or copy the whole ensemble
print(report.summary())                                    # accuracy, loss, ECE, Brier, cost/1k
slm.save("slm.json")

router = Router(reg, strategies=[SLMStrategy(RouterSLM.from_file("slm.json"))])
d = router.route(RouteRequest(text="prove that sqrt(2) is irrational"))
print(slm.predict(d.request.text, objective=Objective(cost=2.0))[:3])   # calibrated p, quality, $/1k, utility

SLMStrategy sits in the ensemble under weight OSR_WEIGHTS_SLM and, like the other learners, updates the model online from every router.learn(Outcome).

Data comes from three collectors that all degrade gracefully when offline:

  • adapters.ModelCatalogue merges OpenRouter prices and Hugging Face model cards (downloads, likes, model-index benchmarks) into ModelCards, turns them into RouteTargets (targets(min_quality=, max_usd_per_1k=, measured=True)), prints a cost/quality benchmark and the Pareto frontier. refresh(leaderboard=True) adds Open LLM Leaderboard accuracies (IFEval, BBH, MATH, GPQA, MuSR, MMLU-Pro) as measured quality; apply_quality() takes the Bradley-Terry win rates that eval.model_quality derives from collected battles. Third-party descriptions are risk-scored and replaced when they look like injections.
  • eval.DatasetCollector pulls routing datasets from the Hugging Face datasets-server into a JSONL cache. Ten public sources are on by default: the pairwise battles routellm-battles (GPT-4-judged), arena-55k, arena-100k, arena-140k (LMArena human votes, with the code / language / math / hard-prompt tags), ppe-human, webdev-arena, mt-bench-human and reward-bench, the per-model judge scores of ultrafeedback, and routellm-gpt4 (RouteLLM's "was Mixtral good enough for this prompt" labels). RouterBench / RouterEval / the raw arena conversations are known but gated: download them with your own Hub token and load the file with osr collect --file routerbench=routerbench.jsonl --preset routerbench (.jsonl / .json / .csv in the preset's shape, --tier folds model columns onto your targets), and any dataset works through DatasetSource. Battles name a moving population of models, so --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small folds every side onto your tiers as it is parsed (tier_of(): the ARENA_TIERS table, name rules for the 2024-26 generations, then the parameter count in the name) - the RouteLLM question "was the cheaper tier good enough?". Collection is polite and resumable: pages are paced, 429s are retried with backoff, and a run the server cuts short keeps its rows and continues from the recorded offset on the next osr collect (<name>.meta.json). rows_from_feedback turns your FeedbackStore into labelled rows.
  • adapters.WebKnowledge searches Hugging Face (text-generation repos, per-keyword fallback) and DuckDuckGo (Brave with BRAVE_API_KEY) for new models and datasets; fetches are https-only, byte-capped and page text is risk-scored.

learning.SelfImprover closes the loop as a champion/challenger cycle: refresh the catalogue, discover, gather evidence, split, train a challenger, calibrate it on the holdout and promote it only when holdout accuracy improves by at least OSR_SLM_MIN_GAIN (or loss drops at equal accuracy). Every cycle is appended to a JSONL history.

osr catalogue --cache data/catalogue.json --hf "llama instruct" --leaderboard --markdown   # live cost / quality
osr collect --cache-dir data --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small
osr -t targets.yaml slm train --out slm.json --cache-dir data --report          # every *-tiers cache, holdout metrics
osr slm predict slm.json "write a python function to parse csv" --cost-weight 1.0
osr -t targets.yaml improve --slm slm.json --cache-dir data --catalogue data/catalogue.json \
    --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small \
    --search "qwen2.5 instruct" --interval 3600                              # daemon; --offline for air-gapped

Serve the result: osr -t targets.yaml --slm slm.json serve, or in the container OSR_SLM=/config/slm.json (Helm config.slm). With OSR_LLM_BASE_URL (and OSR_LLM_API_KEY / OSR_LLM_MODEL) pointing at any OpenAI-compatible endpoint - OpenAI, Azure OpenAI, vLLM, Ollama, OpenRouter - every LLM target gets a chat handler from metadata.model, so /v1/chat/completions with model: "auto" routes and answers.

Speed, and what the numbers mean#

Training is plain SGD over hashed features; with numpy installed (extra fast, included in the serve image) the same steps run on dense arrays - 18 000 rows train in under a minute, ten times the pure-Python speed - and prediction projects through a dense mirror of the weights. OSR_SLM_BACKEND=auto|numpy|python chooses; both paths seed identically and write the same model file, so a zero-dependency deployment reads a numpy-trained model unchanged. The model carries a learned per-target bias (the base rate a unit-norm dot product cannot express) and trains with inverse-time learning-rate decay (OSR_SLM_LR_DECAY), four epochs and l2 1e-3 by default - on the 18 000-row mixed corpus the old constant-rate, eight-epoch defaults overfit below the constant baseline.

Read --report against that baseline. Human preference between tiers is noisy: on the ten-source corpus "always the mid tier" is acceptable for 64 % of holdout prompts and the SLM reaches 64-66 % with a holdout cross-entropy of 0.98-1.00 (the constant predictor: 1.03), well calibrated (ECE 0.05-0.08). It is much stronger where the label really depends on the prompt - 94 % on routellm-gpt4 (baseline 86 %), +12 points on arena-100k - and no better than the prior on sources where the prompt carries no signal. That is what a learned routing prior is; the ensemble adds rules, capability, similarity and the bandit on top.

A corpus of model battles only knows the models it compared. Scored rows therefore train a softmax over the scored population (every target any row scores, plus OSR_SLM_NULL_TARGETS learned "none of these" logits that keep it from saturating), so a tool, skill or agent the data never mentions keeps its catalogue embedding instead of being pushed to zero, and SLMStrategy abstains on such targets - its confidence is scaled by the share of the request's candidates it has evidence about. Rows that name only an expected target (your own feedback, examples/eval_dataset.jsonl) contrast against the whole catalogue, which is how the model learns the non-LLM targets once outcomes arrive.

Where a transformer helps#

The hashed encoder is a bag of words, bigrams and character grams: it cannot tell "explain this SQL" from "SQL that explains itself". Two opt-in encoders add that context, both persisted in the same JSON file and restored by RouterSLM.from_file:

  • OSR_SLM_ENCODER=attention adds learning.AttentionEncoder, a pure-Python transformer block (hashed token embeddings, sinusoidal positions, OSR_SLM_ATTENTION_HEADS heads of OSR_SLM_ATTENTION_HEAD_DIM, residual, attention pooling) to the query encoder: $q = W,\phi(x) + \operatorname{Attn}(x)$. No dependencies, deterministic, hand-derived gradients, but quadratic in OSR_SLM_ATTENTION_MAX_TOKENS - a few milliseconds per prompt and roughly four times the training time - so measure on your holdout (osr slm train --report) before switching. On the public arena corpora (six thousand pairwise rows, seventeen tiered targets) it gained two points at 1 500 rows but lost five at full size with the default budget: the hashed encoder stays the default, and attention earns its keep on order-sensitive traffic, not on generic chat.
  • OSR_SLM_EMBEDDER=sentence-transformers/all-MiniLM-L6-v2 (extra embeddings) or RouterSLM(embedder=callable) appends a frozen pretrained embedding as dense features (learning.EmbeddingFeaturizer): W then learns a linear head on top of a real transformer while the hashed features keep the surface cues (code fences, length, exact vocabulary). The file records the embedder's name; a custom callable is passed again to from_file(embedder=...).

Self-operation#

osr serve --autopilot runs the SelfImprover inside the server (learning.Autopilot): a cycle every OSR_SLM_AUTOPILOT_INTERVAL_S seconds, an early one when learning.DriftMonitor (Page-Hinkley over the success and quality of every outcome the router learns from, via Router.observers) sees the served quality drop, never closer than OSR_SLM_AUTOPILOT_MIN_GAP_S. Feedback is joined back to prompts through the router's request memory (OSR_SLM_AUTOPILOT_REMEMBER texts), an accepted challenger is hot-swapped into the SLMStrategy and written back to disk, a failing cycle is logged and the server keeps serving. GET /stats reports autopilot (cycles, promotions, drift, last report, next run) and POST /autopilot/cycle schedules one now. In the container: OSR_AUTOPILOT=1 with OSR_STATE mounted (Helm autopilot.enabled); --offline keeps it off the network. Each cycle is traced as an autopilot.cycle span with a learn.improve event (and learn.promote when the challenger wins), so GET /events?name=learn.* shows the loop at work. The hosted platform runs the same loop from OSR_PLATFORM_SLM / OSR_PLATFORM_AUTOPILOT and shows it at /platform/dashboard/learning (PLATFORM.md, section 10).

osr -t targets.yaml --slm slm.json --state state serve --autopilot --autopilot-interval 1800 \
    --catalogue data/catalogue.json --source arena-55k --tier frontier=llm-frontier --tier small=llm-small
curl -s localhost:8000/stats | jq .autopilot

6. Declaring components with decorators#

Targets, strategies, signal extractors, policy rules, middleware and telemetry can be declared where they live and assembled by a ComponentRegistry (details in SDK.md):

import opensmartroute as osr

@osr.tool("weather", domains=["travel"], actions=["lookup"], cost=0.0, examples=["weather in paris"])
def weather(request, **kw):
    """Current weather for a city."""          # docstring -> target.description
    return lookup(request.text)

@osr.strategy(weight=0.8)                      # fn(request, signals, candidates) -> {id: score}
def geo_affinity(request, signals, candidates):
    region = request.constraints.region
    return {t.id: 1.0 if region in t.constraints.regions else 0.4 for t in candidates}

@osr.policy_rule                               # fn(target, request, signals) -> reason | None
def business_hours_only(target, request, signals):
    if target.kind is osr.TargetKind.HUMAN and request.context.get("after_hours"):
        return "human queues are closed"

router = osr.components.router(registry)
app = osr.components.builder(registry).with_auto_learning().build()

Every tunable (ensemble weights, thresholds, scales) lives in one typed Settings object and can be overridden per process (osr.configure(...)) or through OSR_<GROUP>_<FIELD> environment variables; osr settings prints the effective values.

7. Enterprise builder#

from opensmartroute.enterprise import (RouterBuilder, CacheMiddleware, TenantMiddleware,
                                       LoggingTelemetry, MetricsTelemetry, FileAuditSink)
from opensmartroute.security import GuardMiddleware

metrics = MetricsTelemetry()
app = (
    RouterBuilder(registry)
    .with_rules(rules)                              # declarative preferences
    .with_defaults()                                # capability + similarity + Thompson bandit
    .with_auto_learning(state_dir=".osr-state")     # IRT + Bradley-Terry + LinUCB + Markov lookahead
    .with_health(latency_slo_ms=3000)               # circuit breakers, rate limits, budgets
    .with_middleware(GuardMiddleware(redact=True),  # gadget defence + PII redaction
                     TenantMiddleware({"acme": {"max_cost_per_1k": 0.005, "data_boundary": "private"}}),
                     CacheMiddleware(ttl_s=30))
    .with_telemetry(LoggingTelemetry(), metrics)
    .with_audit(FileAuditSink("audit.jsonl"))
    .build()
)

d = app.route(RouteRequest("Summarize this contract", constraints=RequestConstraints(tenant="acme")))
app.learn(Outcome(d.request_id, d.target.id, success=True, quality=0.9, cost_usd=0.002, latency_ms=1200,
                  domains=d.trace.signals.domains, complexity=d.trace.signals.complexity))
print(metrics.snapshot()["route_latency_ms"], app.health_snapshot())

with_retrieval(narrow_above=, narrow_to=) switches to retrieve-then-rank for large catalogues, with_queue_awareness() replaces static latency with a queue-aware estimate, with_shadow(candidate, mode="shadow" | "ab") compares policies with SPRT, with_calibration() adds temperature scaling and conformal candidate sets, with_fair_share() enforces tenant weights and with_state_store() accepts Redis, SQL, encrypted, versioned or batched stores. Ports (state, telemetry, audit) are ABCs; see ENTERPRISE.md.

8. Research-track modules#

Each module below implements one roadmap item with tests; papers are cited in RESEARCH.md and formulas in MATH.md.

Signals from a draft or a small model#

from opensmartroute.signals import (DEFAULT_EXTRACTORS, VerbalisedDifficultySignal,
                                    DraftResponseSignal, extract_signals)

extractors = [*DEFAULT_EXTRACTORS,
              VerbalisedDifficultySignal(source=lambda req: small_model_difficulty(req.text)),
              DraftResponseSignal(drafter=lambda req: cheap_model(req.text))]
router = Router(registry, extractors=extractors)

VerbalisedDifficultySignal accepts numbers, 7/10 or words (trivial ... expert) from context["difficulty"] or a callable and blends them into complexity. DraftResponseSignal measures hedging, self-corrections, query overlap and length ratio of a cheap draft.

Response-side uncertainty#

from opensmartroute.signals import UncertaintyGate, EventTrigger, TriggerRule, response_uncertainty
from opensmartroute.strategies import Cascade

gate = UncertaintyGate(sampler=lambda req, n: [cheap_model(req.text) for _ in range(n)], n_samples=4)
cascade = Cascade(quality_gate=gate, threshold=0.6)                   # escalate when samples disagree
result = cascade.run(req, router.route(req).alternatives)

feats = response_uncertainty(samples, p_true=judge_p_true)
trigger = EventTrigger([TriggerRule("semantic_entropy", 0.5, "reroute"),
                        TriggerRule("agreement", 0.5, "ask_human", below=True)])
trigger.check(feats)                                                  # ["reroute", ...]

Hidden-state Dirichlet probe#

from opensmartroute.strategies import HiddenStateStrategy

probe = HiddenStateStrategy(state_fn=lambda req: host_model.hidden_state(req.text),
                            targets=["small", "large"], dim=4096)
probe.fit(logged_states, logged_winners, epochs=20)                   # warm start
router = Router(registry, strategies=[*default_strategies(), probe])

The probe's confidence is 1 - K/S (epistemic uncertainty), so unfamiliar states defer to the other strategies. math.DirichletProbe is usable on its own with any feature vector.

Elastic models and token budgets#

from opensmartroute.strategies import BudgetVariant, TokenBudgetStrategy, expand_elastic

for sib in expand_elastic(registry.get("reasoner"),
                          [BudgetVariant(1024, quality=0.7, cost_scale=0.4),
                           BudgetVariant(8192, quality=0.9)]):
    registry.add(sib)                                                 # reasoner@1024, reasoner@8192
router = Router(registry, strategies=[*default_strategies(), TokenBudgetStrategy()])

Edge and cloud tiers#

Tag targets with metadata: {tier: edge} or {tier: cloud} and add EdgeCloudStrategy(). Edge success is a Thompson posterior per complexity bucket, penalised when decode time eats into the latency SLO; the cloud tier is penalised by upload time and excluded when the request's data_boundary is private or on_prem and the target is public.

Auctions#

AuctionStrategy(bid_fn=None, value=0.01) lets each target bid a claimed success probability and a price (default_bid derives both from the catalogue). Claims are corrected by each bidder's observed bias; the highest corrected surplus wins and pays the second price. Supply a bid_fn for multi-vendor catalogues where providers self-report.

Protocol selection, aggregation, self-escalation, hand-off#

from opensmartroute.strategies import ProtocolPolicy, MixtureOfAgents, SelfEscalation, wrap_stream
from opensmartroute.learning import HandoffPolicy, MixtureCureModel

policy = ProtocolPolicy()                                             # risk / budget -> protocol
choice = policy.choose(req, decision, decision.trace.signals, uncertainty=gate(req, draft))
if choice.protocol == "aggregate":
    res = MixtureOfAgents(k=3, budget_usd=0.05).run(decision, req)   # top-k answers, majority vote
    for o in res.outcomes(req):
        router.learn(o)

monitor = SelfEscalation.for_target(decision.target)                # streaming competence posterior
streamed = wrap_stream(provider.stream(req.text), monitor)
if streamed.escalated:
    ...                                                               # restart on a stronger target

handoff = HandoffPolicy(MixtureCureModel(), fallback_target="senior-agent", pins=task_pins)
if handoff.step(task_id, teacher_risk):                               # permanent hand-off
    target = handoff.target_for(task_id, decision.target.id)
handoff.finish(task_id, failed=False)                                 # censored observation

Personalisation#

learning.UserAdaptiveStrategy reads context["user_id"] (or profile["user_id"]) and keeps a Beta posterior per user and target, shrunk toward similar users and the global posterior; observe_user() warm-starts from logged interactions. learning.SkillAffinity.relevance(profile, skills) feeds retrieval.select_skill_set(relevance=) so skill sets depend on who is asking.

Multi-turn conversations#

from opensmartroute.learning import HistoryTargetStrategy

router = Router(registry, strategies=[*default_strategies(), HistoryTargetStrategy(turns=6)])
d = router.route(RouteRequest("can you explain more?", history=chat_history,
                              context={"last_target": "llm-small", "last_failed": True}))

The same follow-up message routes differently depending on what the conversation has been about: the strategy learns a logistic model over the joint embedding of the recent history and each target's catalogue embedding, so a target that keeps failing in legal threads but succeeds in coding threads is scored accordingly, and an unseen target is still scored through its description and examples. last_target gives the incumbent a small continuity bonus that turns into a penalty when the previous turn failed.

Discovery beyond text similarity#

from opensmartroute.discovery import SchemaAwareStrategy, CachePreservingSelector, SkillGraph

router = Router(registry, strategies=[*default_strategies(), SchemaAwareStrategy()])

selector = CachePreservingSelector(prefix_size=8, evict_after=5)
tools = selector.select(session_id, candidates, needed=[d.target.id])   # prefix-stable tool order

graph = SkillGraph(load_skills("skills"))                               # osr-requires / osr-conflicts
ordered, dropped = graph.compose(["report-builder"])

MCP server recommendation and semantic-router import#

from opensmartroute.adapters import recommend_servers, load_semantic_router_config

for rec in recommend_servers("open a pull request", server_cards, k=2,
                             constraints=req.constraints, allowed_auth=["none", "token"]):
    print(rec.server.name, rec.score, rec.rationale)

imported = load_semantic_router_config("config.yaml")                  # vLLM semantic-router
router = Router(imported.registry, strategies=[RulesStrategy(imported.rules), *default_strategies()])

Energy and carbon#

math.EnergyModel fits Wh per target from measured samples; math.hardware_profile("h100-sxm") gives priors when no meter exists. Both convert to gCO2 with a grid factor and feed wh_per_1k_tokens / gco2_per_1k_tokens on the target's cost, which Objective(energy=, carbon=) trades against quality.

9. Command line#

Install osr with the platform installer (curl -LsSf https://opensmartroute.ai/install.sh | sh on Linux and macOS, irm https://opensmartroute.ai/install.ps1 | iex in PowerShell) or with uv tool install 'opensmartroute[yaml,server]' (add fast for numpy-accelerated SLM training); osr --version confirms the install.

osr takes the catalogue with -t targets.yaml, optional rules with -r rules.yaml and an optional routing SLM with --slm slm.json; --help on any subcommand lists its options. Grouped by task:

# Sign in and manage access tokens
osr login                                                            # browser sign-in to the hosted platform
osr login --url https://osr.example.com --profile work               # another deployment, kept as a profile
osr login --url http://router:8000 --token osr_local_...             # self-hosted osr serve token
echo "$TOKEN" | osr login --with-token                               # non-interactive (CI)
osr whoami                                                           # workspace, plan, edition, token source
osr token generate                                                   # mint an osr_local_ token for osr serve
osr token create --name ci                                           # new platform API key (shown once)
osr token list                                                       # keys of the signed-in workspace
osr token revoke <key-id>
osr logout                                                           # forget the profile (--all for every one)
# Route, quote and inspect
osr -t examples/targets.yaml -r examples/rules.yaml route "I want a refund for order #123" --plan
osr -t examples/targets.yaml route "Summarise this HR complaint" \
    --constraint data_boundary=on_prem --constraint region=eu     # hard constraints; --kinds agent,tool
osr -t examples/targets.yaml -r examples/rules.yaml route "" --event order.shipped --plan   # textless event
osr -t examples/targets.yaml estimate "Translate this contract" --monthly 100000   # cost per candidate + projection
osr -t examples/targets.yaml targets                                 # the catalogue as the router sees it
osr -t examples/targets.yaml stats                                   # feedback statistics per target
osr settings --json                                                  # effective tunables

# Evaluate and audit
osr -t examples/targets.yaml -r examples/rules.yaml eval examples/eval_dataset.jsonl --frontier --calibration
osr -t examples/targets.yaml audit traffic.jsonl --baseline llm-frontier --monthly 3000000 --markdown
                                                                     # Routing Audit: savings + violations
osr -t examples/targets.yaml ope decisions.jsonl                     # off-policy estimate (IPS/SNIPS/DR) of logged decisions
osr -t examples/targets.yaml safety --learned-guard                  # red-team suite

# Learn
osr train --from rows.jsonl --out models.json                        # fit signal models
osr catalogue --cache data/catalogue.json --hf "llama instruct"      # live model prices + quality
osr collect --cache-dir data --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small   # Hub datasets
osr collect --cache-dir data --file routerbench=routerbench.jsonl --preset routerbench             # a gated file you hold
osr -t examples/targets.yaml slm train --out slm.json --report       # routing SLM (see section 5)
osr slm info slm.json                                                # rows, sources, encoder, calibration, fit history
osr -t examples/targets.yaml --slm slm.json route "parse this csv"   # SLM inside the ensemble
osr slm eval slm.json rows.jsonl                                     # SLM accuracy on labelled rows
osr slm predict slm.json "parse this csv" --top 3                    # what the SLM alone would rank
osr slm info slm.json                                                # SLM metadata, targets and size
osr -t examples/targets.yaml improve --slm slm.json --cache-dir data # self-improvement cycle

# Serve and integrate
osr -t examples/targets.yaml -r examples/rules.yaml serve            # http://127.0.0.1:8000/docs (open, local dev)
osr -t examples/targets.yaml serve --generate-token                  # prints an osr_local_ token; all routes need it
osr -t examples/targets.yaml serve --require-auth --token "$TOKEN"   # production: refuse to start without a token
osr -t examples/targets.yaml mcp                                     # MCP server over stdio for an IDE
osr mcp --remote                                                     # stdio bridge to the platform you signed in to
osr mcp --url https://<platform>/mcp --api-key osr_...               # ...or an explicit endpoint and key
osr mcp-manifest tools.json --key-env OSR_MCP_KEY --out signed.json  # sign an MCP tools/list (--verify to check)

# Manifests, stacks and skills
osr validate examples/ocm                                            # Open Capability Manifests
osr -t examples/targets.yaml export-ocm --out manifests/             # catalogue -> OCM manifests
osr stack init --out stack.yaml                                      # starter declarative stack
osr stack validate examples/stack.yaml                               # check (imports resolved)
osr stack plan examples/stack.yaml --against deployed.yaml           # diff against what runs today
osr stack apply examples/stack.yaml --route "refund order 42?"       # build the router, route once
osr stack apply registry://support-desk@1.0.0 --registry $OSR        # marketplace template
osr skills                                                           # validate SKILL.md packages

osr login without --token runs the OAuth 2.0 device authorization grant against the hosted platform (community or enterprise edition alike): the CLI prints a short code, opens /platform/cli/authorize in the browser, and after you approve it there the platform mints a workspace API key (osr_live_...) for that machine. Credentials are stored per profile in ~/.config/opensmartroute/credentials.json (%APPDATA%\opensmartroute on Windows, OSR_CONFIG_DIR overrides); flags, then OSR_API_URL / OSR_API_KEY, then the saved profile decide which one a command uses. A self-hosted osr serve has no accounts: osr token generate (or serve --generate-token) mints an osr_local_... token, the server accepts it as Authorization: Bearer or X-API-Key on every route except the health, metrics, /whoami and OpenAPI paths, and OSR_SERVER_AUTH_TOKENS / OSR_SERVER_REQUIRE_AUTH configure the same thing from the environment (see :class:ServerSettings in SDK.md).

The Routing Audit replays logged traffic ({"text": ..., "model": ..., "cost_usd": ...} per line) through the catalogue and reports savings against a baseline target, policy violations the logged route would have committed, and the route mix; --min-savings makes it a CI gate. In production the same numbers come from RouterBuilder.with_savings(baseline="max") and ep.savings.report().to_markdown().

10. Routing latency#

Routing is pure Python on the request path. Measured with python scripts/bench.py --n 200 --scale on the 16-target example catalogue, CPython 3.11, x86-64 laptop:

Configurationp50p95p99
signals + policy + rules + capability + similarity + bandit5.8 ms6.2 ms6.5 ms
+ IRT + preference + LinUCB + Markov + health8.6 ms12.9 ms17.8 ms
+ LLM judge (only when confidence < escalate_llm_judge_below)+ one provider round-tripn/an/a

Catalogue size: 64 targets route in 14.5 ms p50 and 256 in 16.7 ms with RouterBuilder.with_retrieval(narrow_above=32, narrow_to=24) (versus 36 ms and 142 ms scoring every target); 1024 targets route in 24 ms. Cost is linear in scored targets x strategies; signal extraction is under 1 ms and independent of catalogue size. RouteDecision.trace.elapsed_ms records the routing time per request, MetricsTelemetry exposes route_latency_ms percentiles and the OpenTelemetry adapter emits the osr.route_latency_ms histogram. Full tables in ARCHITECTURE.md.

11. Repository layout#

src/opensmartroute/
  core/        types (RouteTarget, RouteRequest, RouteDecision, Outcome, ...), registry
  signals/     lexicon extractors, task ontology, learned hashed models, verbalised difficulty,
               draft-response and response-uncertainty features
  policy/      hard-constraint filter
  strategies/  rules, capability, similarity, bandit, llm_judge, cascade, task_table, defer, progress,
               probe (Dirichlet), elastic, edge, auction, protocol, escalation, aggregate
  learning/    IRT, preference, LinUCB, Markov, AutoLearner, cold start, task credit,
               personal (user adaptation, skill affinity), handoff (mixture cure),
               multiturn (history-target joint embeddings)
  math/        bandits, irt, preference, markov, calibration, estimators, decision, dirichlet, energy
  realtime/    circuit breaker, rate limit, budget, health
  enterprise/  RouterBuilder, EnterpriseRouter, middleware, telemetry, audit, stores, ops
  security/    InputGuard, learned gadget detector, injection risk, redaction, limits, provenance
  adapters/    OpenAI-compatible client, harnesses, MCP (tools + server recommendation), A2A,
               frameworks, personas, SKILL.md, semantic-router import, embeddings, OpenTelemetry
  eval/        harness, baselines, robustness, frontiers, off-policy evaluation, dataset presets
  discovery.py schema-aware matching, cache-preserving selection, skill graph
  retrieval.py BM25 + dense retrieval, reciprocal-rank fusion, skill-set selection, meta-tools
  execution.py plan executor
  sdk.py       decorator SDK and plugin discovery
  settings.py  typed, environment-overridable tunables
  config.py    load_targets / load_rules (JSON, YAML)
  feedback/    outcome store and preference export
  router.py    orchestrator and plan builder
  aio.py       AsyncRouter
  server.py    FastAPI app and OpenAI-compatible proxy
  cli.py       osr command
deploy/        Dockerfile, entrypoint, Helm chart
examples/      targets.yaml, rules.yaml, skills/, eval_dataset.jsonl, demos
.claude/skills Agent-Skills packages for coding agents; also routable skill targets
tests/         unit and integration tests, public API snapshot, docs coverage gate (test_docs.py)
scripts/       bench.py, release.py, brand_build.py, api_reference.py, exit_criteria.py
docs/          GUIDE, REFERENCE (generated), RESEARCH, ARCHITECTURE, MATH, ENTERPRISE, SECURITY, SDK, ROADMAP, BRAND