# OpenSmartRoute documentation > OpenSmartRoute routes every request to the best LLM, agent, skill, persona, tool or human under hard constraints, explains the decision and learns from outcomes. Open source, zero core dependencies. Every published page of https://opensmartroute.ai/docs as Markdown, in reading order. The short index is https://opensmartroute.ai/llms.txt. ---

OpenSmartRoute

# OpenSmartRoute > **An open, intelligent route to the right decision, solution, or destination.**

CI CodeQL Release PyPI Python 3.10+ Container image License: Apache-2.0 Ruff Typed Zero runtime dependencies

OpenSmartRoute is an open-source **AI decision control plane**. For each request it picks the best **LLM, agent harness, skill, persona, tool, workflow or human**, honours hard constraints (privacy, region, budget, latency), explains the choice, executes the resulting plan and **learns from every outcome**. The core is pure Python with zero runtime dependencies. ```mermaid flowchart LR R([request]) --> G["guard
redact PII"] G --> S["signals
< 1 ms"] S --> P["policy
hard constraints"] P --> ST["strategies + ensemble utility
rules, capability, similarity, bandit, LLM judge"] ST --> D["decision
trace + plan"] D --> X["execute
persona -> skill -> model / agent / tool / human"] X -. outcomes .-> ST ``` ## Why - **Everything is a route target.** One `RouteTarget` contract for models, agent harnesses, SKILL.md packages, personas, MCP tools, workflows and human queues; one policy layer; one learning loop. - **Plans, not just picks.** `route(plan=True)` composes persona, skill and model; `run()` executes the plan and records an `Outcome` per participant. An instructions-only skill or persona runs on the plan's model with its body disclosed in the system prompt. - **Constraints are never traded off.** PII, data boundary, region, tenant, cost and latency SLOs are filtered before any score is computed, with the rejection reason in the trace. Redacted PII is restored only for targets allowed to hold it; every other target, and every log, sees placeholders. - **Learns in production.** Thompson bandits, Item Response Theory, Bradley-Terry preferences, LinUCB, Markov lookahead, task-level credit assignment, drift detection and forgetting. - **Honest evaluation.** Baselines, oracle, label-noise floor, paraphrase robustness, calibration (ECE, Brier, conformal sets), off-policy estimators, public benchmark presets. - **Secure by design.** Learned and heuristic guards against rerouting gadgets and prompt injection, PII redaction, resource limits, signed MCP manifests, encrypted state, hash-chained audit. - **Enterprise-ready.** Builder, middleware, telemetry / state / audit ports, Redis and SQL stores, shadow and A/B routing with SPRT, tenant fair share, async facade, container image and Helm chart. ## Install The `osr` command line and the Python package ship together. On Linux and macOS: ```bash curl -LsSf https://opensmartroute.ai/install.sh | sh ``` On Windows (PowerShell): ```powershell powershell -ExecutionPolicy ByPass -c "irm https://opensmartroute.ai/install.ps1 | iex" ``` The installer puts `osr` in an isolated environment (uv, pipx or a private venv - whichever is available, never the system Python) and adds it to your PATH. `OSR_VERSION=1.0.0` pins a release, `OSR_EXTRAS=all` installs every optional dependency, `OSR_INSTALLER=uv|pipx|venv` forces a backend and `OSR_NO_MODIFY_PATH=1` leaves your shell configuration alone. The scripts are [install.sh](https://github.com/isathish/OpenSmartRoute/blob/main/install.sh) and [install.ps1](https://github.com/isathish/OpenSmartRoute/blob/main/install.ps1) in this repository and attached to every GitHub release. If you already manage Python tools yourself: ```bash uv tool install 'opensmartroute[yaml,server]' # or: pipx install 'opensmartroute[yaml,server]' pip install opensmartroute # library only, zero runtime dependencies pip install 'opensmartroute[yaml]' # + YAML catalogues and rules pip install 'opensmartroute[server]' # + FastAPI server and OpenAI-compatible proxy pip install 'opensmartroute[embeddings]' # + sentence-transformers similarity pip install 'opensmartroute[otel]' # + OpenTelemetry telemetry pip install 'opensmartroute[crypto]' # + AES-GCM encrypted state ``` Container image: `ghcr.io/isathish/opensmartroute:` ([deploy/README.md](https://opensmartroute.ai/docs/deploy.md)). From a checkout: `pip install -e '.[dev]'`. Then sign in. The hosted platform (community or enterprise edition) uses a browser hand-shake; a self-hosted `osr serve` accepts a token you generate yourself: ```bash osr login # opens https://opensmartroute.ai/cli/authorize osr login --url https://osr.example.com # your own platform deployment osr login --url http://router:8000 --token osr_local_... # self-hosted server (see `osr serve --generate-token`) osr whoami # workspace, plan, edition, key ``` Credentials live in `~/.config/opensmartroute/credentials.json` (`%APPDATA%\opensmartroute` on Windows), one profile per `--profile`; `OSR_API_URL` / `OSR_API_KEY` override them in CI. ## Quick start ```python from opensmartroute import Router, TargetRegistry, RouteTarget, TargetKind, Capabilities, Outcome registry = TargetRegistry([ RouteTarget("llm-small", TargetKind.LLM, capabilities=Capabilities(max_complexity=0.45), cost={"usd_per_1k_tokens": 0.0002}, latency_ms=300, quality_prior=0.55, examples=["Hi, how are you?", "What is the capital of France?"]), RouteTarget("llm-frontier", TargetKind.LLM, capabilities=Capabilities(min_complexity=0.5, domains=["math", "coding"]), cost={"usd_per_1k_tokens": 0.015}, latency_ms=2500, quality_prior=0.93, examples=["Prove the theorem step by step."]), RouteTarget("human", TargetKind.HUMAN, capabilities=Capabilities(actions=["escalate"], tags=["safety"]), cost={"usd_per_1k_tokens": 0.5}, latency_ms=300_000), ]) router = Router(registry) d = router.route("Prove that sqrt(2) is irrational, step by step.") print(d.target.id, f"{d.confidence:.2f}") # llm-frontier 0.97 print(d.trace.explain()) # per-strategy scores and rationales router.learn(Outcome(request_id=d.request_id, target_id=d.target.id, success=True, quality=0.9, cost_usd=0.002, latency_ms=1800, domains=d.trace.signals.domains)) ``` Hard constraints and a per-request objective: ```python from opensmartroute import RouteRequest, RequestConstraints, Objective req = RouteRequest("Summarize this patient intake note.", constraints=RequestConstraints(region="eu", data_boundary="private", max_cost_per_1k=0.005), objective=Objective(quality=1.0, cost=0.5, latency=0.1, quality_floor=0.6)) d = router.route(req) print(d.trace.policy_rejections) # {'llm-frontier': 'cost 0.015 > budget 0.005', ...} ``` Catalogue and rules from YAML, evaluated and served from the command line: ```bash osr -t examples/targets.yaml -r examples/rules.yaml route "I want a refund for order #123" --plan osr -t examples/targets.yaml -r examples/rules.yaml eval examples/eval_dataset.jsonl --frontier osr -t examples/targets.yaml -r examples/rules.yaml serve # http://127.0.0.1:8000/docs ``` Wiring real providers, executing plans with `router.run()`, agent harnesses, SKILL.md and MCP catalogues, the decorator SDK and the enterprise builder are covered step by step in [docs/GUIDE.md](https://opensmartroute.ai/docs/GUIDE.md). ## What it routes | Kind | Examples | Executed by | |---|---|---| | `llm` | model endpoints, per reasoning effort or token budget | OpenAI-compatible client or any callable | | `agent` | coding, research and support harnesses with tools and memory | callable, HTTP or subprocess harness | | `skill` | deterministic capabilities, SKILL.md packages | your function; instructions disclosed to the model | | `persona` | system-prompt layers composed on top of the primary target | `run()` prompt composition | | `tool` | MCP or function tools, with schema-aware matching | tool call | | `workflow` | fixed multi-step pipelines | workflow engine | | `human` | queues and experts with Erlang-C capacity maths | ticketing or hand-off; also the abstention target | Catalogues load from YAML or JSON, MCP `tools/list` payloads, A2A agent cards, SKILL.md and persona directories and vLLM semantic-router configurations; the router plugs into LangGraph and Agent Framework graphs as a node and into any chat loop as an OpenAI tool. ## How it decides 1. **Signals** (under 1 ms): task type, domains, complexity, reasoning need, PII, language, modality, history; optional verbalised difficulty, cheap-draft features and hidden-state probes. 2. **Policy**: hard constraints filter targets before any scoring. 3. **Strategies**: rules, capability fit, example similarity, task table, Thompson and LinUCB bandits, IRT, Bradley-Terry, Markov lookahead, multi-turn history embeddings, learning-to-defer, edge/cloud tiers, token budgets, auctions, user adaptation and an LLM judge consulted only below a confidence threshold; optionally a routing SLM (`RouterSLM`) distilled from the whole ensemble that keeps improving on its own from outcomes, public routing datasets and a live model catalogue ([guide](https://opensmartroute.ai/docs/GUIDE.md#the-routing-slm-and-the-self-improvement-loop)). 4. **Utility**: confidence-weighted ensemble, then `w_q * quality - w_c * norm(cost) - w_l * norm(latency)` with a hard quality floor; temperature scaling and conformal candidate sets calibrate the confidence; the router abstains when nothing is safe enough. 5. **After the answer**: cascades and self-escalation stop or reroute on response-side uncertainty (semantic entropy, P(True), streaming competence posterior); mixture-of-agents aggregation and permanent hand-off policies cover agentic trajectories. Formulas and citations: [docs/MATH.md](https://opensmartroute.ai/docs/MATH.md), [docs/RESEARCH.md](https://opensmartroute.ai/docs/RESEARCH.md). ## Production ```python from opensmartroute import RouteRequest, RequestConstraints from opensmartroute.enterprise import RouterBuilder, MetricsTelemetry, FileAuditSink, TenantMiddleware from opensmartroute.security import GuardMiddleware app = (RouterBuilder(registry) .with_defaults().with_auto_learning(state_dir=".osr-state") .with_health(latency_slo_ms=3000) .with_middleware(GuardMiddleware(redact=True), TenantMiddleware({"acme": {"deny_targets": ["llm-frontier"]}, "globex": {"data_boundary": "on_prem"}})) .with_telemetry(MetricsTelemetry()).with_audit(FileAuditSink("audit.jsonl")) .build()) req = RouteRequest("Prove that sqrt(2) is irrational.", constraints=RequestConstraints(tenant="acme")) d = app.route(req) # guard -> tenant -> policy -> strategies; audited and measured result = app.run(req) # ...then executes the plan once targets carry handlers, and learns print(d.target.id, app.health_snapshot()) ``` Routing overhead is 6 ms p50 with the default strategies and 9 ms with every learner enabled on a 16-target catalogue; retrieve-then-rank keeps it flat for thousands of tools ([measurements](https://opensmartroute.ai/docs/GUIDE.md#10-routing-latency)). `osr serve` (or `create_app(app)` with the router above) exposes a FastAPI app and an OpenAI-compatible `/v1/chat/completions` proxy: point any OpenAI client at it with `model="auto"`; guard and tenant violations return 400, no admissible target 422, an unreachable provider 503. Circuit breakers open on provider failures and recover on their own; learner state survives restarts from `state_dir` or a Redis / SQL store. Deployment references: [deploy/README.md](https://opensmartroute.ai/docs/deploy.md), [docs/ENTERPRISE.md](https://opensmartroute.ai/docs/ENTERPRISE.md). ## Hosted platform [platform/](https://github.com/isathish/OpenSmartRoute/blob/main/platform/README.md) packages the router as a service in two containers: `osr-platform-api` (FastAPI: self-serve signup with hashed API keys, plans and quotas, usage metering, the metered `/api/v1/route` family with the full trace, per-tenant constraints and workspace policy, the audit trail, per-request traces and a live event stream, and the OpenAI-compatible `/v1/chat/completions` proxy) and `osr-platform-web` (Next.js: landing page, the rendered documentation, pricing, a live playground and the account dashboard with activity, events, governance and health pages; it proxies `/api` and `/v1` so the browser only talks to one origin). `OSR_PLATFORM_EDITION=community` runs the core `Router`; `enterprise` runs the builder above with auto-learning, health, guard, metrics, a hash-chained audit log and per-tenant constraints. `azd up` deploys both to Azure Container Apps with Azure OpenAI from [infra/](https://github.com/isathish/OpenSmartRoute/blob/main/infra/main.bicep); the reference deployment is https://osr-web.gentlepebble-235bed4c.swedencentral.azurecontainerapps.io. The end-user guide is [docs/PLATFORM.md](https://opensmartroute.ai/docs/PLATFORM.md); the REST API reference is generated from [platform/api/openapi.json](https://github.com/isathish/OpenSmartRoute/blob/main/platform/api/openapi.json). ```bash curl -s -XPOST $OSR/api/v1/signup -H 'content-type: application/json' -d '{"email":"you@example.com"}' curl -s $OSR/api/v1/route -H "authorization: Bearer $KEY" -H 'content-type: application/json' \ -d '{"text":"Prove that sqrt(2) is irrational."}' ``` ## Security Routing is a control plane; its integrity is a security property. `InputGuard` combines a heuristic and a learned detector for confounder gadgets that reroute queries, an injection-risk scorer, PII redaction and prompt sanitisation for the LLM judge. Per-task resource limits, an origin policy for sensitive tool parameters, signed MCP manifests, AES-GCM state, content-free logs and hash-chained audit complete the model. CI runs `ruff -S`, `mypy`, `bandit`, CodeQL and the `osr safety` red-team suite. Threat model: [docs/SECURITY.md](https://opensmartroute.ai/docs/SECURITY.md). Reporting: [SECURITY.md](https://opensmartroute.ai/docs/security-policy.md). ## Documentation The user documentation is a searchable, versioned site at https://osr-web.gentlepebble-235bed4c.swedencentral.azurecontainerapps.io/docs (built from these files at release time, with a REST API reference generated from the platform's OpenAPI document). It covers the hosted platform, the Python SDK and the pip package; the internal planning, go-to-market, brand and sales documents below stay in the repository only. | Document | Contents | |---|---| | [docs/GUIDE.md](https://opensmartroute.ai/docs/GUIDE.md) | User guide: targets, routing, execution, providers, learning, SDK, enterprise builder, research-track modules, CLI, latency, layout | | [docs/PLATFORM.md](https://opensmartroute.ai/docs/PLATFORM.md) | Platform guide: authentication, `/api/v1/route`, execution, the OpenAI-compatible endpoint, feedback, plans and quotas, organizations, tenants, governance, observability (traces, events, readiness), the MCP server, dashboard | | [docs/MARKETPLACE.md](https://opensmartroute.ai/docs/MARKETPLACE.md) | Marketplace: find, install, buy and publish agents, skills, personas, prompts and stack templates; ratings, review lifecycle, `osr stack` | | [docs/MCP.md](https://opensmartroute.ai/docs/MCP.md) | Cost estimates (`estimate`, `POST /api/v1/estimate`), recommended models per use case and the MCP server for VS Code, Cursor, Claude, Windsurf and agents | | [docs/SDK.md](https://opensmartroute.ai/docs/SDK.md) | API reference for the decorator SDK, components and settings | | [docs/REFERENCE.md](https://opensmartroute.ai/docs/REFERENCE.md) | Generated API reference: every module and exported name (`python scripts/api_reference.py`) | | [docs/ARCHITECTURE.md](https://opensmartroute.ai/docs/ARCHITECTURE.md) | Request path, module boundaries, performance envelope | | [docs/ENTERPRISE.md](https://opensmartroute.ai/docs/ENTERPRISE.md) | Ports, stores, middleware, shadow and A/B, multi-replica operation | | [docs/OBSERVABILITY.md](https://opensmartroute.ai/docs/OBSERVABILITY.md) | Tracing and observability: spans and events for every stage, sinks (memory, metrics, log, file, OpenTelemetry), `/events`, `/trace`, `/metrics`, `OSR_OBSERVABILITY_*`; the platform's per-workspace `/api/v1/trace`, `/api/v1/events`, `/api/v1/status` and dashboard pages | | [docs/MATH.md](https://opensmartroute.ai/docs/MATH.md) | Every formula the router uses, with derivations | | [docs/RESEARCH.md](https://opensmartroute.ai/docs/RESEARCH.md) | Literature survey and the idea-to-module map | | [docs/SECURITY.md](https://opensmartroute.ai/docs/SECURITY.md) | Threat model and hardening checklist | | [docs/SECURITY_REVIEW.md](https://opensmartroute.ai/docs/SECURITY_REVIEW.md) | External security review pack: scope, trust boundaries, evidence, reviewer questions, review log | | [docs/ROADMAP.md](https://opensmartroute.ai/docs/ROADMAP.md) | Per-version exit criteria and status | | [docs/PLATFORM_PLAN.md](https://github.com/isathish/OpenSmartRoute/blob/main/docs/PLATFORM_PLAN.md) | Platform strategy: agentic routing, Open Capability Manifest, marketplace, editions and pricing | | [docs/GO_TO_MARKET.md](https://github.com/isathish/OpenSmartRoute/blob/main/docs/GO_TO_MARKET.md) | Go-to-market plan; the sales enablement kit lives in [docs/sales/](https://github.com/isathish/OpenSmartRoute/blob/main/docs/sales/README.md) | | [spec/ocm/README.md](https://opensmartroute.ai/docs/ocm.md) | Open Capability Manifest specification and JSON Schema | | [deploy/README.md](https://opensmartroute.ai/docs/deploy.md) | Container image, Helm chart, reference deployments | | [platform/README.md](https://github.com/isathish/OpenSmartRoute/blob/main/platform/README.md) | Hosted platform: `api/` (FastAPI) and `web/` (Next.js), playground, API keys, plans, editions, `OSR_PLATFORM_*` settings, Azure deployment | | [docs/BRAND.md](https://github.com/isathish/OpenSmartRoute/blob/main/docs/BRAND.md) | Logo system and naming conventions | [.claude/skills/](https://github.com/isathish/OpenSmartRoute/blob/main/.claude/skills) ships Agent-Skills packages that teach coding assistants how to use and extend the project; they are also valid routing targets (`osr skills` validates them) and each one is published on the documentation site under `/docs/skills/`. Coding agents working in this repository start from [AGENTS.md](https://github.com/isathish/OpenSmartRoute/blob/main/AGENTS.md). ## Roadmap | Release | Theme | Status | |---|---|---| | 0.3 | Real integrations: OpenAI-compatible client, adapters, config loaders | Shipped | | 0.4 | Learned signals and honest evaluation | Shipped | | 0.5 | Target representations, effort and personalisation | Complete | | 0.6 | Multi-step and agentic routing | Complete | | 0.7 | Catalogue interop and discovery at scale | Complete | | 0.8 | Operations, risk control and economics | Complete | | 0.9 | Security hardening of the control plane | Complete | | 1.0 | Stable API, reference deployments, security review pack | Complete | Every item in the research track is implemented on `main` with tests; a version is **Complete** when its measured exit criterion is published (`python scripts/exit_criteria.py`, 8/8 met) and **Shipped** when it is in a tagged release. 1.0 means what the repository can prove: frozen public API with a deprecation path, published performance envelope, reference deployments, security controls with a red-team suite in CI and a published review pack ([docs/SECURITY_REVIEW.md](https://opensmartroute.ai/docs/SECURITY_REVIEW.md)), the public leaderboard run ([examples/leaderboard/results](https://opensmartroute.ai/docs/leaderboard-results.md), produced by the recipe in [examples/leaderboard](https://opensmartroute.ai/docs/leaderboard.md)) and two consecutive releases without a breaking change. Adoption evidence - an independent security report, accepted leaderboard listings, two production users ([ADOPTERS.md](https://github.com/isathish/OpenSmartRoute/blob/main/ADOPTERS.md)) - is tracked, never claimed: `python scripts/release.py readiness` prints the whole table computed from the repository, and the release tooling refuses any `1.x` version while a release row is open. Details in [docs/ROADMAP.md](https://opensmartroute.ai/docs/ROADMAP.md) and [CHANGELOG.md](https://opensmartroute.ai/docs/changelog.md). ## Contributing Issues and pull requests are welcome; see [CONTRIBUTING.md](https://opensmartroute.ai/docs/contributing.md). CI runs `ruff`, `mypy`, `bandit` and `pytest` on CPython 3.10 to 3.13 (Linux and Windows), a routing-accuracy gate and the safety suite on the example catalogue, builds the distribution and the container image, and scans with CodeQL. The test suite has no mocks of the router itself: [tests/test_e2e_scenarios.py](https://github.com/isathish/OpenSmartRoute/blob/main/tests/test_e2e_scenarios.py) drives the shipped example catalogue through an in-process OpenAI-compatible provider fleet over HTTP, a real MCP server subprocess over stdio, the CLI, the FastAPI service, provider outages with breaker recovery, agentic multi-round loops and concurrent traffic. Releases are automated: a release pull request bumps the version and rolls the changelog; merging it tags, publishes to PyPI with attestations and pushes the image. ## License Apache-2.0 --- # User guide Everything the README leaves out: the full quick-start sequence, running plans end to end, wiring real providers, the enterprise builder, the research-track modules, routing latency and the repository layout. API reference material lives in [SDK.md](https://opensmartroute.ai/docs/SDK.md) (decorator SDK, components, settings) and [REFERENCE.md](https://opensmartroute.ai/docs/REFERENCE.md) (generated; every module and exported name); design in [ARCHITECTURE.md](https://opensmartroute.ai/docs/ARCHITECTURE.md). Contents 1. [Targets](#1-targets) 2. [Routing](#2-routing) 3. [Executing a plan](#3-executing-a-plan) 4. [Real providers](#4-real-providers) 5. [Learning from outcomes](#5-learning-from-outcomes) 6. [Declaring components with decorators](#6-declaring-components-with-decorators) 7. [Enterprise builder](#7-enterprise-builder) 8. [Research-track modules](#8-research-track-modules) 9. [Command line](#9-command-line) 10. [Routing latency](#10-routing-latency) 11. [Repository layout](#11-repository-layout) ## 1. Targets Every routable thing is a `RouteTarget` with the same contract: an `id`, a `kind`, declared `capabilities`, `cost`, `latency_ms`, a `quality_prior`, optional `examples`, `instructions` and a `handler`. Adding a target never requires retraining; it is routable from its declaration and learns from its outcomes. | Kind | What it is | Typical `capabilities` | Wins the primary route? | Executed by | |---|---|---|---|---| | `llm` | A model endpoint (optionally per reasoning effort) | `domains`, `min/max_complexity`, `languages`, `context_window` | yes | `chat_handler` / any callable | | `skill` | A deterministic or narrow capability (SQL, translate, summarise, a SKILL.md package) | `actions`, `domains`, token footprint as `cost` | yes, or fills the `skill` plan slot | your function; `instructions` are disclosed to the model | | `persona` | System-prompt / behaviour layer (`primary: false`) | `domains`, `actions` | no; fills the `persona` slot on top of a model or agent | `instructions` become the system prompt in `run()` | | `agent` | An agent harness: coding, research or support runtime with tools, memory and its own loop | `supports_tools`, `tags: [autonomous]`, `actions` | yes | `CallableHarness` / `HTTPHarness` / `SubprocessHarness` | | `tool` | A single MCP / function tool | `actions`, `input_schema` in metadata | yes | tool call | | `workflow` | A fixed multi-step pipeline | `actions`, `domains` | yes | workflow engine | | `human` | Queue, team or expert (Erlang-C capacity maths) | `actions: [escalate]`, `tags: [safety]` | yes; also the abstention target | ticketing / hand-off | ```yaml # examples/targets.yaml (excerpt) - id: coding-agent # agent harness: repo access, tests, review loop kind: agent capabilities: { domains: [coding], actions: [code_generation, code_review, action], min_complexity: 0.3, supports_tools: true, tags: [autonomous] } cost: { usd_per_1k_tokens: 0.008 } latency_ms: 6000 examples: ["Refactor this class to remove the circular import and run the test suite."] - id: skill-sql # deterministic skill; `instructions` is its SKILL.md-style body kind: skill capabilities: { domains: [data_analysis, coding], actions: [code_generation, extract] } cost: { usd_per_1k_tokens: 0.001 } instructions: "Always qualify table names with the schema; never SELECT *." - id: persona-legal-counsel # never primary; layered on whichever model/agent wins kind: persona primary: false capabilities: { domains: [legal], actions: [reasoning, summarize, qa] } instructions: "You are cautious in-house counsel. Cite the clause, flag jurisdiction risk." ``` Catalogues load from JSON or YAML (`config.load_targets`), from MCP `tools/list` payloads (`adapters.tools_from_mcp`), A2A agent cards (`adapters.agent_from_card`), SKILL.md directories (`adapters.load_skills`), persona folders (`adapters.load_personas`) and vLLM semantic-router configurations (`adapters.load_semantic_router_config`). ### Agent skills for coding assistants [.claude/skills/](https://github.com/isathish/OpenSmartRoute/blob/main/.claude/skills) ships nine Agent-Skills packages (`SKILL.md`, agentskills.io format) that teach a coding agent how to use and extend OpenSmartRoute: `osr-routing-catalogue`, `osr-decorator-sdk`, `osr-enterprise-builder`, `osr-evaluation`, `osr-security-hardening`, `osr-integrations`, `osr-deploy-serve`, `osr-platform` and `osr-contributing`. Claude Code discovers them automatically in this repository; other agents can load the directory, and each package is published on the documentation site under `/docs/skills/`. They are also valid routing targets: the same loader the SDK uses (`load_skills`) validates them in CI and the router fills the plan's skill slot with the right one. ```bash osr skills # validate + list (default root .claude/skills) osr -t examples/targets.yaml --skills .claude/skills route "Create a JSONL eval dataset and gate CI on accuracy" --plan # plan: llm:llm-mid -> skill:osr-evaluation ``` ## 2. Routing ```python from opensmartroute import Router, RouteRequest, RequestConstraints, Objective router = Router(registry) # plain text d = router.route("Prove that sqrt(2) is irrational, step by step.") print(d.target.id, f"{d.confidence:.2f}") print(d.trace.explain()) # per-strategy scores and rationales # hard constraints and a per-request objective req = RouteRequest( text="Summarize this patient intake note.", constraints=RequestConstraints(region="eu", data_boundary="private", max_cost_per_1k=0.005), objective=Objective(quality=1.0, cost=0.5, latency=0.1, quality_floor=0.6), ) d = router.route(req) print(d.trace.policy_rejections) # {'llm-frontier': 'cost 0.015 > budget 0.005', ...} # a composed plan: persona -> skill -> model d = router.route("Review this NDA clause for GDPR liability.", plan=True) for slot in d.plan.slots: print(slot.role, slot.target.id, f"{slot.confidence:.2f}") ``` Policy runs before scoring and is never traded off: enable flags, allow / deny lists, kind, region, data boundary, PII, tenant, cost / latency / token SLOs, modality, language and jailbreak risk. Strategies then score the survivors and a confidence-weighted ensemble produces a quality estimate; utility is `w_q * quality - w_c * norm(cost) - w_l * norm(latency)` with a hard `quality_floor`. With a `ConformalCalibrator`, `RouteDecision.candidate_set` is the smallest set with the requested coverage and `abstain` is set when no target is safe enough. A generative judge can be consulted only when the ensemble is unsure: ```python from opensmartroute import LLMJudgeStrategy router = Router(registry, llm_judge=LLMJudgeStrategy(llm=lambda prompt: my_llm(prompt)), escalate_llm_judge_below=0.6) ``` ## 3. Executing a plan `route()` decides; `run()` decides and executes the whole plan: skill pre-processors run first, persona and skill `instructions` are composed into the system prompt, the primary target is called, and an `Outcome` is recorded for every participant so each one learns from the result. ```python from opensmartroute.adapters import (CallableHarness, HTTPHarness, SubprocessHarness, chat_handler, harness_handler, load_skills) registry.get("llm-frontier").handler = chat_handler(openai, model="gpt-4o") registry.get("coding-agent").handler = harness_handler( SubprocessHarness(["my-coding-agent", "--repo", "."], system_flag="--system")) registry.get("research-agent").handler = harness_handler( HTTPHarness("https://agents.internal/research/run", api_key_env="AGENT_TOKEN")) registry.get("skill-sql").handler = lambda req: lookup_schema(req.text) # runs BEFORE the model for skill in load_skills("examples/skills"): # SKILL.md folders registry.add(skill) res = router.run("Is this indemnification clause enforceable in California?", task_id="ticket-812") res.text # the primary target's answer res.steps # [persona:persona-legal-counsel, llm:llm-frontier] with per-step latency res.system_prompt # composed persona + skill instructions res.outcomes # one Outcome per participant, role="persona"/"skill"/None, shared task_id ``` A skill handler may return a `RouteRequest` (rewrite the request), a `str` (attached as `context["skill_output"]` and shown to the model) or `None` (instructions only). Async handlers work through `AsyncRouter.run()`; `EnterpriseRouter.run()` adds health, budgets and telemetry. See [examples/end_to_end.py](https://github.com/isathish/OpenSmartRoute/blob/main/examples/end_to_end.py) and [SDK.md](https://opensmartroute.ai/docs/SDK.md#executing-end-to-end). ## 4. Real providers The bundled client is stdlib only and speaks the OpenAI chat and embeddings API, which covers OpenAI, Azure OpenAI, vLLM, Ollama, LiteLLM, OpenRouter, Groq and Mistral. ```python from opensmartroute import Router, LLMJudgeStrategy from opensmartroute.adapters import OpenAICompatClient, chat_handler, embedder, judge_fn from opensmartroute.strategies import CapabilityStrategy, SimilarityStrategy openai = OpenAICompatClient() # OPENAI_API_KEY from env ollama = OpenAICompatClient("http://localhost:11434/v1", api_key="ollama") registry.get("llm-frontier").handler = chat_handler(openai, model="gpt-4o") registry.get("llm-small").handler = chat_handler(ollama, model="llama3.1") router = Router(registry, strategies=[CapabilityStrategy(), SimilarityStrategy(embedder=embedder(openai, model="text-embedding-3-small"))], llm_judge=LLMJudgeStrategy(judge_fn(openai, model="gpt-4o-mini")), escalate_llm_judge_below=0.5) d = router.route(req) result = router.execute(d, req) # ChatResult(text, prompt_tokens, completion_tokens, latency_ms) res = router.run(req) # route + execute the full plan ``` `adapters.sentence_transformers_embedder()` and `adapters.OpenTelemetryTelemetry` are lazy imports behind the `embeddings` and `otel` extras. `osr serve` exposes a FastAPI app with an OpenAI-compatible `/v1/chat/completions` proxy that routes, executes and learns. ## 5. Learning from outcomes ```python from opensmartroute import Outcome router.learn(Outcome(request_id=d.request_id, target_id=d.target.id, success=True, quality=0.9, cost_usd=0.002, latency_ms=1800, domains=d.trace.signals.domains)) ``` One `Outcome` fans out to every learner: Thompson bandits per domain, IRT ability and difficulty, Bradley-Terry preferences, LinUCB on the signal vector, the Markov / MDP lookahead, the task table and the personalisation posteriors. `Outcome.task_id` and `step` join delayed, task-level rewards to the routing decisions that produced them (`learning.TaskCredit`, `learning.TaskPins`). Page-Hinkley drift detection flags targets whose success rate degrades; every learner forgets with a configurable `decay` and can be merged across replicas (`learning.merge_learners`). Each `learn()` also records a `learn.outcome` event on the request's trace and, with `with_audit(sink, outcomes=True)`, an `outcome` record in the audit chain, so a decision and its result stay joined by `request_id` ([OBSERVABILITY.md](https://opensmartroute.ai/docs/OBSERVABILITY.md); on the hosted platform `POST /api/v1/feedback` does the same). Offline warm starts: `learning.warm_start_from_matrix()` replays a `(prompt, {target: reward})` matrix, `learning.warm_start()` copies posteriors from the nearest declared neighbours and `osr train --from rows.jsonl --out models.json` fits the hashed signal models. ### The routing SLM and the self-improvement loop `learning.RouterSLM` is a small routing model in one JSON file: a hashed dual encoder (`SLMSettings.dim` x `feature_dim`), a temperature calibrator and a snapshot of every target's price, latency and quality prior. It has no dependencies, predicts in microseconds and can be trained from any `EvalRow` source - labelled datasets, your own outcomes, synthetic ontology prompts or the full router's decisions (`distill_router`, which compresses rules + capability fit + learners + judge into one model). ```python from opensmartroute import RouterSLM, SLMStrategy, Router, RouteRequest from opensmartroute.eval import DatasetCollector, synthetic_rows from opensmartroute.learning import distill_router slm = RouterSLM(reg.all(), seed=0) slm.fit(synthetic_rows(reg.all())) # no data yet: seed prompts scored by capability fit slm.fit(DatasetCollector("data").corpus()) # collected battle datasets (RouteLLM, LMArena) student, report = distill_router(Router(reg), texts) # or copy the whole ensemble print(report.summary()) # accuracy, loss, ECE, Brier, cost/1k slm.save("slm.json") router = Router(reg, strategies=[SLMStrategy(RouterSLM.from_file("slm.json"))]) d = router.route(RouteRequest(text="prove that sqrt(2) is irrational")) print(slm.predict(d.request.text, objective=Objective(cost=2.0))[:3]) # calibrated p, quality, $/1k, utility ``` `SLMStrategy` sits in the ensemble under weight `OSR_WEIGHTS_SLM` and, like the other learners, updates the model online from every `router.learn(Outcome)`. Data comes from three collectors that all degrade gracefully when offline: - `adapters.ModelCatalogue` merges OpenRouter prices and Hugging Face model cards (downloads, likes, `model-index` benchmarks) into `ModelCard`s, turns them into `RouteTarget`s (`targets(min_quality=, max_usd_per_1k=, measured=True)`), prints a cost/quality benchmark and the Pareto frontier. `refresh(leaderboard=True)` adds Open LLM Leaderboard accuracies (IFEval, BBH, MATH, GPQA, MuSR, MMLU-Pro) as measured quality; `apply_quality()` takes the Bradley-Terry win rates that `eval.model_quality` derives from collected battles. Third-party descriptions are risk-scored and replaced when they look like injections. - `eval.DatasetCollector` pulls routing datasets from the Hugging Face datasets-server into a JSONL cache. Ten public sources are on by default: the pairwise battles `routellm-battles` (GPT-4-judged), `arena-55k`, `arena-100k`, `arena-140k` (LMArena human votes, with the code / language / math / hard-prompt tags), `ppe-human`, `webdev-arena`, `mt-bench-human` and `reward-bench`, the per-model judge scores of `ultrafeedback`, and `routellm-gpt4` (RouteLLM's "was Mixtral good enough for this prompt" labels). RouterBench / RouterEval / the raw arena conversations are known but gated: download them with your own Hub token and load the file with `osr collect --file routerbench=routerbench.jsonl --preset routerbench` (`.jsonl` / `.json` / `.csv` in the preset's shape, `--tier` folds model columns onto your targets), and any dataset works through `DatasetSource`. Battles name a moving population of models, so `--tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small` folds every side onto your tiers as it is parsed (`tier_of()`: the `ARENA_TIERS` table, name rules for the 2024-26 generations, then the parameter count in the name) - the RouteLLM question "was the cheaper tier good enough?". Collection is polite and resumable: pages are paced, 429s are retried with backoff, and a run the server cuts short keeps its rows and continues from the recorded offset on the next `osr collect` (`.meta.json`). `rows_from_feedback` turns your `FeedbackStore` into labelled rows. - `adapters.WebKnowledge` searches Hugging Face (text-generation repos, per-keyword fallback) and DuckDuckGo (Brave with `BRAVE_API_KEY`) for new models and datasets; fetches are https-only, byte-capped and page text is risk-scored. `learning.SelfImprover` closes the loop as a champion/challenger cycle: refresh the catalogue, discover, gather evidence, split, train a challenger, calibrate it on the holdout and promote it only when holdout accuracy improves by at least `OSR_SLM_MIN_GAIN` (or loss drops at equal accuracy). Every cycle is appended to a JSONL history. ```bash osr catalogue --cache data/catalogue.json --hf "llama instruct" --leaderboard --markdown # live cost / quality osr collect --cache-dir data --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small osr -t targets.yaml slm train --out slm.json --cache-dir data --report # every *-tiers cache, holdout metrics osr slm predict slm.json "write a python function to parse csv" --cost-weight 1.0 osr -t targets.yaml improve --slm slm.json --cache-dir data --catalogue data/catalogue.json \ --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small \ --search "qwen2.5 instruct" --interval 3600 # daemon; --offline for air-gapped ``` Serve the result: `osr -t targets.yaml --slm slm.json serve`, or in the container `OSR_SLM=/config/slm.json` (Helm `config.slm`). With `OSR_LLM_BASE_URL` (and `OSR_LLM_API_KEY` / `OSR_LLM_MODEL`) pointing at any OpenAI-compatible endpoint - OpenAI, Azure OpenAI, vLLM, Ollama, OpenRouter - every LLM target gets a chat handler from `metadata.model`, so `/v1/chat/completions` with `model: "auto"` routes *and* answers. #### Speed, and what the numbers mean Training is plain SGD over hashed features; with numpy installed (extra `fast`, included in the serve image) the same steps run on dense arrays - 18 000 rows train in under a minute, ten times the pure-Python speed - and prediction projects through a dense mirror of the weights. `OSR_SLM_BACKEND=auto|numpy|python` chooses; both paths seed identically and write the same model file, so a zero-dependency deployment reads a numpy-trained model unchanged. The model carries a learned per-target bias (the base rate a unit-norm dot product cannot express) and trains with inverse-time learning-rate decay (`OSR_SLM_LR_DECAY`), four epochs and `l2` 1e-3 by default - on the 18 000-row mixed corpus the old constant-rate, eight-epoch defaults overfit *below* the constant baseline. Read `--report` against that baseline. Human preference between tiers is noisy: on the ten-source corpus "always the mid tier" is acceptable for 64 % of holdout prompts and the SLM reaches 64-66 % with a holdout cross-entropy of 0.98-1.00 (the constant predictor: 1.03), well calibrated (ECE 0.05-0.08). It is much stronger where the label really depends on the prompt - 94 % on `routellm-gpt4` (baseline 86 %), +12 points on `arena-100k` - and no better than the prior on sources where the prompt carries no signal. That is what a learned routing prior is; the ensemble adds rules, capability, similarity and the bandit on top. A corpus of model battles only knows the models it compared. Scored rows therefore train a softmax over the *scored population* (every target any row scores, plus `OSR_SLM_NULL_TARGETS` learned "none of these" logits that keep it from saturating), so a tool, skill or agent the data never mentions keeps its catalogue embedding instead of being pushed to zero, and `SLMStrategy` abstains on such targets - its confidence is scaled by the share of the request's candidates it has evidence about. Rows that name only an `expected` target (your own feedback, `examples/eval_dataset.jsonl`) contrast against the whole catalogue, which is how the model learns the non-LLM targets once outcomes arrive. #### Where a transformer helps The hashed encoder is a bag of words, bigrams and character grams: it cannot tell "explain this SQL" from "SQL that explains itself". Two opt-in encoders add that context, both persisted in the same JSON file and restored by `RouterSLM.from_file`: - `OSR_SLM_ENCODER=attention` adds `learning.AttentionEncoder`, a pure-Python transformer block (hashed token embeddings, sinusoidal positions, `OSR_SLM_ATTENTION_HEADS` heads of `OSR_SLM_ATTENTION_HEAD_DIM`, residual, attention pooling) to the query encoder: $q = W\,\phi(x) + \operatorname{Attn}(x)$. No dependencies, deterministic, hand-derived gradients, but quadratic in `OSR_SLM_ATTENTION_MAX_TOKENS` - a few milliseconds per prompt and roughly four times the training time - so measure on your holdout (`osr slm train --report`) before switching. On the public arena corpora (six thousand pairwise rows, seventeen tiered targets) it gained two points at 1 500 rows but *lost* five at full size with the default budget: the hashed encoder stays the default, and attention earns its keep on order-sensitive traffic, not on generic chat. - `OSR_SLM_EMBEDDER=sentence-transformers/all-MiniLM-L6-v2` (extra `embeddings`) or `RouterSLM(embedder=callable)` appends a *frozen* pretrained embedding as dense features (`learning.EmbeddingFeaturizer`): `W` then learns a linear head on top of a real transformer while the hashed features keep the surface cues (code fences, length, exact vocabulary). The file records the embedder's name; a custom callable is passed again to `from_file(embedder=...)`. #### Self-operation `osr serve --autopilot` runs the `SelfImprover` inside the server (`learning.Autopilot`): a cycle every `OSR_SLM_AUTOPILOT_INTERVAL_S` seconds, an early one when `learning.DriftMonitor` (Page-Hinkley over the success and quality of every outcome the router learns from, via `Router.observers`) sees the served quality drop, never closer than `OSR_SLM_AUTOPILOT_MIN_GAP_S`. Feedback is joined back to prompts through the router's request memory (`OSR_SLM_AUTOPILOT_REMEMBER` texts), an accepted challenger is hot-swapped into the `SLMStrategy` and written back to disk, a failing cycle is logged and the server keeps serving. `GET /stats` reports `autopilot` (cycles, promotions, drift, last report, next run) and `POST /autopilot/cycle` schedules one now. In the container: `OSR_AUTOPILOT=1` with `OSR_STATE` mounted (Helm `autopilot.enabled`); `--offline` keeps it off the network. Each cycle is traced as an `autopilot.cycle` span with a `learn.improve` event (and `learn.promote` when the challenger wins), so `GET /events?name=learn.*` shows the loop at work. The hosted platform runs the same loop from `OSR_PLATFORM_SLM` / `OSR_PLATFORM_AUTOPILOT` and shows it at `/platform/dashboard/learning` ([PLATFORM.md](https://opensmartroute.ai/docs/PLATFORM.md), section 10). ```bash osr -t targets.yaml --slm slm.json --state state serve --autopilot --autopilot-interval 1800 \ --catalogue data/catalogue.json --source arena-55k --tier frontier=llm-frontier --tier small=llm-small curl -s localhost:8000/stats | jq .autopilot ``` ## 6. Declaring components with decorators Targets, strategies, signal extractors, policy rules, middleware and telemetry can be declared where they live and assembled by a `ComponentRegistry` (details in [SDK.md](https://opensmartroute.ai/docs/SDK.md)): ```python import opensmartroute as osr @osr.tool("weather", domains=["travel"], actions=["lookup"], cost=0.0, examples=["weather in paris"]) def weather(request, **kw): """Current weather for a city.""" # docstring -> target.description return lookup(request.text) @osr.strategy(weight=0.8) # fn(request, signals, candidates) -> {id: score} def geo_affinity(request, signals, candidates): region = request.constraints.region return {t.id: 1.0 if region in t.constraints.regions else 0.4 for t in candidates} @osr.policy_rule # fn(target, request, signals) -> reason | None def business_hours_only(target, request, signals): if target.kind is osr.TargetKind.HUMAN and request.context.get("after_hours"): return "human queues are closed" router = osr.components.router(registry) app = osr.components.builder(registry).with_auto_learning().build() ``` Every tunable (ensemble weights, thresholds, scales) lives in one typed `Settings` object and can be overridden per process (`osr.configure(...)`) or through `OSR__` environment variables; `osr settings` prints the effective values. ## 7. Enterprise builder ```python from opensmartroute.enterprise import (RouterBuilder, CacheMiddleware, TenantMiddleware, LoggingTelemetry, MetricsTelemetry, FileAuditSink) from opensmartroute.security import GuardMiddleware metrics = MetricsTelemetry() app = ( RouterBuilder(registry) .with_rules(rules) # declarative preferences .with_defaults() # capability + similarity + Thompson bandit .with_auto_learning(state_dir=".osr-state") # IRT + Bradley-Terry + LinUCB + Markov lookahead .with_health(latency_slo_ms=3000) # circuit breakers, rate limits, budgets .with_middleware(GuardMiddleware(redact=True), # gadget defence + PII redaction TenantMiddleware({"acme": {"max_cost_per_1k": 0.005, "data_boundary": "private"}}), CacheMiddleware(ttl_s=30)) .with_telemetry(LoggingTelemetry(), metrics) .with_audit(FileAuditSink("audit.jsonl")) .build() ) d = app.route(RouteRequest("Summarize this contract", constraints=RequestConstraints(tenant="acme"))) app.learn(Outcome(d.request_id, d.target.id, success=True, quality=0.9, cost_usd=0.002, latency_ms=1200, domains=d.trace.signals.domains, complexity=d.trace.signals.complexity)) print(metrics.snapshot()["route_latency_ms"], app.health_snapshot()) ``` `with_retrieval(narrow_above=, narrow_to=)` switches to retrieve-then-rank for large catalogues, `with_queue_awareness()` replaces static latency with a queue-aware estimate, `with_shadow(candidate, mode="shadow" | "ab")` compares policies with SPRT, `with_calibration()` adds temperature scaling and conformal candidate sets, `with_fair_share()` enforces tenant weights and `with_state_store()` accepts Redis, SQL, encrypted, versioned or batched stores. Ports (state, telemetry, audit) are ABCs; see [ENTERPRISE.md](https://opensmartroute.ai/docs/ENTERPRISE.md). ## 8. Research-track modules Each module below implements one roadmap item with tests; papers are cited in [RESEARCH.md](https://opensmartroute.ai/docs/RESEARCH.md) and formulas in [MATH.md](https://opensmartroute.ai/docs/MATH.md). ### Signals from a draft or a small model ```python from opensmartroute.signals import (DEFAULT_EXTRACTORS, VerbalisedDifficultySignal, DraftResponseSignal, extract_signals) extractors = [*DEFAULT_EXTRACTORS, VerbalisedDifficultySignal(source=lambda req: small_model_difficulty(req.text)), DraftResponseSignal(drafter=lambda req: cheap_model(req.text))] router = Router(registry, extractors=extractors) ``` `VerbalisedDifficultySignal` accepts numbers, `7/10` or words (`trivial` ... `expert`) from `context["difficulty"]` or a callable and blends them into `complexity`. `DraftResponseSignal` measures hedging, self-corrections, query overlap and length ratio of a cheap draft. ### Response-side uncertainty ```python from opensmartroute.signals import UncertaintyGate, EventTrigger, TriggerRule, response_uncertainty from opensmartroute.strategies import Cascade gate = UncertaintyGate(sampler=lambda req, n: [cheap_model(req.text) for _ in range(n)], n_samples=4) cascade = Cascade(quality_gate=gate, threshold=0.6) # escalate when samples disagree result = cascade.run(req, router.route(req).alternatives) feats = response_uncertainty(samples, p_true=judge_p_true) trigger = EventTrigger([TriggerRule("semantic_entropy", 0.5, "reroute"), TriggerRule("agreement", 0.5, "ask_human", below=True)]) trigger.check(feats) # ["reroute", ...] ``` ### Hidden-state Dirichlet probe ```python from opensmartroute.strategies import HiddenStateStrategy probe = HiddenStateStrategy(state_fn=lambda req: host_model.hidden_state(req.text), targets=["small", "large"], dim=4096) probe.fit(logged_states, logged_winners, epochs=20) # warm start router = Router(registry, strategies=[*default_strategies(), probe]) ``` The probe's confidence is `1 - K/S` (epistemic uncertainty), so unfamiliar states defer to the other strategies. `math.DirichletProbe` is usable on its own with any feature vector. ### Elastic models and token budgets ```python from opensmartroute.strategies import BudgetVariant, TokenBudgetStrategy, expand_elastic for sib in expand_elastic(registry.get("reasoner"), [BudgetVariant(1024, quality=0.7, cost_scale=0.4), BudgetVariant(8192, quality=0.9)]): registry.add(sib) # reasoner@1024, reasoner@8192 router = Router(registry, strategies=[*default_strategies(), TokenBudgetStrategy()]) ``` ### Edge and cloud tiers Tag targets with `metadata: {tier: edge}` or `{tier: cloud}` and add `EdgeCloudStrategy()`. Edge success is a Thompson posterior per complexity bucket, penalised when decode time eats into the latency SLO; the cloud tier is penalised by upload time and excluded when the request's `data_boundary` is `private` or `on_prem` and the target is `public`. ### Auctions `AuctionStrategy(bid_fn=None, value=0.01)` lets each target bid a claimed success probability and a price (`default_bid` derives both from the catalogue). Claims are corrected by each bidder's observed bias; the highest corrected surplus wins and pays the second price. Supply a `bid_fn` for multi-vendor catalogues where providers self-report. ### Protocol selection, aggregation, self-escalation, hand-off ```python from opensmartroute.strategies import ProtocolPolicy, MixtureOfAgents, SelfEscalation, wrap_stream from opensmartroute.learning import HandoffPolicy, MixtureCureModel policy = ProtocolPolicy() # risk / budget -> protocol choice = policy.choose(req, decision, decision.trace.signals, uncertainty=gate(req, draft)) if choice.protocol == "aggregate": res = MixtureOfAgents(k=3, budget_usd=0.05).run(decision, req) # top-k answers, majority vote for o in res.outcomes(req): router.learn(o) monitor = SelfEscalation.for_target(decision.target) # streaming competence posterior streamed = wrap_stream(provider.stream(req.text), monitor) if streamed.escalated: ... # restart on a stronger target handoff = HandoffPolicy(MixtureCureModel(), fallback_target="senior-agent", pins=task_pins) if handoff.step(task_id, teacher_risk): # permanent hand-off target = handoff.target_for(task_id, decision.target.id) handoff.finish(task_id, failed=False) # censored observation ``` ### Personalisation `learning.UserAdaptiveStrategy` reads `context["user_id"]` (or `profile["user_id"]`) and keeps a Beta posterior per user and target, shrunk toward similar users and the global posterior; `observe_user()` warm-starts from logged interactions. `learning.SkillAffinity.relevance(profile, skills)` feeds `retrieval.select_skill_set(relevance=)` so skill sets depend on who is asking. ### Multi-turn conversations ```python from opensmartroute.learning import HistoryTargetStrategy router = Router(registry, strategies=[*default_strategies(), HistoryTargetStrategy(turns=6)]) d = router.route(RouteRequest("can you explain more?", history=chat_history, context={"last_target": "llm-small", "last_failed": True})) ``` The same follow-up message routes differently depending on what the conversation has been about: the strategy learns a logistic model over the joint embedding of the recent history and each target's catalogue embedding, so a target that keeps failing in legal threads but succeeds in coding threads is scored accordingly, and an unseen target is still scored through its description and examples. `last_target` gives the incumbent a small continuity bonus that turns into a penalty when the previous turn failed. ### Discovery beyond text similarity ```python from opensmartroute.discovery import SchemaAwareStrategy, CachePreservingSelector, SkillGraph router = Router(registry, strategies=[*default_strategies(), SchemaAwareStrategy()]) selector = CachePreservingSelector(prefix_size=8, evict_after=5) tools = selector.select(session_id, candidates, needed=[d.target.id]) # prefix-stable tool order graph = SkillGraph(load_skills("skills")) # osr-requires / osr-conflicts ordered, dropped = graph.compose(["report-builder"]) ``` ### MCP server recommendation and semantic-router import ```python from opensmartroute.adapters import recommend_servers, load_semantic_router_config for rec in recommend_servers("open a pull request", server_cards, k=2, constraints=req.constraints, allowed_auth=["none", "token"]): print(rec.server.name, rec.score, rec.rationale) imported = load_semantic_router_config("config.yaml") # vLLM semantic-router router = Router(imported.registry, strategies=[RulesStrategy(imported.rules), *default_strategies()]) ``` ### Energy and carbon `math.EnergyModel` fits Wh per target from measured samples; `math.hardware_profile("h100-sxm")` gives priors when no meter exists. Both convert to gCO2 with a grid factor and feed `wh_per_1k_tokens` / `gco2_per_1k_tokens` on the target's `cost`, which `Objective(energy=, carbon=)` trades against quality. ## 9. Command line Install `osr` with the platform installer (`curl -LsSf https://opensmartroute.ai/install.sh | sh` on Linux and macOS, `irm https://opensmartroute.ai/install.ps1 | iex` in PowerShell) or with `uv tool install 'opensmartroute[yaml,server]'` (add `fast` for numpy-accelerated SLM training); `osr --version` confirms the install. `osr` takes the catalogue with `-t targets.yaml`, optional rules with `-r rules.yaml` and an optional routing SLM with `--slm slm.json`; `--help` on any subcommand lists its options. Grouped by task: ```bash # Sign in and manage access tokens osr login # browser sign-in to the hosted platform osr login --url https://osr.example.com --profile work # another deployment, kept as a profile osr login --url http://router:8000 --token osr_local_... # self-hosted osr serve token echo "$TOKEN" | osr login --with-token # non-interactive (CI) osr whoami # workspace, plan, edition, token source osr token generate # mint an osr_local_ token for osr serve osr token create --name ci # new platform API key (shown once) osr token list # keys of the signed-in workspace osr token revoke osr logout # forget the profile (--all for every one) # Route, quote and inspect osr -t examples/targets.yaml -r examples/rules.yaml route "I want a refund for order #123" --plan osr -t examples/targets.yaml route "Summarise this HR complaint" \ --constraint data_boundary=on_prem --constraint region=eu # hard constraints; --kinds agent,tool osr -t examples/targets.yaml -r examples/rules.yaml route "" --event order.shipped --plan # textless event osr -t examples/targets.yaml estimate "Translate this contract" --monthly 100000 # cost per candidate + projection osr -t examples/targets.yaml targets # the catalogue as the router sees it osr -t examples/targets.yaml stats # feedback statistics per target osr settings --json # effective tunables # Evaluate and audit osr -t examples/targets.yaml -r examples/rules.yaml eval examples/eval_dataset.jsonl --frontier --calibration osr -t examples/targets.yaml audit traffic.jsonl --baseline llm-frontier --monthly 3000000 --markdown # Routing Audit: savings + violations osr -t examples/targets.yaml ope decisions.jsonl # off-policy estimate (IPS/SNIPS/DR) of logged decisions osr -t examples/targets.yaml safety --learned-guard # red-team suite # Learn osr train --from rows.jsonl --out models.json # fit signal models osr catalogue --cache data/catalogue.json --hf "llama instruct" # live model prices + quality osr collect --cache-dir data --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small # Hub datasets osr collect --cache-dir data --file routerbench=routerbench.jsonl --preset routerbench # a gated file you hold osr -t examples/targets.yaml slm train --out slm.json --report # routing SLM (see section 5) osr slm info slm.json # rows, sources, encoder, calibration, fit history osr -t examples/targets.yaml --slm slm.json route "parse this csv" # SLM inside the ensemble osr slm eval slm.json rows.jsonl # SLM accuracy on labelled rows osr slm predict slm.json "parse this csv" --top 3 # what the SLM alone would rank osr slm info slm.json # SLM metadata, targets and size osr -t examples/targets.yaml improve --slm slm.json --cache-dir data # self-improvement cycle # Serve and integrate osr -t examples/targets.yaml -r examples/rules.yaml serve # http://127.0.0.1:8000/docs (open, local dev) osr -t examples/targets.yaml serve --generate-token # prints an osr_local_ token; all routes need it osr -t examples/targets.yaml serve --require-auth --token "$TOKEN" # production: refuse to start without a token osr -t examples/targets.yaml mcp # MCP server over stdio for an IDE osr mcp --remote # stdio bridge to the platform you signed in to osr mcp --url https:///mcp --api-key osr_... # ...or an explicit endpoint and key osr mcp-manifest tools.json --key-env OSR_MCP_KEY --out signed.json # sign an MCP tools/list (--verify to check) # Manifests, stacks and skills osr validate examples/ocm # Open Capability Manifests osr -t examples/targets.yaml export-ocm --out manifests/ # catalogue -> OCM manifests osr stack init --out stack.yaml # starter declarative stack osr stack validate examples/stack.yaml # check (imports resolved) osr stack plan examples/stack.yaml --against deployed.yaml # diff against what runs today osr stack apply examples/stack.yaml --route "refund order 42?" # build the router, route once osr stack apply registry://support-desk@1.0.0 --registry $OSR # marketplace template osr skills # validate SKILL.md packages ``` `osr login` without `--token` runs the OAuth 2.0 device authorization grant against the hosted platform (community or enterprise edition alike): the CLI prints a short code, opens `/platform/cli/authorize` in the browser, and after you approve it there the platform mints a workspace API key (`osr_live_...`) for that machine. Credentials are stored per profile in `~/.config/opensmartroute/credentials.json` (`%APPDATA%\opensmartroute` on Windows, `OSR_CONFIG_DIR` overrides); flags, then `OSR_API_URL` / `OSR_API_KEY`, then the saved profile decide which one a command uses. A self-hosted `osr serve` has no accounts: `osr token generate` (or `serve --generate-token`) mints an `osr_local_...` token, the server accepts it as `Authorization: Bearer` or `X-API-Key` on every route except the health, metrics, `/whoami` and OpenAPI paths, and `OSR_SERVER_AUTH_TOKENS` / `OSR_SERVER_REQUIRE_AUTH` configure the same thing from the environment (see :class:`ServerSettings` in [SDK.md](https://opensmartroute.ai/docs/SDK.md#settings)). The Routing Audit replays logged traffic (`{"text": ..., "model": ..., "cost_usd": ...}` per line) through the catalogue and reports savings against a baseline target, policy violations the logged route would have committed, and the route mix; `--min-savings` makes it a CI gate. In production the same numbers come from `RouterBuilder.with_savings(baseline="max")` and `ep.savings.report().to_markdown()`. ## 10. Routing latency Routing is pure Python on the request path. Measured with `python scripts/bench.py --n 200 --scale` on the 16-target example catalogue, CPython 3.11, x86-64 laptop: | Configuration | p50 | p95 | p99 | |---|---|---|---| | signals + policy + rules + capability + similarity + bandit | 5.8 ms | 6.2 ms | 6.5 ms | | + IRT + preference + LinUCB + Markov + health | 8.6 ms | 12.9 ms | 17.8 ms | | + LLM judge (only when `confidence < escalate_llm_judge_below`) | + one provider round-trip | n/a | n/a | Catalogue size: 64 targets route in 14.5 ms p50 and 256 in 16.7 ms with `RouterBuilder.with_retrieval(narrow_above=32, narrow_to=24)` (versus 36 ms and 142 ms scoring every target); 1024 targets route in 24 ms. Cost is linear in `scored targets x strategies`; signal extraction is under 1 ms and independent of catalogue size. `RouteDecision.trace.elapsed_ms` records the routing time per request, `MetricsTelemetry` exposes `route_latency_ms` percentiles and the OpenTelemetry adapter emits the `osr.route_latency_ms` histogram. Full tables in [ARCHITECTURE.md](https://opensmartroute.ai/docs/ARCHITECTURE.md#performance-envelope). ## 11. Repository layout ``` src/opensmartroute/ core/ types (RouteTarget, RouteRequest, RouteDecision, Outcome, ...), registry signals/ lexicon extractors, task ontology, learned hashed models, verbalised difficulty, draft-response and response-uncertainty features policy/ hard-constraint filter strategies/ rules, capability, similarity, bandit, llm_judge, cascade, task_table, defer, progress, probe (Dirichlet), elastic, edge, auction, protocol, escalation, aggregate learning/ IRT, preference, LinUCB, Markov, AutoLearner, cold start, task credit, personal (user adaptation, skill affinity), handoff (mixture cure), multiturn (history-target joint embeddings) math/ bandits, irt, preference, markov, calibration, estimators, decision, dirichlet, energy realtime/ circuit breaker, rate limit, budget, health enterprise/ RouterBuilder, EnterpriseRouter, middleware, telemetry, audit, stores, ops security/ InputGuard, learned gadget detector, injection risk, redaction, limits, provenance adapters/ OpenAI-compatible client, harnesses, MCP (tools + server recommendation), A2A, frameworks, personas, SKILL.md, semantic-router import, embeddings, OpenTelemetry eval/ harness, baselines, robustness, frontiers, off-policy evaluation, dataset presets discovery.py schema-aware matching, cache-preserving selection, skill graph retrieval.py BM25 + dense retrieval, reciprocal-rank fusion, skill-set selection, meta-tools execution.py plan executor sdk.py decorator SDK and plugin discovery settings.py typed, environment-overridable tunables config.py load_targets / load_rules (JSON, YAML) feedback/ outcome store and preference export router.py orchestrator and plan builder aio.py AsyncRouter server.py FastAPI app and OpenAI-compatible proxy cli.py osr command deploy/ Dockerfile, entrypoint, Helm chart examples/ targets.yaml, rules.yaml, skills/, eval_dataset.jsonl, demos .claude/skills Agent-Skills packages for coding agents; also routable skill targets tests/ unit and integration tests, public API snapshot, docs coverage gate (test_docs.py) scripts/ bench.py, release.py, brand_build.py, api_reference.py, exit_criteria.py docs/ GUIDE, REFERENCE (generated), RESEARCH, ARCHITECTURE, MATH, ENTERPRISE, SECURITY, SDK, ROADMAP, BRAND ``` --- # Platform guide The hosted platform runs OpenSmartRoute as a service. A workspace has an API key and a plan; each request to the API is routed to a target (model, agent, skill, persona or tool) and answered with the decision, its trace and, on request, the target's output. An OpenAI-compatible endpoint accepts existing chat-completion clients. This page covers calling the platform: authentication, routing, execution, the OpenAI-compatible endpoint, feedback, plans, organizations, tenants, governance (workspace policy and budgets), observability (probes, metrics, per-request traces and the event stream), caching, the dashboard and the MCP server. The [marketplace](https://opensmartroute.ai/docs/MARKETPLACE.md) - shared agents, skills, personas, prompts and stack templates - has its own page. To run the router on your own infrastructure see [Deploy with Docker and Helm](https://opensmartroute.ai/docs/deploy.md); to use it as a Python library see the [user guide](https://opensmartroute.ai/docs/GUIDE.md) and the [SDK guide](https://opensmartroute.ai/docs/SDK.md). Every endpoint below is listed in the REST API reference (`/docs/api`) and in `/openapi.json`. ## 1. Authentication 1. Sign up at `/platform/signup` with GitHub, Google or an email address - with a password if you want to sign in by email later (`OSR_PLATFORM_PASSWORD_LOGIN`, on by default). Signup creates a personal workspace on the **free** plan and issues an API key. 2. The key is shown once. Store it in a secret manager. Keys are created, renamed, rotated and revoked in the dashboard (`/platform/dashboard/keys`) or with `POST /api/v1/keys`, `PATCH /api/v1/keys/{key_id}` `{name}`, `POST /api/v1/keys/{key_id}/rotate` (a new secret for the same key, shown once; the old one stops working immediately) and `DELETE /api/v1/keys/{key_id}`. 3. Send the key with every metered request in either header: ```http Authorization: Bearer osr_... X-API-Key: osr_... ``` Public endpoints (`/api/v1/info`, `/api/v1/models`, `/api/v1/llms`, `/api/v1/rankings`, `/api/v1/catalogue`, `/api/v1/estimate`, `/api/v1/stats/public`, `/api/v1/registry`) need no key. ### Email and password `POST /api/v1/auth/password/login` `{email, password}` returns the same browser session as a single sign-on callback (`session_token`, `user`, `account`, `plan`); the dashboard stores it and sends it as a bearer token. Passwords are at least 8 characters (no composition rules) and are stored as salted scrypt hashes. A signed-in user sets or changes theirs with `POST /api/v1/auth/password` `{password, current_password}` - the current one is required once set and every other browser is signed out - and removes it with `DELETE /api/v1/auth/password` as long as a provider identity remains. Sign-in attempts are limited per address and per email; `has_password` on the user object says which users have one. `/platform/dashboard/account` has the form. ### Account lifecycle: confirmation, recovery, deletion Anyone can create an account with any email address. What happens around it: | Step | How | Endpoints | |---|---|---| | Email confirmation | Signup sends a link to `/platform/verify-email?token=...` (valid three days). Signing in through a provider that vouches for the address, accepting an invitation sent to it or completing a password reset also confirms it. `email_verified` on the user object; `/platform/dashboard/account` shows a banner with *Send again* until it is done. | `POST /api/v1/auth/verify` `{token}`, `POST /api/v1/auth/verify/send` (signed in, once a minute) | | Forgot password | `/platform/forgot-password` emails a single-use link to `/platform/reset-password?token=...` (one hour). The answer is always `202` so it never reveals whether an address has an account; limited per address and per email. Resetting revokes every session, confirms the address, signs the person in and sends a notice. | `POST /api/v1/auth/password/forgot` `{email}`, `POST /api/v1/auth/password/reset` `{token, password}` | | Invitations | `POST /api/v1/workspace/invites` emails the invitee a link to `/platform/invite/` (fourteen days) and returns the same link once to the inviter. | see [Organizations](#7-organizations) | | Account history | The person's own sign-ins, password and email changes and memberships, newest first. | `GET /api/v1/me/audit` | | Download my data | One JSON document with everything held about the person and the current workspace: profile, identities, sessions, memberships and history; the workspace with plan, members, key prefixes, tenants, policy, SSO, usage, the request log and reported outcomes of up to a year. Prompt text is never stored, so none is included. *Download my data* on `/platform/dashboard/account`. | `GET /api/v1/me/export?days=` | | Delete account | *Delete my account* on `/platform/dashboard/account`: type the email (and the password when one is set). Sessions, linked identities and memberships go, workspaces only this user belonged to are disabled and their keys revoked; an organization with other members needs another owner first. A notice is emailed. | `POST /api/v1/me/delete` `{confirm, password}` | | Getting started | The checklist of a workspace: e-mail confirmed, API key created, first request routed, first outcome reported, a model provider connected and - for organizations - a teammate invited; each step carries the dashboard link and the API call that completes it, `next` is the first open one and `complete` the share done. The dashboard overview shows it until every step is done. | `GET /api/v1/onboarding` | Signup, confirmation, password recovery, invitation links and this checklist form the `onboarding` domain, which a deployment can run as its own service (see [Services](#services)). Every one of these events - plus sign-ins, failed sign-ins, role changes, removals and operator actions - is written to the account audit log. Workspace admins read their workspace's entries with `GET /api/v1/workspace/audit`; operators see everything at `/platform/admin/audit` (`GET /api/v1/admin/audit-log`) and per user on `/platform/admin/users/`, where they can also issue a reset link, re-send or force the email confirmation, disable, sign out everywhere or delete the user. Email is delivered through the SMTP server in `OSR_PLATFORM_SMTP_URL`. Without one, nothing is lost: every message is kept in the outbox and operators pick the links up at `/platform/admin/mail` (`GET /api/v1/admin/mail`); `mail` in `GET /api/v1/info` tells the web app which case applies. ### Command line sign-in Install the CLI (`curl -LsSf https://opensmartroute.ai/install.sh | sh`, or `irm https://opensmartroute.ai/install.ps1 | iex` on Windows) and run `osr login`. The CLI shows an eight-character code and opens `/platform/cli/authorize`; sign in there, check that the code and computer name match, choose the key name and approve. The platform mints a new workspace API key for that machine and the CLI stores it in `~/.config/opensmartroute/credentials.json`. This is the OAuth 2.0 device authorization grant (RFC 8628) and works the same for the community and enterprise editions: | Step | Endpoint | Notes | |---|---|---| | CLI asks for a code | `POST /api/v1/auth/device/code` `{client_name}` | anonymous, rate limited per IP; returns `device_code`, `user_code`, `verification_uri[_complete]`, `expires_in` (900 s), `interval` (5 s) | | Browser looks it up | `GET /api/v1/auth/device/{user_code}` | signed in; shows client name, status and expiry | | Browser decides | `POST /api/v1/auth/device/approve` / `deny` `{user_code, name}` | approving needs the `admin` role and a free key slot in the plan | | CLI polls | `POST /api/v1/auth/device/token` `{device_code}` | `authorization_pending`, `slow_down`, `access_denied`, `expired_token` as RFC 8628 errors; success returns `access_token` (the `osr_live_` key), `account`, `plan`, `edition` | Only approve codes from a terminal you started yourself. Keys minted this way appear on `/platform/dashboard/keys` with the chosen name and can be revoked like any other. Pasting a key works too (`osr login --token osr_live_...`, or `--with-token` from stdin in CI); `osr whoami` shows the workspace, plan and edition behind the stored credential, `osr token create|list|revoke` manage keys and `osr mcp --remote` bridges an IDE to this workspace's MCP server without copying the key around. ## 2. Route a request `POST /api/v1/route` returns a decision: the chosen target, the confidence, ranked alternatives and the signals extracted from the request. The target is not called unless `execute` is set. ```bash curl -s "$OSR_URL/api/v1/route" \ -H "Authorization: Bearer $OSR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"text": "Prove that sqrt(2) is irrational.", "top_k": 3}' ``` ```json { "request_id": "7f3c...", "target": {"id": "llm-frontier", "kind": "llm", "name": "Frontier model"}, "confidence": 0.81, "alternatives": [{"id": "llm-mid", "kind": "llm", "utility": 0.62}], "signals": {"complexity": 0.74, "domains": ["math"], "reasoning_need": 0.9, "contains_pii": false, "...": "..."}, "ranked": [{"id": "llm-frontier", "utility": 0.79, "quality_estimate": 0.91, "breakdown": {"...": "..."}}], "policy_rejections": {}, "explanation": "llm-frontier: high reasoning need, math domain, quality weight dominates ...", "executable": true, "elapsed_ms": 1.9 } ``` Request fields: | Field | Type | Meaning | |---|---|---| | `text` | string, required | The user request. | | `history` | list of `{role, content}` | Prior turns; the router reads them for intent, task type and session affinity. | | `context` | object | Free-form facts (`app`, `session_id`, `user_locale`, ...). Keys starting with `_` are dropped. | | `objective` | object | Relative weights: `quality` (default 1.0), `cost` (0.15), `latency` (0.05), plus `quality_floor`, `energy`, `carbon`. `cost_weight` is accepted as a synonym of `cost`. | | `constraints` | object | Hard constraints, never traded off: see below. | | `profile` | object | Opaque user/tenant features (locale, expertise, preferences). Hashed into the learners; never logged raw. | | `kinds` | list | Restrict to target kinds: `llm`, `agent`, `skill`, `persona`, `tool`, `human`. | | `top_k` | int, default 3 | How many alternatives to return. | | `plan` | bool | Build a persona → skill → model plan instead of choosing a single target (**pro** plan and above). | | `execute` | bool | Route and run the chosen target or plan; the output is returned in `result` (**pro** and above). | Constraints (`constraints` object): | Key | Effect | |---|---| | `max_cost_per_1k` | Drop targets whose price per 1k tokens is above this. | | `max_latency_ms` | Drop targets slower than this. | | `preferred_max_latency_ms` | Soft target: targets whose observed p90 latency is above it are penalised, not dropped. | | `region` | Only targets served from this region. | | `data_boundary` | Only targets at least this strict (`public` < `private` < `on_prem`). | | `contains_pii` | Force the PII handling on or off (`null` lets the sensitivity signal decide). | | `allow_targets` / `deny_targets` | Explicit allow / deny lists of target ids. | | `allowed_kinds` | Same as `kinds`. | | `require_tools` | Only targets that can call tools. | ### Quote before routing `POST /api/v1/estimate` takes the same body and returns the price of each candidate without calling anything: `targets[]` with `cost_usd`, `input_tokens`, `output_tokens`, vendor and model, plus the router's `recommended` pick and the `cheapest`, `best_quality` and `fastest` candidates. `output_tokens` overrides the assumed reply length; `monthly_requests` adds a `monthly` projection (`recommended_usd`, `cheapest_usd`, `dearest_usd`, `savings_usd`). Without a key it is rate limited per client and ignores tenant rules; with a key it is metered as `estimate` and applies your plan and tenants. `GET /api/v1/models/recommended` returns the router's picks per use case from live quotes. Details, the SDK form and the `/estimate` page: [MCP.md](https://opensmartroute.ai/docs/MCP.md). ## 3. Execute the chosen target With `"execute": true` the platform runs the chosen target (or plan) with the configured providers and returns the output, the steps that ran and the metered cost: ```bash curl -s "$OSR_URL/api/v1/route" -H "Authorization: Bearer $OSR_API_KEY" -H "Content-Type: application/json" \ -d '{"text": "Summarise this contract clause for a non-lawyer: ...", "execute": true, "objective": {"cost": 0.4}}' ``` The response adds `result` with `text`, `target_id`, `steps[]` (`role`, `target_id`, `ok`, `latency_ms`), `tokens` and `cost_usd`. ### Model providers A target executes when it is mapped to a model on an **OpenAI-compatible endpoint** - OpenAI, Azure OpenAI, OpenRouter, Mistral, Groq, Together, Fireworks, DeepSeek, a local Ollama, vLLM or LiteLLM, anything that speaks `/v1/chat/completions`. Two sources define the mapping and are merged (the console wins on the same provider name or target): * the **file** the deployment mounts, `OSR_PLATFORM_PROVIDERS` (YAML/JSON path or inline JSON) with `providers.` (`base_url`, `api_key` or `api_key_env`, optional `api_key_header`, `extra_headers`, `timeout_s`, `max_retries`) and `models.` (`provider`, `model`, optional `system_prompt` and request defaults). On first start it is imported into the console once, so it can be edited there. * the **operator console**, `/platform/admin/providers`: add an endpoint from a preset, store the key inline or name the environment variable that holds it, *Check* it (`GET /models` with its credentials records reachability, latency and the upstream model ids, which the mapping form then offers), map targets to upstream models, disable or delete. Every write re-binds the handlers in the API at once, and every other API replica re-reads the store within `OSR_PLATFORM_PROVIDERS_RELOAD_S` (30 s) - no restart, no redeploy. `GET /api/v1/providers` is the public, OpenRouter-style view of what a deployment executes on, rendered by the site at `/providers`: one row per executable target with `provider`, `model`, the recognised reference model, `input_usd_per_1m` / `output_usd_per_1m` (from the reference catalogue when the upstream model is known, else the target's declared cost), context window, tool and reasoning support, circuit-breaker `health` and seven-day traffic; plus each provider's host, kind, health and model count. Credentials never appear: the console shows only whether a key is set and its last four characters (`api_key_set`, `api_key_hint`). Keys stored through the console live in the platform database next to the organization SSO secrets; prefer `api_key_env` when the host injects secrets. Endpoints: `GET|POST /api/v1/admin/providers`, `GET|PATCH|DELETE /api/v1/admin/providers/{id}`, `POST /api/v1/admin/providers/{id}/check`, `PUT|DELETE /api/v1/admin/providers/models/{target_id}`. Targets without a mapping still route (decisions, plans, traces, quotes) and report `executable: false`. ### Response headers | Header | On | Meaning | |---|---|---| | `X-Request-Id` | every response | The id to quote in feedback and support requests; echoed back when you send one. | | `Server-Timing` | every response | `app;dur=` - time spent inside the platform. | | `X-OSR-Target` | `/v1/chat/completions` | The target id that answered. | | `X-OSR-Confidence` | `/v1/chat/completions` | The router's confidence in that choice. | | `Retry-After` | `429` | Seconds until the per-minute window, the daily quota or the budget period resets. | | `X-Quota-Limit` | `429` (daily) | Your plan's requests per day. | | `X-Budget-Limit`, `X-Budget-Used`, `X-Budget-Period` | `429` (budget) | The exhausted workspace or tenant budget in USD and its period (`daily` / `monthly`). | | `X-Upgrade: true` | `403` | The feature exists but is not in your plan. | | `ETag`, `Cache-Control` | public catalogue reads | Conditional requests: send `If-None-Match` and receive `304` when nothing changed. | ## 4. OpenAI-compatible endpoint Point an OpenAI SDK at `$OSR_URL/v1` and set `model` to `auto`. The router chooses the target per request, runs it and returns a standard `chat.completion` whose `model` field is the target that answered. Decision metadata is added in an `opensmartroute` object that other clients ignore. ```python from openai import OpenAI client = OpenAI(base_url=f"{OSR_URL}/v1", api_key=OSR_API_KEY) reply = client.chat.completions.create( model="auto", messages=[{"role": "user", "content": "Write a haiku about routing."}], ) print(reply.model) # e.g. "llm-small" - the target the router chose print(reply.choices[0].message.content) ``` ```json "opensmartroute": { "request_id": "7f3c...", "target": "llm-small", "kind": "llm", "confidence": 0.88, "plan": [], "alternatives": ["llm-mid"], "cost_usd": 0.00012, "latency_ms": 412.5, "fallback_from": [] } ``` Options outside the OpenAI schema go in the request body and are ignored by other providers: | Field | Meaning | |---|---| | `model` | `auto` (default) lets the router choose; a target id pins that target. | | `models` | Candidate list: the router chooses among them and falls back down the list when a target fails. | | `osr.objective` / `osr.constraints` | Same as on `/api/v1/route`. | | `osr.tenant` | Tenant slug (or the `X-OSR-Tenant` header); applies the tenant's constraints (**enterprise**). | | `osr.plan` | `true`/`false` to force or suppress plan composition; omitted = whatever the plan tier allows. | | `osr.fallbacks` | `false` disables the automatic retry on the next best target (default `true`). | | `stream` | `true` streams server-sent events in the OpenAI format: each provider token is relayed as it arrives; the last chunk carries `usage` and the `opensmartroute` metadata. Quota, routing and provider errors before the first token are ordinary HTTP errors; a fallback to another target happens only before the first token. | `GET /v1/models` lists `auto` and every executable target. ## 5. Feedback `POST /api/v1/feedback` reports how a routed answer turned out. The learners (Bradley-Terry, IRT, LinUCB) update on the workspace's own traffic. Available on the **pro** plan and above. ```bash curl -s "$OSR_URL/api/v1/feedback" -H "Authorization: Bearer $OSR_API_KEY" -H "Content-Type: application/json" \ -d '{"request_id": "7f3c...", "target_id": "llm-frontier", "success": true, "quality": 0.9, "latency_ms": 2100}' ``` Fields: `request_id`, `target_id`, `success` (required); `quality` (0..1), `cost_usd`, `latency_ms`, `domains`, `complexity`, `preferred_over` (the id of a target this one beat in a comparison). Every report is kept against its request: `GET /api/v1/activity` summarises it per row (`outcome`), `GET /api/v1/trace/{request_id}` lists the reports (`outcomes`) next to the spans, and in the enterprise edition the audit chain records an `outcome` entry after the `decision` entry (section 10). The dashboard has the same form without curl: expand a row on `/platform/dashboard/activity` (or open its trace) and report good / poor, a quality score and, optionally, the target that would have done better - the latter is recorded as a pairwise preference for the Bradley-Terry learner. ## 6. Plans and quotas | Plan | Requests / day | Requests / min | Keys | Features | |---|---|---|---|---| | **free** | 500 | 30 | 2 | route + trace, OpenAI-compatible proxy | | **pro** | 20,000 | 300 | 10 | + plans (`plan`/`execute`), feedback | | **enterprise** | 500,000 | 3,000 | 100 | + tenants, `/stats`, hash-chained `/audit`, organization SSO, 50 tenants, 500 seats | `GET /api/v1/info` (`plans`) returns the values for the deployment you are calling; `GET /api/v1/me` returns your plan. Upgrades are made in the dashboard (`/platform/dashboard/billing`). * `429 rate limit exceeded` - per-minute window; wait `Retry-After` seconds. * `429 daily quota exhausted` - the day's budget; `X-Quota-Limit` shows the ceiling. * `403` with `X-Upgrade: true` - the feature is not in your plan. * `404 ... requires the enterprise edition` - the deployment runs the community image, which does not expose tenants, statistics or audit regardless of plan. ## 7. Organizations Signup creates a personal workspace. An **organization** is a second kind of workspace with its own members, roles, API keys, tenants, plan and usage; one user can belong to several and switches between them in the dashboard (`/platform/dashboard/workspace`, `POST /api/v1/workspaces/{account_id}/switch`). Creating one (`POST /api/v1/workspaces` with `name` and an optional `slug`) makes you its owner and issues its first API key. Roles are `member` < `admin` < `owner`: | Action | Role | |---|---| | Route, read usage, savings and activity | `member` | | Create and revoke API keys, change tenants, invite and remove members, rename the workspace | `admin` | | Grant or remove `owner`, configure organization SSO, delete the connection | `owner` | API keys belong to the workspace and act with its full rights; the role model applies to signed-in browser sessions. Members are invited by email (`POST /api/v1/workspace/invites`): the invitee gets a message with the link when the deployment has a mail server, and the platform returns the same link once to the inviter so it can be shared directly. Seats and tenants are limited per plan (`max_members`, `max_tenants` in `GET /api/v1/info`). An organization always keeps at least one owner; owners transfer ownership before leaving. **Organization SSO** (enterprise) lets people sign in with the company identity provider: `PUT /api/v1/workspace/sso` with `kind` (`github`, `google`, `microsoft`, `gitlab` or `oidc` with an `issuer`), `client_id`, `client_secret`, the allowed email `domains` and the `default_role` given to anyone who signs in from one of those domains. The workspace needs a slug first; the sign-in page shows the provider as `org-`. ## 8. Tenants Tenants (enterprise) apply a fixed set of hard constraints per customer, team or region without the caller repeating them. Create a tenant, then send its slug in `X-OSR-Tenant` (or `osr.tenant`). ```bash curl -s -X PUT "$OSR_URL/api/v1/tenants" -H "Authorization: Bearer $OSR_API_KEY" -H "Content-Type: application/json" \ -d '{"slug": "eu-health", "config": {"region": "eu", "data_boundary": "private", "max_cost_per_1k": 0.01, "deny_targets": ["llm-frontier"]}}' curl -s "$OSR_URL/api/v1/route" -H "Authorization: Bearer $OSR_API_KEY" -H "X-OSR-Tenant: eu-health" \ -H "Content-Type: application/json" -d '{"text": "Anonymise this discharge summary ..."}' ``` Tenant config keys: `region`, `data_boundary`, `max_cost_per_1k`, `max_latency_ms`, `preferred_max_latency_ms`, `deny_targets`, `allow_targets`, `contains_pii`, `daily_budget_usd`, `monthly_budget_usd`. Explicit request constraints win over the tenant defaults; the workspace policy (next section) applies on top of both. Executed spend is attributed to the tenant, so `GET /api/v1/governance` shows spend and budgets per tenant. ## 9. Governance: workspace policy, budgets and controls A **workspace policy** applies to every request of the workspace, whichever key or tenant sent it. It takes the same keys as a tenant plus budgets, and needs the `admin` role: ```bash curl -s -X PUT "$OSR_URL/api/v1/policy" -H "Authorization: Bearer $OSR_API_KEY" -H "Content-Type: application/json" \ -d '{"config": {"data_boundary": "private", "deny_targets": ["llm-frontier"], "monthly_budget_usd": 250}}' ``` `GET /api/v1/policy` returns the policy, the accepted keys and the current budget lines; `DELETE` removes it. Constraints merge deterministically: a boundary or region fills in when the request has none, cost and latency caps take the stricter value, deny lists are merged and allow lists intersected. Unknown keys, unknown target ids and an id present in both `allow_targets` and `deny_targets` are rejected with `400`. **Budgets** count executed spend (`execute: true`, `/v1/chat/completions`, MCP `ask`) - quotes and route-only calls are free. `daily_budget_usd` resets at midnight UTC, `monthly_budget_usd` on the first of the month. Once a workspace or tenant budget is exhausted, routed calls return `429 ... budget ... exhausted` with `Retry-After` and the `X-Budget-*` headers until the period rolls over or the limit is raised. `GET /api/v1/governance` is the compliance view of the workspace in one payload: | Block | Contents | |---|---| | `controls` | What the deployment enforces: `input_guard`, `pii_redaction`, `steering_strip`, `metrics`, `tracing`, `audit`, `health_breakers`, `decision_cache`, `learning` (`persisted` or `in-process`). | | `audit` | Whether the hash-chained audit trail is on, its length and the result of the last verification. | | `retention_days` | How long usage rows are kept before the background sweep purges them. | | `policy`, `policy_keys`, `budgets`, `spend` | The workspace policy, its accepted keys, budget lines with used and remaining amounts, spend today and this month. | | `quota` | Requests today against the plan's daily and per-minute limits. | | `tenants` | Each tenant with its monthly spend and budget lines. | | `catalogue` | Targets per `data_boundary`, targets that refuse PII, executable targets, providers, and which targets remain reachable under the policy. | The dashboard renders the same data at `/platform/dashboard/governance`, with a policy editor for admins. ## 10. Observability and caching The platform observes itself: every span, event, counter and alert below is produced, stored and served by the platform - there is no external metrics system, tracing backend or alerting tool to run. Every deployment exposes liveness and readiness probes and the counters without a key: | Endpoint | Meaning | |---|---| | `GET /healthz` | Process is up: edition, version, target count, uptime. | | `GET /readyz` | `200` when the database, catalogue, router and (enterprise) audit file all answer; `503` with the failing check otherwise. Also reports `database_dialect` (`sqlite` or `postgres`), `redis` and `events` (the event bus) - those two are informational and never fail readiness. | | `GET /metrics` | The counters in text exposition format: `osr_platform_http_requests_total{method,route,status}`, `osr_platform_http_request_duration_ms_*`, `osr_platform_http_in_flight`, plus the router's `osr_route_decisions_total`, `osr_route_latency_ms_*`, `osr_outcome_*` and `osr_decision_cache_*`. | Responses carry `X-Request-Id` (send your own to correlate with your logs) and `Server-Timing`; the API writes one JSON access-log line per request (`osr.platform.access`) with the route template, status, latency and account - never the prompt. `GET /api/v1/stats` (enterprise) adds `http`, `controls`, `cache` and `observability` snapshots; `/platform/dashboard/health` shows them live. ### Time series and alerts | Endpoint | Returns | |---|---| | `GET /api/v1/telemetry/series?window=24h` | The workspace's metered traffic bucketed over `1h` (per minute), `6h` (5 min), `24h` (15 min), `7d` (3 h) or `30d` (per day): requests, failures, `error_rate`, p50 / p95 latency, cost and tokens per bucket, totals per target and per endpoint, and the window total. Computed from the platform's own usage records, so it agrees with the activity log to the row. Plans with `stats` also get `http`: the deployment's requests, 4xx, 5xx and p50 / p95 / p99 latency per bucket from the per-minute counters the process keeps for 24 hours. | | `GET /api/v1/alerts` | Active conditions, most severe first, each with the dashboard page that shows or fixes it. Workspace rules on every plan: a daily or monthly budget of the workspace or a tenant at 80 % (`warning`) or exhausted (`critical`), and 20 % or more of the last 15 minutes' metered requests failing (10+ requests). Deployment rules on plans with `stats`: a failing readiness check, 5 % or more 5xx over the last five minutes (20+ requests), p95 above 2 000 ms, an open or half-open circuit breaker per target, a routing-drift alarm in the last hour, a failed autopilot cycle, a routing SLM last fitted more than 30 days ago, and tracing being off. | The overview shows the last 24 hours (traffic, latency) and the alerts; `/platform/dashboard/health` adds the deployment series and the same alerts, refreshed with the page. ### Notifications: alert delivery Alerts are evaluated by the API itself - no external scheduler. Every `OSR_PLATFORM_ALERTS_INTERVAL_S` seconds (default 60; `0` turns the loop off and leaves *Evaluate now* to operators) one replica - the one holding the `osr:alerts:leader` lease in Redis, or the only one without Redis - re-evaluates the workspace rules for every workspace that was active in the last 15 minutes or has a channel or an open alert, plus the deployment rules. Each condition becomes a **notification episode**: opened when the rule starts firing, refreshed while it stays on (an escalation from `warning` to `critical` is delivered again), resolved when it clears. Episodes land in an inbox with unread counts per severity, are published to the `alerts` Kafka topic in cluster mode, and are delivered to the workspace's **channels**: | Kind | Target | Delivery | |---|---|---| | `email` | An address | Through the platform mailer (`OSR_PLATFORM_SMTP_URL`, otherwise the outbox at `/platform/admin/mail`). | | `webhook` | An `https://` URL | `POST` of `{event, alert, workspace, url}` with `X-OSR-Event` (`alert.firing`, `alert.escalated`, `alert.resolved`, `alert.test`), `X-OSR-Delivery` and, when the channel has a secret, `X-OSR-Signature: t=,v1=` = HMAC-SHA256 of `.`; verify like Stripe signatures. | | `slack` | An incoming-webhook URL | Block Kit message with severity, rule, value against threshold and a link to the page that fixes it. | | `teams` | An incoming-webhook URL | MessageCard with the same facts. | A channel filters by minimum severity (`info`, `warning`, `critical`) and optionally by rule names, can be disabled, records its last delivery and last error, and has *Send test*. Failed webhook deliveries are retried on the following evaluation rounds up to three attempts. Up to 20 channels per workspace; creating or changing them takes the `admin` role. Operators have the same inbox and channels for the **deployment** scope (readiness, 5xx rate, p95, circuit breakers, drift, autopilot, SLM age, tracing) at `/platform/admin/notifications`. | Endpoint | Purpose | |---|---| | `GET /api/v1/notifications?state=firing&unread=true&limit=100` | The workspace inbox (`items`) and unread counts per severity (`unread`). | | `GET /api/v1/notifications/unread` | Unread counts only - the dashboard bell polls this. | | `POST /api/v1/notifications/read` | `{"ids": [...]}` marks those read, `{"ids": null}` marks everything read. | | `GET|POST /api/v1/notifications/channels` | List channels (with the rule names you can filter on) or create one: `kind`, `name`, `target`, optional `secret`, `min_severity`, `rules`, `enabled`. | | `PATCH|DELETE /api/v1/notifications/channels/{id}` | Change or remove a channel. | | `POST /api/v1/notifications/channels/{id}/test` | Deliver a test notification and return `ok`, `status`, `error`. | | `GET /api/v1/notifications/deliveries` | The last delivery attempts (channel, event, attempt, status, error). | | `GET|POST /api/v1/admin/notifications` and `.../channels`, `.../deliveries` | The operator (deployment scope) equivalents; `POST /api/v1/admin/notifications/evaluate` runs one round now and returns what it opened, resolved, delivered and failed. | ### Shared state and replicas A single API replica keeps everything on its data volume (SQLite, learner state, audit chain). A deployment that runs several replicas moves the shared parts to services that are themselves containers - no managed cloud database, cache or broker is required, and the same images run on a laptop, a Kubernetes cluster or Azure Container Apps: | State | Service | Setting | |---|---|---| | Accounts, users, keys, tenants, policies, usage, feedback, marketplace | PostgreSQL | `OSR_PLATFORM_DATABASE_URL` | | Rate-limit windows, learner state (enterprise) | Redis | `OSR_PLATFORM_REDIS_URL` | | `usage`, `feedback` and `admin` events as JSON topics `osr.` | Kafka (any Kafka-protocol broker) | `OSR_PLATFORM_KAFKA_BOOTSTRAP`, `OSR_PLATFORM_KAFKA_TOPIC_PREFIX` | `GET /api/v1/info` -> `storage` says which mode a deployment runs (`{database, clustered, redis, events}`) and the operator console shows it on `/platform/admin`. Event payloads carry the same fields as the activity log and never the prompt text. `platform/docker-compose.yml` in the repository starts the complete stack; the Azure template provisions the three services as container apps with internal TCP ingress next to the API and web app. In that stack the routing SLM is its own service too: the API publishes each routed prompt to the `training` topic (opt-in, `OSR_PLATFORM_TRAINING_EVENTS`), the SLM service pairs prompts with the outcomes reported through `POST /api/v1/feedback`, trains a challenger on a schedule, keeps it only when it beats the champion on a held-out split, and writes the promotion to the shared volume from which every API replica reloads it. Operators see the model in service, the evidence ingested, every cycle and a prompt probe on `/platform/admin/slm` (`GET /api/v1/admin/slm`, `/slm/reports`, `POST /slm/cycle`, `/slm/predict`). ### Services The API is one image that can run as one process or as several: the documentation JSON (`docs`), the public model catalogue and rankings (`rankings`), the marketplace (`marketplace`), provider management (`providers`), signup and account links (`onboarding`), identity and workspaces (`accounts`: sign-in, sessions, organizations, keys, tenants, policy, governance, usage), Stripe billing (`billing`), the operator console API (`admin`), the MCP endpoint (`mcp`), the OpenAI-compatible proxy (`openai`), the routing core with its trace, event and learning reads (`routing`) and the routing SLM (`slm`) each have their own entry point and port. `/api/v1/info`, `/api/v1/status` and `/api/v1/estimate` always stay with the API. The API remains the single public origin and forwards a domain's paths to its service when `OSR_PLATFORM__URL` is set (streamed completions are relayed as they arrive); the answer carries `X-OSR-Service` naming the process that served it. `GET /api/v1/info` -> `services` lists the topology; operators see live health per service on `/platform/admin/services` (`GET /api/v1/admin/services`). The Compose stack and the Azure template deploy every service as its own container next to the gateway. ### Traces and events The router records every request as a tree of **spans** (`request`, `route`, `plan`, `execute`) with **events** inside them (`route.signals`, `route.policy`, `route.rank`, `route.fallback`, `guard.*`, `execute.step`, `learn.outcome`, `cache.hit`, ...). The newest ones stay in an in-memory buffer (1000 by default, `OSR_OBSERVABILITY_MEMORY_EVENTS`) and every one of them is also written to the platform database by the platform's own telemetry store (`OSR_PLATFORM_TELEMETRY_STORE`, on by default) - off the request path, from a background writer, so tracing never slows routing. Events and traces therefore survive restarts, can be read by time range and are purged after `OSR_PLATFORM_TELEMETRY_RETENTION_DAYS` (14). They are readable per workspace: | Endpoint | Returns | |---|---| | `GET /api/v1/trace/{request_id}` | The activity row plus every span and event of one request, in time order, with `trace_id`, `spans` and the root `duration_ms`; `outcomes` (the feedback reported for the request) and `audit` (its hash-chained records, plans with `audit`). `source` says whether the rows came from the buffer or the store. `404` when the request is not the workspace's; `events: []` once the request is older than the telemetry retention. | | `GET /api/v1/events?name=route.*&kind=&level=&request_id=&since=&until=&limit=200` | Events of the workspace's requests (oldest first, `limit` up to 2000; `name` may end with `*`; `since` / `until` are epoch seconds, default the last 6 hours). Deployment-wide events without a request id (breaker transitions, autopilot) are included on plans with `stats`. `source`, `persisted`, `retention_start` and `retention_days` describe the store. `404` when tracing is off entirely. | | `GET /api/v1/status` | Public readiness: the `/readyz` checks plus edition, versions, uptime, controls and tracing state; `503` while a check fails. `Cache-Control: no-store`. | `POST /api/v1/route` responses include `trace_id` and the `X-OSR-Trace-Id` header when the request was traced, and `GET /api/v1/activity` marks rows whose spans are kept (in memory or in the store) with `traced: true` and summarises the feedback reported for each request in `outcome` (`POST /api/v1/feedback` is stored per request, so the loop from decision to result is visible per row). Attributes hold ids, numbers and short labels; the request text is represented only by its digest and length. The dashboard shows the history at `/platform/dashboard/events` (window of 1 hour to 7 days, search by request or trace id), opens a trace from any activity row (with a *How it was routed* summary of signals, policy, ranking and decision, the outcome and the audit records) and adds a *Trace* tab to the playground. Public catalogue reads (`/api/v1/info`, `/api/v1/models*`, `/api/v1/rankings`, `/api/v1/llms*`, `/api/v1/catalogue`, `/api/v1/stats/public`) return an `ETag` and `Cache-Control: public, max-age=60, stale-while-revalidate=60`; send `If-None-Match` to get `304` when nothing changed. Operators tune this and the decision cache with: | Setting | Default | Effect | |---|---|---| | `OSR_PLATFORM_METRICS_PUBLIC` | `true` | `false` restricts `/metrics` to requests carrying `X-Admin-Token`. | | `OSR_PLATFORM_HTTP_CACHE_MAX_AGE_S` | `60` | `Cache-Control` lifetime of public reads; `0` sends `no-store`. | | `OSR_PLATFORM_ROUTE_CACHE_TTL_S` | `0` | Reuse a routing decision for identical requests within this window (`0` = off). | | `OSR_PLATFORM_RETENTION_DAYS` | `365` | Usage rows older than this are purged every six hours; `0` keeps everything. | | `OSR_PLATFORM_TELEMETRY_STORE` | `true` | Persist every span and event to the platform database (the events page, traces and `since` / `until` queries outlive the in-memory buffer). | | `OSR_PLATFORM_TELEMETRY_RETENTION_DAYS` | `14` | Persisted spans and events older than this are purged with the same sweep; `0` keeps them. | ### Learning: the routing SLM and the autopilot Every deployment learns from `POST /api/v1/feedback`: the bandit, IRT, Bradley-Terry, LinUCB and Markov strategies update on each outcome (persisted under the data volume in the enterprise edition). A deployment can also serve a **routing SLM** - the small routing model `osr slm train` / `osr improve` write to a file - as one more strategy in the ensemble, and run the **autopilot** in-process so the platform's own feedback keeps retraining it: | Setting | Default | Effect | |---|---|---| | `OSR_PLATFORM_SLM` | none | Path of the model file; it joins the ensemble as the `slm` strategy (weight `OSR_WEIGHTS_SLM`, default 1.0). | | `OSR_PLATFORM_AUTOPILOT` | `false` | Run `SelfImprover` cycles on a thread: outcomes and remembered prompts become rows, a challenger is trained on a split and promoted only when it beats the champion on the holdout. The promoted model is written to `DATA_DIR/autopilot/slm.json` and outranks the mounted file after a restart. | | `OSR_PLATFORM_AUTOPILOT_OFFLINE` | `true` | No catalogue refresh, dataset download or web search from the API process; `false` also refreshes `OSR_PLATFORM_AUTOPILOT_SOURCES` (public dataset presets, comma separated). | | `OSR_PLATFORM_AUTOPILOT_INTERVAL_S` | `OSR_SLM_AUTOPILOT_INTERVAL_S` (3600) | Schedule; drift alarms from the outcome stream trigger a cycle early (`OSR_SLM_AUTOPILOT_DRIFT_*`, rate-limited by `OSR_SLM_AUTOPILOT_MIN_GAP_S`). | | `OSR_PLATFORM_AUTOPILOT_MIN_ROWS` | `20` | A cycle needs at least this many rows before it trains a challenger. | `GET /api/v1/learning` (any plan) returns the strategy ensemble with its weights, the learner state (persistence, drift resets, quarantined state), per-target outcome statistics, the SLM in service (rows, sources, encoder, calibration, fit history - never the weights) and the autopilot status with its recent champion-vs-challenger reports. `controls.slm` / `controls.autopilot` appear in `GET /api/v1/governance` and `GET /api/v1/info`. Operators trigger a cycle with `POST /api/v1/admin/autopilot/cycle` (`X-Admin-Token`; 404 without an autopilot). Cycles are traced as an `autopilot.cycle` span with `learn.improve` / `learn.promote` events, so they appear in `GET /api/v1/events` and on `/platform/dashboard/events`. The dashboard renders all of it at `/platform/dashboard/learning`. ## 11. Marketplace The marketplace (`/marketplace`) is where agents, skills, personas, prompts, tools, model profiles and stack templates are shared. Every listing is a manifest the SDK, CLI and this API understand; installing records it on your account and returns the manifest and ready-made snippets, and `registry://` references in a stack file resolve to `GET /api/v1/registry/{slug}/manifest`. Browsing is public; installing, rating and publishing need an account. See the [marketplace guide](https://opensmartroute.ai/docs/MARKETPLACE.md) for finding, installing, publishing and the review lifecycle; the endpoints are in the [REST API reference](https://github.com/isathish/OpenSmartRoute/blob/main/platform/api/openapi.json) under *Marketplace*. ## 12. MCP server The platform is also a [Model Context Protocol](https://modelcontextprotocol.io) server, so an IDE or agent (VS Code, Cursor, Claude, Windsurf) can route, quote and explain through it. `GET /mcp` describes the server without a key; `POST /mcp` accepts JSON-RPC 2.0 messages with the same `Authorization` header as the REST API. Tools: `route`, `estimate`, `recommend`, `explain`, `list_targets`, `feedback`, `ask` (pro plan and above; routes and executes) and `marketplace_search` / `marketplace_get`. Routed calls are metered like `/api/v1/route`. Clients that only speak stdio use the CLI as a bridge: ```json {"mcpServers": {"opensmartroute": {"command": "osr", "args": ["mcp", "--url", "https:///mcp", "--api-key", "osr_..."]}}} ``` Per-client configuration, the tool reference and the SDK side are in [MCP.md](https://opensmartroute.ai/docs/MCP.md). ## 13. Dashboard The site has two halves on one hostname. The **website** - landing page, `/docs`, `/models`, `/vendors`, `/rankings`, `/pricing`, `/marketplace`, `/compare`, `/roi`, `/estimate` - needs no account. The **platform** is everything under `/platform`: sign-in and sign-up (`/platform/login`, `/platform/signup`, password recovery, invitations, `/platform/cli/authorize`), the dashboard (`/platform/dashboard/...`), the operator console (`/platform/admin/...`), the playground (`/platform/playground`) and the marketplace publish flow (`/platform/marketplace/publish`). On the hosted service the two are separate deployments; the older paths without the prefix (for example /dashboard/keys, /login or /auth/callback) redirect permanently to their `/platform` counterpart with the query string intact, so bookmarks, emailed links and OAuth redirect URIs registered before the split keep working. Signed-in users have a dashboard for the data the API exposes: | Page | What it shows | |---|---| | `/platform/dashboard` | Requests, tokens and spend for the last 30 days; plan and quota usage; the last 24 hours as a traffic and latency series (`GET /api/v1/telemetry/series`) and the active alerts (`GET /api/v1/alerts`). | | `/platform/dashboard/usage` | Per-day and per-target usage (`GET /api/v1/usage`); CSV export of the daily series and both breakdowns. | | `/platform/dashboard/savings` | Routed cost against a baseline model (`GET /api/v1/savings?baseline=`); CSV export per day and per target for finance. | | `/platform/dashboard/activity` | Request-level activity log: endpoint, target, domain, complexity, tokens, reported outcome - never the prompt text. Rows still in the tracer buffer open a span waterfall with the decision story, outcome and audit records. CSV export of the loaded rows. | | `/platform/dashboard/events` | The router's event history for the workspace (`GET /api/v1/events`, persisted by the platform): a window of 1 hour to 7 days, spans and events with level and stage filters, search by request id / trace id / name, most frequent names, click-through to the request trace, CSV export. | | `/platform/dashboard/learning` | How the router learns (`GET /api/v1/learning`): strategy weights, outcomes per target, the routing SLM (rows, sources, the targets it ranks, accuracy history, calibration) and the autopilot (schedule, drift, cycles, a retraining-in-progress indicator, champion vs challenger; operators can run a cycle). Fits and cycles export as CSV. | | `/platform/dashboard/keys` | Create, name and revoke API keys. | | `/platform/dashboard/tenants` | Tenant constraints and budgets (enterprise). | | `/platform/dashboard/governance` | Controls in force, workspace policy editor, budgets and spend, audit status, retention, catalogue data boundaries. | | `/platform/dashboard/health` | Deployment readiness (`GET /api/v1/status`), the active alerts, HTTP traffic and latency percentiles over time (`GET /api/v1/telemetry/series`), live counters, routing decisions and latency, circuit breakers, decision cache (enterprise). | | `/platform/dashboard/notifications` | The alert inbox (firing and resolved episodes, mark read), delivery channels (email, webhook, Slack, Teams) with severity and rule filters and *Send test*, and the delivery log. The bell in the header shows the unread count. | | `/platform/dashboard/audit` | Hash-chained decision trail with chain verification (enterprise); export as CSV or as the JSONL the offline verifier reads. | | `/platform/dashboard/billing` | Plan, invoices and upgrades. | | `/platform/dashboard/workspace` | The current workspace: name, slug, plan; create or switch to an organization; organization SSO. | | `/platform/dashboard/members` | Members, roles and pending invitations of an organization. | | `/platform/dashboard/listings` | Marketplace listings you published and everything you installed. | | `/platform/dashboard/integrations` | MCP configuration for VS Code, Cursor, Windsurf, Claude Code and Claude Desktop with your key. | | `/platform/dashboard/account` | Your sign-in identities, password and browser sessions. | Every table with an *Export CSV* button downloads exactly the rows shown (UTF-8 with BOM, RFC 4180 quoting, cells that start with `=`, `+`, `-` or `@` are prefixed so spreadsheets do not run them as formulas). A failed read shows what it means - session expired, feature not on the plan, budget exhausted with the amounts, rate limit with the retry delay, API unreachable - with a *Retry* button; an expired session returns to sign-in and back to the page afterwards. The public pages `/platform/playground` (route a request in the browser and copy the equivalent `curl`), `/models` (every routing target with request, token, cost, latency and health statistics; also `GET /api/v1/models`, and the reference LLM catalogue with vendor prices and context windows at `GET /api/v1/llms`), `/vendors` (one page per LLM vendor with prices, context windows and benchmarks), `/rankings` (targets ordered by observed quality per domain), `/roi` (estimated saving against always calling one model), `/compare` (alternatives side by side) and `/marketplace` need no account. Machine-readable companions: `/llms.txt` (documentation map for AI assistants, llmstxt.org; every documentation page is also served as Markdown at `/docs/.md` and advertised with a `text/markdown` alternate link), `/llms-full.txt` (the whole documentation as one Markdown file), `/feed.xml` (Atom feed of releases), `/sitemap.xml` and `/openapi.json` (the complete OpenAPI 3.1 document of the API - the committed snapshot the REST reference renders, not the subset one gateway process happens to serve). ### Operator console The people who run a deployment sign in at `/platform/admin/login` with an operator **username and password** (`POST /api/v1/admin/auth/login` returns an `osr_op_` session token valid for twelve hours; send it as `Authorization: Bearer`). The first operator comes from the environment: set `OSR_PLATFORM_ADMIN_USERNAME` and `OSR_PLATFORM_ADMIN_PASSWORD` and the API creates it - or resets its password - at start-up as a `superadmin`. The static `OSR_PLATFORM_ADMIN_TOKEN` (`X-Admin-Token`) keeps working for automation and counts as a superadmin. Operators are separate from workspace users. | Page | What it does | Endpoints | |---|---|---| | `/platform/admin` | Deployment counters: users, workspaces, keys, tenants, operators, sessions, requests, errors and provider cost of the last seven days, plan mix, sign-in configuration. | `GET /api/v1/admin/overview`, `GET /api/v1/admin/plans` | | `/platform/admin/analytics` | The whole deployment over 7, 30 or 90 days: requests and failures per day with mean latency, cost per day, endpoint share, workspaces per plan, signups per day, the busiest workspaces and the targets that carried the traffic. | `GET /api/v1/admin/analytics?days=` | | `/platform/admin/health` | This replica of the routing process: readiness checks, HTTP traffic and latency series (1 h / 6 h / 24 h), the last five minutes, circuit breakers per target, the deployment alerts in force and the alert evaluator; auto-refresh. | `GET /api/v1/admin/health?window=` | | `/platform/admin/activity` | Every metered request across workspaces, newest first: workspace, endpoint, target, status, latency, tokens, cost, request id; filter by endpoint, failures, workspace or request id; load older pages; CSV export. | `GET /api/v1/admin/activity?limit=&before=&account_id=&endpoint=&target=&failed=&request_id=` | | `/platform/admin/users` | The user directory: search by email or name, filter by status, email confirmation and sign-in method (password or SSO / key only), sort by joined, last sign-in, email or name, export the matching rows as CSV. Select rows for a bulk action: disable, enable, mark the email verified, sign out everywhere, send a password reset, delete (each user is reported as done or failed). Create one with a personal workspace, plan, optional password and first key (shown once; without a password a reset link is issued so the person chooses one). Per user: workspaces and roles, linked identities, sessions, lifecycle history; rename, disable, reset password or send a reset link, re-send or force the email confirmation, sign out everywhere, delete (workspaces they alone belonged to are disabled and their keys revoked). | `GET|POST /api/v1/admin/users` (`q`, `disabled`, `verified`, `method`, `sort`, `order`, `format=csv`), `POST /api/v1/admin/users/bulk`, `GET|PATCH|DELETE /api/v1/admin/users/{id}`, `DELETE /api/v1/admin/users/{id}/sessions`, `POST /api/v1/admin/users/{id}/reset-link`, `POST /api/v1/admin/users/{id}/verify-link`, `GET /api/v1/admin/users/{id}/audit` | | `/platform/admin/workspaces` | The workspace directory: search, filter by kind, plan and status, sort by name, plan or created, export CSV; create an organization for an owner email. Per workspace: plan, name, slug, disable, delete; members with roles (add by email, change role, remove - the last owner stays); API keys (mint, revoke); tenants (create or replace a validated configuration, delete); policy, SSO and 30-day usage. *Open dashboard* (superadmins) mints a one-hour browser session on the workspace as its owner and switches to the workspace portal, which shows an *Operator view* banner with the way back; the view sees every plan feature (statistics, audit, tenants) whatever the workspace's plan, while quotas stay the workspace's; the owner sees the session (provider `operator`) in their session list and the audit log records `workspace.open`. | `GET|POST /api/v1/admin/accounts` (`q`, `kind`, `plan`, `disabled`, `sort`, `order`, `format=csv`), `GET|PATCH|DELETE /api/v1/admin/accounts/{id}`, `.../usage`, `.../members[/{user_id}]`, `.../keys[/{key_id}]`, `.../tenants/{slug}`, `POST /api/v1/admin/accounts/{id}/session` | | `/platform/admin/tenants` | Every tenant across workspaces with its configuration. | `GET /api/v1/admin/tenants` | | `/platform/admin/leads` | Product-qualified leads: workspaces scored on volume, tenants, EU or private data boundaries, agentic traffic, several teams and a company email domain; CSV export. | `GET /api/v1/admin/pql?days=&qualified_only=` | | `/platform/admin/plans` | The plan catalogue - limits, price, feature matrix on this edition - and the workspaces on each plan. | `GET /api/v1/admin/plans`, `GET /api/v1/admin/overview` | | `/platform/admin/slm` | The routing SLM service: model in service, prompts and outcomes ingested, improvement cycles (champion vs challenger), run a cycle, probe the model with a prompt. | `GET /api/v1/admin/slm`, `GET /api/v1/admin/slm/reports`, `POST /api/v1/admin/slm/cycle`, `POST /api/v1/admin/slm/predict` | | `/platform/admin/providers` | Model providers: add an OpenAI-compatible endpoint from a preset, check it (latency, upstream model list), map catalogue targets to upstream models with an optional system prompt, disable or delete; shows the mounted file, what is attached on this replica and the targets that cannot execute yet. | `GET|POST /api/v1/admin/providers`, `GET|PATCH|DELETE /api/v1/admin/providers/{id}`, `POST /api/v1/admin/providers/{id}/check`, `PUT|DELETE /api/v1/admin/providers/models/{target_id}` | | `/platform/admin/services` | Where each domain runs (in the API or as its own service) with a live health probe of every remote service. | `GET /api/v1/admin/services` | | `/platform/admin/marketplace` | Marketplace moderation: the review queue and every other status, approve, reject or unpublish with a note, featured and verified flags, the imported catalogue per source and the scheduled refresh (run it now). | `GET /api/v1/admin/registry?status=`, `POST /api/v1/admin/registry/{slug}/approve`, `POST /api/v1/admin/registry/{slug}/reject`, `POST /api/v1/admin/registry/{slug}/unpublish`, `POST /api/v1/admin/registry/{slug}/flags`, `GET /api/v1/admin/registry/import`, `POST /api/v1/admin/registry/import/refresh` | | `/platform/admin/notifications` | Deployment alerts as an inbox (readiness, error rate, latency, breakers, drift, autopilot, SLM, tracing), operator channels (email, webhook, Slack, Teams), the delivery log and *Evaluate now*. | `GET /api/v1/admin/notifications`, `POST /api/v1/admin/notifications/read`, `POST /api/v1/admin/notifications/evaluate`, `GET|POST /api/v1/admin/notifications/channels`, `PATCH|DELETE /api/v1/admin/notifications/channels/{id}`, `POST /api/v1/admin/notifications/channels/{id}/test`, `GET /api/v1/admin/notifications/deliveries` | | `/platform/admin/operators` | Operator accounts (superadmins create, disable, promote and delete them; the last enabled superadmin cannot be removed), your own password and sessions. Superadmins open any operator's console sessions and revoke one or all of them - a compromised console account is locked out at once. | `GET|POST /api/v1/admin/operators`, `PATCH|DELETE /api/v1/admin/operators/{id}`, `GET|DELETE /api/v1/admin/operators/{id}/sessions`, `POST /api/v1/admin/auth/password`, `GET /api/v1/admin/auth/sessions`, `POST /api/v1/admin/auth/logout` | | `/platform/admin/audit` | Deployment-wide account-lifecycle log: signups, sign-ins and failed attempts, password and email changes, invitations, roles, removals, deletions and every operator action (actor `operator`). Filter by action (the menu lists every action with its count), actor type, time window, user, workspace or free text over actor, IP and details; page back with *Load older*; export the matching events as CSV. | `GET /api/v1/admin/audit-log` (`action`, `actor_type`, `actor_id`, `user_id`, `account_id`, `q`, `after`, `before`, `limit`, `format=csv`) | | `/platform/admin/mail` | Every email the platform composed with its delivery state and counters (composed, awaiting delivery, failed); filter by recipient, kind or delivery; open a message to read it, copy its link, resend it (SMTP only) or delete it; send a test message to prove the SMTP settings. Without `OSR_PLATFORM_SMTP_URL` this is the delivery channel. | `GET /api/v1/admin/mail` (`to`, `kind`, `unsent`, `failed`, `limit`, `offset`), `GET|DELETE /api/v1/admin/mail/{id}`, `POST /api/v1/admin/mail/{id}/resend`, `POST /api/v1/admin/mail/test` | | `/platform/admin/settings` | The effective configuration grouped (sign-in, storage, services, learning, marketplace, mail, billing, observability) with the variable that sets each value; secrets show only whether they are set. Retention: request rows, persisted telemetry, audit log and outbox sizes, and *Run the sweep now*. | `GET /api/v1/admin/settings`, `GET /api/v1/admin/retention`, `POST /api/v1/admin/retention/purge` | | `/platform/admin/search` | Support search from the header box: users by email or name, workspaces by name, slug or email, API keys by prefix, operators, tenants by slug and request ids - each hit links to its page. | `GET /api/v1/admin/lookup?q=` | Every `/api/v1/admin/*` call is written to the access log with `operator:` (or `operator:admin-token`) as the actor. The console is served by the web app but every action is an API call, so the same administration works from scripts with the static token. Operators keep both portals open at once: the console's account menu leads to the workspace dashboard (a superadmin opens any workspace from `/platform/admin/workspaces`) and the dashboard's account menu leads back to the console while an operator session exists. ## 14. Privacy and data handling * Prompt text is routed in memory and, when execution is requested, forwarded to the configured provider. The platform stores request metadata (endpoint, target, domain, complexity, token and cost counts), not the prompt or the answer. * `profile` values are hashed before they reach the learners. * PII detection runs on every request; `contains_pii` together with a tenant `data_boundary` keeps sensitive requests on private or on-prem targets. Details: [security model](https://opensmartroute.ai/docs/SECURITY.md). * The public web pages load Google Analytics 4 in consent mode: no analytics or advertising cookies until the visitor accepts the banner, IP addresses truncated, nothing on the dashboard. The choice is stored in `localStorage` (`osr-consent`) and can be changed on the privacy page. Besides page views the tag records Core Web Vitals, clicks on outbound links and calls to action, code-snippet copies and the sign-up / sign-in events. Operators of a self-hosted web image leave `NEXT_PUBLIC_GA_MEASUREMENT_ID` empty to ship no tag at all. ## 15. Errors All errors are JSON: `{"detail": "..."}` with a conventional status code and an `X-Request-Id` to quote. | Status | Typical cause | |---|---| | `400` | Invalid `objective`/`constraints`/policy keys, unknown target id in a policy, malformed email, an organization-only call on a personal workspace. | | `401` | Missing, invalid or revoked API key (`WWW-Authenticate: Bearer`). | | `403` | Feature not in plan (`X-Upgrade`), role too low, signup disabled. | | `404` | Unknown target, tenant, model or listing; enterprise-only feature on a community deployment. | | `409` | Duplicate signup, tenant or workspace slug; removing the last owner. | | `422` | Body failed validation (the response lists the offending fields). | | `429` | Rate limit, daily quota or an exhausted workspace/tenant budget (`Retry-After`, `X-Budget-*`). | | `501` | `/v1/chat/completions` on a deployment with no model provider configured. | | `502` | The web app could not reach the API. | | `503` | `/readyz` when a dependency is not ready (the body names the failing check). | ## Related * [REST API reference](https://github.com/isathish/OpenSmartRoute/blob/main/platform/api/openapi.json) - every endpoint, parameter and schema (rendered at `/docs/api`). * [Marketplace](https://opensmartroute.ai/docs/MARKETPLACE.md) - find, install and publish agents, skills, personas, prompts and templates. * [User guide](https://opensmartroute.ai/docs/GUIDE.md) - targets, rules, plans and learning. * [Deploy](https://opensmartroute.ai/docs/deploy.md) - run the same router on your own infrastructure with `osr serve`. --- # Marketplace The marketplace is the part of the [hosted platform](https://opensmartroute.ai/docs/PLATFORM.md) where routing building blocks are shared: **agents, skills, personas, prompts, tools, model profiles and stack templates**, with versions, ratings and a short review lifecycle. Every listing is a plain YAML or JSON document that the OpenSmartRoute SDK, CLI and hosted API understand, so what you install is what you run; there is no marketplace-only format. Browsing, installing, rating and publishing prompts and personas work from the website without writing code. Templates, skills and agents come with copy-paste snippets for the CLI and the SDK. ## What is on it | Kind | What it is | Manifest | |---|---|---| | **Template** (stack) | A complete routing setup - models, rules, weights - in one file. Apply it and you have a working router. | `kind: stack` document ([`opensmartroute.stack`](https://opensmartroute.ai/docs/REFERENCE.md#opensmartroutestack)) | | **Agent** | A multi-step worker the router can hand a whole task to. | Open Capability Manifest ([spec](https://opensmartroute.ai/docs/ocm.md)) | | **Skill** | An Agent-Skills package (`SKILL.md`) that is also a routable target. | OCM, generated from the `SKILL.md` | | **Persona** | A system prompt with a voice, a domain and a complexity band. | OCM with `kind: persona` | | **Tool** | A function or MCP tool with typed parameters. | OCM with `kind: tool` | | **Model profile** (llm) | A model with cost, latency, context window and quality prior. | OCM with `kind: llm` | | **Prompt** | A reusable prompt with `{{variables}}`. | `{"prompt": "...", "variables": [...]}` | Every listing has a **slug** (its address, `/marketplace/`), a **version** (`1.2.3`), a publisher, a licence, tags, a Markdown description, an install count and a star rating. ## Finding and installing 1. Open **Marketplace** in the top navigation. Filter by kind, tag, price or search; sort by popular, top rated, newest or name. 2. Open a listing. The **About** tab is the publisher's description; **Use it** shows ready-made snippets (CLI, Python, HTTP); **Manifest** is the raw document; **Reviews** lists ratings. 3. Click **Install** (free) or **Buy** (paid). You need an account - signing up is free. Free installs complete at once; paid ones go through the checkout and are recorded on your account when the payment completes. 4. Everything you installed is listed under **Dashboard -> Marketplace -> Installed by you**, together with the version you installed and whether a newer one exists. Installing records the listing on your account and returns the manifest and snippets. Nothing is executed on your behalf. ### Using an installed template ```bash osr stack apply registry://support-desk@1.0.0 --registry https:// # build and summarise osr stack apply registry://support-desk --registry https:// --route "Refund for order 42?" osr stack apply registry://support-desk --registry https:// --out stack.yaml # flatten & keep ``` Or reference it from your own stack and let imports do the work: ```yaml osr: "1" kind: stack name: my-desk imports: - registry://support-desk@1.0.0 targets: - id: llm-onprem kind: llm ``` `osr stack validate my-desk.yaml`, `osr stack plan my-desk.yaml --against deployed.yaml` and `osr stack apply my-desk.yaml` are the Terraform-style verbs: check, diff, apply. ### Using an installed agent, skill, persona, tool or model An OCM manifest becomes a target with one call: ```python from opensmartroute import Router from opensmartroute.ocm import target_from_capability manifest = {...} # GET /api/v1/registry//manifest router = Router([target_from_capability(manifest)]) ``` Prompts are text: read `manifest["prompt"]` and fill in `manifest["variables"]`. The HTTP endpoints are public for published listings: ``` GET /api/v1/registry?kind=template&q=support&sort=popular GET /api/v1/registry/{slug} GET /api/v1/registry/{slug}/manifest?version=1.0.0 GET /api/v1/registry/{slug}/reviews ``` ## Ratings and reviews Anyone with an account can rate a listing from 1 to 5 stars and leave a short review. One review per account per listing (writing again replaces the previous one). Reviews from accounts that installed the listing are marked as such. A listing shows its average rating, the number of ratings and a star histogram. ## Publishing **Marketplace -> Publish** walks through four steps: choose the kind, describe it, paste or write the manifest, review and submit. Prompts and personas need no code at all; for templates the wizard offers a starter you can edit. You can also post the same payload to `POST /api/v1/registry`. ### The lifecycle ``` draft -> review -> published -> archived ^ | +--- rejected ``` - **Draft** - only you can see it. Edit anything, including the manifest. - **Submit** - the manifest is validated (OCM schema, stack schema or prompt shape). Free listings are published immediately; paid listings go to **review**. (Operators can require review for everything with `OSR_PLATFORM_REGISTRY_AUTO_PUBLISH=false`.) - **Review** - a platform reviewer approves or rejects with a note. Rejected listings go back to you with the note; fix and submit again. - **Published** - live. Name, summary, description, tags, price and licence stay editable. The manifest is frozen: ship a **new version** instead, with a changelog. Old versions remain downloadable (`?version=`). - **Archived** - hidden from the marketplace; existing installs keep working. You can restore it as a draft. Your listings, their status and any reviewer note are under **Dashboard -> Marketplace -> Published by you**; each one has an edit page with the version form. ### Paid listings Set a price in USD when publishing (Pro and Enterprise plans). Buyers pay through the platform's checkout; the purchase is confirmed by the payment webhook and the listing is then recorded on their account. Free plans can publish a limited number of free listings. ### Writing a good listing - A one-line summary that says what it does and for whom. - A description that covers what it needs (models, keys, data) and how to use it - Markdown, no raw HTML or images. - Tags people would search for (`support`, `pii`, `sql`, `on-prem`). - Test templates locally first: `osr stack validate stack.yaml`. - Skills: keep `SKILL.md` front matter complete (`name`, `description`); `osr skills validate` checks it. ## Built-in content Every platform ships seeded with the repository's own Agent-Skills packages, a starter template, the `examples/stack.yaml` support-desk template, a persona and a couple of prompts, all published by `OpenSmartRoute` and marked verified. Set `OSR_PLATFORM_REGISTRY_SEED=false` to run an empty marketplace. ## Imported catalogue The hosted marketplace also carries the public Agent Skills ecosystem and the official MCP server registry, imported by `platform/api/scripts/harvest_skills.py` (module `osr_platform.harvest`): | Source | What is imported | Listing kind | |---|---|---| | [skills.sh](https://skills.sh) | every `SKILL.md` in each repository on the leaderboard (about 2 400 repositories) | `skill` | | GitHub code search | the long tail of `SKILL.md` files (needs a GitHub token; optional) | `skill` | | [MCP registry](https://registry.modelcontextprotocol.io) | the latest active version of every server, with its remote endpoints and packages | `tool` | Imported listings are published under the upstream owner's name, are not marked verified, and carry `metadata.source` in their manifest (`provider`, `repository`, `path`, `ref`, `url`) so the listing page links back to the original file; the readme starts with the upstream install command (`npx skills add owner/repo --skill name`). Copyright and licence stay with the author (`license` is taken from the frontmatter when present). Domains are inferred from the description with the router's own lexicon; the `skill-md` / `mcp` tags and the source name are added to the author's tags. ```bash # collect into a resumable JSONL cache (one file per source) python -X utf8 platform/api/scripts/harvest_skills.py collect --work-dir .osr-harvest \ --source skills.sh --source mcp-registry [--source github] [--limit N] # publish into a running platform (batches of 200 through the admin import endpoint) ... OSR_PLATFORM_ADMIN_TOKEN=... python -X utf8 platform/api/scripts/harvest_skills.py publish --work-dir .osr-harvest \ --url https://api.example.com # ... or straight into a local platform database python -X utf8 platform/api/scripts/harvest_skills.py publish --work-dir .osr-harvest --data-dir .osr-platform ``` `collect` only fetches what the cache does not have yet, so re-running it picks up new skills; `publish --refresh` overwrites listings that already exist (same version, manifest replaced in place), without it existing slugs are skipped. Slugs are `owner-repo-skill` (or `mcp-`), truncated with a short hash when longer than 63 characters. For a platform whose SQLite file lives on a network share (the Azure Container Apps deployment mounts Azure Files), publish tens of thousands of rows **offline**: build the database locally with `publish --data-dir`, stop the API, replace `platform.sqlite3` on the share, start it again. Sustained bulk writes through the HTTP endpoint over SMB have corrupted the listings table before; the endpoint is meant for incremental updates of a few hundred rows. The hosted marketplace stays current from inside the platform process - no CI job or cron is involved. Set `OSR_PLATFORM_MARKETPLACE_REFRESH_S` (seconds between runs; `604800` = weekly, `0` = off, the Azure deployment defaults to weekly) and the process that owns the marketplace harvests only rows that are new upstream (`OSR_PLATFORM_MARKETPLACE_REFRESH_SOURCES`, default `skills.sh,mcp-registry`; `OSR_PLATFORM_MARKETPLACE_REFRESH_LIMIT` new rows per source per run, default 2000) into `/harvest/` and publishes the ones the catalogue lacks in batches of 100. Set `OSR_PLATFORM_GITHUB_TOKEN` so the GitHub API allows the repository listings. `GET /api/v1/admin/registry/import` reports the last run and the next run time under `refresh`; `POST /api/v1/admin/registry/import/refresh` runs it now. From a machine, `publish --url ... --only-new` does the same: it asks `POST /api/v1/admin/registry/import/check {"slugs": [...]}` which slugs already exist and sends only the rest. ## For operators | Setting | Default | Effect | |---|---|---| | `OSR_PLATFORM_REGISTRY_AUTO_PUBLISH` | `true` | Free listings go live on submit; `false` sends everything to review. | | `OSR_PLATFORM_REGISTRY_SEED` | `true` | Seed built-in skills, templates, personas and prompts at startup. | Moderation uses the admin token (`X-Admin-Token`): ``` GET /api/v1/admin/registry?status=review POST /api/v1/admin/registry/{slug}/approve {"note": "..."} POST /api/v1/admin/registry/{slug}/reject {"note": "why"} POST /api/v1/admin/registry/{slug}/flags {"featured": true, "verified": true} POST /api/v1/admin/registry/{slug}/unpublish {"note": "..."} GET /api/v1/admin/registry/import published listings per source POST /api/v1/admin/registry/import {"items": [...], "refresh": false} (up to 500 harvested rows) POST /api/v1/admin/registry/import/check {"slugs": [...]} -> {"existing": [...]} (up to 2000 slugs) POST /api/v1/admin/registry/import/refresh run the scheduled harvest now (202; 409 while running) ``` Paid purchases need billing to be configured (Stripe keys); without it `POST .../install` on a paid listing answers `402`. ## Security notes - Manifests are validated against the OCM / stack schemas and size-limited; descriptions are rendered without raw HTML, scripts or images and external links carry `rel="nofollow"`. - Installing never runs anything: it returns a document. Review what a template or agent points at (endpoints, tools) before applying it, the same way you would review a Terraform module. - Slugs are stable; a new version never changes the slug, so pin `@version` in production stacks. ## Related * [Platform guide](https://opensmartroute.ai/docs/PLATFORM.md) - authentication, routing, plans, organizations and the dashboard. * [REST API reference](https://github.com/isathish/OpenSmartRoute/blob/main/platform/api/openapi.json) - the *Marketplace* group lists every endpoint (rendered at `/docs/api/marketplace`). * [User guide](https://opensmartroute.ai/docs/GUIDE.md) - stack files, `osr stack` and the Open Capability Manifest. --- # Cost estimates, recommended models and the MCP server Three ways to ask the router "what will this cost and what should I use?" before spending tokens: a quote API and page, a recommended-models digest, and a [Model Context Protocol](https://modelcontextprotocol.io) server that puts all of it inside VS Code, Cursor, Claude, Windsurf or any agent framework. The same pieces exist in the SDK (`opensmartroute.estimate`, `opensmartroute.mcp_server`, `osr estimate`, `osr mcp`) and on the hosted platform (`POST /api/v1/estimate`, `GET /api/v1/models/recommended`, `/mcp`, the `/estimate` page and *Dashboard -> Integrations*). ## 1. Quotes A quote routes the request exactly as `route()` would - signals, policy, strategies, learned quality - and then prices every ranked candidate instead of executing one: - **Tokens.** Input tokens come from the text (word / digit / punctuation / CJK / code heuristic, `estimate_tokens`) plus the history (`estimate_messages_tokens`, 4 tokens of framing per message). Output tokens are the caller's `output_tokens`, else the signals' predicted answer length, else `DEFAULT_OUTPUT_TOKENS` (256); `output_tokens_source` says which. - **Prices.** `usd_per_1k_input` / `usd_per_1k_output` on the target when declared, else the blended `usd_per_1k_tokens`; a fixed `usd_per_call` is added for agents, tools and humans. On the platform a target that resolves to a reference model is priced from the live catalogue (`priced_from: "catalogue"`), so quotes follow vendor list prices. - **Picks.** `recommended` is the router's decision; `best_quality` the highest quality estimate; `cheapest` and `fastest` are chosen among candidates within `quality_tolerance` (0.1) of the best that fit their context window, so a free-but-useless target never wins. `savings_usd` is the gap between the dearest candidate and the recommended one. - **Nothing is executed and the text is never stored.** ```python from opensmartroute.estimate import estimate q = estimate(router, "Summarise this contract in five bullets", output_tokens=300) print(q.input_tokens, q.output_tokens, q.output_tokens_source) # 9 300 caller print(q.recommended.name, q.recommended.cost_usd) # Small model 0.000062 for t in q.targets: # every candidate, ranked print(t.rank, t.target_id, f"${t.cost_usd:.6f}", t.quality_estimate, t.latency_ms, t.fits_context) print(q.cheapest.target_id, q.best_quality.target_id, q.fastest.target_id, q.savings_usd) q.to_dict() # the JSON the API returns ``` `estimate(router, request, *, output_tokens=None, kinds=None, prices=None, quality_tolerance=0.1, decision=None)` accepts a `RouteRequest` or a string, a `kinds` filter and a `prices(target) -> (usd_per_1k_in, usd_per_1k_out) | None` hook for live price feeds; pass `decision=` to quote what an earlier `route()` already ranked. ```bash osr -t targets.yaml estimate "Translate this contract into German" --monthly 100000 osr -t targets.yaml estimate "..." --output-tokens 500 --cost-weight 1.0 --json ``` ### `POST /api/v1/estimate` Same body as `POST /api/v1/route` (`text`, `history`, `objective`, `constraints`, `kinds`) plus `output_tokens` and `monthly_requests`. Without a key the endpoint is rate limited per client (20 per minute, 8 000 characters) and ignores tenant rules; with a key it is metered as `estimate`, applies your plan and tenants and returns `X-Request-Id`. ```bash curl -s https:///api/v1/estimate -H "Authorization: Bearer $OSR_KEY" \ -H "Content-Type: application/json" \ -d '{"text": "Summarise this contract...", "monthly_requests": 50000}' ``` ```json { "request_id": "…", "input_tokens": 9, "output_tokens": 256, "output_tokens_source": "signals", "total_tokens": 265, "confidence": 0.71, "savings_usd": 0.00209, "recommended": {"target_id": "llm-small", "name": "Small model", "vendor": "openai", "model": "openai/gpt-4.1-nano", "input_cost_usd": 0.0000009, "output_cost_usd": 0.0001, "cost_usd": 0.000103, "latency_ms": 400, "quality_estimate": 0.62, "fits_context": true, "priced_from": "catalogue", "rationale": {"semantic": "…", "cost": "…"}}, "cheapest": {"…": "…"}, "best_quality": {"…": "…"}, "fastest": {"…": "…"}, "targets": [{"…": "…"}], "signals": {"complexity": 0.31, "domains": ["legal"], "task_type": "summarise", "contains_pii": false, "…": "…"}, "policy_rejections": {}, "monthly": {"requests": 50000, "recommended_usd": 5.15, "cheapest_usd": 5.15, "best_quality_usd": 109.6, "dearest_usd": 109.6, "savings_usd": 104.45, "tokens": 13250000}, "anonymous": false } ``` The `/estimate` page is the same call with a form: paste a prompt, choose what to optimise for, set the expected answer length and monthly volume, and read the four picks, the signals the router saw and the full candidate table with input / output cost bars. ## 2. Recommended models `GET /api/v1/models/recommended` (no key) quotes a representative prompt per use case - `chat`, `code`, `reasoning`, `summarise`, `extraction`, `pii` - against the live catalogue and returns, for each, the `recommended`, `cheapest`, `best_quality` and `fastest` pick with cost per request, quality estimate, latency, 30-day traffic, the reference model's intelligence index and the reasons (`why`). A `leaderboard` counts how often each target was the router's pick. The digest is recomputed at most once a minute and follows the deployment's targets, prices and learned quality rather than a hand-maintained list. It renders on `/models` ("Recommended per use case") and on `/estimate`. ## 3. MCP server ### What it exposes | Tool | Purpose | Side effects | |---|---|---| | `route` | Pick the best target: decision, confidence, alternatives, signals, ranked candidates with reasons | none | | `estimate` | The quote above for a request | none | | `recommend` | One-paragraph recommendation for a task; `priority` = balanced, cost, quality or speed | none | | `explain` | Human-readable trace: signals, policy filtering, per-strategy breakdown | none | | `list_targets` | The catalogue (optionally one `kind`) with prices, latency, quality prior, boundary | none | | `feedback` | Report an outcome (`request_id`, `target_id`, `success`, `quality`, `cost_usd`, `latency_ms`) | learners update | | `ask` | Route **and** run the chosen target / plan; returns the answer, tokens and cost | executes | | `marketplace_search`, `marketplace_get` | Search listings; fetch one with its manifest and snippets | none | Resources: `osr://targets` (the catalogue as JSON) and `osr://stats` (feedback statistics). The server speaks MCP `2025-06-18` (also accepts `2025-03-26` and `2024-11-05`) over JSON-RPC 2.0: `initialize`, `ping`, `tools/list`, `tools/call`, `resources/list`, `resources/read`, `prompts/list`, `logging/setLevel`; notifications get no reply; unknown methods return `-32601`, unknown tools `-32602`, and tool failures come back as `isError` results, never as protocol errors. `ask` and `marketplace_*` appear only when the server was given an `execute` / `marketplace` hook: `osr serve` and the platform provide both; a bare `MCPServer(router)` exposes the read-only tools and `feedback`. ### Hosted platform `GET /mcp` describes the server (tools, resources, protocol version, connection snippets) without a key. `POST /mcp` takes one JSON-RPC message or a batch with the same `Authorization: Bearer ` (or `X-API-Key`) header as the REST API; notifications return `202`. Every routed call carries the workspace tenant and is metered as `mcp` against the plan's quotas; `feedback` needs the feedback feature and `ask` the plan feature (Pro and above). Quota, plan and routing errors surface as tool errors such as `429: rate limit exceeded`. *Dashboard -> Integrations* renders ready-to-paste configuration for each client with the workspace's key. HTTP transport (the client talks to `https:///mcp` directly): ```jsonc // VS Code: .vscode/mcp.json {"servers": {"opensmartroute": {"type": "http", "url": "https:///mcp", "headers": {"Authorization": "Bearer osr_..."}}}} // Cursor: .cursor/mcp.json {"mcpServers": {"opensmartroute": {"url": "https:///mcp", "headers": {"Authorization": "Bearer osr_..."}}}} // Windsurf: ~/.codeium/windsurf/mcp_config.json {"mcpServers": {"opensmartroute": {"serverUrl": "https:///mcp", "headers": {"Authorization": "Bearer osr_..."}}}} ``` ```bash claude mcp add --transport http opensmartroute https:///mcp --header "Authorization: Bearer osr_..." ``` Stdio bridge for clients that only launch local processes (Claude Desktop and older clients): the CLI forwards each line to the platform with your key. ```json {"mcpServers": {"opensmartroute": {"command": "osr", "args": ["mcp", "--url", "https:///mcp", "--api-key", "osr_..."]}}} ``` ### Self-hosted and local `osr serve` mounts the same server at `POST /mcp` (and `GET /mcp` for the description) next to `/route` and `/v1`, with `ask` backed by the configured providers. Without a service, the CLI runs the server over stdio straight from a catalogue - useful for a laptop IDE setup: ```bash osr -t targets.yaml -r rules.yaml mcp # newline-delimited JSON-RPC on stdin/stdout osr -t targets.yaml mcp --list-tools # what the IDE will see osr mcp --url https:///mcp --api-key osr_... # bridge to a hosted platform ``` ```json {"servers": {"opensmartroute": {"type": "stdio", "command": "osr", "args": ["-t", "targets.yaml", "-r", "rules.yaml", "mcp"]}}} ``` ### In Python ```python from opensmartroute.mcp_server import MCPServer, serve_stdio server = MCPServer(router, execute=my_execute, marketplace=my_marketplace, prices=my_prices) server.tools() # tool specs (tools/list) server.call("estimate", {"text": "…"}) # {"content": [...], "structuredContent": {...}} reply = server.handle({"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "recommend", "arguments": {"task": "…", "priority": "cost"}}}) serve_stdio(server) # block on stdin/stdout ``` `execute(request) -> dict` enables `ask`; `marketplace(action, args)` with `action` in `search` / `get` enables the marketplace tools; `prices` is the quote hook above. `RemoteMCP(url, api_key)` and `bridge_stdio(url, api_key)` are the client side used by `osr mcp --url`. Everything is standard library only; `osr serve` needs the `server` extra (FastAPI). ### Things to ask your assistant - "Estimate what this prompt will cost and which model to use." - "Recommend the cheapest model that can refactor this file." - "Explain how OpenSmartRoute would route this request." - "Search the marketplace for a SQL reporting skill and show me its manifest." - "Route and answer this with the best model, then record that the answer was good." --- # SDK Guide ## Stability & versioning - **SemVer.** Everything exported from `opensmartroute` (top-level) is public API. `opensmartroute.enterprise`, `.security`, `.learning`, `.realtime`, `.math`, `.eval`, `.aio`, `.adapters`, `.config` are public too but may add fields between minor versions. - Modules prefixed `_` and anything not in an `__all__` are private. - The complete list of public modules and names, with one-line summaries, is generated into [REFERENCE.md](https://opensmartroute.ai/docs/REFERENCE.md) from the source (`python scripts/api_reference.py`). - Deprecations are announced one minor version ahead with `DeprecationWarning`. ## Installation matrix `curl -LsSf https://opensmartroute.ai/install.sh | sh` (Linux, macOS) and `irm https://opensmartroute.ai/install.ps1 | iex` (Windows) install the `osr` CLI with the `yaml` and `server` extras in an isolated tool environment (uv, pipx or a private venv). For the library, pick the extras you need: | Extra | Adds | Use when | |---|---|---| | *(none)* | pure stdlib | embedding in any Python 3.10+ service | | `yaml` | PyYAML | loading `targets.yaml` / `rules.yaml` | | `server` | FastAPI, uvicorn, pydantic | running `osr serve` | | `embeddings` | sentence-transformers | semantic `SimilarityStrategy` | | `otel` | opentelemetry-api/sdk | `adapters.OpenTelemetryTelemetry` | | `crypto` | cryptography | AES-GCM encrypted state stores | | `dev` | pytest, ruff, mypy, bandit | contributing | | `all` | every extra above | the installer's `OSR_EXTRAS=all` | ## Connecting real providers `adapters.OpenAICompatClient` is a stdlib HTTP client for any OpenAI-compatible API (OpenAI, Azure OpenAI, vLLM, Ollama, LiteLLM, OpenRouter, Groq, Mistral). It has explicit timeouts, exponential backoff on 408/429/5xx, and maps failures to SDK errors (`ConfigurationError` for 401/403, `TargetUnavailableError` for connectivity/5xx, `ExecutionError` for the rest). The API key is read from `OPENAI_API_KEY` (or `OPENAI_API_KEY_FILE`) and never appears in logs or exceptions. ```python from opensmartroute.adapters import OpenAICompatClient, chat_handler, embedder, judge_fn openai = OpenAICompatClient() # https://api.openai.com/v1 local = OpenAICompatClient("http://localhost:11434/v1", api_key="ollama") # Ollama registry = TargetRegistry([ RouteTarget("gpt-4o-mini", TargetKind.LLM, cost={"usd_per_1k_tokens": 0.00015}, handler=chat_handler(openai, model="gpt-4o-mini")), RouteTarget("llama-local", TargetKind.LLM, cost={"usd_per_1k_tokens": 0.0}, constraints=TargetConstraints(data_boundary="on_prem"), handler=chat_handler(local, model="llama3.1")), ]) router = Router( registry, strategies=[CapabilityStrategy(), SimilarityStrategy(embedder=embedder(openai, model="text-embedding-3-small"))], llm_judge=LLMJudgeStrategy(judge_fn(openai, model="gpt-4o-mini")), escalate_llm_judge_below=0.5, ) decision = router.route(request) result = router.execute(decision, request) # ChatResult(text, tokens, latency_ms); outcome auto-recorded ``` `chat_handler` builds the message list from `request.history`, and uses `request.context["system"]` as the system prompt — which is how a **persona** slot in a `RoutePlan` changes model behaviour without a separate deployment. Optional adapters (lazy-imported): ```python from opensmartroute.adapters import sentence_transformers_embedder, OpenTelemetryTelemetry SimilarityStrategy(embedder=sentence_transformers_embedder("all-MiniLM-L6-v2")) RouterBuilder(reg).with_telemetry(OpenTelemetryTelemetry()) ``` ## Loading configuration ```python from opensmartroute.config import load_targets, load_rules registry = load_targets("targets.yaml") # or .json rules = load_rules("rules.yaml") ``` Both raise `ConfigurationError` with the file, the offending entry and the reason. Only `yaml.safe_load` is used. ## Executing end to end `Router.route()` returns a decision; `Router.run()` (also on `AsyncRouter` and `EnterpriseRouter`) routes **and** executes the resulting plan, returning an `ExecutionResult`. `execute(decision, request)` is still available and returns just the primary handler's raw response. ```python res = router.run("Write SQL for monthly active users by region", task_id="job-17") res.text # best-effort text of the primary response res.response # the raw handler response (ChatResult, HarnessResult, dict, str, ...) res.target_id # primary target res.steps # [ExecutionStep(role, target_id, latency_ms, ok, note), ...] res.system_prompt # what was composed into context["system"] (None if nothing was added) res.effective_request # the RouteRequest the primary handler actually received res.outcomes # one Outcome per participant, already passed to router.learn() ``` What `run()` does, in order: 1. `route(request, plan=True)` builds the plan: primary + optional `persona` and `skill` slots. A slot is only filled when its best candidate's quality estimate is at least `Router(slot_quality_floor=0.5)` and (for `run`) its confidence is at least `min_slot_confidence`. 2. **Skill pre-processing.** If the `skill` slot's target has a `handler`, it is called with the request and may return - a `RouteRequest` — replaces the request the primary sees (rewrite / enrich); - a `str` — attached as `context["skill_output"]` and appended to the system prompt; - `None` — the skill contributes instructions only. Exceptions are recorded as a failed step (and a failed `Outcome`) but do not abort the run. 3. **Prompt composition.** `context["system"]` is set to: any existing `system` + `# Persona: ` + `RouteTarget.instructions` of the persona + `# Skill: ` + skill instructions + the primary's own `instructions` (for `llm`/`agent` targets) + `# Skill output`. `chat_handler` and the harness adapters read this key. 4. **Primary call.** `target.handler(request, **kw)`. Missing handler → `ExecutionError`. In the sync path a coroutine result raises `ExecutionError("use aexecute()")`; `AsyncRouter.run()` awaits both skill and primary handlers. 5. **Outcomes.** Success is read from `response.success` / `response["success"]` (default `True`); cost from `response.cost_usd` or `total_tokens × unit_cost`; quality from `response.quality`. One `Outcome` is created for the primary (`role=None`) and one per slot (`role="persona"|"skill"`), all carrying `task_id` (argument or `request.context["task_id"]`) so delayed feedback can be joined later. Each is passed to `learn`; on a handler exception, failure outcomes are recorded and the exception re-raised. ### Targets carry `instructions` ```yaml - id: persona-legal-counsel kind: persona primary: false instructions: | You are cautious in-house legal counsel. Cite the governing clause... - id: skill-sql kind: skill instructions: "Always qualify table names with the schema; never SELECT *." ``` `instructions` is the persona's system prompt, a skill's body (SKILL.md style) or an agent's standing instructions. It is disclosed to the model only when the target is part of the plan. ### Agent harnesses An agent harness is any runtime that takes a task and drives its own loop (coding agent, research agent, OpenHands, SWE-agent, Claude Code, LangGraph graph, ...). Adapters live in `opensmartroute.adapters` and all produce a `HarnessResult(text, success, steps, prompt_tokens, completion_tokens, cost_usd, latency_ms, quality, artifacts, raw)`: ```python from opensmartroute.adapters import CallableHarness, HTTPHarness, SubprocessHarness, harness_handler # in-process: fn(task, context, history) -> str | dict | HarnessResult registry.get("research-agent").handler = harness_handler(CallableHarness(my_graph.invoke)) # HTTP: POST {"task", "context", "history"} -> JSON {"text"|"output", "success", "usage", "cost_usd"} registry.get("support-agent").handler = harness_handler( HTTPHarness("https://agents.internal/support/run", api_key_env="AGENT_TOKEN", timeout_s=300)) # CLI: task on stdin, answer on stdout (or JSON with json_output=True); system prompt via a flag registry.get("coding-agent").handler = harness_handler( SubprocessHarness(["my-agent", "--repo", "."], system_flag="--system", timeout_s=900)) ``` Non-zero exit codes and `"success": false` become failed outcomes; HTTP 408/429/5xx, timeouts and unreachable hosts raise `TargetUnavailableError` (so `HealthRegistry` breakers trip); other HTTP errors raise `ExecutionError`; a missing binary or non-http URL raises `ConfigurationError`. Implement the `AgentHarness` protocol (`run(task, *, context, history) -> HarnessResult`) for anything else. ### Skills from `SKILL.md` Folders following the [Agent Skills](https://agentskills.io/specification) layout load directly: ```python from opensmartroute.adapters import load_skill, load_skills, skill_from_markdown for skill in load_skills("skills/"): # skills//SKILL.md, name must match the folder registry.add(skill) ``` Frontmatter `name`/`description` become `id`/`examples`; the Markdown body becomes `instructions` (progressive disclosure: only the description is used for routing, the body is shown to the model once selected). Routing metadata can be added under `metadata:` — `osr-domains`, `osr-actions`, `osr-languages`, `osr-tags`, `osr-quality-prior`, `osr-primary` (`"false"` for slot-only skills), `osr-latency-ms`. `license`, `compatibility`, `allowed-tools` and other metadata are preserved in `RouteTarget.metadata`. Give the returned target a `handler` to make it an executable pre-processor or primary skill. ## Learning from outcomes Every learner (bandits, IRT, preference, LinUCB, Markov, task table, judge calibration) consumes the same `Outcome`: ```python from opensmartroute import Outcome d = app.route(req) resp = call_provider(d.target, req) app.learn(Outcome(request_id=req.request_id, target_id=d.target.id, success=True, quality=grade(resp), cost_usd=resp.cost, latency_ms=resp.ms)) ``` - **Plans**: `Router.run()` records one `Outcome` per participant with `role="skill" | "persona" | "primary"`; learners keep a target's record as a skill separate from its record as the primary answerer (`signals.extra["role"]`). - **Cascades**: `Cascade.run(request, d.trace.ranked)` returns a `CascadeResult`; feed every step to the learners with `app.learn_cascade(request, result)` (rejected steps are failures with the gate score as quality, the accepted step a success). `result.total_cost_usd` is the spend across steps. - **Delayed / task-level rewards**: set `Outcome.task_id` (and `step`) so a reward observed at the end of a multi-step task credits every call in the trajectory (`Router(task_credit=...)`). - **Offline warm start**: `learning.warm_start_from_matrix(rows, strategies, weight=0.5)` replays a full-information reward matrix `(prompt, {target_id: reward})` from an offline evaluation into every learner before serving traffic; `learning.warm_start()` seeds a brand-new target from its neighbours. - **Replicas**: `learning.merge_learners(local, remote)` folds another replica's learners into yours (sufficient statistics only, no prompts); `AutoLearner.refresh()` re-loads a writer's snapshot from a shared `StateStore`. Corrupt state is quarantined on load (`AutoLearner.quarantined`) instead of raising. - **Objective**: `Objective(quality, cost, latency, quality_floor, energy, carbon)`; `energy` and `carbon` weigh `RouteTarget.unit_energy` (`cost["wh_per_1k_tokens"]`) and `unit_carbon` (`cost["gco2_per_1k_tokens"]` or energy x `cost["gco2_per_wh"]`). - **Cold-start exploration**: a target declared with only `id`, `kind` and `cost` is never ranked first and so never earns an outcome. `Router(explore_rate=0.1, explore_min_samples=200)` (or `RoutingSettings.explore_rate` / `explore_min_samples`) serves the least-seen viable candidate with probability `explore_rate` until it has that many outcomes; `decision.propensities` includes the exploration mass (so off-policy estimates stay unbiased) and `signals.extra["explored"]` marks the request. Off by default. - **Roadmap exit criteria**: `opensmartroute.eval.criteria` turns each offline-measurable exit criterion in [ROADMAP.md](https://opensmartroute.ai/docs/ROADMAP.md) into a simulation of the real router (`cold_start_ratio()`, `effort_token_savings()`, `multi_round_vs_best_single()`, `match_at_1_at_scale()`, `conformal_coverage()`, `knapsack_never_exceeds_cap()`, `ope_within_live_ci()`) and `opensmartroute.eval.agentic` adds the tau-bench-style one (`AgentTask`, `load_agentic_tasks()`, `synthetic_agentic_tasks()`, `task_routing_frontier()`: per-step routing through `ProgressRouter` against every always-use-agent-X policy on the accuracy-latency frontier). `python scripts/exit_criteria.py [--full]` runs them all; on your own data, `osr eval DATASET --multi-round` runs the multi-round comparison on scored rows, `osr eval DATASET --effort` the effort-routing token comparison on rows that carry per-target `scores` and `tokens` (`EvalRow.tokens`), and `osr eval TASKS.jsonl --agentic` the task-routing frontier on tasks with per-agent per-step `success` and `latency_ms` tables. - **Contrastive and policy-gradient learners**: `learning.ContrastiveRouter().fit(rows, targets)` trains on scored `EvalRow`s against each row's acceptable set (`objective="distilled"` uses reward-softened labels); wrap it in `learning.ContrastiveStrategy(model)` to keep learning online. `learning.PolicyGradientStrategy()` learns a routing policy directly from `decision_reward()` (quality minus weighted cost and latency) with no quality predictor in between; `learning.decision_regret(rows, targets, choose, predict)` compares decision regret with prediction error on the same rows. - **Headroom before you learn**: `eval.headroom.routing_headroom(rows, targets)` tells you how much of the oracle gap is measurable on your data; `target_diversity`, `min_catalogue`, `scaling_curve` and `learnability_by_difficulty` answer whether, and where, a learned router can beat the best single target. - **Other target kinds**: `strategies.SemanticCache.target()` turns a semantic cache into a `DESTINATION` target scored by `SemanticCacheStrategy`; `strategies.AnnotatorPool` + `HumanRoutingStrategy` route among `HUMAN` annotators with Dawid-Skene skill estimates and `select_quorum()`; `strategies.MemoryRouter` routes *what to remember* across memory tiers under a token budget; `strategies.ModalityStrategy` / `ModalityEscalation` handle text-first escalation to multimodal targets; `strategies.SpeculativeCascade` runs draft and strong models in parallel when the learned acceptance rate makes that cheaper. ## Error handling All exceptions derive from `OpenSmartRouteError` and carry `.code` and `.details`: ```python from opensmartroute import OpenSmartRouteError, NoRouteError, SecurityError try: d = app.route(req) except NoRouteError as e: # e.details["rejections"] -> {target_id: reason} fallback() except SecurityError as e: reject_request(e.to_dict()) except OpenSmartRouteError: log.exception("routing failed") ``` The SDK never raises bare `Exception`/`RuntimeError`; strategy failures (e.g. a broken LLM judge) degrade to neutral scores rather than failing the route. ## Threading & async - `Router` and `EnterpriseRouter` are safe to share across threads. `Router.lock` (a re-entrant lock) guards strategy scoring, every learner update (`learn`, `learn_correction`, `credit_task`), task pins, the request memory, retrieval narrowing and the exploration RNG, so a strategy's `score` never observes a half-applied `update`. The optional LLM judge is scored *outside* the lock (network I/O) - put it behind `escalate_llm_judge_below` so it runs rarely, and set `TimeoutMiddleware`. - `AutoLearner` takes the same lock (`EnterpriseRouter` / `RouterBuilder` pass `router.lock`; pass it yourself when wiring by hand) so `learn`, `reset_target`, `save`, `load` and `refresh` serialise against routing. `save()` deep-copies a sequence-numbered snapshot under the lock and writes it outside; a separate I/O lock orders writers and readers so a slow save never overwrites a newer snapshot and two saves never race on the state files. - Per-module state elsewhere has its own lock: `SemanticCache`, `MemoryRouter`, `AnnotatorPool`, `ModalityEscalation` (posterior) and `SpeculativeCascade` (acceptance posteriors, history, failure counters). Embedding / hashing runs outside those locks. - `SpeculativeCascade` runs `concurrency` speculative requests at once (pool of `2 * concurrency` workers); `draft_timeout_s` abandons a slow draft and serves the strong answer; a draft that raises escalates instead of failing (`details["draft_error"]`); a strong future that is no longer needed is cancelled or its late failure swallowed. `stats()` reports runs, modes, failure counts and per-bucket acceptance. - `AsyncRouter` wraps any router for `await`-based apps and awaits coroutine handlers; `route` / `learn` run in the default executor under the same lock. - Routing has no I/O of its own. ### Multi-round execution `ProgressRouter.run_step` routes through `router.route` (middleware and telemetry apply) and executes with `execution.execute`; a handler that raises is recorded as a failed step for its target *before* the exception propagates, so the next step already denies it after `failures_before_switch`. Step cost falls back to `target.unit_cost x estimated tokens` when the handler reports none, so `budget_usd` is tracked even for plain-string handlers. `arun_step` is the coroutine twin. `MultiRoundExecutor` (route -> execute -> judge -> refine) learns **once per round** with the judged quality (no provisional learn + correction double count); a round whose handler raised becomes a `Round` with `error` set, learns a failure for that target and the loop continues on a different target. `stop_reason` is one of `accepted`, `budget`, `deadline` (`deadline_ms`) or `max_rounds`; `return_best=True` (default) returns the highest-quality round when nothing reached the threshold; `credit_task=True` delivers the final quality as the task-level reward through `router.credit_task`, which re-credits every step (learners then see one judged update per round plus one credited update). `arun` awaits coroutine handlers and an `async` judge. ## Type checking The package ships `py.typed`; `mypy --strict`-friendly signatures on the public surface. ## Configuration ```yaml # targets.yaml targets: - id: llm-frontier kind: llm capabilities: { domains: [math, coding], min_complexity: 0.5, supports_tools: true } constraints: { data_boundary: public, pii_allowed: false, regions: [us, eu] } cost: { usd_per_1k_tokens: 0.015 } latency_ms: 2500 quality_prior: 0.93 examples: ["Prove that sqrt(2) is irrational."] ``` ```yaml # rules.yaml (Arch-Router-style preferences) rules: - name: escalation-intent when: { actions: [escalate] } prefer: [human-escalation] pin: true # restrict candidates, not just boost ``` Secrets are **never** in YAML. Handlers read them at call time: ```python from opensmartroute.security import load_secret api_key = load_secret("OPENAI_API_KEY") # env var or OPENAI_API_KEY_FILE ``` ## Extending ### Decorator SDK Every routing component - targets, strategies, signal extractors, policy rules, middleware and telemetry sinks - can be declared with a decorator and assembled by a `ComponentRegistry`. The decorators are transparent (they return the decorated object unchanged) and the registry stores *blueprints* (factories), so one declaration can back many independent routers. ```python import opensmartroute as osr @osr.tool("weather", domains=["travel"], actions=["lookup"], cost=0.0, examples=["weather in paris"]) def weather(request, **kw): """Current weather for a city.""" # first paragraph -> target.description return lookup(request.text) @osr.agent("planner", domains=["travel"], min_complexity=0.3, quality_prior=0.9) def planner(request, **kw): """Plan a multi-day trip.""" return plan(request.text) @osr.strategy(weight=0.8) # fn(request, signals, candidates) -> {id: score} def geo_affinity(request, signals, candidates): region = request.constraints.region return {t.id: 1.0 if region in t.constraints.regions else 0.4 for t in candidates} @osr.strategy # or a Strategy subclass; name inferred -> "recency" class RecencyStrategy(osr.Strategy): def score(self, request, signals, candidates): ... @osr.signal(order=-1) # runs before the built-in extractors def urgency(request, signals): return {"urgent": "asap" in request.text.lower()} # unknown keys land in signals.extra @osr.policy_rule # fn(target, request, signals) -> reason | None def business_hours_only(target, request, signals): if target.kind is osr.TargetKind.HUMAN and request.context.get("after_hours"): return "human queues are closed" return None @osr.middleware # fn(request, next_) -> RouteDecision def tag_tenant(request, next_): request.context.setdefault("tenant", "public") return next_(request) router = osr.components.router() # Router: targets + strategies + signals + rules app = osr.components.builder().with_auto_learning().build() # EnterpriseRouter variant ``` | Decorator | Accepts | Materialised as | |---|---|---| | `@target(id, kind, ...)`, `@tool`, `@skill`, `@agent` | handler `fn(request, **kw)` | `RouteTarget(handler=fn)` | | `@strategy(name=, weight=, order=)` | `Strategy` subclass or scorer function | `Strategy` / `FunctionStrategy` | | `@signal(name=, order=)` | `SignalExtractor` subclass or `fn(request, signals)` | `SignalExtractor` / `FunctionSignal` | | `@policy_rule(name=, order=)` | `fn(target, request, signals) -> str \| None` or `Policy` subclass | rule appended to `Policy.with_rules` | | `@middleware(name=, order=)` | `Middleware` subclass or `fn(request, next_)` | `Middleware` / `FunctionMiddleware` | | `@telemetry(name=, order=)` | `Telemetry` subclass | `Telemetry` | - `osr.components` is the process-wide registry the top-level decorators bind to. Create your own `ComponentRegistry()` for isolation (tests, multi-tenant apps) - its methods are the same decorators. - Duplicate names raise `ConfigurationError` (`register(..., replace=True)` to override); `unregister`, `clear`, `merge` and `add(*instances)` manage the catalogue. - `RouterBuilder.with_components(registry, targets=, strategies=, signals=, policy=, middleware=, telemetry=)` wires a registry into an existing builder selectively. - Plugins: `registry.include("pkg.module")` imports a module so its decorators run (`"pkg.module:setup"` calls `setup(registry)` afterwards); `registry.discover()` loads every installed distribution advertising an `opensmartroute.plugins` entry point. ### Subclassing ```python from opensmartroute import Strategy, StrategyScore class GeoAffinity(Strategy): name = "geo" def score(self, request, signals, candidates): region = request.context.get("region") return {t.id: StrategyScore(1.0 if region in t.constraints.regions else 0.5, f"region={region}") for t in candidates} app = RouterBuilder(reg).with_defaults().with_strategy(GeoAffinity(), weight=0.8).build() ``` Custom `Policy`, `SignalExtractor`, `Middleware`, `Telemetry`, `StateStore`, `AuditSink` follow the same pattern — subclass, implement, pass to the builder (or register with a decorator). ## Settings Every tunable lives in one immutable `Settings` object, grouped by consumer (`routing`, `policy`, `rules`, `capability`, `bandit`, `weights`, `slm`, `server`, `observability`). Components resolve the process-wide settings unless given an explicit instance. ```python import opensmartroute as osr from opensmartroute.settings import RoutingSettings, WeightSettings osr.configure(routing=RoutingSettings(narrow_above=200), weights=WeightSettings(rules=3.0)) router = osr.Router(reg) # uses the configured settings isolated = osr.Router(reg, settings=osr.Settings()) # library defaults, unaffected print(osr.get_settings().env_keys()) # every honoured OSR_* variable ``` Environment overlay: `OSR__` (e.g. `OSR_ROUTING_SOFTMAX_TEMPERATURE=0.2`, `OSR_WEIGHTS_CAPABILITY=1.5`, `OSR_POLICY_DATA_BOUNDARIES=public,internal,confidential`). Values are parsed against the default's type; a bad value raises `ConfigurationError` with `details={"env": key}`. `osr.configure()` with no arguments re-reads the environment. `osr settings [--json]` prints the effective values, their `OSR_*` keys and which ones the environment overrides. ## Naming conventions `opensmartroute.branding` is the single source of truth for every brand-derived name: | Convention | Value | Helper | |---|---|---| | Package / import | `opensmartroute` | `branding.PACKAGE` | | CLI | `osr` | `branding.CLI` | | Environment variables | `OSR__` | `branding.env_key("routing", "narrow_to")` | | Error codes | `OSR_` (`OSR_NO_ROUTE`, `OSR_CONFIG`, ...) | `branding.error_code("no_route")` | | Metadata / frontmatter keys | `osr-` (`osr-quality-prior`) | `branding.metadata_key("quality_prior")` | | State directory | `.osr-state` | `branding.STATE_DIR` | | Loggers | `opensmartroute[.component]` | `branding.logger("enterprise")` | | HTTP `User-Agent` | `opensmartroute/` | `branding.user_agent()` | | Plugin entry-point group | `opensmartroute.plugins` | `branding.ENTRY_POINT_GROUP` | | Config directory | `~/.config/opensmartroute` (`%APPDATA%\opensmartroute`), `OSR_CONFIG_DIR` | `credentials.config_dir()` | | Platform URL / key | `OSR_API_URL`, `OSR_API_KEY` | `branding.platform_url()`, `branding.API_KEY_ENV` | | Access tokens | `osr_live_` (platform key), `osr_local_` (self-hosted server) | `credentials.token_kind()`, `generate_token()` | Strategy names are snake_case nouns (`capability`, `llm_judge`); decorator-registered classes drop the `Strategy`/`Signal`/`Middleware`/`Telemetry` suffix automatically (`GeoAffinityStrategy` -> `geo_affinity`). ## HTTP API (`osr serve`) | Method | Path | Body / response | |---|---|---| | POST | `/route` | `{text, context, objective, constraints, kinds, top_k, plan}` → `RouteDecision.to_dict()` | | POST | `/feedback` | `Outcome` fields → `{status}` | | GET | `/targets` | catalogue | | GET | `/stats` | feedback aggregates (+ `health` snapshot when the router has one) | | GET | `/healthz` | liveness | | GET | `/whoami` | `{service, edition: "self-hosted", version, auth_required, authenticated, targets}` - what `osr whoami` shows | | GET | `/v1/models` | OpenAI-compatible model list (`auto` + every primary target) | | POST | `/v1/chat/completions` | OpenAI-compatible completion; `model: "auto"` routes, a target id pins | `create_app(router)` takes a plain `Router` or an `EnterpriseRouter` (middleware, health, telemetry and audit then apply to every request). Errors map to HTTP statuses: `SecurityError` / `ValidationError` → 400, `NoRouteError` → 422, `TargetUnavailableError` → 503, `ExecutionError` → 502, unknown pinned model → 404. `create_app(router, auth_tokens=[...])` (CLI: `serve --token`, `--generate-token`, or `OSR_SERVER_AUTH_TOKENS=a,b`) turns on access control: every path except `/healthz`, `/readyz`, `/metrics`, `/whoami` and the OpenAPI documents needs `Authorization: Bearer ` or `X-API-Key: `, compared in constant time; anything else is `401` with `WWW-Authenticate: Bearer`. `ServerSettings.require_auth` (`OSR_SERVER_REQUIRE_AUTH=1`, `serve --require-auth`) refuses to start without a token. Tokens are opaque strings; `osr token generate` mints `osr_local_...` ones and `osr login --url http://host:8000 --token ...` stores one for the CLI. ## CLI ``` osr login [--url URL] [--token T | --with-token] [--no-browser] [--profile P] [--json] osr whoami [--json] | osr logout [--all] | osr token generate|create|list|revoke osr -t targets.yaml -r rules.yaml route "text" [--plan] [--json] [--cost-weight 0.3] osr -t targets.yaml -r rules.yaml eval dataset.jsonl [--frontier] [--multi-round --threshold 0.8 --max-rounds 3] osr -t targets.yaml eval rows.jsonl --effort # rows with per-target scores + tokens vs always-think osr -t targets.yaml eval tasks.jsonl --agentic [--retries 1] # per-step task routing vs best single agent osr -t targets.yaml targets | stats | serve [--host --port] [--token T ...] [--generate-token] [--require-auth] osr -t targets.yaml --skills .claude/skills route "text" --plan # add SKILL.md packages as slot targets osr skills [ROOT] [--json] # validate + list SKILL.md packages osr settings [--json] ``` `osr login` signs in to the hosted platform with the OAuth 2.0 device authorization grant (the browser opens `/platform/cli/authorize`, you approve the code, the platform mints a workspace API key) or stores a pasted token; `opensmartroute.credentials` (`CredentialStore`, `device_login`, `whoami`, `PlatformClient`) is the library behind it and works with any injected transport. ## Interop roadmap See [ROADMAP.md](https://opensmartroute.ai/docs/ROADMAP.md) — v0.7 covers MCP tool catalogues, A2A agent cards, the OpenAI-compatible proxy mode, LangGraph / Microsoft Agent Framework nodes, and Redis/Postgres `StateStore` adapters. --- # Enterprise Architecture ## Design principles | Principle | How it shows up | |---|---| | **Hexagonal / ports and adapters** | `StateStore`, `Telemetry`, `AuditSink`, `Policy`, `Strategy`, `SignalExtractor` are ports. In-memory and file adapters ship in `enterprise`; `enterprise.stores` adds Redis, SQL, versioned, encrypted, batched and namespaced stores; `adapters` provides the OpenAI-compatible client, agent harnesses, MCP / A2A / framework glue, sentence-transformers and OpenTelemetry. | | **Separation of concerns** | Signals (what is asked), Policy (what is allowed), Strategies (what is good), Ensemble (decide), Execution, Learning. Each stage is a module with one job. | | **Fail fast** | `RouterBuilder.build()` validates configuration (no targets, no strategies, duplicate names) before serving traffic. | | **Fail safe** | Hard constraints are never traded off; if nothing is admissible you get `NoRouteError` with the rejection reasons, not a silent bad route. | | **Explainability by default** | Every decision carries a `RouteTrace`; every strategy returns a rationale. | | **Inductive learning** | Every strategy works for a brand-new target from its declared capabilities; learners start silent (confidence 0) and gain weight with evidence. | | **Zero-dependency core** | Pure stdlib. Deterministic given a seed. Sub-millisecond per decision. | | **Twelve-factor** | Config via YAML / env (`OSR__` overrides every tunable in `opensmartroute.settings`), secrets via env or `*_FILE`, stateless router plus pluggable state store. | ## Design patterns used - **Strategy**: `Strategy` subclasses are interchangeable scorers. - **Chain of responsibility**: `Middleware` (guard, tenant, fair share, cache, timeout, shadow, router). - **Builder**: `RouterBuilder` fluent assembly with validation. - **Observer**: `Telemetry.on_decision / on_outcome / on_error`. - **Registry**: `TargetRegistry`, `HealthRegistry`. - **Circuit breaker / bulkhead / rate limiter**: `realtime`. - **Template method**: `Policy.check` with `HealthPolicy` decorating the base policy. - **Decorator**: `HealthPolicy(inner)`, `CostAwareBandit(inner)`, every `*StateStore(inner)` wrapper. - **Facade**: `EnterpriseRouter`, `AsyncRouter`. - **Memento**: `state()/load()` on every learner; `AutoLearner` atomic snapshots. ## Component view ```mermaid flowchart TB subgraph Edge API[HTTP / gRPC / SDK call] end subgraph Middleware chain G[GuardMiddleware
size, gadget, redaction] T[TenantMiddleware
defaults, limits] F[FairShareMiddleware
dominant-resource fairness] C[CacheMiddleware
LRU + TTL] TO[TimeoutMiddleware] SH[ShadowMiddleware
shadow / A-B + SPRT] end subgraph Core router S[Signals] P[Policy + HealthPolicy] RT[Retriever
BM25 + dense, RRF] ST[Strategies
rules, capability, similarity, task table, irt, preference, linucb, markov, health, queue, bandit] E[Ensemble + Objective + calibration] D[Decision + Trace + Plan + candidate set] end subgraph Learning loop O[Outcome] AL[AutoLearner
fan-out, drift, persist] H[HealthRegistry
breaker, rate, budget] IF[InflightTracker] end subgraph Ports SS[(StateStore
memory, file, Redis, SQL
versioned, encrypted, batched)] TEL[Telemetry] AU[AuditSink] FB[(FeedbackStore)] end API --> G --> T --> F --> C --> TO --> SH --> S --> P --> RT --> ST --> E --> D D --> TEL D --> AU O --> AL --> ST O --> H --> P IF --> ST AL --> SS O --> FB ``` ## Builder surface | Method | Wires | |---|---| | `with_rules(rules)` | `RulesStrategy` (pin / prefer / avoid) evaluated first | | `with_defaults()` | `CapabilityStrategy`, `SimilarityStrategy`, `BanditStrategy` | | `with_strategy(s, weight)` | any `Strategy`, optional ensemble weight | | `with_auto_learning(state_dir)` | `IRTStrategy`, `PreferenceStrategy`, `LinUCBStrategy`, `MarkovStrategy` behind an `AutoLearner` with drift detection and atomic persistence | | `with_state_store(store, save_every)` | persist learner state through any `StateStore` instead of files | | `with_health(latency_slo_ms)` | `HealthRegistry`, `HealthPolicy`, `HealthStrategy` | | `with_queue_awareness(slo_ms)` | `InflightTracker` + `QueueAwareStrategy`; `EnterpriseRouter.execute()` brackets each call with `acquire` / `release` | | `with_calibration(conformal_alpha)` | `TemperatureScaler` for confidence, `ConformalCalibrator` for `RouteDecision.candidate_set`; `router.calibrate()` replays remembered outcomes | | `with_retrieval(narrow_above, narrow_to, skill_set_size)` | `Retriever` (BM25 + dense, reciprocal-rank fusion) for large catalogues; submodular skill-set selection | | `with_fair_share(weights, ...)` | `FairShareMiddleware` | | `with_shadow(candidate, mode)` | `ShadowMiddleware` in `shadow` (log only) or `ab` (SPRT-judged) mode; `EnterpriseRouter.learn()` feeds outcomes to it | | `with_middleware(...)`, `with_telemetry(...)`, `with_audit(sink)`, `with_feedback(store)` | chain, observers, audit trail, outcome log | | `with_llm_judge(judge, escalate_below)`, `with_objective(obj)`, `with_policy(policy)`, `with_router_options(**kw)` | judge escalation, default objective, custom policy, any other `Router` argument | `build()` validates the configuration (no targets, no strategies, duplicate strategy names) and returns an `EnterpriseRouter` with `route()`, `execute()`, `run()`, `learn()` and `health_snapshot()`. ## State stores All stores implement the three-method `StateStore` port (`get` / `put` / `delete` of JSON dicts) and compose as decorators. External clients are duck-typed, so the core imports no driver. | Store | Backend | Notes | |---|---|---| | `InMemoryStateStore` | process memory | tests, single replica | | `FileStateStore` | one JSON file per key | keys hashed (no path traversal), atomic `replace` | | `RedisStateStore(client, prefix, ttl_s)` | any client with `get` / `set(ex=)` / `delete` (redis-py, valkey, fakeredis) | shared state across replicas | | `SQLStateStore(connection, table)` | any DB-API 2.0 connection (psycopg, sqlite3, pg8000, mysql-connector) | dialect detected; `ON CONFLICT` / `ON DUPLICATE KEY` upserts; table name validated | | `VersionedStateStore(inner, current, migrations)` | wrapper | stamps `{"_schema": n}`, runs forward migrations on read, refuses newer state | | `EncryptedStateStore(inner, key \| key_env)` | wrapper (`crypto` extra) | AES-256-GCM, per-record nonce, key id in the envelope for rotation, plaintext records still readable | | `BatchedStateStore(inner, interval_s, max_pending)` | wrapper | write-behind; reads see pending writes; `flush()` on shutdown or as a context manager | | `NamespacedStateStore(inner, namespace)` | wrapper | key prefix for several routers or tenants on one backend | ```python import redis from opensmartroute.enterprise import RouterBuilder from opensmartroute.enterprise.stores import BatchedStateStore, EncryptedStateStore, RedisStateStore, VersionedStateStore store = BatchedStateStore( VersionedStateStore( EncryptedStateStore(RedisStateStore(redis.Redis.from_url(url), prefix="osr:prod:"), key_env="OSR_STATE_KEY"), current=1, ), interval_s=2.0, ) app = RouterBuilder(registry).with_defaults().with_auto_learning().with_state_store(store).build() ``` ## Operational controls - **Shadow and A/B** (`ShadowMiddleware`, `ABTest`, `SPRT`): a candidate router runs beside production. In shadow mode only agreement rate and cost delta are logged. In A/B mode a deterministic hash of the request id assigns a traffic share to the candidate and outcomes are compared with Wald's sequential probability ratio test, so a bad candidate is stopped early and a good one is promoted with a controlled error rate. `on_verdict` fires once with the summary. - **Tenant fairness** (`FairShareMiddleware`): dominant-resource fairness across tenants over a sliding window; tenants above their fair share are steered to cheaper targets (cost weight boost) and, above the hard factor, capped out of premium targets. Nobody is starved. - **Queue-aware latency** (`InflightTracker`, `QueueAwareStrategy`): live in-flight counts and observed service times feed Erlang-C (when `metadata.concurrency` is set) or Kingman G/G/1 wait estimates, so the latency in the utility is the current time-to-first-token, not the catalogue number. ## Deployment topologies 1. **Library**: import `Router` / `EnterpriseRouter` in-process. Lowest latency; state on local disk. 2. **Sidecar**: the container image (`ghcr.io/isathish/opensmartroute:`, built from `deploy/Dockerfile`) next to each app; shared state via a `StateStore` adapter. 3. **Central control plane**: the Helm chart in `deploy/helm/opensmartroute` behind a gateway (LiteLLM, Envoy, aisix); the router returns a decision, the gateway executes. Scale horizontally; state in Redis / SQL. See [deploy/README.md](https://opensmartroute.ai/docs/deploy.md). ### Scalability notes - Routing is CPU-bound and O(|targets| x |strategies|) with tiny constants; 10k decisions/s per core is typical for <= 50 targets. Above `narrow_above` admissible targets the retriever keeps only `narrow_to` candidates, so cost stays flat for catalogues with thousands of tools. - All in-memory structures are bounded (LRU caches, capped lists, session maps cleared past a limit). - Learner updates are O(1) (bandits, IRT, BT) or O(d^2) with d ~ 21 (LinUCB, Sherman-Morrison); value iteration is bounded to 30 sweeps over a small state space. - Multi-replica consistency: learners are *eventually consistent*. Two supported patterns: 1. **Writer / readers** - one replica learns and persists through a shared `StateStore`; the others call `AutoLearner.refresh()` on a timer and adopt the snapshot wholesale (no double counting). 2. **Independent replicas** - every replica learns from the outcomes it sees and snapshots are folded together with `learning.merge_learners(local, remote)`; every learner merges sufficient statistics (`merge()` on `ThompsonBeta`, `LinUCB`, `IRTModel`, `BradleyTerry`, `MarkovChain` / `RoutingMDP`, `BanditStrategy`, `TaskTableStrategy`). Merge deltas, not the same snapshot twice. - Corrupt or incompatible persisted state is quarantined on load (`*.corrupt-` file or `quarantine/...` store key) and listed in `AutoLearner.quarantined`; the learner starts blank and routing continues. ## Reliability - Circuit breaker per target; half-open probes. - Token-bucket rate limits and rolling budgets per target (and per tenant via `TenantMiddleware`). The policy only *checks* availability for each candidate; one token is consumed for the target that is actually chosen, so evaluating 16 candidates does not drain 16 tokens. - `HealthStrategy` demotes slow/unreliable targets softly; `HealthPolicy` excludes them hard. Each target keeps a rolling window of its last 256 latencies (`LatencyWindow`; p50 / p90 / p99 in `health_snapshot()` and on the platform's Health page); once five calls are in the window the strategy judges the **p90**, so a target with a good mean and a bad tail is penalised. The SLO is the request's hard `max_latency_ms`, else its soft `preferred_max_latency_ms` (penalise, never exclude), else `with_health(latency_slo_ms=...)`. - Drift detection flags targets whose quality degrades; wire `AutoLearner.drifted` to alerts. - Atomic state writes (`os.replace`): a crash never leaves a half-written model. - Per-task resource limits (`security.limits.ResourceLimitMiddleware`) cap steps, tool calls, depth, tokens, cost and wall-clock for agentic plans. ## Observability - Tracing: every stage of every request (signals, policy, rank, plan, execute, learn, health, cache, guard, shadow, fair share, autopilot) emits spans and events into the `Tracer` - `RouterBuilder.with_tracing(...)`, `MemorySink` / `MetricsSink` / `LoggingSink` / `FileSink` / `OpenTelemetrySink`, `GET /events`, `GET /trace/{request_id}`, `X-OSR-Trace-Id`. The hosted platform exposes the same buffer per workspace (`GET /api/v1/trace/{request_id}` with the reported outcomes and the request's audit records, `GET /api/v1/events`, `/platform/dashboard/events`). See [OBSERVABILITY.md](https://opensmartroute.ai/docs/OBSERVABILITY.md). - `LoggingTelemetry`: structured JSON (request hash, never raw text). - `MetricsTelemetry`: counters, p50 / p95 / p99 routing latency, mean confidence; expose via `/metrics`. - `OpenTelemetryTelemetry` (`otel` extra): spans and the `osr.route_latency_ms` histogram. - `FileAuditSink`: hash-chained audit log (tamper-evident); the chain resumes across restarts and `FileAuditSink.verify(path)` re-walks it, naming the first edited, removed or unparseable line. Records carry the `request_id`, so a decision and the outcomes reported for it can be joined back to the request (`with_audit(sink, outcomes=True)`; the platform shows them on the trace). - `RouteTrace.explain()`: human-readable decision explanation for support tooling. - `ShadowMiddleware.summary()`, `FairShareMiddleware.snapshot()`, `InflightTracker.snapshot()` and `EnterpriseRouter.health_snapshot()` for dashboards. ## Multi-tenancy `TenantMiddleware` enforces tenant presence, applies per-tenant cost ceilings, data boundaries and deny lists. `FairShareMiddleware` keeps one tenant from crowding out the others. `TargetConstraints.tenants` restricts a target to specific tenants. Learner state can be namespaced per tenant with `NamespacedStateStore` or a per-tenant `RouterBuilder`. ## Compliance hooks - Data boundary (`public` < `private` < `on_prem`) is a hard constraint. - Region allow-lists per target. - PII never reaches a `pii_allowed=False` target; `Redactor` masks before caching / logging. - Audit trail with request id, target, confidence, tenant; no content. - Learner state encrypted at rest with `EncryptedStateStore`; key from the environment or a `*_FILE` secret via `load_secret`. - Signed MCP manifests (`adapters.mcp.verify_manifest`) and an origin policy for sensitive tool parameters (`security.provenance.OriginPolicy`) for plans that execute state-changing tools. --- # Tracing and observability Every stage of a request - signal extraction, policy filtering, ranking, planning, execution, learning, health, cache, guard, shadow rollout, fair share and the autopilot - emits **spans** and **events** into a `Tracer`. Sinks decide what happens to them: keep the last N in memory for `/trace`, aggregate them into Prometheus counters, log them as JSON lines, append them to a file or forward them to OpenTelemetry. Instrumentation is always on and free when no sink is attached (a shared no-op span, no allocation), so the SDK stays zero-dependency and the cost of capturing everything is opt-in. ```python from opensmartroute import Router, configure_tracing from opensmartroute.observability import MemorySink, MetricsSink tracer = configure_tracing(MemorySink(), MetricsSink()) # process-wide; or configure_tracing() -> settings router = Router(reg) # picks up the global tracer d = router.route("Prove that sqrt(2) is irrational") for e in tracer.find(MemorySink).trace(d.request_id): # spans + events of that request, in time order print(e["kind"], e["name"], e.get("duration_ms"), e["attributes"]) print(tracer.find(MetricsSink).prometheus()) # osr_* exposition text ``` ``` span route 2.31ms {'text_sha256': '9f2c…', 'text_len': 35, 'target': 'large', 'confidence': 0.82, 'elapsed_ms': 2.1, ...} event route.signals {'duration_ms': 0.4, 'complexity': 0.71, 'domains': ['math'], 'reasoning_need': 0.9, 'contains_pii': False, ...} event route.policy {'pool': 2, 'admissible': 2, 'rejections': {}} event route.rank {'role': None, 'candidates': 2, 'strategies': {'capability': 0.05, ...}, 'top': [{'id': 'large', 'utility': 0.82, ...}]} ``` `osr route "text" --events` prints the same trace after the decision; `osr serve` exposes it over HTTP and the hosted platform shows it per workspace in the dashboard (activity drawer, `/platform/dashboard/events`, playground *Trace* tab - see the platform endpoints below). ## What is captured | Span | Opened by | Attributes (end of span) | |---|---|---| | `request` | `EnterpriseRouter.route` | `middleware` (class names), `tenant`, `target`, `confidence`, `audited` | | `http.request` | `osr serve` middleware | `method`, `path`, `app`, `status_code`, `target`, inbound `traceparent` | | `route` | `Router.route` | `text_sha256`, `text_len`, `tenant`, `objective`, `kinds`, `candidates`, `exclude`, `plan`, `task_id`, `session_id`, `target`, `kind`, `confidence`, `elapsed_ms`, `alternatives`, `abstain`, `plan_slots` | | `plan` | `Router._build_plan` (inside `route`) | `primary`, `kind`, `slots` | | `execute` | `execution.execute` / `aexecute` | `target`, `kind`, `plan_slots`, `task_id`, `runner`, `ok`, `latency_ms`, `cost_usd`, `tokens`, `outcomes`, `system_prompt_len`, `pii_redacted`, `pii_restored` | | `autopilot.cycle` | `Autopilot.run_once` | `reason`, `online`, `accepted`, `outcome`, `cycle` | | Event | Emitted when | |---|---| | `route.signals` | signals extracted: `duration_ms`, complexity, domains, actions, language, token estimate, PII, jailbreak risk, tools, task type, reasoning need | | `route.policy` | hard constraints applied: `pool`, `admissible`, `rejections` (target -> reason) | | `route.no_route` | no admissible target (before `NoRouteError` is raised) | | `route.pinned` | a rule pinned a target (`kept` tells whether it survived the policy) | | `route.narrowed` | the retriever shortlisted a large catalogue | | `route.shortlist` | a strategy shortlisted candidates | | `route.rank` | strategies scored (`strategies` with per-strategy ms, `top` 3, `excluded_by_floor`) | | `route.escalate` | the LLM judge was consulted because confidence was low | | `route.explore` | the bandit chose to explore (`served` tells whether the exploration was taken) | | `route.abstain` | the calibrated confidence was below the abstention threshold | | `route.fallback` | a fallback target replaced an unavailable one | | `plan.slot` | a plan slot was filled (or not: `filled=False`, `reason`) | | `execute.step` | one execution step (persona, skill, primary) finished | | `learn.outcome` | `Router.learn` / `learn_correction` recorded an outcome (`corrected`, `learner`) | | `learn.task_credit` | task-level credit assigned to the plan's targets | | `learn.calibrate` | the confidence calibrator was refit (temperature) | | `learn.improve` | a `SelfImprover` cycle finished (`rows`, `champion_accuracy`, `challenger_accuracy`, `accepted`, `reason`; inside an `autopilot.cycle` span when the autopilot ran it) | | `learn.promote` | a challenger SLM replaced the serving champion (`rows`, `trained_at`, `targets`, `saved`) | | `health.breaker` | a circuit breaker changed state (`previous`, `state`; level `warning` when it opens) | | `cache.hit` / `cache.miss` | `CacheMiddleware` lookup (`target`, `age_s` on a hit) | | `tenant.rejected` | `TenantMiddleware` refused a request (`reason`, `tenant`) | | `route.slow` | `TimeoutMiddleware` saw a decision exceed its budget | | `guard.blocked` / `guard.flagged` / `guard.redacted` | `GuardMiddleware` verdicts (`reasons`, `gadget`, `entities`) | | `shadow.compare` / `shadow.verdict` | `ShadowMiddleware` compared production and candidate; the A/B test reached a verdict | | `fairshare.throttled` / `fairshare.capped` | `FairShareMiddleware` delayed or refused a tenant | | `autopilot.drift` | the autopilot detected quality drift and scheduled a cycle | `opensmartroute.observability.SPAN_NAMES` and `EVENT_NAMES` list the vocabulary; a test asserts every emitted name belongs to it. **Privacy.** Request text never enters a span or event: the `route` span carries `text_sha256` (16 hex characters) and `text_len` (`text_digest()`); attribute strings are truncated to `OSR_OBSERVABILITY_ATTRIBUTE_MAX_LEN` (500) and nested values are JSON-cleaned. Sinks receive ids, numbers, target ids, reasons and scores. ## Spans, events, trace ids An `Event` is a flat record: `name`, `kind` (`event` or `span`), `ts`, `trace_id`, `span_id`, `parent_id`, `request_id`, `level`, `status` and `duration_ms` (spans only) and `attributes`. A span's closing record is the same shape as an event so every sink handles one type. The trace id of a request is derived once, at the root span: an inbound W3C `traceparent` wins, otherwise a UUID request id is reused, otherwise the request id is hashed (so a `request_id` supplied by the caller yields a stable trace id). Nested spans inherit trace id, request id and the sampling decision through a `contextvars` context, so threads and `asyncio` tasks spawned by the router carry the right parent. `Tracer.current()` returns the innermost open span; `current_tracer()` resolves the tracer to report to (the open span's, else one bound with `use_tracer`, else the global one), which is how middleware, the health registry and the executor emit without holding a reference. ## Sinks | Sink | Purpose | |---|---| | `MemorySink(max_events=1000)` | ring buffer; `events(request_id=, trace_id=, name=, kind=, level=, limit=)` (`name` may end with `*`), `trace(request_id)`; backs `/events`, `/trace/{id}` and `osr route --events` | | `MetricsSink()` | counters and p50/p95/p99 per span name; `snapshot()` for `/stats`, `prometheus()` for `/metrics` (`osr_events_total`, `osr_spans_total`, `osr_span_duration_ms`, `osr_span_decisions_total`, `osr_executions_total`, `osr_execution_cost_usd_total`, `osr_learn_outcomes_total`, `osr_breaker_transitions_total`, `osr_cache_events_total`, `osr_span_errors_total`) | | `LoggingSink(log=None)` | one JSON line per event on the `opensmartroute.events` logger, level follows the event | | `FileSink(path)` | append-only JSONL, one event per line (ship with any log forwarder) | | `OpenTelemetrySink()` (`adapters.optional`, `otel` extra) | live bridge: each span becomes an OTel span (parented through the OTel context, joined to `traceparent`), each event a span event and an `osr.events` counter, durations an `osr.span_duration_ms` histogram; the host configures the SDK, exporter and resource | Write your own by subclassing `EventSink` and implementing `emit(event)` (optionally `span_start`, `span_end`, `snapshot`). A sink that raises is counted in `Tracer.sink_errors` and logged (first ten occurrences); it never breaks routing. ## Configuration `configure_tracing(*sinks, sample_rate=None)` installs sinks on the process-wide tracer that every `Router` uses by default. With no sinks it reads `ObservabilitySettings` (`OSR_OBSERVABILITY_*`): | Setting | Env | Default | Effect | |---|---|---|---| | `metrics` | `OSR_OBSERVABILITY_METRICS` | `true` | attach a `MetricsSink` | | `memory_events` | `OSR_OBSERVABILITY_MEMORY_EVENTS` | `1000` | attach a `MemorySink` of that size (`0` disables) | | `log_events` | `OSR_OBSERVABILITY_LOG_EVENTS` | `false` | attach a `LoggingSink` | | `events_file` | `OSR_OBSERVABILITY_EVENTS_FILE` | `""` | attach a `FileSink` at that path | | `otel` | `OSR_OBSERVABILITY_OTEL` | `false` | attach an `OpenTelemetrySink` (needs the `otel` extra) | | `sample_rate` | `OSR_OBSERVABILITY_SAMPLE_RATE` | `1.0` | share of root spans that are recorded | | `attribute_max_len` | `OSR_OBSERVABILITY_ATTRIBUTE_MAX_LEN` | `500` | longest attribute string kept | `Router(reg, tracer=Tracer([...]))` gives one router a private tracer; `RouterBuilder(...).with_tracing(*sinks_or_tracer, sample_rate=)` does the same for an `EnterpriseRouter` (no arguments: build from settings). `osr`, `osr serve`, the Docker image and the hosted platform call `configure_tracing()` / `Tracer.from_settings()` at start-up, so the environment variables above are all that is needed in a container. ## HTTP endpoints (`osr serve`) | Endpoint | Returns | |---|---| | `GET /metrics` | `MetricsTelemetry` exposition followed by the tracer's `MetricsSink` counters (names are disjoint) | | `GET /events?request_id=&trace_id=&name=route.*&kind=&level=&limit=200` | recent events from the memory buffer (404 when `OSR_OBSERVABILITY_MEMORY_EVENTS=0`) | | `GET /trace/{request_id}` | every span and event of one request, in time order, with its `trace_id` | | `GET /stats` | adds an `observability` block: sinks, sample rate, sink errors and each sink's snapshot | Every traced response carries `X-OSR-Trace-Id`; a client that sends `traceparent` gets its own trace id back, so the routing spans line up with the caller's trace in Jaeger, Tempo, Azure Monitor or any OTel backend. Probes (`/healthz`, `/readyz`), `/metrics`, `/events`, `/trace/*`, `/whoami` and the OpenAPI pages are never traced. The endpoints follow the server's access control: with `OSR_SERVER_AUTH_TOKENS` set, `/events`, `/trace/*` and `/stats` need a token (`/metrics` stays open for scrapers). ## HTTP endpoints (hosted platform) The platform API (`platform/api`) exposes the same tracer per workspace, so a tenant only ever sees its own requests - and it is the platform's own observability: it persists every span and event to its database (`osr_platform.telemetry`, `OSR_PLATFORM_TELEMETRY_STORE`, on by default, off the request path through a queued writer; `OSR_PLATFORM_TELEMETRY_RETENTION_DAYS`), keeps per-minute HTTP counters for a day, computes time series from its usage records and evaluates alert rules in-process (`osr_platform.alerts`). Nothing has to be exported to a metrics, tracing or alerting system. | Endpoint | Returns | |---|---| | `GET /api/v1/trace/{request_id}` | the activity row of the request plus its spans and events (from the buffer, or from the store once evicted - `source` says which), the feedback reported for it (`outcomes`) and its audit records (`audit`); `404` for another workspace's request | | `GET /api/v1/events?name=&kind=&level=&request_id=&since=&until=&limit=` | events joined to the workspace through its request ids (deployment-wide events on plans with `stats`); `since` / `until` select a time range from the store (default the last 6 hours) | | `GET /api/v1/telemetry/series?window=` | the workspace's requests, failures, p50 / p95 latency, cost and tokens per bucket (`1h` .. `30d`) with per-target and per-endpoint totals; `http` (plans with `stats`) adds the deployment's requests, 4xx / 5xx and latency percentiles per bucket | | `GET /api/v1/alerts` | active conditions - budgets, failing requests, and on plans with `stats` readiness, 5xx rate, latency SLO, breakers, drift, autopilot errors, a stale SLM, tracing off - each with the dashboard page that shows or fixes it | | `GET /api/v1/status` | public readiness with the tracing state, for status pages (`503` while a check fails) | | `GET /api/v1/activity` | request log with `traced: true` on rows whose spans are kept and `outcome` summarising the feedback per request | `POST /api/v1/route` returns `trace_id` and `X-OSR-Trace-Id`. The dashboard renders the history at `/platform/dashboard/events` (1 hour to 7 days), a span waterfall from every traced activity row and a *Trace* tab in the playground; the overview shows the 24-hour series and the alerts, `/platform/dashboard/health` the deployment series, the *Systems* card (`/api/v1/status`) and the same alerts. ## OpenTelemetry end to end ```python from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter trace.set_tracer_provider(TracerProvider()) trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(ConsoleSpanExporter())) from opensmartroute import configure_tracing from opensmartroute.adapters.optional import OpenTelemetrySink configure_tracing(OpenTelemetrySink()) # or OSR_OBSERVABILITY_OTEL=true ``` OTel spans are named `osr.route`, `osr.plan`, `osr.execute`, ...; attributes carry the `osr.` prefix (`osr.target`, `osr.confidence`, `osr.request_id`, `osr.trace_id`). `OpenTelemetryTelemetry` (the older `Telemetry` port) still records the `osr.route_latency_ms` histogram per decision; the sink is the richer, per-stage view and the two can run side by side. ## Relationship to telemetry and audit `Telemetry` sinks (`LoggingTelemetry`, `MetricsTelemetry`, `OpenTelemetryTelemetry`) observe *decisions and outcomes* on an `EnterpriseRouter`; `AuditSink` keeps a hash-chained, tamper-evident record of *who got what*. The tracer observes *how* the decision was made - every stage, with timings, on plain and enterprise routers alike. Use audit for compliance, telemetry for dashboards you already have and the tracer for debugging, SLOs per stage and distributed tracing. `RouteTrace.explain()` remains the human-readable explanation attached to a `RouteDecision`. `MetricsTelemetry` is the dashboard store: `snapshot()` returns `decisions_total`, `decisions_by_target`, `policy_rejections_total`, `errors_by_type`, `route_latency_ms` (`p50`, `p95`, `p99`, `mean`, `count`), `route_latency_histogram_ms`, `mean_confidence`, `outcomes_by_target` (count, success rate, cost, quality, latency) and `uptime_s`; `prometheus(namespace="osr", extra={...})` renders the same as Prometheus text (`osr_route_decisions_total{target="..."}`, `osr_route_latency_ms_bucket{le="..."}`, `osr_outcome_cost_usd_total`, `osr_decision_cache_hits_total`, extras as gauges). `osr serve` and the hosted platform expose it at `GET /metrics`; `RouterBuilder.with_cache(ttl_s, max_size)` adds a decision cache whose `CacheMiddleware.stats()` feed the `osr_decision_cache_*` series, and `with_audit(sink, outcomes=True)` chains outcomes into the audit trail alongside decisions. --- # Reproducible leaderboard runs Everything the public-suite numbers in [docs/ROADMAP.md](https://opensmartroute.ai/docs/ROADMAP.md) (v0.4 measurement) are made of, as commands you can re-run. The same recipe is what an OpenSmartRoute entry on RouterArena, LLMRouterBench or xRouteBench is built from; the runs themselves are tracked under v1.0 in the roadmap and recorded, with their submission status, in [results/README.md](https://opensmartroute.ai/docs/leaderboard-results.md#listings). Requirements: `pip install "opensmartroute[yaml]"`, network access to the Hugging Face datasets-server (public suites only; RouterBench / RouterEval are gated and need a Hub token plus `--preset`). ## 1. Catalogue [targets.yaml](https://github.com/isathish/OpenSmartRoute/blob/main/examples/leaderboard/targets.yaml): three tiers, `llm-small` / `llm-mid` / `llm-frontier`. Battle datasets name 2023-24 models; `osr collect --tier` maps each model to the tier it played in (`eval.collect.ARENA_TIERS`), so every row's label is "which tier was good enough for this prompt" - the RouteLLM framing. ## 2. Collect the public suites ```bash osr collect --cache-dir data --limit 4000 \ --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small ``` Writes one `-tiers.jsonl` (+ `.meta.json`) per public source: `arena-55k`, `arena-100k`, `arena-140k`, `routellm-battles`, `ppe-human`, `webdev-arena`, `mt-bench-human`, `reward-bench`, `ultrafeedback`, `routellm-gpt4`. `--source NAME` restricts the run; `--refresh` ignores the cache. ## 3. Train the routing SLM on the training suites only ```bash osr -t examples/leaderboard/targets.yaml slm train --out slm.json --cache-dir data \ --source arena-55k-tiers --source routellm-battles-tiers --report --seed 0 ``` `--report` holds out 10 % and prints its accuracy. The suites in step 4 are **not** in `--source`, so they are held out at the suite level, not just the row level. ## 4. Evaluate on the held-out suites ```bash for suite in mt-bench-human ppe-human webdev-arena; do osr -t examples/leaderboard/targets.yaml --slm slm.json eval data/$suite-tiers.jsonl \ --baselines --calibration --robustness --limit 1000 --json > results/$suite.json done # declarative router (no learned signals) for the ablation column osr -t examples/leaderboard/targets.yaml eval data/mt-bench-human-tiers.jsonl --baselines --limit 1000 ``` What each flag reports and where it lands in the table: | Flag | Reports | Table column | |---|---|---| | (default) | accuracy, mean quality, cost, latency, mean confidence, raw ECE / Brier | Router, ECE raw | | `--baselines` | random / cheapest / best-prior / static task table / oracle, gap to oracle, noise floor | Static task table, Random, Cheapest | | `--robustness` | repeat flip rate, paraphrase stability, profile-swap dependence (first 200 rows) | Paraphrase | | `--calibration` | ECE, Brier, reliability bins, and `held_out`: temperature and isotonic calibrators fitted on the even rows, scored on the odd rows | ECE held-out | The published table used `--limit 1000` per suite and seed 0 throughout; the SLM bundle and the `Router` are deterministic, so re-running gives the same numbers for the same cache. ## 5. Gated suites (RouterBench, RouterEval, xRouteBench, RouterXBench) Download the benchmark file with your Hub token, then evaluate it directly through the layout preset: ```bash osr -t examples/leaderboard/targets.yaml --slm slm.json eval routerbench_0shot.pkl.jsonl \ --preset routerbench --baselines --calibration --limit 5000 ``` These rows carry several sampled answers per model, so `--baselines` can also report the label-noise floor (`noise_floor()`), which the pairwise suites above cannot. ## 6. Publish Commit `results/*.json`, the exact `osr --version`, the dataset ids, splits and row counts, and the SHA-256 of `slm.json` together; that triple is the reproducible config a leaderboard entry links to. `python scripts/exit_criteria.py --full` prints the synthetic exit criteria alongside, so the two tables can be published from one run. The run published with 0.5.0 is in [results/README.md](https://opensmartroute.ai/docs/leaderboard-results.md) (one JSON per suite, learned and declarative); it is the entry submitted to RouterArena, LLMRouterBench and xRouteBench. --- # Published runs - opensmartroute 0.5.0 Produced with the recipe in [../README.md](https://opensmartroute.ai/docs/leaderboard.md) on 2026-09-06, `osr 0.5.0`, seed 0, `--limit 1000` per suite. One JSON per suite: `.json` is the learned router (`--slm slm.json --baselines --calibration --robustness`), `.declarative.json` is the same catalogue without the SLM (`--baselines` only). The per-row arrays `calibration.conformal_pairs` and `calibration.temperature_samples` were removed from the committed files; every aggregate they feed (`reliability`, `held_out`) is kept. ## Data | Role | Cache name | Hugging Face dataset | Rows after `--tier` relabelling | |---|---|---|---| | train | `arena-55k-tiers` | `lmarena-ai/arena-human-preference-55k` | 1 986 | | train | `routellm-battles-tiers` | `routellm/gpt4_judge_battles` | 4 000 | | held out | `mt-bench-human-tiers` | `lmsys/mt_bench_human_judgments` (`human`) | 2 228 | | held out | `ppe-human-tiers` | `lmarena-ai/PPE-Human-Preference-V1` (`test`) | 2 360 | | held out | `webdev-arena-tiers` | `lmarena-ai/webdev-arena-preference-10k` (`test`) | 1 070 | Catalogue: [../targets.yaml](https://github.com/isathish/OpenSmartRoute/blob/main/examples/leaderboard/targets.yaml) (`llm-small` / `llm-mid` / `llm-frontier`). SLM bundle: `osr -t examples/leaderboard/targets.yaml slm train --out slm.json --cache-dir data --source arena-55k-tiers --source routellm-battles-tiers --report --seed 0` -> 5 956 rows, 10 % holdout accuracy 0.721, ECE 0.056, `sha256 8faf8466902b753a9f22382de24bfd0e1d5271c7fe5d6564dec8105310dcbb1e` (4.0 MB, not committed; the command is deterministic for the same cache). ## Results | Suite (n=1000) | Router (learned) | Declarative | Static task table | Random | Cheapest | Best prior | Paraphrase | Repeat flips | ECE raw | ECE held-out temp. | ECE held-out isotonic | |---|---|---|---|---|---|---|---|---|---|---|---| | MT-Bench human | **0.523** | 0.421 | 0.541 | 0.441 | 0.372 | 0.343 | 1.000 | 0.000 | 0.328 | 0.056 | 0.048 | | PPE human | **0.552** | 0.418 | 0.651 | 0.445 | 0.174 | 0.651 | 0.898 | 0.000 | 0.298 | 0.121 | 0.035 | | WebDev Arena | **0.573** | 0.192 | 0.670 | 0.391 | 0.000 | 0.670 | 0.883 | 0.000 | 0.283 | 0.032 | 0.047 | Cost and latency per 1k tokens for the learned router: MT-Bench $0.00363 / 968 ms, PPE $0.00390 / 1005 ms, WebDev $0.00324 / 925 ms (static task table: $0.003 / 900 ms on MT-Bench, $0.015 / 2500 ms on PPE and WebDev, since it degenerates to "always the best single tier"). Reading: the learned router beats the declarative router by +10 to +38 pp and random / cheapest everywhere, at a fifth of the cost of "always frontier" on PPE and WebDev. It does **not** beat the best single tier on any suite (-2 to -10 pp): a pairwise battle between two named models is a weak label for a three-tier decision, and the label-noise floor cannot be measured on these rows (one sample per prompt). Raw confidence is over-confident (ECE 0.28-0.33); a calibrator fitted on the even rows and scored on the odd rows brings held-out ECE to 0.03-0.05 (isotonic). These are the numbers a RouterArena / LLMRouterBench / xRouteBench entry should quote; the gated multi-sample suites (step 5 of the recipe) are what would let the router be scored against the noise floor. ## Listings Where this run has been submitted. `python scripts/release.py readiness` counts rows whose status is `accepted` for the v1.0 adoption item "Leaderboard listings accepted" and reports it as open until one listing is accepted. A submission links the exact commit, the SLM SHA-256 above and this page. | Leaderboard | Entry | Status | Notes | |---|---|---|---| | RouterArena | - | not submitted | submission = a PR to the RouterArena repository with `runs/*.json` | | LLMRouterBench | - | not submitted | needs the gated multi-sample suites (step 5) for the noise-floor comparison | | xRouteBench | - | not submitted | component ablation report from `osr eval --ablation` | --- # API reference Every module under `src/opensmartroute` and every public name it exports, generated from the source by `python scripts/api_reference.py` (checked by `tests/test_docs.py`; do not edit by hand). The first line of each docstring is the summary; open the module for the full contract. Narrative documentation: [GUIDE.md](https://opensmartroute.ai/docs/GUIDE.md) (usage), [SDK.md](https://opensmartroute.ai/docs/SDK.md) (stability, extension points), [ARCHITECTURE.md](https://opensmartroute.ai/docs/ARCHITECTURE.md), [MATH.md](https://opensmartroute.ai/docs/MATH.md), [ENTERPRISE.md](https://opensmartroute.ai/docs/ENTERPRISE.md), [SECURITY.md](https://opensmartroute.ai/docs/SECURITY.md), [RESEARCH.md](https://opensmartroute.ai/docs/RESEARCH.md). Stability: names re-exported from `opensmartroute` (the top-level package) are frozen by `tests/public_api.json` and follow SemVer. Sub-module names are public but may change in a minor release with a CHANGELOG entry. ## Modules - [`opensmartroute`](#opensmartroute) - OpenSmartRoute — an open, intelligent route to the right decision, solution, or destination. (69 names) - [`opensmartroute.adapters`](#opensmartrouteadapters) - Adapters connect OpenSmartRoute to real providers and infrastructure. (70 names) - [`opensmartroute.adapters.a2a`](#opensmartrouteadaptersa2a) - Import A2A (Agent-to-Agent protocol) **Agent Cards** as ``TargetKind.AGENT`` targets. (4 names) - [`opensmartroute.adapters.catalogue`](#opensmartrouteadapterscatalogue) - Live model catalogue: collect model cards (price, context, modalities, benchmarks) from public sources. (11 names) - [`opensmartroute.adapters.frameworks`](#opensmartrouteadaptersframeworks) - Drop-in nodes for agent frameworks. (7 names) - [`opensmartroute.adapters.handlers`](#opensmartrouteadaptershandlers) - Executors for non-LLM targets: HTTP endpoints, MCP tools and asynchronous queues. (7 names) - [`opensmartroute.adapters.harness`](#opensmartrouteadaptersharness) - Agent-harness adapters: route to a *runtime*, not just a model. (7 names) - [`opensmartroute.adapters.mcp`](#opensmartrouteadaptersmcp) - Import MCP (Model Context Protocol) tools as ``TargetKind.TOOL`` targets. (11 names) - [`opensmartroute.adapters.mcp_servers`](#opensmartrouteadaptersmcp_servers) - MCP *server* recommendation (MCP-Zero 2506.01056; ToolRet 2603.06467). (3 names) - [`opensmartroute.adapters.openai_compat`](#opensmartrouteadaptersopenai_compat) - OpenAI-compatible HTTP client (stdlib only). (6 names) - [`opensmartroute.adapters.optional`](#opensmartrouteadaptersoptional) - Optional adapters that need extra dependencies. Everything is lazily imported so the. (3 names) - [`opensmartroute.adapters.personas`](#opensmartrouteadapterspersonas) - Import persona catalogues as ``TargetKind.PERSONA`` targets. (4 names) - [`opensmartroute.adapters.semantic_router`](#opensmartrouteadapterssemantic_router) - Import a vLLM *semantic-router* configuration (vllm-project/semantic-router). (3 names) - [`opensmartroute.adapters.skills`](#opensmartrouteadaptersskills) - Load Agent-Skills ``SKILL.md`` packages as ``TargetKind.SKILL`` targets. (4 names) - [`opensmartroute.adapters.websearch`](#opensmartrouteadapterswebsearch) - Web knowledge for the self-improving router: stdlib HTTP fetch, search providers, page text. (14 names) - [`opensmartroute.aio`](#opensmartrouteaio) - Async façade. Routing itself is CPU-bound and sub-millisecond, so we run it in. (1 names) - [`opensmartroute.branding`](#opensmartroutebranding) - OpenSmartRoute naming conventions: one place for every brand-bound identifier. (27 names) - [`opensmartroute.cli`](#opensmartroutecli) - ``osr`` command-line interface. (2 names) - [`opensmartroute.config`](#opensmartrouteconfig) - Configuration loading: targets and rules from JSON or YAML. (3 names) - [`opensmartroute.core`](#opensmartroutecore) - see module (16 names) - [`opensmartroute.core.registry`](#opensmartroutecoreregistry) - Target registry: the catalogue of everything a request may be routed to. (1 names) - [`opensmartroute.core.types`](#opensmartroutecoretypes) - Core data model for OpenSmartRoute. (16 names) - [`opensmartroute.credentials`](#opensmartroutecredentials) - Credentials for the ``osr`` CLI: where the access token lives and how it is obtained. (18 names) - [`opensmartroute.discovery`](#opensmartroutediscovery) - Tool discovery beyond text similarity. (5 names) - [`opensmartroute.enterprise`](#opensmartrouteenterprise) - Enterprise integration layer: ports (hexagonal architecture), middleware and telemetry. (18 names) - [`opensmartroute.enterprise.ops`](#opensmartrouteenterpriseops) - Operational controls: shadow / A-B routing, tenant fairness and queue-aware latency. (7 names) - [`opensmartroute.enterprise.savings`](#opensmartrouteenterprisesavings) - Savings ledger - the always-on savings report that backs the ROI story and the dashboard. (3 names) - [`opensmartroute.enterprise.stores`](#opensmartrouteenterprisestores) - Production state-store backends and wrappers. (7 names) - [`opensmartroute.errors`](#opensmartrouteerrors) - Exception hierarchy. Every error raised by the SDK derives from :class:`OpenSmartRouteError`. (11 names) - [`opensmartroute.estimate`](#opensmartrouteestimate) - Token, cost and latency estimates *before* a request is sent anywhere. (9 names) - [`opensmartroute.eval`](#opensmartrouteeval) - RouterBench-style evaluation harness. (19 names) - [`opensmartroute.eval.agentic`](#opensmartrouteevalagentic) - tau-bench-style agentic task evaluation: does per-step routing beat the best single agent?. (4 names) - [`opensmartroute.eval.audit`](#opensmartrouteevalaudit) - Routing Audit - shadow-mode replay of logged LLM traffic to quantify what the router would change. (6 names) - [`opensmartroute.eval.baselines`](#opensmartrouteevalbaselines) - Baselines every router must beat, plus oracle ceilings and the sampling noise floor. (13 names) - [`opensmartroute.eval.collect`](#opensmartrouteevalcollect) - Collect routing datasets from the Hugging Face Hub, your own feedback log and synthetic seeds. (29 names) - [`opensmartroute.eval.criteria`](#opensmartrouteevalcriteria) - Offline realisations of the ROADMAP exit criteria. (15 names) - [`opensmartroute.eval.datasets`](#opensmartrouteevaldatasets) - Adapters from public routing benchmarks to :class:`EvalRow`. (7 names) - [`opensmartroute.eval.frontier`](#opensmartrouteevalfrontier) - Three-objective frontier and ablations. (3 names) - [`opensmartroute.eval.headroom`](#opensmartrouteevalheadroom) - Routing headroom: when does routing pay, and how much catalogue does it need?. (7 names) - [`opensmartroute.eval.ope`](#opensmartrouteevalope) - Off-policy evaluation from logged routing decisions. (6 names) - [`opensmartroute.eval.robustness`](#opensmartrouteevalrobustness) - Robustness and fairness checks for a router. (6 names) - [`opensmartroute.execution`](#opensmartrouteexecution) - Plan-aware execution: turn a :class:`RouteDecision` into a real answer. (5 names) - [`opensmartroute.feedback`](#opensmartroutefeedback) - Feedback store: append-only outcome log that closes the learning loop. (1 names) - [`opensmartroute.learning`](#opensmartroutelearning) - Auto-learning strategies built on :mod:`opensmartroute.math`. (47 names) - [`opensmartroute.learning.attention`](#opensmartroutelearningattention) - A pure-Python transformer block for the routing SLM's query encoder. (2 names) - [`opensmartroute.learning.autopilot`](#opensmartroutelearningautopilot) - Self-operation: the routing SLM runs its own improvement loop inside the live process. (2 names) - [`opensmartroute.learning.coldstart`](#opensmartroutelearningcoldstart) - Cold start for new targets and self-improving target descriptions. (8 names) - [`opensmartroute.learning.contrastive`](#opensmartroutelearningcontrastive) - Contrastive and reward-distilled router training (RouterDC, NeurIPS 2024; Zooter 2311.08692). (4 names) - [`opensmartroute.learning.credit`](#opensmartroutelearningcredit) - Delayed, task-level credit assignment for agentic trajectories. (2 names) - [`opensmartroute.learning.embed`](#opensmartroutelearningembed) - Pretrained transformer embeddings as frozen features for the routing SLM. (2 names) - [`opensmartroute.learning.handoff`](#opensmartroutelearninghandoff) - Permanent-handoff policy from censored teacher signals (TACIT-Switch 2608.27911). (3 names) - [`opensmartroute.learning.multiturn`](#opensmartroutelearningmultiturn) - Multi-turn routing with history-target joint embeddings (MTRouter 2604.23530). (3 names) - [`opensmartroute.learning.personal`](#opensmartroutelearningpersonal) - Few-shot personalisation (GMTRouter 2511.08590; SkillFeed 2608.28241). (4 names) - [`opensmartroute.learning.policy_gradient`](#opensmartroutelearningpolicy_gradient) - End-to-end policy-gradient routing (Router-R1 2506.09033; RLCascadeRouter 2608.15817). (4 names) - [`opensmartroute.learning.self_improve`](#opensmartroutelearningself_improve) - Self-improvement loop: refresh the catalogue, gather evidence, train a challenger, promote it only if better. (2 names) - [`opensmartroute.learning.slm`](#opensmartroutelearningslm) - The OpenSmartRoute routing SLM: a small, self-contained model that picks the target for a prompt. (5 names) - [`opensmartroute.math`](#opensmartroutemath) - Mathematical toolkit behind OpenSmartRoute's decisions. (46 names) - [`opensmartroute.math.bandits`](#opensmartroutemathbandits) - Multi-armed and contextual bandits for online routing decisions. (8 names) - [`opensmartroute.math.calibration`](#opensmartroutemathcalibration) - Calibration and distribution-free risk control for routing confidence. (6 names) - [`opensmartroute.math.decision`](#opensmartroutemathdecision) - Multi-objective decision helpers and queueing theory for real-time routing. (9 names) - [`opensmartroute.math.dirichlet`](#opensmartroutemathdirichlet) - Dirichlet probe over host hidden states (ProbeDirichlet, RouterXBench 2602.11877). (2 names) - [`opensmartroute.math.energy`](#opensmartroutemathenergy) - Hardware-aware energy characterisation (HW-Router 2608.14575; 2608.28044). (3 names) - [`opensmartroute.math.estimators`](#opensmartroutemathestimators) - Streaming estimators, drift detection and calibration. (11 names) - [`opensmartroute.math.irt`](#opensmartroutemathirt) - Item Response Theory for routing (IRT-Router, ACL 2025). (3 names) - [`opensmartroute.math.markov`](#opensmartroutemathmarkov) - Markov chains and MDPs for conversational / multi-step routing. (2 names) - [`opensmartroute.math.preference`](#opensmartroutemathpreference) - Bradley–Terry pairwise preference model (RouteLLM, Prompt-to-Leaderboard). (2 names) - [`opensmartroute.mcp_server`](#opensmartroutemcp_server) - Model Context Protocol server: the router as a set of tools for any IDE or agent. (8 names) - [`opensmartroute.observability`](#opensmartrouteobservability) - Tracing and observability: every step of routing, execution and learning as a structured event. (18 names) - [`opensmartroute.ocm`](#opensmartrouteocm) - Open Capability Manifest (OCM) - a vendor-neutral description of any routable capability. (12 names) - [`opensmartroute.policy`](#opensmartroutepolicy) - Policy layer: hard constraints that are never traded off against utility. (20 names) - [`opensmartroute.realtime`](#opensmartrouterealtime) - Real-time operational controls: health, circuit breaking, rate & budget limits. (9 names) - [`opensmartroute.retrieval`](#opensmartrouteretrieval) - Retrieval-based candidate narrowing for very large target pools (ToolRet / Skill-RAG). (12 names) - [`opensmartroute.router`](#opensmartrouterouter) - The Router: signals -> policy -> strategies -> ensemble -> decision (+plan). (3 names) - [`opensmartroute.sdk`](#opensmartroutesdk) - OpenSmartRoute SDK: decorator-driven registration of routing components. (16 names) - [`opensmartroute.security`](#opensmartroutesecurity) - Security controls for the routing control plane. (27 names) - [`opensmartroute.security.gadget`](#opensmartroutesecuritygadget) - Learned confounder-gadget detector (Rerouting LLM Routers, Shafran et al. 2025). (6 names) - [`opensmartroute.security.injection`](#opensmartroutesecurityinjection) - Instruction-injection detection for *text that is not the user's request*. (4 names) - [`opensmartroute.security.limits`](#opensmartroutesecuritylimits) - Resource-amplification limits ("Beyond Max Tokens"). (8 names) - [`opensmartroute.security.provenance`](#opensmartroutesecurityprovenance) - Origin (provenance) policy for tool parameters – ROPE-style control-flow integrity. (12 names) - [`opensmartroute.security.safety`](#opensmartroutesecuritysafety) - Safety-routing regression suite ("When Safety Routing Breaks"). (4 names) - [`opensmartroute.server`](#opensmartrouteserver) - Optional FastAPI server exposing the router over HTTP. (14 names) - [`opensmartroute.settings`](#opensmartroutesettings) - Typed, environment-overridable settings: the only home for OpenSmartRoute's tunable constants. (13 names) - [`opensmartroute.signals`](#opensmartroutesignals) - Signal extraction: cheap, deterministic features computed from the request. (44 names) - [`opensmartroute.signals.events`](#opensmartroutesignalsevents) - Event- and workflow-driven signals for agentic inputs. (7 names) - [`opensmartroute.signals.models`](#opensmartroutesignalsmodels) - Learned signal models: hashed n-gram linear models with no dependencies. (7 names) - [`opensmartroute.signals.ontology`](#opensmartroutesignalsontology) - Task ontology: families -> types -> subtypes, with an orthogonal *domain* axis. (4 names) - [`opensmartroute.signals.uncertainty`](#opensmartroutesignalsuncertainty) - Uncertainty signals that come from *outside the query text*. (12 names) - [`opensmartroute.stack`](#opensmartroutestack) - Declarative *stacks*: one document that describes a whole routing setup. (12 names) - [`opensmartroute.strategies`](#opensmartroutestrategies) - see module (66 names) - [`opensmartroute.strategies.aggregate`](#opensmartroutestrategiesaggregate) - Routing / aggregation switch (Mixture-of-Agents, Wang et al. 2024; JiSi 2601.01330). (5 names) - [`opensmartroute.strategies.auction`](#opensmartroutestrategiesauction) - Error-aware reverse auction across providers (EA-RAM 2608.12719). (5 names) - [`opensmartroute.strategies.bandit`](#opensmartroutestrategiesbandit) - Online-learning strategy: contextual Thompson-sampling bandit. (1 names) - [`opensmartroute.strategies.base`](#opensmartroutestrategiesbase) - Strategy interface. (3 names) - [`opensmartroute.strategies.capability`](#opensmartroutestrategiescapability) - Capability-fit strategy: match request signals to a target's declared capabilities. (1 names) - [`opensmartroute.strategies.cascade`](#opensmartroutestrategiescascade) - Cascade execution (FrugalGPT / Router-R1 multi-round / AutoMix POMDP). (5 names) - [`opensmartroute.strategies.defer`](#opensmartroutestrategiesdefer) - Defer-to-human and effort ("think or not") strategies. (3 names) - [`opensmartroute.strategies.edge`](#opensmartroutestrategiesedge) - Edge-cloud token-aware routing (Pro-Router 2608.28726; RelayLLM 2601.05167). (1 names) - [`opensmartroute.strategies.elastic`](#opensmartroutestrategieselastic) - Token-budget-aware routing to elastic / many-in-one models (Nemotron Elastic 2511.16664; Star 2605.07182). (3 names) - [`opensmartroute.strategies.escalation`](#opensmartroutestrategiesescalation) - Bayesian self-escalation during generation (2608.24087). (5 names) - [`opensmartroute.strategies.human`](#opensmartroutestrategieshuman) - Routing among human annotators and experts (QUORUM 2608.27974; Dawid-Skene 1979). (5 names) - [`opensmartroute.strategies.llm_judge`](#opensmartroutestrategiesllm_judge) - Generative routing: let an LLM act as the router (Router-R1 / LLM-as-judge). (2 names) - [`opensmartroute.strategies.memory`](#opensmartroutestrategiesmemory) - Memory-tier routing for agents (BudgetMem 2602.06025; Gated-Memory Routing 2609.00237). (5 names) - [`opensmartroute.strategies.modality`](#opensmartroutestrategiesmodality) - Multimodal routing and modality escalation (LatentRouter 2605.11301; modality escalation. (4 names) - [`opensmartroute.strategies.probe`](#opensmartroutestrategiesprobe) - Hidden-state routing with a Dirichlet probe (ProbeDirichlet, RouterXBench 2602.11877). (2 names) - [`opensmartroute.strategies.progress`](#opensmartroutestrategiesprogress) - Agentic trajectories: per-step routing and multi-round execution. (5 names) - [`opensmartroute.strategies.protocol`](#opensmartroutestrategiesprotocol) - Collaboration-protocol selection (2608.14927). (7 names) - [`opensmartroute.strategies.rules`](#opensmartroutestrategiesrules) - Declarative rule-based routing (Arch-Router-style domain/action preferences). (2 names) - [`opensmartroute.strategies.semantic_cache`](#opensmartroutestrategiessemantic_cache) - Semantic caching as a routing target (GPTCache; vLLM semantic router 2603.04444). (4 names) - [`opensmartroute.strategies.session`](#opensmartroutestrategiessession) - Session affinity: keep a conversation with the target that is already serving it. (2 names) - [`opensmartroute.strategies.similarity`](#opensmartroutestrategiessimilarity) - Similarity-based routing (UniRoute / GraphRouter flavour). (3 names) - [`opensmartroute.strategies.speculative`](#opensmartroutestrategiesspeculative) - Speculative (draft-based) cascades (speculative cascades, Narasimhan et al. 2024; Differential. (3 names) - [`opensmartroute.strategies.task_table`](#opensmartroutestrategiestask_table) - Static task table strategy (SCX Router, 2609.02292). (1 names) ## `opensmartroute` Source: [src/opensmartroute/__init__.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/__init__.py) OpenSmartRoute — an open, intelligent route to the right decision, solution, or destination. | Name | Kind | Summary | |---|---|---| | `AsyncRouter` | re-export of [`opensmartroute.aio.AsyncRouter`](#opensmartrouteaio) | ``await``-able wrapper around a :class:`Router`: route / learn / execute / run off the event loop. | | `BanditStrategy` | re-export of [`opensmartroute.strategies.bandit.BanditStrategy`](#opensmartroutestrategiesbandit) | Thompson-sampling Beta bandit per (context, target); context = dominant domain (+ plan role). | | `Capabilities` | re-export of [`opensmartroute.core.types.Capabilities`](#opensmartroutecoretypes) | Declarative description of what a target is good at. | | `CapabilityStrategy` | re-export of [`opensmartroute.strategies.capability.CapabilityStrategy`](#opensmartroutestrategiescapability) | Declarative fit: domain / action overlap, complexity band, language, modality and quality prior. | | `Cascade` | re-export of [`opensmartroute.strategies.cascade.Cascade`](#opensmartroutestrategiescascade) | Execute ranked targets in planner order (cheapest / MDP / POMDP), stopping when the quality gate passes. | | `CascadePlanner` | re-export of [`opensmartroute.strategies.cascade.CascadePlanner`](#opensmartroutestrategiescascade) | Finite-horizon MDP over an ordered cascade with a stop action after each step. | | `ComponentRegistry` | re-export of [`opensmartroute.sdk.ComponentRegistry`](#opensmartroutesdk) | Blueprints for every routing component, with decorators that register into it. | | `ConfigurationError` | re-export of [`opensmartroute.errors.ConfigurationError`](#opensmartrouteerrors) | Invalid catalogue, rules, SKILL.md, settings or a missing optional dependency. | | `DeferStrategy` | re-export of [`opensmartroute.strategies.defer.DeferStrategy`](#opensmartroutestrategiesdefer) | Learning-to-defer: scores human targets by risk, PII, escalation intent, frustration and model uncertainty. | | `EffortStrategy` | re-export of [`opensmartroute.strategies.defer.EffortStrategy`](#opensmartroutestrategiesdefer) | Match a target's reasoning ``effort_level`` to ``signals.reasoning_need``; penalise over- and under-thinking. | | `Event` | re-export of [`opensmartroute.observability.Event`](#opensmartrouteobservability) | One captured span or event: flat, JSON-friendly, never carries request text. | | `EventSink` | re-export of [`opensmartroute.observability.EventSink`](#opensmartrouteobservability) | Receiver port: override :meth:`emit`; live bridges may also implement :meth:`span_start` / :meth:`span_end`. | | `ExecutionError` | re-export of [`opensmartroute.errors.ExecutionError`](#opensmartrouteerrors) | A target handler failed while executing a plan. | | `ExecutionResult` | re-export of [`opensmartroute.execution.ExecutionResult`](#opensmartrouteexecution) | What happened when a decision was executed. | | `ExecutionStep` | re-export of [`opensmartroute.execution.ExecutionStep`](#opensmartrouteexecution) | One executed plan slot: role, target, latency and whether it succeeded. | | `FeedbackStore` | re-export of [`opensmartroute.feedback.FeedbackStore`](#opensmartroutefeedback) | Append-only :class:`Outcome` log (in memory or JSONL file) with per-target statistics. | | `LLMJudgeStrategy` | re-export of [`opensmartroute.strategies.llm_judge.LLMJudgeStrategy`](#opensmartroutestrategiesllm_judge) | LLM-as-router with optional **score calibration**. | | `NoRouteError` | re-export of [`opensmartroute.errors.NoRouteError`](#opensmartrouteerrors) | No target satisfied the hard constraints. | | `Objective` | re-export of [`opensmartroute.core.types.Objective`](#opensmartroutecoretypes) | What the caller wants to optimise. Weights are relative. | | `OpenSmartRouteError` | re-export of [`opensmartroute.errors.OpenSmartRouteError`](#opensmartrouteerrors) | Base class for all SDK errors. | | `Outcome` | re-export of [`opensmartroute.core.types.Outcome`](#opensmartroutecoretypes) | Feedback about how a routed request actually went. | | `PlanSlot` | re-export of [`opensmartroute.core.types.PlanSlot`](#opensmartroutecoretypes) | One filled slot of a multi-target plan (persona -> skill -> model). | | `Policy` | re-export of [`opensmartroute.policy.Policy`](#opensmartroutepolicy) | Ordered chain of :data:`PolicyRule`; returns the first rejection reason or ``None``. | | `ProgressRouter` | re-export of [`opensmartroute.strategies.progress.ProgressRouter`](#opensmartroutestrategiesprogress) | Route each step of a task with trajectory context. | | `RankedTarget` | re-export of [`opensmartroute.core.types.RankedTarget`](#opensmartroutecoretypes) | A scored candidate: utility, ensemble quality estimate and the per-strategy breakdown. | | `RequestConstraints` | re-export of [`opensmartroute.core.types.RequestConstraints`](#opensmartroutecoretypes) | Hard constraints on the request (never traded off). | | `RouteDecision` | re-export of [`opensmartroute.core.types.RouteDecision`](#opensmartroutecoretypes) | The answer to ``route()``: chosen target, confidence, alternatives, optional plan, trace and propensities. | | `RoutePlan` | re-export of [`opensmartroute.core.types.RoutePlan`](#opensmartroutecoretypes) | A composed route (MasRouter-style): several targets working together. | | `RouteRequest` | re-export of [`opensmartroute.core.types.RouteRequest`](#opensmartroutecoretypes) | The customer need. | | `RouteTarget` | re-export of [`opensmartroute.core.types.RouteTarget`](#opensmartroutecoretypes) | A routable destination: an LLM, agent, skill, persona, tool, workflow or human. | | `RouteTrace` | re-export of [`opensmartroute.core.types.RouteTrace`](#opensmartroutecoretypes) | Everything needed to explain a decision. | | `Router` | re-export of [`opensmartroute.router.Router`](#opensmartrouterouter) | Signals -> policy -> strategies -> ensemble -> decision (+ optional plan). | | `RouterSLM` | re-export of [`opensmartroute.learning.slm.RouterSLM`](#opensmartroutelearningslm) | Small routing model: dual encoder + target catalogue snapshot + calibration, in one JSON file. | | `Rule` | re-export of [`opensmartroute.strategies.rules.Rule`](#opensmartroutestrategiesrules) | If all `when` conditions match, boost `prefer` targets and penalise `avoid`. | | `RulesStrategy` | re-export of [`opensmartroute.strategies.rules.RulesStrategy`](#opensmartroutestrategiesrules) | Applies declarative :class:`Rule` preferences (prefer / avoid / pin) when their ``when`` conditions match. | | `SLMStrategy` | re-export of [`opensmartroute.learning.slm.SLMStrategy`](#opensmartroutelearningslm) | Ensemble member backed by a :class:`RouterSLM`; scores are its probabilities, and it keeps learning online. | | `SecurityError` | re-export of [`opensmartroute.errors.SecurityError`](#opensmartrouteerrors) | Request rejected by an input guard (prompt injection, oversize, etc.). | | `SelfImprover` | re-export of [`opensmartroute.learning.self_improve.SelfImprover`](#opensmartroutelearningself_improve) | Closed loop that keeps a :class:`RouterSLM` current with the model market and its own traffic. | | `Settings` | re-export of [`opensmartroute.settings.Settings`](#opensmartroutesettings) | All tunables, grouped by consumer. Immutable; derive variants with :meth:`replace`. | | `Signals` | re-export of [`opensmartroute.core.types.Signals`](#opensmartroutecoretypes) | Cheap deterministic features extracted from a request. | | `SimilarityStrategy` | re-export of [`opensmartroute.strategies.similarity.SimilarityStrategy`](#opensmartroutestrategiessimilarity) | Embed the request and each target's examples / description; score by best and top-k mean similarity. | | `Span` | re-export of [`opensmartroute.observability.Span`](#opensmartrouteobservability) | An open unit of work; a context manager that records duration, status and nested events. | | `StateStoreError` | re-export of [`opensmartroute.errors.StateStoreError`](#opensmartrouteerrors) | A learner-state store failed to load, save or migrate. | | `Strategy` | re-export of [`opensmartroute.strategies.base.Strategy`](#opensmartroutestrategiesbase) | Scores each candidate target in [0, 1] and explains why. | | `StrategyScore` | re-export of [`opensmartroute.core.types.StrategyScore`](#opensmartroutecoretypes) | One strategy's opinion about one target. | | `TargetConstraints` | re-export of [`opensmartroute.core.types.TargetConstraints`](#opensmartroutecoretypes) | Where / for whom a target may be used. Checked by the policy layer. | | `TargetKind` | re-export of [`opensmartroute.core.types.TargetKind`](#opensmartroutecoretypes) | Kinds of things a request can be routed to. | | `TargetRegistry` | re-export of [`opensmartroute.core.registry.TargetRegistry`](#opensmartroutecoreregistry) | In-memory catalogue of :class:`RouteTarget` by id: add / upsert / remove, filtered listing, (de)serialisation. | | `TargetUnavailableError` | re-export of [`opensmartroute.errors.TargetUnavailableError`](#opensmartrouteerrors) | A remote target or catalogue source could not be reached. | | `TaskTableStrategy` | re-export of [`opensmartroute.strategies.task_table.TaskTableStrategy`](#opensmartroutestrategiestask_table) | Static ``task_type -> target -> quality`` table with family and prior fallbacks; learns from outcomes. | | `Tracer` | re-export of [`opensmartroute.observability.Tracer`](#opensmartrouteobservability) | Opens spans, records events and fans them out to sinks; ``sample_rate`` < 1 traces a share of requests. | | `ValidationError` | re-export of [`opensmartroute.errors.ValidationError`](#opensmartrouteerrors) | A request, outcome or target failed validation. | | `__version__` | constant | | | `agent` | re-export of [`opensmartroute.sdk.agent`](#opensmartroutesdk) | @agent(id, ...): shorthand for an ``agent`` target. | | `components` | re-export of [`opensmartroute.sdk.components`](#opensmartroutesdk) | : The process-wide registry that the top-level decorators (``opensmartroute.strategy`` ...) bind to. | | `configure` | re-export of [`opensmartroute.settings.configure`](#opensmartroutesettings) | Install process-wide settings. ``configure()`` with no arguments re-reads the environment;. | | `configure_tracing` | re-export of [`opensmartroute.observability.configure_tracing`](#opensmartrouteobservability) | Install sinks on the process-wide tracer. With no sinks, use the ones named by settings. | | `get_settings` | re-export of [`opensmartroute.settings.get_settings`](#opensmartroutesettings) | The process-wide :class:`Settings` (environment overlay applied once, lazily). | | `get_tracer` | re-export of [`opensmartroute.observability.get_tracer`](#opensmartrouteobservability) | The process-wide tracer (disabled until :func:`configure_tracing` adds sinks). | | `load_rules` | re-export of [`opensmartroute.config.load_rules`](#opensmartrouteconfig) | Load a rules file (top-level list or ``rules:`` key) into a :class:`RulesStrategy`. | | `load_targets` | re-export of [`opensmartroute.config.load_targets`](#opensmartrouteconfig) | Load a catalogue file (top-level list or ``targets:`` key) into a :class:`TargetRegistry`. | | `middleware` | re-export of [`opensmartroute.sdk.middleware`](#opensmartroutesdk) | @middleware: register a Middleware class or ``fn(request, next_route)``. | | `policy_rule` | re-export of [`opensmartroute.sdk.policy_rule`](#opensmartroutesdk) | @policy_rule: register ``fn(target, request, signals) -> reason \| None``. | | `signal` | re-export of [`opensmartroute.sdk.signal`](#opensmartroutesdk) | @signal: register a SignalExtractor class or ``fn(request, signals) -> mapping``. | | `skill` | re-export of [`opensmartroute.sdk.skill`](#opensmartroutesdk) | @skill(id, ...): shorthand for a ``skill`` target. | | `strategy` | re-export of [`opensmartroute.sdk.strategy`](#opensmartroutesdk) | @strategy(weight=, name=): register a Strategy class or ``fn(request, signals, candidates)``. | | `target` | re-export of [`opensmartroute.sdk.target`](#opensmartroutesdk) | @target(id, kind, ...): the decorated callable becomes a RouteTarget handler. | | `telemetry` | re-export of [`opensmartroute.sdk.telemetry`](#opensmartroutesdk) | @telemetry: register a Telemetry sink class or factory. | | `tool` | re-export of [`opensmartroute.sdk.tool`](#opensmartroutesdk) | @tool(id, ...): shorthand for a ``tool`` target. | ## `opensmartroute.adapters` Source: [src/opensmartroute/adapters/__init__.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/__init__.py) Adapters connect OpenSmartRoute to real providers and infrastructure. | Name | Kind | Summary | |---|---|---| | `AgentHarness` | re-export of [`opensmartroute.adapters.harness.AgentHarness`](#opensmartrouteadaptersharness) | Structural interface of an agent runtime: ``run(task, context=, history=) -> HarnessResult``. | | `CallableHarness` | re-export of [`opensmartroute.adapters.harness.CallableHarness`](#opensmartrouteadaptersharness) | Wrap an in-process agent: ``fn(task, context, history) -> str \| dict \| HarnessResult``. | | `ChatResult` | re-export of [`opensmartroute.adapters.openai_compat.ChatResult`](#opensmartrouteadaptersopenai_compat) | Result of a chat completion: text, model, token counts, latency and the raw response. | | `HTTPHarness` | re-export of [`opensmartroute.adapters.harness.HTTPHarness`](#opensmartrouteadaptersharness) | POST the task as JSON to an agent endpoint. | | `HarnessResult` | re-export of [`opensmartroute.adapters.harness.HarnessResult`](#opensmartrouteadaptersharness) | What an agent harness returns: text, success, token / cost / latency usage and optional self-graded quality. | | `InMemoryQueue` | re-export of [`opensmartroute.adapters.handlers.InMemoryQueue`](#opensmartrouteadaptershandlers) | Reference :class:`Queue`: FIFO in memory, resolves into :class:`~opensmartroute.Outcome`. | | `ModelCard` | re-export of [`opensmartroute.adapters.catalogue.ModelCard`](#opensmartrouteadapterscatalogue) | What the catalogue knows about one model: identity, price, limits, evidence of quality, provenance. | | `ModelCatalogue` | re-export of [`opensmartroute.adapters.catalogue.ModelCatalogue`](#opensmartrouteadapterscatalogue) | Merged, persisted model cards from every source; the SLM's view of the target universe. | | `OpenAICompatClient` | re-export of [`opensmartroute.adapters.openai_compat.OpenAICompatClient`](#opensmartrouteadaptersopenai_compat) | Stdlib-only client for the OpenAI chat / embeddings API (OpenAI, Azure, vLLM, Ollama, LiteLLM) with retries. | | `OpenTelemetrySink` | re-export of [`opensmartroute.adapters.optional.OpenTelemetrySink`](#opensmartrouteadaptersoptional) | Live bridge from the tracer to OpenTelemetry: every OpenSmartRoute span becomes an OTel span. | | `OpenTelemetryTelemetry` | re-export of [`opensmartroute.adapters.optional.OpenTelemetryTelemetry`](#opensmartrouteadaptersoptional) | Emits one span per decision and counters/histograms via the OTel API. | | `PendingResult` | re-export of [`opensmartroute.adapters.handlers.PendingResult`](#opensmartrouteadaptershandlers) | Immediate answer of a queued target: the request was accepted and will be answered later. | | `Queue` | re-export of [`opensmartroute.adapters.handlers.Queue`](#opensmartrouteadaptershandlers) | Structural interface of an asynchronous queue target (ticketing, human tier, workflow run). | | `QueuedItem` | re-export of [`opensmartroute.adapters.handlers.QueuedItem`](#opensmartrouteadaptershandlers) | One request waiting in (or resolved from) a queue. | | `SearchHit` | re-export of [`opensmartroute.adapters.websearch.SearchHit`](#opensmartrouteadapterswebsearch) | One search result: where it came from, what it says, and when it was seen. | | `SemanticRouterImport` | re-export of [`opensmartroute.adapters.semantic_router.SemanticRouterImport`](#opensmartrouteadapterssemantic_router) | Result of importing a vLLM semantic-router config: registry, rules, default model, categories, warnings. | | `ServerCard` | re-export of [`opensmartroute.adapters.mcp_servers.ServerCard`](#opensmartrouteadaptersmcp_servers) | Description of an MCP server (tools, tags, auth, latency, cost, region, data boundary) for recommendation. | | `ServerRecommendation` | re-export of [`opensmartroute.adapters.mcp_servers.ServerRecommendation`](#opensmartrouteadaptersmcp_servers) | A ranked server from ``recommend_servers`` with its fused score, rationale and matched tool names. | | `StdioMCPClient` | re-export of [`opensmartroute.adapters.mcp.StdioMCPClient`](#opensmartrouteadaptersmcp) | Tiny JSON-RPC-over-stdio MCP client (newline-delimited). Thread-safe, blocking. | | `SubprocessHarness` | re-export of [`opensmartroute.adapters.harness.SubprocessHarness`](#opensmartrouteadaptersharness) | Run a CLI agent: task on stdin, answer on stdout, exit code 0 = success. | | `WebKnowledge` | re-export of [`opensmartroute.adapters.websearch.WebKnowledge`](#opensmartrouteadapterswebsearch) | Fan a query out to search providers, de-duplicate by URL and cache the hits as JSON. | | `a2a_handler` | re-export of [`opensmartroute.adapters.a2a.a2a_handler`](#opensmartrouteadaptersa2a) | Return a handler that sends the request text to an A2A agent and returns its text. | | `agent_from_card` | re-export of [`opensmartroute.adapters.a2a.agent_from_card`](#opensmartrouteadaptersa2a) | Build an ``agent`` RouteTarget from an A2A agent card (name, skills, tags, input modes). | | `brave_search` | re-export of [`opensmartroute.adapters.websearch.brave_search`](#opensmartrouteadapterswebsearch) | Brave Search API web results; the key is read from ``BRAVE_API_KEY`` (or ``BRAVE_API_KEY_FILE``). | | `card_to_target` | re-export of [`opensmartroute.adapters.catalogue.card_to_target`](#opensmartrouteadapterscatalogue) | A :class:`RouteTarget` for a model card; risky third-party descriptions are replaced by the name. | | `attach_chat_handlers` | re-export of [`opensmartroute.adapters.openai_compat.attach_chat_handlers`](#opensmartrouteadaptersopenai_compat) | Give every LLM target a :func:`chat_handler` on ``client``; returns the ids that got one. | | `chat_handler` | re-export of [`opensmartroute.adapters.openai_compat.chat_handler`](#opensmartrouteadaptersopenai_compat) | Adapter for ``RouteTarget.handler``: turns a RouteRequest into a chat call. | | `connect_mcp` | re-export of [`opensmartroute.adapters.mcp.connect_mcp`](#opensmartrouteadaptersmcp) | Spawn a stdio MCP server, list its tools and return ``(client, targets)``. | | `duckduckgo_search` | re-export of [`opensmartroute.adapters.websearch.duckduckgo_search`](#opensmartrouteadapterswebsearch) | DuckDuckGo instant-answer API (abstract + related topics). Keyless; shallow but good for definitions. | | `embedder` | re-export of [`opensmartroute.adapters.openai_compat.embedder`](#opensmartrouteadaptersopenai_compat) | Adapter for ``SimilarityStrategy(embedder=...)``. | | `enrich_description` | re-export of [`opensmartroute.adapters.mcp.enrich_description`](#opensmartrouteadaptersmcp) | Return ``(routing_description, examples, capabilities)`` for a tool. | | `fetch_agent_card` | re-export of [`opensmartroute.adapters.a2a.fetch_agent_card`](#opensmartrouteadaptersa2a) | Download an agent card over HTTPS (plain HTTP is refused); raises TargetUnavailableError on failure. | | `fetch_bytes` | re-export of [`opensmartroute.adapters.websearch.fetch_bytes`](#opensmartrouteadapterswebsearch) | GET ``url`` over https with a timeout and a body cap; transport errors become ``TargetUnavailableError``. | | `fetch_huggingface_models` | re-export of [`opensmartroute.adapters.catalogue.fetch_huggingface_models`](#opensmartrouteadapterscatalogue) | Model cards from the Hugging Face Hub search (downloads, likes, tags, ``model-index`` benchmarks). | | `fetch_json` | re-export of [`opensmartroute.adapters.websearch.fetch_json`](#opensmartrouteadapterswebsearch) | GET a JSON document (see :func:`fetch_bytes`); malformed bodies raise ``TargetUnavailableError``. | | `fetch_leaderboard_quality` | re-export of [`opensmartroute.adapters.catalogue.fetch_leaderboard_quality`](#opensmartrouteadapterscatalogue) | ``{hub model id: {benchmark: accuracy}}`` from the Open LLM Leaderboard table (official rows, unflagged). | | `fetch_openrouter_models` | re-export of [`opensmartroute.adapters.catalogue.fetch_openrouter_models`](#opensmartrouteadapterscatalogue) | Model cards from OpenRouter's public listing (prices per token, context, modalities, tool support). | | `fetch_page_text` | re-export of [`opensmartroute.adapters.websearch.fetch_page_text`](#opensmartrouteadapterswebsearch) | Fetch a page and return ``{url, title, text, risk}``; ``risk`` is the injection/gadget risk of the text. | | `harness_handler` | re-export of [`opensmartroute.adapters.harness.harness_handler`](#opensmartrouteadaptersharness) | Adapter for ``RouteTarget.handler``: a RouteRequest becomes a harness task. | | `html_to_text` | re-export of [`opensmartroute.adapters.websearch.html_to_text`](#opensmartrouteadapterswebsearch) | ``(title, text)`` of an HTML document with scripts/styles removed and whitespace collapsed. | | `http_handler` | re-export of [`opensmartroute.adapters.handlers.http_handler`](#opensmartrouteadaptershandlers) | ``RouteTarget.handler`` that POSTs the request to ``url`` and returns a :class:`HarnessResult`. | | `huggingface_search` | re-export of [`opensmartroute.adapters.websearch.huggingface_search`](#opensmartrouteadapterswebsearch) | Search the Hugging Face Hub (``what`` = ``models`` or ``datasets``), ranked by downloads. No key needed. | | `judge_fn` | re-export of [`opensmartroute.adapters.openai_compat.judge_fn`](#opensmartrouteadaptersopenai_compat) | Adapter for ``LLMJudgeStrategy(llm=...)``: prompt in, completion text out. | | `langgraph_condition` | re-export of [`opensmartroute.adapters.frameworks.langgraph_condition`](#opensmartrouteadaptersframeworks) | Edge selector for ``add_conditional_edges``: routes on target id (``by='target'``) or kind. | | `langgraph_node` | re-export of [`opensmartroute.adapters.frameworks.langgraph_node`](#opensmartrouteadaptersframeworks) | Return a LangGraph-compatible node ``state -> dict`` (partial state update). | | `load_personas` | re-export of [`opensmartroute.adapters.personas.load_personas`](#opensmartrouteadapterspersonas) | Load personas from a directory of markdown files or a JSON/JSONL/CSV catalogue. | | `load_semantic_router_config` | re-export of [`opensmartroute.adapters.semantic_router.load_semantic_router_config`](#opensmartrouteadapterssemantic_router) | Convert a vLLM semantic-router ``model_config`` / ``categories`` document into targets and rules. | | `load_skill` | re-export of [`opensmartroute.adapters.skills.load_skill`](#opensmartrouteadaptersskills) | Load one skill directory (must contain ``SKILL.md``). | | `load_skills` | re-export of [`opensmartroute.adapters.skills.load_skills`](#opensmartrouteadaptersskills) | Load every ``*/SKILL.md`` under ``root`` (one level deep, sorted by name). | | `maf_router_executor` | re-export of [`opensmartroute.adapters.frameworks.maf_router_executor`](#opensmartrouteadaptersframeworks) | Agent Framework style: ``executor(message, ctx) -> target id \| response``; ``handoffs`` maps. | | `manifest_from_targets` | re-export of [`opensmartroute.adapters.mcp.manifest_from_targets`](#opensmartrouteadaptersmcp) | Reverse: dump MCP-shaped tool dicts (for signing / publishing a catalogue). | | `model_key` | re-export of [`opensmartroute.adapters.catalogue.model_key`](#opensmartrouteadapterscatalogue) | Vendor-agnostic key for matching model names across sources: lowercase, no vendor prefix, no punctuation. | | `mcp_tool_handler` | re-export of [`opensmartroute.adapters.handlers.mcp_tool_handler`](#opensmartrouteadaptershandlers) | ``RouteTarget.handler`` that invokes one MCP tool via ``call(name, arguments)``. | | `openai_tool_handler` | re-export of [`opensmartroute.adapters.frameworks.openai_tool_handler`](#opensmartrouteadaptersframeworks) | Callable behind ``openai_tool_spec``: ``(text, objective?, kinds?) -> RouteDecision.to_dict()``. | | `openai_tool_spec` | re-export of [`opensmartroute.adapters.frameworks.openai_tool_spec`](#opensmartrouteadaptersframeworks) | OpenAI function-calling tool definition that lets a model ask the router for a target. | | `persona_from_markdown` | re-export of [`opensmartroute.adapters.personas.persona_from_markdown`](#opensmartrouteadapterspersonas) | Parse a ``*.agent.md`` / ``*.chatmode.md`` / front-matter markdown file into a persona target. | | `persona_target` | re-export of [`opensmartroute.adapters.personas.persona_target`](#opensmartrouteadapterspersonas) | Build a ``persona`` RouteTarget whose ``instructions`` is the system prompt (non-primary by default). | | `personas_from_records` | re-export of [`opensmartroute.adapters.personas.personas_from_records`](#opensmartrouteadapterspersonas) | Persona targets from JSON / CSV-style records (``name`` + ``prompt``/``system`` keys); others skipped. | | `quality_from_benchmarks` | re-export of [`opensmartroute.adapters.catalogue.quality_from_benchmarks`](#opensmartrouteadapterscatalogue) | Mean of normalised benchmark scores (percentages are divided by 100); None when there are none. | | `quality_from_popularity` | re-export of [`opensmartroute.adapters.catalogue.quality_from_popularity`](#opensmartrouteadapterscatalogue) | Weak prior in [0.35, 0.75] from log-scaled downloads and likes (popularity is not quality; it is a hint). | | `queue_handler` | re-export of [`opensmartroute.adapters.handlers.queue_handler`](#opensmartrouteadaptershandlers) | ``RouteTarget.handler`` that enqueues the request and returns a :class:`PendingResult`. | | `recommend_servers` | re-export of [`opensmartroute.adapters.mcp_servers.recommend_servers`](#opensmartrouteadaptersmcp_servers) | Rank MCP servers for a task; constraint violations are excluded, not down-weighted. | | `route_and_execute` | re-export of [`opensmartroute.adapters.frameworks.route_and_execute`](#opensmartrouteadaptersframeworks) | Route ``text`` and optionally execute the plan; returns ``(decision, ExecutionResult \| None)``. | | `sentence_transformers_embedder` | re-export of [`opensmartroute.adapters.optional.sentence_transformers_embedder`](#opensmartrouteadaptersoptional) | Semantic embedder for ``SimilarityStrategy``. Requires ``opensmartroute[embeddings]``. | | `sign_manifest` | re-export of [`opensmartroute.adapters.mcp.sign_manifest`](#opensmartrouteadaptersmcp) | Wrap a tool list in a signed manifest. ``algorithm`` = ``hmac-sha256`` (key = shared. | | `skill_from_markdown` | re-export of [`opensmartroute.adapters.skills.skill_from_markdown`](#opensmartrouteadaptersskills) | Parse one SKILL.md (Agent-Skills front matter + body) into a ``skill`` RouteTarget; strict validation. | | `skills_from_card` | re-export of [`opensmartroute.adapters.a2a.skills_from_card`](#opensmartrouteadaptersa2a) | One non-primary ``skill`` target per skill declared on an A2A agent card (``/``). | | `tools_from_manifest` | re-export of [`opensmartroute.adapters.mcp.tools_from_manifest`](#opensmartrouteadaptersmcp) | Import tools from a (signed) manifest; ``kw`` goes to :func:`tools_from_mcp`. | | `tools_from_mcp` | re-export of [`opensmartroute.adapters.mcp.tools_from_mcp`](#opensmartrouteadaptersmcp) | Convert ``tools/list`` output into targets. ``call(name, arguments)`` becomes the handler. | | `verify_manifest` | re-export of [`opensmartroute.adapters.mcp.verify_manifest`](#opensmartrouteadaptersmcp) | Constant-time verification. ``key`` = shared secret (HMAC) or 32-byte Ed25519 public key. | ## `opensmartroute.adapters.a2a` Source: [src/opensmartroute/adapters/a2a.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/a2a.py) Import A2A (Agent-to-Agent protocol) **Agent Cards** as ``TargetKind.AGENT`` targets. | Name | Kind | Summary | |---|---|---| | `a2a_handler` | function `(url: str, token: str \| None=None, timeout_s: float=60.0, method: str='message/send')` | Return a handler that sends the request text to an A2A agent and returns its text. | | `agent_from_card` | function `(card: dict[str, Any], *, call: Callable[[str, RouteRequest], Any] \| None=None, cost_per_1k_tokens: float=0.0, latency_ms: float=2000.0, quality_prior: float=0.65, languages: list[str] \| None=None)` | Build an ``agent`` RouteTarget from an A2A agent card (name, skills, tags, input modes). | | `fetch_agent_card` | function `(base_url: str, timeout_s: float=10.0, path: str='/.well-known/agent.json')` | Download an agent card over HTTPS (plain HTTP is refused); raises TargetUnavailableError on failure. | | `skills_from_card` | function `(card: dict[str, Any], agent_id: str \| None=None)` | One non-primary ``skill`` target per skill declared on an A2A agent card (``/``). | ## `opensmartroute.adapters.catalogue` Source: [src/opensmartroute/adapters/catalogue.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/catalogue.py) Live model catalogue: collect model cards (price, context, modalities, benchmarks) from public sources. | Name | Kind | Summary | |---|---|---| | `LEADERBOARD_DATASET` | constant | Open LLM Leaderboard v2 results table on the Hub. | | `OPENROUTER_MODELS` | constant | public model + pricing listing (no key). | | `ModelCard` | class | What the catalogue knows about one model: identity, price, limits, evidence of quality, provenance. | | `ModelCatalogue` | class | Merged, persisted model cards from every source; the SLM's view of the target universe. | | `card_to_target` | function `(card: ModelCard, *, default_latency_ms: float=1500.0, settings: Settings \| None=None)` | A :class:`RouteTarget` for a model card; risky third-party descriptions are replaced by the name. | | `fetch_huggingface_models` | function `(query: str='', *, limit: int=50, pipeline: str='text-generation', timeout_s: float \| None=None, settings: Settings \| None=None)` | Model cards from the Hugging Face Hub search (downloads, likes, tags, ``model-index`` benchmarks). | | `fetch_leaderboard_quality` | function `(*, limit: int=5000, dataset: str=LEADERBOARD_DATASET, timeout_s: float \| None=None, settings: Settings \| None=None)` | ``{hub model id: {benchmark: accuracy}}`` from the Open LLM Leaderboard table (official rows, unflagged). | | `fetch_openrouter_models` | function `(*, url: str=OPENROUTER_MODELS, timeout_s: float \| None=None, settings: Settings \| None=None)` | Model cards from OpenRouter's public listing (prices per token, context, modalities, tool support). | | `model_key` | function `(model_id: str)` | Vendor-agnostic key for matching model names across sources: lowercase, no vendor prefix, no punctuation. | | `quality_from_benchmarks` | function `(benchmarks: dict[str, float])` | Mean of normalised benchmark scores (percentages are divided by 100); None when there are none. | | `quality_from_popularity` | function `(downloads: int, likes: int)` | Weak prior in [0.35, 0.75] from log-scaled downloads and likes (popularity is not quality; it is a hint). | ## `opensmartroute.adapters.frameworks` Source: [src/opensmartroute/adapters/frameworks.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/frameworks.py) Drop-in nodes for agent frameworks. | Name | Kind | Summary | |---|---|---| | `langgraph_condition` | function `(by: str='target', default: str='__end__')` | Edge selector for ``add_conditional_edges``: routes on target id (``by='target'``) or kind. | | `langgraph_node` | function `(router: Router, *, execute: bool=False, plan: bool=False, context_key: str='route_context', on_no_route: str='__no_route__')` | Return a LangGraph-compatible node ``state -> dict`` (partial state update). | | `last_user_text` | function `(state: dict[str, Any])` | (last user utterance, prior history as role/content dicts). | | `maf_router_executor` | function `(router: Router, *, execute: bool=False)` | Agent Framework style: ``executor(message, ctx) -> target id \| response``; ``handoffs`` maps. | | `openai_tool_handler` | function `(router: Router)` | Callable behind ``openai_tool_spec``: ``(text, objective?, kinds?) -> RouteDecision.to_dict()``. | | `openai_tool_spec` | function `(name: str='route_request')` | OpenAI function-calling tool definition that lets a model ask the router for a target. | | `route_and_execute` | function `(router: Router, text: str, *, history: list[dict[str, str]] \| None=None, context: dict[str, Any] \| None=None, objective: Objective \| None=None, execute: bool=False, plan: bool=False, learn: bool=True)` | Route ``text`` and optionally execute the plan; returns ``(decision, ExecutionResult \| None)``. | ## `opensmartroute.adapters.handlers` Source: [src/opensmartroute/adapters/handlers.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/handlers.py) Executors for non-LLM targets: HTTP endpoints, MCP tools and asynchronous queues. | Name | Kind | Summary | |---|---|---| | `InMemoryQueue` | class | Reference :class:`Queue`: FIFO in memory, resolves into :class:`~opensmartroute.Outcome`. | | `PendingResult` | class | Immediate answer of a queued target: the request was accepted and will be answered later. | | `Queue` | class | Structural interface of an asynchronous queue target (ticketing, human tier, workflow run). | | `QueuedItem` | class | One request waiting in (or resolved from) a queue. | | `http_handler` | function `(url: str, *, api_key: str \| None=None, api_key_env: str \| None=None, timeout_s: float=300.0, headers: Mapping[str, str] \| None=None, payload: Callable[[str, Mapping[str, Any], list[dict[str, str]]], dict[str, Any]] \| None=None, body: Mapping[str, Any] \| None=None)` | ``RouteTarget.handler`` that POSTs the request to ``url`` and returns a :class:`HarnessResult`. | | `mcp_tool_handler` | function `(call: Callable[[str, dict[str, Any]], Any], tool: str, input_schema: dict[str, Any] \| None=None, arguments: Callable[[RouteRequest], dict[str, Any]] \| None=None)` | ``RouteTarget.handler`` that invokes one MCP tool via ``call(name, arguments)``. | | `queue_handler` | function `(queue: Queue, target_id: str \| None=None)` | ``RouteTarget.handler`` that enqueues the request and returns a :class:`PendingResult`. | ## `opensmartroute.adapters.harness` Source: [src/opensmartroute/adapters/harness.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/harness.py) Agent-harness adapters: route to a *runtime*, not just a model. | Name | Kind | Summary | |---|---|---| | `AgentHarness` | class | Structural interface of an agent runtime: ``run(task, context=, history=) -> HarnessResult``. | | `CallableHarness` | class | Wrap an in-process agent: ``fn(task, context, history) -> str \| dict \| HarnessResult``. | | `HTTPHarness` | class | POST the task as JSON to an agent endpoint. | | `HarnessResult` | class | What an agent harness returns: text, success, token / cost / latency usage and optional self-graded quality. | | `SubprocessHarness` | class | Run a CLI agent: task on stdin, answer on stdout, exit code 0 = success. | | `coerce_result` | function `(out: Any)` | Normalise ``str \| dict \| HarnessResult \| object-with-text`` into a HarnessResult. | | `harness_handler` | function `(harness: AgentHarness)` | Adapter for ``RouteTarget.handler``: a RouteRequest becomes a harness task. | ## `opensmartroute.adapters.mcp` Source: [src/opensmartroute/adapters/mcp.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/mcp.py) Import MCP (Model Context Protocol) tools as ``TargetKind.TOOL`` targets. | Name | Kind | Summary | |---|---|---| | `MCPTool` | class | A tool as listed by an MCP server: name, description, JSON input schema and annotations. | | `StdioMCPClient` | class | Tiny JSON-RPC-over-stdio MCP client (newline-delimited). Thread-safe, blocking. | | `canonical_json` | function `(obj: Any)` | Deterministic JSON encoding (sorted keys, no whitespace) used for manifest digests and signatures. | | `connect_mcp` | function `(command: list[str], server: str='', **kw: Any)` | Spawn a stdio MCP server, list its tools and return ``(client, targets)``. | | `enrich_description` | function `(name: str, description: str, input_schema: dict[str, Any] \| None, server: str='', enricher: Callable[[str], list[str]] \| None=None)` | Return ``(routing_description, examples, capabilities)`` for a tool. | | `manifest_digest` | function `(tools: list[dict[str, Any]])` | SHA-256 hex digest of a tool list in canonical JSON. | | `manifest_from_targets` | function `(targets: Iterable[RouteTarget])` | Reverse: dump MCP-shaped tool dicts (for signing / publishing a catalogue). | | `sign_manifest` | function `(tools: list[dict[str, Any]], key: bytes, *, server: str='', algorithm: str='hmac-sha256', key_id: str='')` | Wrap a tool list in a signed manifest. ``algorithm`` = ``hmac-sha256`` (key = shared. | | `tools_from_manifest` | function `(manifest: dict[str, Any], key: bytes \| None, *, require_signature: bool=True, max_age_s: float \| None=None, **kw: Any)` | Import tools from a (signed) manifest; ``kw`` goes to :func:`tools_from_mcp`. | | `tools_from_mcp` | function `(payload: Any, server: str='', call: Callable[[str, dict[str, Any]], Any] \| None=None, enricher: Callable[[str], list[str]] \| None=None, cost_per_call_usd: float=0.0, latency_ms: float=300.0, quality_prior: float=0.6, max_description_risk: float=0.6, guard: InputGuard \| None=None)` | Convert ``tools/list`` output into targets. ``call(name, arguments)`` becomes the handler. | | `verify_manifest` | function `(manifest: dict[str, Any], key: bytes, max_age_s: float \| None=None)` | Constant-time verification. ``key`` = shared secret (HMAC) or 32-byte Ed25519 public key. | ## `opensmartroute.adapters.mcp_servers` Source: [src/opensmartroute/adapters/mcp_servers.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/mcp_servers.py) MCP *server* recommendation (MCP-Zero 2506.01056; ToolRet 2603.06467). | Name | Kind | Summary | |---|---|---| | `ServerCard` | class | Description of an MCP server (tools, tags, auth, latency, cost, region, data boundary) for recommendation. | | `ServerRecommendation` | class | A ranked server from ``recommend_servers`` with its fused score, rationale and matched tool names. | | `recommend_servers` | function `(request: RouteRequest \| str, servers: Iterable[ServerCard \| dict[str, Any]], k: int=3, constraints: RequestConstraints \| None=None, allowed_auth: Iterable[str] \| None=None, min_score: float=0.0)` | Rank MCP servers for a task; constraint violations are excluded, not down-weighted. | ## `opensmartroute.adapters.openai_compat` Source: [src/opensmartroute/adapters/openai_compat.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/openai_compat.py) OpenAI-compatible HTTP client (stdlib only). | Name | Kind | Summary | |---|---|---| | `ChatResult` | class | Result of a chat completion: text, model, token counts, latency and the raw response. | | `OpenAICompatClient` | class | Stdlib-only client for the OpenAI chat / embeddings API (OpenAI, Azure, vLLM, Ollama, LiteLLM) with retries. | | `attach_chat_handlers` | function `(targets: Iterable[Any], client: OpenAICompatClient, *, model_key: str='model', default_model: str \| None=None, overwrite: bool=False)` | Give every LLM target a :func:`chat_handler` on ``client``; returns the ids that got one. | | `chat_handler` | function `(client: OpenAICompatClient, model: str, system_prompt: str \| None=None, **defaults: Any)` | Adapter for ``RouteTarget.handler``: turns a RouteRequest into a chat call. | | `embedder` | function `(client: OpenAICompatClient, model: str, batch: int=64)` | Adapter for ``SimilarityStrategy(embedder=...)``. | | `judge_fn` | function `(client: OpenAICompatClient, model: str, max_tokens: int=400)` | Adapter for ``LLMJudgeStrategy(llm=...)``: prompt in, completion text out. | ## `opensmartroute.adapters.optional` Source: [src/opensmartroute/adapters/optional.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/optional.py) Optional adapters that need extra dependencies. Everything is lazily imported so the. | Name | Kind | Summary | |---|---|---| | `OpenTelemetrySink` | class | Live bridge from the tracer to OpenTelemetry: every OpenSmartRoute span becomes an OTel span. | | `OpenTelemetryTelemetry` | class | Emits one span per decision and counters/histograms via the OTel API. | | `sentence_transformers_embedder` | function `(model_name: str='sentence-transformers/all-MiniLM-L6-v2', device: str \| None=None, normalize: bool=True)` | Semantic embedder for ``SimilarityStrategy``. Requires ``opensmartroute[embeddings]``. | ## `opensmartroute.adapters.personas` Source: [src/opensmartroute/adapters/personas.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/personas.py) Import persona catalogues as ``TargetKind.PERSONA`` targets. | Name | Kind | Summary | |---|---|---| | `load_personas` | function `(path: str \| Path, **kw: Any)` | Load personas from a directory of markdown files or a JSON/JSONL/CSV catalogue. | | `persona_from_markdown` | function `(text: str, *, path: Path \| None=None, **kw: Any)` | Parse a ``*.agent.md`` / ``*.chatmode.md`` / front-matter markdown file into a persona target. | | `persona_target` | function `(name: str, prompt: str, description: str='', *, id_prefix: str='persona:', domains: list[str] \| None=None, tags: list[str] \| None=None, languages: list[str] \| None=None, primary: bool=False, quality_prior: float=0.6, source: str='', extra: dict[str, Any] \| None=None)` | Build a ``persona`` RouteTarget whose ``instructions`` is the system prompt (non-primary by default). | | `personas_from_records` | function `(records: Iterable[dict[str, Any]], **kw: Any)` | Persona targets from JSON / CSV-style records (``name`` + ``prompt``/``system`` keys); others skipped. | ## `opensmartroute.adapters.semantic_router` Source: [src/opensmartroute/adapters/semantic_router.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/semantic_router.py) Import a vLLM *semantic-router* configuration (vllm-project/semantic-router). | Name | Kind | Summary | |---|---|---| | `DEFAULT_DOMAIN_MAP` | constant | semantic-router's MMLU-style categories -> OpenSmartRoute ontology domains. | | `SemanticRouterImport` | class | Result of importing a vLLM semantic-router config: registry, rules, default model, categories, warnings. | | `load_semantic_router_config` | function `(source: str \| Path \| dict[str, Any], domain_map: dict[str, list[str]] \| None=None, rule_weight: float=0.8)` | Convert a vLLM semantic-router ``model_config`` / ``categories`` document into targets and rules. | ## `opensmartroute.adapters.skills` Source: [src/opensmartroute/adapters/skills.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/skills.py) Load Agent-Skills ``SKILL.md`` packages as ``TargetKind.SKILL`` targets. | Name | Kind | Summary | |---|---|---| | `load_skill` | function `(skill_dir: str \| Path, **kw: Any)` | Load one skill directory (must contain ``SKILL.md``). | | `load_skills` | function `(root: str \| Path, **kw: Any)` | Load every ``*/SKILL.md`` under ``root`` (one level deep, sorted by name). | | `parse_frontmatter` | function `(text: str)` | Split ``---`` frontmatter from the body. Returns ``(frontmatter, body)``. | | `skill_from_markdown` | function `(text: str, *, path: Path \| None=None, cost_per_1k_tokens: float=0.0)` | Parse one SKILL.md (Agent-Skills front matter + body) into a ``skill`` RouteTarget; strict validation. | ## `opensmartroute.adapters.websearch` Source: [src/opensmartroute/adapters/websearch.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/adapters/websearch.py) Web knowledge for the self-improving router: stdlib HTTP fetch, search providers, page text. | Name | Kind | Summary | |---|---|---| | `BRAVE_API` | constant | Brave web search (needs BRAVE_API_KEY). | | `DATASETS_SERVER` | constant | Hugging Face datasets-server REST root (/rows). | | `DDG_API` | constant | DuckDuckGo instant-answer JSON endpoint (no key). | | `HF_API` | constant | Hugging Face Hub REST root (models, datasets, search). | | `SearchHit` | class | One search result: where it came from, what it says, and when it was seen. | | `SearchProvider` | constant | (query, limit) -> hits. | | `WebKnowledge` | class | Fan a query out to search providers, de-duplicate by URL and cache the hits as JSON. | | `brave_search` | function `(query: str, limit: int=10, *, api_key_env: str='BRAVE_API_KEY', timeout_s: float \| None=None, settings: Settings \| None=None)` | Brave Search API web results; the key is read from ``BRAVE_API_KEY`` (or ``BRAVE_API_KEY_FILE``). | | `duckduckgo_search` | function `(query: str, limit: int=10, *, timeout_s: float \| None=None, settings: Settings \| None=None)` | DuckDuckGo instant-answer API (abstract + related topics). Keyless; shallow but good for definitions. | | `fetch_bytes` | function `(url: str, *, timeout_s: float \| None=None, max_bytes: int \| None=None, headers: Mapping[str, str] \| None=None, settings: Settings \| None=None)` | GET ``url`` over https with a timeout and a body cap; transport errors become ``TargetUnavailableError``. | | `fetch_json` | function `(url: str, *, timeout_s: float \| None=None, headers: Mapping[str, str] \| None=None, settings: Settings \| None=None)` | GET a JSON document (see :func:`fetch_bytes`); malformed bodies raise ``TargetUnavailableError``. | | `fetch_page_text` | function `(url: str, *, max_chars: int=20000, timeout_s: float \| None=None, settings: Settings \| None=None)` | Fetch a page and return ``{url, title, text, risk}``; ``risk`` is the injection/gadget risk of the text. | | `html_to_text` | function `(html: str)` | ``(title, text)`` of an HTML document with scripts/styles removed and whitespace collapsed. | | `huggingface_search` | function `(query: str, limit: int=10, *, what: str='models', pipeline_tag: str \| None='text-generation', timeout_s: float \| None=None, settings: Settings \| None=None)` | Search the Hugging Face Hub (``what`` = ``models`` or ``datasets``), ranked by downloads. No key needed. | ## `opensmartroute.aio` Source: [src/opensmartroute/aio.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/aio.py) Async façade. Routing itself is CPU-bound and sub-millisecond, so we run it in. | Name | Kind | Summary | |---|---|---| | `AsyncRouter` | class | ``await``-able wrapper around a :class:`Router`: route / learn / execute / run off the event loop. | ## `opensmartroute.branding` Source: [src/opensmartroute/branding.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/branding.py) OpenSmartRoute naming conventions: one place for every brand-bound identifier. | Name | Kind | Summary | |---|---|---| | `API_KEY_ENV` | constant | ``OSR_API_KEY`` - access token the CLI sends (overrides the credentials file). | | `API_URL_ENV` | constant | ``OSR_API_URL`` - platform / server URL the CLI talks to. | | `BRAND` | constant | product name, one word. | | `CLI` | constant | console script name. | | `CONFIG_DIR_ENV` | constant | ``OSR_CONFIG_DIR`` - overrides the per-user config folder. | | `CONFIG_DIR_NAME` | constant | per-user config folder name (``~/.config/opensmartroute``, ``%APPDATA%\opensmartroute``). | | `ENTRY_POINT_GROUP` | constant | importlib.metadata entry-point group for plugins. | | `ENV_PREFIX` | constant | ``OSR_`` - every settings environment variable starts with this. | | `ERROR_CODE_PREFIX` | constant | ``OSR_`` - machine-readable error codes (``OSR_NO_ROUTE``). | | `INSTALL_SCRIPT_PS1` | constant | Windows installer: ``irm .../install.ps1 \| iex``. | | `INSTALL_SCRIPT_SH` | constant | Linux / macOS installer: ``curl -fsSL .../install.sh \| sh``. | | `LOCAL_TOKEN_PREFIX` | constant | self-hosted ``osr serve`` access tokens (``osr_local_...``). | | `METADATA_PREFIX` | constant | ``osr-`` - SKILL.md / persona front-matter keys (``osr-domains``). | | `PACKAGE` | constant | Python package / distribution / logger root. | | `REPOSITORY` | constant | source repository (installer fallback URLs). | | `SHORT_NAME` | constant | short form used for env-var and error-code prefixes. | | `SKILLS_DIR` | constant | : Default Agent-Skills root (``*/SKILL.md``); the location Claude Code discovers project skills in. | | `STATE_DIR` | constant | default learner-state directory. | | `TOKEN_PREFIX` | constant | hosted-platform API keys (``osr_live_...``). | | `WEBSITE` | constant | public website; also the default hosted-platform URL of ``osr login``. | | `env_key` | function `(*parts: str)` | ``env_key("routing", "softmax_temperature") -> "OSR_ROUTING_SOFTMAX_TEMPERATURE"``. | | `error_code` | function `(kind: str)` | ``error_code("no_route") -> "OSR_NO_ROUTE"``. | | `logger` | function `(component: str \| None=None)` | ``logger() -> "opensmartroute"``; ``logger("enterprise") -> "opensmartroute.enterprise"``. | | `metadata_key` | function `(field: str)` | ``metadata_key("quality_prior") -> "osr-quality-prior"`` (SKILL.md / persona frontmatter). | | `platform_url` | function `(environ: Mapping[str, str] \| None=None)` | The platform / server URL the CLI talks to: ``OSR_API_URL`` or the hosted platform (:data:`WEBSITE`). | | `user_agent` | function `()` | ``"opensmartroute/"`` for outbound HTTP clients. | | `version` | function `()` | The package version (``opensmartroute.__version__``), resolved at call time. | ## `opensmartroute.cli` Source: [src/opensmartroute/cli.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/cli.py) ``osr`` command-line interface. | Name | Kind | Summary | |---|---|---| | `build_router` | function `(targets: str, rules: str \| None=None, state: str \| None=None, models: str \| None=None, skills: str \| None=None, slm: str \| None=None, remember_requests: int=0)` | Assemble the CLI's router from targets / rules / state / models / SKILL.md / SLM paths (every command). | | `main` | function `(argv: list[str] \| None=None)` | Entry point of the ``osr`` console script; returns the process exit status. | ## `opensmartroute.config` Source: [src/opensmartroute/config.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/config.py) Configuration loading: targets and rules from JSON or YAML. | Name | Kind | Summary | |---|---|---| | `load_document` | function `(path: str \| Path)` | Read a JSON or YAML file (YAML needs the ``yaml`` extra); raises ConfigurationError on any problem. | | `load_rules` | function `(path: str \| Path)` | Load a rules file (top-level list or ``rules:`` key) into a :class:`RulesStrategy`. | | `load_targets` | function `(path: str \| Path)` | Load a catalogue file (top-level list or ``targets:`` key) into a :class:`TargetRegistry`. | ## `opensmartroute.core` Source: [src/opensmartroute/core/__init__.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/core/__init__.py) | Name | Kind | Summary | |---|---|---| | `Capabilities` | re-export of [`opensmartroute.core.types.Capabilities`](#opensmartroutecoretypes) | Declarative description of what a target is good at. | | `Objective` | re-export of [`opensmartroute.core.types.Objective`](#opensmartroutecoretypes) | What the caller wants to optimise. Weights are relative. | | `Outcome` | re-export of [`opensmartroute.core.types.Outcome`](#opensmartroutecoretypes) | Feedback about how a routed request actually went. | | `PlanSlot` | re-export of [`opensmartroute.core.types.PlanSlot`](#opensmartroutecoretypes) | One filled slot of a multi-target plan (persona -> skill -> model). | | `RankedTarget` | re-export of [`opensmartroute.core.types.RankedTarget`](#opensmartroutecoretypes) | A scored candidate: utility, ensemble quality estimate and the per-strategy breakdown. | | `RequestConstraints` | re-export of [`opensmartroute.core.types.RequestConstraints`](#opensmartroutecoretypes) | Hard constraints on the request (never traded off). | | `RouteDecision` | re-export of [`opensmartroute.core.types.RouteDecision`](#opensmartroutecoretypes) | The answer to ``route()``: chosen target, confidence, alternatives, optional plan, trace and propensities. | | `RoutePlan` | re-export of [`opensmartroute.core.types.RoutePlan`](#opensmartroutecoretypes) | A composed route (MasRouter-style): several targets working together. | | `RouteRequest` | re-export of [`opensmartroute.core.types.RouteRequest`](#opensmartroutecoretypes) | The customer need. | | `RouteTarget` | re-export of [`opensmartroute.core.types.RouteTarget`](#opensmartroutecoretypes) | A routable destination: an LLM, agent, skill, persona, tool, workflow or human. | | `RouteTrace` | re-export of [`opensmartroute.core.types.RouteTrace`](#opensmartroutecoretypes) | Everything needed to explain a decision. | | `Signals` | re-export of [`opensmartroute.core.types.Signals`](#opensmartroutecoretypes) | Cheap deterministic features extracted from a request. | | `StrategyScore` | re-export of [`opensmartroute.core.types.StrategyScore`](#opensmartroutecoretypes) | One strategy's opinion about one target. | | `TargetConstraints` | re-export of [`opensmartroute.core.types.TargetConstraints`](#opensmartroutecoretypes) | Where / for whom a target may be used. Checked by the policy layer. | | `TargetKind` | re-export of [`opensmartroute.core.types.TargetKind`](#opensmartroutecoretypes) | Kinds of things a request can be routed to. | | `TargetRegistry` | re-export of [`opensmartroute.core.registry.TargetRegistry`](#opensmartroutecoreregistry) | In-memory catalogue of :class:`RouteTarget` by id: add / upsert / remove, filtered listing, (de)serialisation. | ## `opensmartroute.core.registry` Source: [src/opensmartroute/core/registry.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/core/registry.py) Target registry: the catalogue of everything a request may be routed to. | Name | Kind | Summary | |---|---|---| | `TargetRegistry` | class | In-memory catalogue of :class:`RouteTarget` by id: add / upsert / remove, filtered listing, (de)serialisation. | ## `opensmartroute.core.types` Source: [src/opensmartroute/core/types.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/core/types.py) Core data model for OpenSmartRoute. | Name | Kind | Summary | |---|---|---| | `TargetKind` | class | Kinds of things a request can be routed to. | | `Capabilities` | class | Declarative description of what a target is good at. | | `TargetConstraints` | class | Where / for whom a target may be used. Checked by the policy layer. | | `RouteTarget` | class | A routable destination: an LLM, agent, skill, persona, tool, workflow or human. | | `EFFORT_LEVELS` | constant | Named reasoning-effort levels -> numeric effort in [0, 1] (``RouteTarget.effort``). | | `Objective` | class | What the caller wants to optimise. Weights are relative. | | `RequestConstraints` | class | Hard constraints on the request (never traded off). | | `RouteRequest` | class | The customer need. | | `Signals` | class | Cheap deterministic features extracted from a request. | | `StrategyScore` | class | One strategy's opinion about one target. | | `RankedTarget` | class | A scored candidate: utility, ensemble quality estimate and the per-strategy breakdown. | | `RouteTrace` | class | Everything needed to explain a decision. | | `PlanSlot` | class | One filled slot of a multi-target plan (persona -> skill -> model). | | `RoutePlan` | class | A composed route (MasRouter-style): several targets working together. | | `RouteDecision` | class | The answer to ``route()``: chosen target, confidence, alternatives, optional plan, trace and propensities. | | `Outcome` | class | Feedback about how a routed request actually went. | ## `opensmartroute.credentials` Source: [src/opensmartroute/credentials.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/credentials.py) Credentials for the ``osr`` CLI: where the access token lives and how it is obtained. | Name | Kind | Summary | |---|---|---| | `CREDENTIALS_FILE` | constant | file name inside :func:`config_dir`. | | `DEFAULT_PROFILE` | constant | profile used when ``--profile`` is not given. | | `DEVICE_CODE_PATH` | constant | RFC 8628 device-authorization endpoint of the platform. | | `DEVICE_TOKEN_PATH` | constant | noqa: S105 # nosec B105 - RFC 8628 token endpoint (a URL path). | | `ME_PATH` | constant | platform: who am I (workspace, plan, edition). | | `TOKEN_BYTES` | constant | entropy of :func:`generate_token`. | | `WHOAMI_PATH` | constant | self-hosted ``osr serve``: is this token accepted. | | `Credential` | class | One saved sign-in: where (``url``), what (``token``) and what the platform said about it. | | `CredentialStore` | class | Profiles in ``/credentials.json`` (owner-only permissions on POSIX). | | `PlatformClient` | class | Minimal JSON client for the platform / server API used by the CLI (stdlib only, injectable transport). | | `Transport` | constant | : ``(method, url, headers, body, timeout) -> (status, json)``; tests inject a fake instead of urllib. | | `apply_identity` | function `(cred: Credential, info: Mapping[str, Any])` | Copy what :func:`whoami` learned (kind, workspace, plan, edition) onto ``cred``. | | `config_dir` | function `(environ: Mapping[str, str] \| None=None)` | Per-user configuration directory (``OSR_CONFIG_DIR`` > ``%APPDATA%`` > ``$XDG_CONFIG_HOME`` > ``~/.config``). | | `device_login` | function `(url: str \| None=None, *, client_name: str \| None=None, open_browser: bool=True, out: Callable[[str], None]=print, transport: Transport \| None=None, sleep: Callable[[float], None]=time.sleep, timeout_s: float \| None=None)` | Sign in to the hosted platform with the device authorization grant and return the credential. | | `generate_token` | function `(prefix: str=LOCAL_TOKEN_PREFIX)` | A fresh random access token (``osr_local_<43 chars>``) for a self-hosted ``osr serve``. | | `redact` | function `(token: str, keep: int=6)` | ``osr_live_abc123...`` - the prefix plus a few characters, never the whole token. | | `token_kind` | function `(token: str)` | ``"platform"`` for ``osr_live_`` keys, ``"server"`` for ``osr_local_`` tokens, else ``"unknown"``. | | `whoami` | function `(cred: Credential, *, transport: Transport \| None=None, timeout: float=30.0)` | Validate ``cred`` against its server and describe the identity behind it. | ## `opensmartroute.discovery` Source: [src/opensmartroute/discovery.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/discovery.py) Tool discovery beyond text similarity. | Name | Kind | Summary | |---|---|---| | `CachePreservingSelector` | class | Keep the serialized tool list prefix-stable across turns of a session. | | `SchemaAwareStrategy` | class | Scores tools by how many of their ``input_schema`` parameters the request can fill (see ``schema_match``). | | `SkillGraph` | class | Dependency / conflict / composition graph over skill targets. | | `extract_entities` | function `(text: str)` | Typed entity mentions found in ``text`` (kind -> values). | | `schema_match` | function `(request: RouteRequest \| str, target: RouteTarget)` | Return ``(coverage, filled, missing)`` for the target's input schema. | ## `opensmartroute.enterprise` Source: [src/opensmartroute/enterprise/__init__.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/enterprise/__init__.py) Enterprise integration layer: ports (hexagonal architecture), middleware and telemetry. | Name | Kind | Summary | |---|---|---| | `AuditSink` | class | Tamper-evident audit trail port (hash-chained). | | `CacheMiddleware` | class | LRU decision cache keyed on (text, constraints, objective, route options). TTL in seconds. | | `EnterpriseRouter` | class | Router + middleware chain + telemetry + auto-learning + audit. Thread-safe façade. | | `FileAuditSink` | class | Append-only JSONL, each line ``{prev, hash, record}`` with. | | `FileStateStore` | class | One JSON file per key under ``root``; keys are hashed so they can't traverse paths. | | `InMemoryStateStore` | class | Thread-safe dict-backed :class:`StateStore`; values are deep-copied through JSON on read and write. | | `LoggingTelemetry` | class | Structured JSON logs; never logs raw request text (only a hash + length). | | `MetricsTelemetry` | class | In-process counters, outcome tallies and a latency histogram. | | `Middleware` | class | Chain-of-responsibility hook around routing: ``__call__(request, next_) -> RouteDecision``. | | `RouteFn` | constant | the "next" callable a middleware wraps. | | `RouterBuilder` | class | Fluent builder that validates configuration and wires all enterprise pieces. | | `SavingsEntry` | re-export of [`opensmartroute.enterprise.savings.SavingsEntry`](#opensmartrouteenterprisesavings) | One routed request in the ledger. | | `SavingsLedger` | re-export of [`opensmartroute.enterprise.savings.SavingsLedger`](#opensmartrouteenterprisesavings) | Telemetry sink that keeps a per-request baseline-vs-routed cost ledger (see module docs). | | `SavingsReport` | re-export of [`opensmartroute.enterprise.savings.SavingsReport`](#opensmartrouteenterprisesavings) | Aggregate savings and the quality they were bought at. | | `StateStore` | class | Key/value persistence port for learner state, caches, breaker state. | | `Telemetry` | class | Observer port with no-op defaults. Implement for OpenTelemetry, Prometheus, Datadog…. | | `TenantMiddleware` | class | Enforces that a tenant is present and applies per-tenant defaults/limits. | | `TimeoutMiddleware` | class | Soft deadline: raise if routing itself exceeded ``budget_ms`` (should never happen. | ## `opensmartroute.enterprise.ops` Source: [src/opensmartroute/enterprise/ops.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/enterprise/ops.py) Operational controls: shadow / A-B routing, tenant fairness and queue-aware latency. | Name | Kind | Summary | |---|---|---| | `SPRT` | class | Wald SPRT for two Bernoulli success rates: H0 p=p0 vs H1 p=p0+delta. | | `ABTest` | class | Traffic split + SPRT comparison of candidate vs control outcomes. | | `FairShareMiddleware` | class | Dominant Resource Fairness across tenants over a sliding window. | | `InflightTracker` | class | Per-target in-flight counters + arrival/service statistics for queueing estimates. | | `QueueAwareStrategy` | class | Scores targets by *current* time-to-first-token = catalogue latency + queueing wait vs. | | `ShadowMiddleware` | class | Run ``candidate`` beside production. ``mode='shadow'``: log only. ``mode='ab'``: serve. | | `TenantUsage` | class | Sliding-window ledger of one tenant's requests, cost and tokens for :class:`FairShareMiddleware`. | ## `opensmartroute.enterprise.savings` Source: [src/opensmartroute/enterprise/savings.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/enterprise/savings.py) Savings ledger - the always-on savings report that backs the ROI story and the dashboard. | Name | Kind | Summary | |---|---|---| | `SavingsEntry` | class | One routed request in the ledger. | | `SavingsLedger` | class | Telemetry sink that keeps a per-request baseline-vs-routed cost ledger (see module docs). | | `SavingsReport` | class | Aggregate savings and the quality they were bought at. | ## `opensmartroute.enterprise.stores` Source: [src/opensmartroute/enterprise/stores.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/enterprise/stores.py) Production state-store backends and wrappers. | Name | Kind | Summary | |---|---|---| | `BatchedStateStore` | class | Write-behind buffer. Reads are served from the pending map first. | | `EncryptedStateStore` | class | AES-256-GCM envelope encryption. ``key`` is 32 raw bytes, or read (base64/hex/raw). | | `Migration` | constant | state-document transform applied by VersionedStateStore. | | `NamespacedStateStore` | class | Prefix every key with ``/`` so several routers or tenants can share one backing store. | | `RedisStateStore` | class | StateStore on any redis-py-compatible client (``get`` / ``set`` / ``delete``), JSON values, optional TTL. | | `SQLStateStore` | class | DB-API 2.0 key/value store (PostgreSQL, SQLite, MySQL). | | `VersionedStateStore` | class | Schema-versioned envelope with forward migrations. | ## `opensmartroute.errors` Source: [src/opensmartroute/errors.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/errors.py) Exception hierarchy. Every error raised by the SDK derives from :class:`OpenSmartRouteError`. | Name | Kind | Summary | |---|---|---| | `OpenSmartRouteError` | class | Base class for all SDK errors. | | `ConfigurationError` | class | Invalid catalogue, rules, SKILL.md, settings or a missing optional dependency. | | `ValidationError` | class | A request, outcome or target failed validation. | | `NoRouteError` | class | No target satisfied the hard constraints. | | `TargetUnavailableError` | class | A remote target or catalogue source could not be reached. | | `ExecutionError` | class | A target handler failed while executing a plan. | | `SecurityError` | class | Request rejected by an input guard (prompt injection, oversize, etc.). | | `StateStoreError` | class | A learner-state store failed to load, save or migrate. | | `AuthenticationError` | class | The CLI has no valid access token for the platform / server, or a sign-in was denied or timed out. | | `OpenSmartRouteDeprecationWarning` | class | Emitted by :func:`deprecated`; filter with ``warnings.simplefilter`` on this class. | | `deprecated` | function `(name: str, *, since: str, removal: str, replacement: str \| None=None, stacklevel: int=3)` | Announce a deprecation according to the policy in CONTRIBUTING.md. | ## `opensmartroute.estimate` Source: [src/opensmartroute/estimate.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/estimate.py) Token, cost and latency estimates *before* a request is sent anywhere. | Name | Kind | Summary | |---|---|---| | `DEFAULT_OUTPUT_TOKENS` | constant | : Output length assumed when neither the caller nor the signals say (a short answer). | | `MESSAGE_OVERHEAD_TOKENS` | constant | : Tokens a chat API adds per message for role / separators (OpenAI-style framing). | | `PriceHook` | constant | : ``(usd_per_1k_input, usd_per_1k_output)`` for a target, or ``None`` to fall back to its declared cost. | | `RequestEstimate` | class | A quote for one request across every candidate, with named picks. | | `TargetEstimate` | class | The quote for one candidate target. | | `estimate` | function `(router: Router, request: RouteRequest \| str, *, output_tokens: int \| None=None, kinds: list[str] \| None=None, prices: PriceHook \| None=None, quality_tolerance: float=0.1, decision: RouteDecision \| None=None)` | Quote ``request`` against every candidate the router would consider - nothing is executed. | | `estimate_messages_tokens` | function `(messages: list[dict[str, Any]])` | Token estimate for an OpenAI-style message list (content plus per-message framing). | | `estimate_tokens` | function `(text: str)` | Approximate the tokenizer count of ``text`` without any tokenizer library. | | `target_prices` | function `(target: RouteTarget)` | ``(usd per 1k input tokens, usd per 1k output tokens)`` from the target's declared cost. | ## `opensmartroute.eval` Source: [src/opensmartroute/eval/__init__.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/eval/__init__.py) RouterBench-style evaluation harness. | Name | Kind | Summary | |---|---|---| | `AuditReport` | re-export of [`opensmartroute.eval.audit.AuditReport`](#opensmartrouteevalaudit) | Aggregate of a shadow replay; ``to_markdown`` renders the Routing Audit deliverable. | | `AuditRow` | re-export of [`opensmartroute.eval.audit.AuditRow`](#opensmartrouteevalaudit) | One logged request: what happened, and what the router would have done. | | `DatasetCollector` | re-export of [`opensmartroute.eval.collect.DatasetCollector`](#opensmartrouteevalcollect) | Cache-backed corpus builder: collect sources, add feedback / synthetic rows, dedupe, split. | | `DatasetSource` | re-export of [`opensmartroute.eval.collect.DatasetSource`](#opensmartrouteevalcollect) | One Hub dataset split to collect: repo id, config, split, the preset that parses it, an optional model map. | | `EvalResult` | class | Aggregate routing metrics over a dataset: accuracy, cost, latency, confidence, calibration, coverage. | | `EvalRow` | class | One labelled prompt: expected / acceptable targets or per-target quality scores (RouterBench style). | | `area_under_frontier` | function `(points: list[dict[str, float]])` | Trapezoidal area under accuracy(cost) — analogous to RouterBench's AIQ. | | `calibration_report` | function `(router: Router, rows: list[EvalRow], objective: Objective \| None=None)` | Confidence calibration of the router on labelled rows: ECE, Brier, reliability bins and. | | `collect_dataset` | re-export of [`opensmartroute.eval.collect.collect_dataset`](#opensmartrouteevalcollect) | Download ``source`` from the Hub and parse it into rows. | | `cost_quality_frontier` | function `(router: Router, rows: list[EvalRow], cost_weights: list[float] \| None=None)` | Sweep the cost weight to trace the accuracy-vs-cost curve (RouterBench Fig. 1 style). | | `evaluate` | function `(router: Router, rows: list[EvalRow], objective: Objective \| None=None)` | Route every row and score accuracy, realised quality, cost, latency, ECE / Brier and conformal coverage. | | `fetch_hf_rows` | re-export of [`opensmartroute.eval.collect.fetch_hf_rows`](#opensmartrouteevalcollect) | One page (max 100) of records from the datasets-server ``/rows`` endpoint, flattened to ``{column: value}``. | | `load_audit_log` | re-export of [`opensmartroute.eval.audit.load_audit_log`](#opensmartrouteevalaudit) | Read a JSONL traffic log (``text`` or ``prompt``/``messages`` per line). | | `load_dataset` | function `(path: str \| Path)` | Read a JSONL evaluation dataset (``text`` or ``prompt`` plus the optional EvalRow keys); a line that is. | | `model_quality` | re-export of [`opensmartroute.eval.collect.model_quality`](#opensmartrouteevalcollect) | Data-derived quality prior per model in [0, 1]: pairwise rows (two scored models) are fitted with. | | `routing_audit` | re-export of [`opensmartroute.eval.audit.routing_audit`](#opensmartrouteevalaudit) | Replay ``rows`` through ``router`` in shadow mode and aggregate an :class:`AuditReport`. | | `rows_from_feedback` | re-export of [`opensmartroute.eval.collect.rows_from_feedback`](#opensmartrouteevalcollect) | Rows from :class:`Outcome` records. ``texts`` maps ``request_id`` to the prompt (outcomes carry no text);. | | `summarize_audit` | re-export of [`opensmartroute.eval.audit.summarize_audit`](#opensmartrouteevalaudit) | Aggregate :class:`AuditRow` records into an :class:`AuditReport`. | | `synthetic_rows` | re-export of [`opensmartroute.eval.collect.synthetic_rows`](#opensmartrouteevalcollect) | Ontology seed prompts labelled with the best capability-fit target (cold-start supervision). | ## `opensmartroute.eval.agentic` Source: [src/opensmartroute/eval/agentic.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/eval/agentic.py) tau-bench-style agentic task evaluation: does per-step routing beat the best single agent?. | Name | Kind | Summary | |---|---|---| | `AgentTask` | class | A multi-step task with the measured per-step success probability and latency of each agent. | | `load_agentic_tasks` | function `(path: str \| Path)` | Read tasks from JSONL: ``{"task_id", "steps", "success", "latency_ms", "cost_usd"?, "domain"?}``. | | `synthetic_agentic_tasks` | function `(n: int=200, *, seed: int=0, hard_share: float=0.3)` | Three agents (fast / balanced / strong) and ``n`` tasks of 3-5 steps whose steps are mostly. | | `task_routing_frontier` | function `(registry: TargetRegistry, tasks: Sequence[AgentTask], *, retries: int=1, accuracy_tolerance: float=0.02, latency_ratio_target: float=0.9, objective: Objective \| None=None, router_factory: Callable[[TargetRegistry], Router] \| None=None, seed: int=0)` | Replay ``tasks`` step by step through :class:`ProgressRouter` (each agent's handler draws the. | ## `opensmartroute.eval.audit` Source: [src/opensmartroute/eval/audit.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/eval/audit.py) Routing Audit - shadow-mode replay of logged LLM traffic to quantify what the router would change. | Name | Kind | Summary | |---|---|---| | `DEFAULT_COMPLETION_TOKENS` | constant | : Default assumed output length when a log row carries no ``completion_tokens``. | | `AuditReport` | class | Aggregate of a shadow replay; ``to_markdown`` renders the Routing Audit deliverable. | | `AuditRow` | class | One logged request: what happened, and what the router would have done. | | `load_audit_log` | function `(path: str \| Path)` | Read a JSONL traffic log (``text`` or ``prompt``/``messages`` per line). | | `routing_audit` | function `(router: Router, rows: list[dict[str, Any]], *, baseline: str \| None=None, monthly_requests: int \| None=None, plan: bool=False, keep_rows: bool=True)` | Replay ``rows`` through ``router`` in shadow mode and aggregate an :class:`AuditReport`. | | `summarize_audit` | function `(rows: list[AuditRow], *, monthly_requests: int \| None=None, keep_rows: bool=True)` | Aggregate :class:`AuditRow` records into an :class:`AuditReport`. | ## `opensmartroute.eval.baselines` Source: [src/opensmartroute/eval/baselines.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/eval/baselines.py) Baselines every router must beat, plus oracle ceilings and the sampling noise floor. | Name | Kind | Summary | |---|---|---| | `Policy` | constant | a baseline: (row, targets) -> chosen target id. | | `PolicyResult` | class | Metrics of one baseline policy on a dataset (same fields as the router's EvalResult headline). | | `baseline_suite` | function `(rows: list[EvalRow], targets: Sequence[RouteTarget], train_rows: list[EvalRow] \| None=None, quality_floor: float=0.7, seed: int=0)` | Run every baseline and both oracles; ``train_rows`` (default: ``rows``) fit the task table. | | `best_prior_policy` | function `(row: EvalRow, targets: Sequence[RouteTarget])` | Always the target with the highest declared ``quality_prior``. | | `cheapest_policy` | function `(row: EvalRow, targets: Sequence[RouteTarget])` | Always the lowest unit cost (ties broken by latency). | | `evaluate_policy` | function `(name: str, policy: Policy, rows: list[EvalRow], targets: Sequence[RouteTarget])` | Run a baseline :data:`Policy` over the rows and aggregate accuracy, quality, cost and latency. | | `fit_task_table` | function `(rows: list[EvalRow], targets: Sequence[RouteTarget])` | ``task_type -> target`` with the highest mean quality on the training rows. | | `most_expensive_policy` | function `(row: EvalRow, targets: Sequence[RouteTarget])` | Always the highest unit cost - the "just use the frontier model" baseline. | | `multi_sample_oracle` | function `(rows: list[EvalRow], targets: Sequence[RouteTarget])` | Mean quality of the oracle that picks by the *mean over all samples* per target. | | `noise_floor` | function `(rows: list[EvalRow], targets: Sequence[RouteTarget], seed: int=0)` | How much accuracy is lost to sampling noise alone. | | `oracle_policy` | function `(quality_floor: float \| None=None)` | Per-row best target. With ``quality_floor`` -> cheapest target reaching the floor. | | `random_policy` | function `(seed: int=0)` | Uniformly random target (seeded). | | `static_task_table_policy` | function `(table: dict[str, str])` | Look the row's ``task_type`` up in a table from :func:`fit_task_table` (``"*"`` = default). | ## `opensmartroute.eval.collect` Source: [src/opensmartroute/eval/collect.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/eval/collect.py) Collect routing datasets from the Hugging Face Hub, your own feedback log and synthetic seeds. | Name | Kind | Summary | |---|---|---| | `ARENA_TIERS` | constant | everything else in those datasets is a 7B-14B chat model -> "small". | | `DATASETS_SERVER` | re-export of [`opensmartroute.adapters.websearch.DATASETS_SERVER`](#opensmartrouteadapterswebsearch) | Hugging Face datasets-server REST root (/rows). | | `DEFAULT_SOURCES` | constant | public ones. | | `HISTORY_FILE` | constant | the self-improver's report log; lives in the cache dir but is not a dataset. | | `KNOWN_SOURCES` | constant | pairwise human / judge preference battles - the winner is the label. | | `PAIRWISE` | constant | preset name for battle datasets: prompt / conversation, model_a, model_b, winner. | | `PARSERS` | constant | collector presets that are not wide / nested tables. | | `REWARD_BENCH` | constant | preset name for RewardBench: prompt, chosen_model, rejected_model, subset. | | `ROUTELLM_GOOD_ENOUGH` | constant | RouteLLM's threshold: a weak-model score >= 4 means the cheap model was good enough. | | `ROUTELLM_GPT4` | constant | preset name for RouteLLM's gpt4_dataset: prompt + GPT-4-judged Mixtral score 1-5. | | `ROUTELLM_STRONG` | constant | the strong model of RouteLLM's gpt4_dataset (its answers are the reference). | | `ROUTELLM_WEAK` | constant | the weak model whose answer GPT-4 scores 1-5 against the reference. | | `ULTRAFEEDBACK` | constant | preset name for UltraFeedback: instruction + N completions with a model and score. | | `DatasetCollector` | class | Cache-backed corpus builder: collect sources, add feedback / synthetic rows, dedupe, split. | | `DatasetSource` | class | One Hub dataset split to collect: repo id, config, split, the preset that parses it, an optional model map. | | `Parser` | constant | record -> row (or None to skip). | | `collect_dataset` | function `(source: DatasetSource, *, timeout_s: float \| None=None, settings: Settings \| None=None)` | Download ``source`` from the Hub and parse it into rows. | | `fetch_hf_rows` | function `(dataset: str, *, config: str='default', split: str='train', offset: int=0, length: int=100, timeout_s: float \| None=None, settings: Settings \| None=None)` | One page (max 100) of records from the datasets-server ``/rows`` endpoint, flattened to ``{column: value}``. | | `from_pairwise_row` | function `(rec: dict[str, Any], model_map: dict[str, str] \| None=None, source: DatasetSource \| None=None)` | A battle record -> row with ``scores`` 1 / 0 for winner / loser (0.5 each on a tie). Understands the. | | `from_reward_bench_row` | function `(rec: dict[str, Any], source: DatasetSource \| None=None)` | A RewardBench record (``prompt``, ``chosen_model``, ``rejected_model``, ``subset``) -> a pairwise row. | | `from_routellm_gpt4_row` | function `(rec: dict[str, Any], source: DatasetSource \| None=None)` | A ``routellm/gpt4_dataset`` record (``prompt`` + ``mixtral_score`` 1-5) -> the strong model scores 1.0 and. | | `from_ultrafeedback_row` | function `(rec: dict[str, Any], source: DatasetSource \| None=None)` | An UltraFeedback record (``instruction`` + ``completions[{model, overall_score}]``) -> a scored row with. | | `iter_hf_rows` | function `(source: DatasetSource, *, offset: int=0, timeout_s: float \| None=None, settings: Settings \| None=None)` | Page through ``source`` from raw record ``offset`` until ``source.limit`` records or the split is exhausted. | | `model_quality` | function `(rows: Iterable[EvalRow], *, min_n: int=20, epochs: int=5, seed: int=0)` | Data-derived quality prior per model in [0, 1]: pairwise rows (two scored models) are fitted with. | | `rows_from_feedback` | function `(outcomes: Iterable[Outcome] \| FeedbackStore, texts: dict[str, str] \| None=None, *, min_quality: float=0.5)` | Rows from :class:`Outcome` records. ``texts`` maps ``request_id`` to the prompt (outcomes carry no text);. | | `rows_from_records` | function `(records: Iterable[dict[str, Any]], source: DatasetSource)` | Parse raw records with the source's preset (``text_key`` is tried first when set). | | `synthetic_rows` | function `(targets: Sequence[RouteTarget], *, per_template: int=2, seed: int=0)` | Ontology seed prompts labelled with the best capability-fit target (cold-start supervision). | | `tier_model_map` | function `(tiers: Mapping[str, str], models: Iterable[str]=())` | ``model_map`` for a :class:`DatasetSource`: every known battle model (plus ``models``) -> the target id. | | `tier_of` | function `(model: str)` | Tier of a battle-dataset model name: the :data:`ARENA_TIERS` entry, else the name rules, else the parameter. | ## `opensmartroute.eval.criteria` Source: [src/opensmartroute/eval/criteria.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/eval/criteria.py) Offline realisations of the ROADMAP exit criteria. | Name | Kind | Summary | |---|---|---| | `CriterionResult` | class | One measured exit criterion. | | `bootstrap_ci` | function `(values: Sequence[float], *, n_boot: int=1000, level: float=0.95, seed: int=0)` | Percentile bootstrap confidence interval for the mean of ``values``. | | `cold_start_ratio` | function `(n_outcomes: int=200, *, domain: str='legal', n_eval: int=200, seed: int=0, target_ratio: float=0.9, generalist_quality: float=0.6, explore_rate: float=0.1, max_requests: int=20000)` | Accuracy on ``domain`` prompts of a target declared with only ``id`` / ``kind`` / ``cost``. | | `conformal_coverage` | function `(alpha: float=0.1, *, n_cal: int=1000, n_test: int=1000, label_noise: float=0.5, seed: int=0, tolerance: float=0.02)` | Fit :class:`ConformalCalibrator` on the real router's propensities over labelled prompts and. | | `domain_expert_registry` | function `(domains: Sequence[str]=_DOMAINS, *, strip: str \| None=None, generalist: bool=False)` | One hand-configured expert per domain. ``strip=`` replaces that expert with a. | | `effort_token_savings` | function `(registry: TargetRegistry, rows: Sequence[EvalRow], *, quality_tolerance: float=0.02, token_ratio_target: float=0.7, objective: Objective \| None=None, router_factory: Callable[[TargetRegistry], Router] \| None=None)` | Route rows that carry per-target ``scores`` *and* ``tokens`` and compare the tokens spent. | | `knapsack_never_exceeds_cap` | function `(steps: int=1000000, *, window: int=1000, drift_every: int=100000, seed: int=0)` | Drive :class:`MultiKnapsackBandit` (``on_capped="abstain"``) for ``steps`` pulls with. | | `match_at_1` | function `(router: Router \| EnterpriseRouter, prompts: Sequence[tuple[str, str]])` | Fraction of ``(text, expected_target_id)`` pairs the router gets right at rank 1. | | `match_at_1_at_scale` | function `(small: int=50, large: int=5000, *, n_prompts: int=200, seed: int=0, max_drop: float=0.05, narrow_above: int=32, narrow_to: int=24)` | Match@1 on a ``small`` catalogue versus a ``large`` one with retrieve-then-rank narrowing. | | `multi_round_vs_best_single` | function `(registry: TargetRegistry, rows: Sequence[EvalRow], *, threshold: float=0.8, max_rounds: int=3, failures_before_switch: int=1, objective: Objective \| None=None, cost_ratio_target: float=0.6, router_factory: Callable[[TargetRegistry], Router] \| None=None)` | Run :class:`MultiRoundExecutor` over rows that carry per-target ``scores`` (RouterBench. | | `ope_within_live_ci` | function `(n_log: int=2000, *, seed: int=0, logging_objective: Objective \| None=None, target_objective: Objective \| None=None, logging_temperature: float=0.3)` | Log decisions from a cost-seeking router (actions *sampled* from its propensities), estimate. | | `run_all` | function `(*, quick: bool=True, seed: int=0)` | Every offline criterion. ``quick`` shrinks the expensive simulations (10^5 steps, 1 000 tools,. | | `synthetic_effort_rows` | function `(n: int=400, *, seed: int=0)` | One reasoning model exposed as two effort siblings (``reasoner@fast`` with ``effort="low"``. | | `synthetic_scored_rows` | function `(n: int=300, *, seed: int=0)` | A three-tier catalogue (small / medium / large) and RouterBench-style rows with a quality. | | `synthetic_tool_catalogue` | function `(n: int, *, seed: int=0)` | ``n`` distinct tools (verb x object x qualifier, up to 5 000 unique combinations) and one. | ## `opensmartroute.eval.datasets` Source: [src/opensmartroute/eval/datasets.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/eval/datasets.py) Adapters from public routing benchmarks to :class:`EvalRow`. | Name | Kind | Summary | |---|---|---| | `PRESETS` | constant | Named layouts accepted by ``load_benchmark`` / ``osr eval --preset``. | | `NestedPreset` | class | Layout of a "nested" benchmark: a per-model mapping under ``container_keys`` holding score / cost / samples. | | `WidePreset` | class | Column layout of a "wide" benchmark: one row per prompt, ```` columns for scores / costs. | | `from_nested_row` | function `(row: dict[str, Any], preset: NestedPreset, model_map: dict[str, str] \| None=None)` | Convert one nested-format record to an :class:`EvalRow`, keeping repeated samples when present. | | `from_wide_row` | function `(row: dict[str, Any], preset: WidePreset, model_map: dict[str, str] \| None=None)` | Convert one wide-format record to an :class:`EvalRow` (None when it has no prompt or scores). | | `load_benchmark` | function `(path: str \| Path, preset: str \| WidePreset \| NestedPreset='routerbench', model_map: dict[str, str] \| None=None, limit: int \| None=None)` | Load a benchmark file (.jsonl / .json / .csv) into :class:`EvalRow` objects. | | `models_in` | function `(rows: Iterable[EvalRow])` | Distinct target ids that carry scores in the rows, in first-seen order. | ## `opensmartroute.eval.frontier` Source: [src/opensmartroute/eval/frontier.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/eval/frontier.py) Three-objective frontier and ablations. | Name | Kind | Summary | |---|---|---| | `ablation_report` | function `(router: Router, rows: list[EvalRow], objective: Objective \| None=None)` | Leave-one-out over strategies (and extractors, when explicitly set on the router). | | `frontier3` | function `(router: Router, rows: list[EvalRow], cost_weights: list[float] \| None=None, latency_weights: list[float] \| None=None)` | Sweep cost / latency objective weights and return the Pareto-optimal (quality, cost, latency) points. | | `hypervolume` | function `(points: list[dict[str, float]], ref: tuple[float, float, float] \| None=None)` | Dominated hypervolume of the Pareto points in normalised (quality, cost, latency) space. | ## `opensmartroute.eval.headroom` Source: [src/opensmartroute/eval/headroom.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/eval/headroom.py) Routing headroom: when does routing pay, and how much catalogue does it need?. | Name | Kind | Summary | |---|---|---| | `DiversityReport` | class | How differently the targets score rows: pairwise disagreement, winner entropy and winner share. | | `HeadroomReport` | class | Oracle vs best-single quality on a dataset, with the label-noise floor that says whether the gap is real. | | `learnability_by_difficulty` | function `(rows: Sequence[EvalRow], targets: Sequence[RouteTarget], buckets: int=4)` | Routing gain (oracle - best single) per difficulty bucket. Difficulty of a row is. | | `min_catalogue` | function `(rows: Sequence[EvalRow], targets: Sequence[RouteTarget], fraction: float=0.95)` | Greedy forward selection: the smallest ordered subset whose oracle reaches ``fraction`` of the. | | `routing_headroom` | function `(rows: Sequence[EvalRow], targets: Sequence[RouteTarget], seed: int=0)` | Oracle minus best-single quality with the label-noise floor for context. | | `scaling_curve` | function `(rows: Sequence[EvalRow], targets: Sequence[RouteTarget], sizes: Sequence[int] \| None=None, *, trials: int=20, seed: int=0)` | Mean oracle / best-single quality and headroom on random subsets of each size. | | `target_diversity` | function `(rows: Sequence[EvalRow], targets: Sequence[RouteTarget])` | How different the targets are on this data: pairwise score disagreement and winner spread. | ## `opensmartroute.eval.ope` Source: [src/opensmartroute/eval/ope.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/eval/ope.py) Off-policy evaluation from logged routing decisions. | Name | Kind | Summary | |---|---|---| | `LoggedDecision` | class | One logged routing event for off-policy evaluation: text, action taken, its propensity and observed reward. | | `OPEResult` | class | IPS, self-normalised IPS and doubly-robust value estimates with effective sample size and clip count. | | `ips_confidence_interval` | function `(logs: list[LoggedDecision], weights: list[float], z: float=1.96)` | Normal-approximation CI for the IPS estimate given per-sample weights. | | `mean_reward_model` | function `(logs: list[LoggedDecision])` | Simplest DR reward model: mean logged reward per action, global mean for unseen actions. | | `off_policy_evaluate` | function `(router: Router, logs: list[LoggedDecision], objective: Objective \| None=None, max_weight: float=20.0, reward_model: Callable[[LoggedDecision, str], float] \| None=None, deterministic: bool=False)` | Estimate the value of ``router`` on logged traffic. | | `target_propensities` | function `(router: Router, lg: LoggedDecision, objective: Objective \| None=None)` | The evaluated router's ``P(target \| x)`` for a logged context (empty when it has no route). | ## `opensmartroute.eval.robustness` Source: [src/opensmartroute/eval/robustness.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/eval/robustness.py) Robustness and fairness checks for a router. | Name | Kind | Summary | |---|---|---| | `coreset` | function `(rows: list[EvalRow], k: int, embedder: Callable[[list[str]], list[list[float]]] \| None=None, seed: int=0)` | Greedy k-center (farthest-first) subset of ``rows`` for a diverse evaluation set. | | `diversity` | function `(rows: list[EvalRow], embedder: Callable[[list[str]], list[list[float]]] \| None=None, sample: int=300)` | Mean pairwise cosine distance of the row texts (sampled), 0 = all identical. | | `paraphrase_robustness` | function `(router: Router, rows: list[EvalRow], n: int=4, objective: Objective \| None=None, seed: int=0)` | Share of rule-based paraphrases that route to the same target as the original (``robustness``). | | `paraphrases` | function `(text: str, n: int=4, seed: int=0)` | ``n`` deterministic surface-level paraphrases of ``text`` (never returns the original). | | `profile_swap_fairness` | function `(router: Router, rows: list[EvalRow], profiles: Sequence[dict[str, Any]] \| None=None, objective: Objective \| None=None)` | Decision agreement across user profiles. ``dependence`` = fraction of rows whose. | | `repeat_flip_rate` | function `(router: Router, rows: list[EvalRow], k: int=5, objective: Objective \| None=None)` | Route each row ``k`` times; ``flip_rate`` = share of rows whose decision is not identical every time. | ## `opensmartroute.execution` Source: [src/opensmartroute/execution.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/execution.py) Plan-aware execution: turn a :class:`RouteDecision` into a real answer. | Name | Kind | Summary | |---|---|---| | `PRELUDE_ROLES` | constant | Roles executed *before* the primary target, in this order. | | `ExecutionResult` | class | What happened when a decision was executed. | | `ExecutionStep` | class | One executed plan slot: role, target, latency and whether it succeeded. | | `aexecute` | async function `(decision: RouteDecision, request: RouteRequest, learn: LearnFn \| None=None, *, min_slot_confidence: float=0.0, task_id: str \| None=None, **kw: Any)` | Async twin of :func:`execute`; awaits coroutine handlers. | | `execute` | function `(decision: RouteDecision, request: RouteRequest, learn: LearnFn \| None=None, *, min_slot_confidence: float=0.0, task_id: str \| None=None, **kw: Any)` | Run the plan (persona -> skill -> primary) and record outcomes via ``learn``. | ## `opensmartroute.feedback` Source: [src/opensmartroute/feedback/__init__.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/feedback/__init__.py) Feedback store: append-only outcome log that closes the learning loop. | Name | Kind | Summary | |---|---|---| | `FeedbackStore` | class | Append-only :class:`Outcome` log (in memory or JSONL file) with per-target statistics. | ## `opensmartroute.learning` Source: [src/opensmartroute/learning/__init__.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/learning/__init__.py) Auto-learning strategies built on :mod:`opensmartroute.math`. | Name | Kind | Summary | |---|---|---| | `DEFAULT_DOMAINS` | constant | Domain one-hot block of the LinUCB context vector (ontology domains + the "general" fallback). | | `AttentionEncoder` | re-export of [`opensmartroute.learning.attention.AttentionEncoder`](#opensmartroutelearningattention) | One self-attention block with attention pooling; encodes text into a ``dim``-vector (unnormalised). | | `AutoLearner` | class | Single entry point for closing the loop. | | `Autopilot` | re-export of [`opensmartroute.learning.autopilot.Autopilot`](#opensmartroutelearningautopilot) | Runs a :class:`SelfImprover` on a schedule and on drift, inside a live process. | | `ContrastiveRouter` | re-export of [`opensmartroute.learning.contrastive.ContrastiveRouter`](#opensmartroutelearningcontrastive) | Dual-encoder router trained with a contrastive or a distillation objective. | | `ContrastiveStrategy` | re-export of [`opensmartroute.learning.contrastive.ContrastiveStrategy`](#opensmartroutelearningcontrastive) | Scores candidates with a trained :class:`ContrastiveRouter`. | | `DriftMonitor` | re-export of [`opensmartroute.learning.autopilot.DriftMonitor`](#opensmartroutelearningautopilot) | Page-Hinkley over outcomes: alarms when the served success rate (or quality) drops for real. | | `EmbeddingFeaturizer` | re-export of [`opensmartroute.learning.embed.EmbeddingFeaturizer`](#opensmartroutelearningembed) | Hashed features plus a frozen dense embedding, indexed above the hashed space (``dim + i``). | | `ExampleMiner` | re-export of [`opensmartroute.learning.coldstart.ExampleMiner`](#opensmartroutelearningcoldstart) | Promote prompts a target handled well into that target's ``examples``. | | `HandoffPolicy` | re-export of [`opensmartroute.learning.handoff.HandoffPolicy`](#opensmartroutelearninghandoff) | Permanent handoff of a task to ``fallback_target`` once the failure risk is too high. | | `HistoryTargetModel` | re-export of [`opensmartroute.learning.multiturn.HistoryTargetModel`](#opensmartroutelearningmultiturn) | Logistic model over ``h * e_t`` with a shared weight vector and per-target biases. | | `HistoryTargetStrategy` | re-export of [`opensmartroute.learning.multiturn.HistoryTargetStrategy`](#opensmartroutelearningmultiturn) | Scores each target by the learned success probability given the conversation so far. | | `IRTStrategy` | class | 2PL Item Response Theory: target ability vs (domain, complexity-bucket) item difficulty, learned online. | | `ImprovementReport` | re-export of [`opensmartroute.learning.self_improve.ImprovementReport`](#opensmartroutelearningself_improve) | What one cycle did: evidence gathered, catalogue changes, champion vs challenger, and the verdict. | | `LinUCBStrategy` | class | Contextual bandit (LinUCB) on the signal vector; keeps the last context per request for the update. | | `MarkovStrategy` | class | Prefers targets that do well on the *predicted next* conversation state too. | | `MixtureCureModel` | re-export of [`opensmartroute.learning.handoff.MixtureCureModel`](#opensmartroutelearninghandoff) | Weibull mixture-cure model on cumulative risk with censoring. | | `PolicyGradientStrategy` | re-export of [`opensmartroute.learning.policy_gradient.PolicyGradientStrategy`](#opensmartroutelearningpolicy_gradient) | REINFORCE-trained softmax routing policy over hashed request features. | | `PreferenceStrategy` | class | Bradley-Terry strengths per domain, fed by ``Outcome.preferred_over`` pairwise comparisons. | | `RegretReport` | re-export of [`opensmartroute.learning.policy_gradient.RegretReport`](#opensmartroutelearningpolicy_gradient) | Decision regret versus prediction error on a scored dataset. | | `RequestMemory` | re-export of [`opensmartroute.learning.coldstart.RequestMemory`](#opensmartroutelearningcoldstart) | Bounded LRU of request texts keyed by request id. | | `RouterSLM` | re-export of [`opensmartroute.learning.slm.RouterSLM`](#opensmartroutelearningslm) | Small routing model: dual encoder + target catalogue snapshot + calibration, in one JSON file. | | `SLMReport` | re-export of [`opensmartroute.learning.slm.SLMReport`](#opensmartroutelearningslm) | How an SLM did on a set of rows: accuracy, realised quality, cost, calibration and the training loss. | | `SLMStrategy` | re-export of [`opensmartroute.learning.slm.SLMStrategy`](#opensmartroutelearningslm) | Ensemble member backed by a :class:`RouterSLM`; scores are its probabilities, and it keeps learning online. | | `SelfImprover` | re-export of [`opensmartroute.learning.self_improve.SelfImprover`](#opensmartroutelearningself_improve) | Closed loop that keeps a :class:`RouterSLM` current with the model market and its own traffic. | | `SimilarityFallback` | re-export of [`opensmartroute.learning.coldstart.SimilarityFallback`](#opensmartroutelearningcoldstart) | For targets with no observations yet, rank by request<->description similarity. | | `SkillAffinity` | re-export of [`opensmartroute.learning.personal.SkillAffinity`](#opensmartroutelearningpersonal) | Profile-conditioned skill relevance: Beta posterior per (profile bucket, skill). | | `TaskCredit` | re-export of [`opensmartroute.learning.credit.TaskCredit`](#opensmartroutelearningcredit) | Buffers per-step outcomes of a task and redistributes the final reward (uniform / discounted / last / blend). | | `TaskPins` | re-export of [`opensmartroute.learning.credit.TaskPins`](#opensmartroutelearningcredit) | Admission-time pinning: ``task_id -> target_id`` while the target keeps succeeding. | | `UserAdaptiveStrategy` | re-export of [`opensmartroute.learning.personal.UserAdaptiveStrategy`](#opensmartroutelearningpersonal) | Per-user Beta posteriors per target, shrunk toward similar users and the global posterior. | | `acceptable_set` | re-export of [`opensmartroute.learning.contrastive.acceptable_set`](#opensmartroutelearningcontrastive) | Targets that are 'fine' for a row: ``expected`` + ``acceptable`` when labelled, otherwise every. | | `decision_regret` | re-export of [`opensmartroute.learning.policy_gradient.decision_regret`](#opensmartroutelearningpolicy_gradient) | Mean decision regret of ``choose`` against the per-row oracle, next to the prediction error of. | | `decision_reward` | re-export of [`opensmartroute.learning.policy_gradient.decision_reward`](#opensmartroutelearningpolicy_gradient) | Scalar decision reward of an outcome under ``objective`` (quality minus normalised cost/latency). | | `distill_router` | re-export of [`opensmartroute.learning.slm.distill_router`](#opensmartroutelearningslm) | Compress the full router into an SLM: route every text, take the ensemble's ranked utilities as soft. | | `history_vector` | re-export of [`opensmartroute.learning.multiturn.history_vector`](#opensmartroutelearningmultiturn) | Recency-weighted joint embedding of the last ``turns`` messages and the current text. | | `load_embedder` | re-export of [`opensmartroute.learning.embed.load_embedder`](#opensmartroutelearningembed) | Build the embedder a model file names: ``sentence-transformers/...`` (or any Hugging Face id) via. | | `merge_learners` | function `(local: Iterable[Strategy], remote: Iterable[Strategy])` | Federated merge: fold the evidence of ``remote`` strategies into the same-named ``local``. | | `nearest_targets` | re-export of [`opensmartroute.learning.coldstart.nearest_targets`](#opensmartroutelearningcoldstart) | The ``k`` most similar existing targets (cosine + same-kind and domain-overlap bonuses) for warm starts. | | `outcome_counts` | re-export of [`opensmartroute.learning.coldstart.outcome_counts`](#opensmartroutelearningcoldstart) | Number of recorded outcomes per target id. | | `profile_bucket` | re-export of [`opensmartroute.learning.personal.profile_bucket`](#opensmartroutelearningpersonal) | Coarse profile bucket used to pool users with the same declared attributes. | | `profile_vector` | re-export of [`opensmartroute.learning.personal.profile_vector`](#opensmartroutelearningpersonal) | Hashed one-hot encoding of ``key=value`` pairs (lists expand to one pair per element). | | `signal_vector` | function `(signals: Signals, domains: list[str] \| None=None)` | Fixed-length numeric context for contextual bandits (dim = 8 + len(domains)). | | `soft_labels` | re-export of [`opensmartroute.learning.contrastive.soft_labels`](#opensmartroutelearningcontrastive) | Zooter soft labels: ``softmax(score / temperature)`` over the scored targets. | | `target_document` | re-export of [`opensmartroute.learning.coldstart.target_document`](#opensmartroutelearningcoldstart) | Text used to embed a target: name, description, domains, actions, tags and up to 12 examples. | | `target_embedding` | re-export of [`opensmartroute.learning.coldstart.target_embedding`](#opensmartroutelearningcoldstart) | Unit-norm centroid of the target document and its examples (hashing embedder by default). | | `warm_start` | re-export of [`opensmartroute.learning.coldstart.warm_start`](#opensmartroutelearningcoldstart) | Seed every learner with shrunk knowledge from the new target's nearest neighbours. | | `warm_start_from_matrix` | re-export of [`opensmartroute.learning.coldstart.warm_start_from_matrix`](#opensmartroutelearningcoldstart) | Offline **full-information reward-matrix** warm start (OrcaRouter 2605.30736). | ## `opensmartroute.learning.attention` Source: [src/opensmartroute/learning/attention.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/learning/attention.py) A pure-Python transformer block for the routing SLM's query encoder. | Name | Kind | Summary | |---|---|---| | `AttentionContext` | class | Everything :meth:`AttentionEncoder.backward` needs from one forward pass. | | `AttentionEncoder` | class | One self-attention block with attention pooling; encodes text into a ``dim``-vector (unnormalised). | ## `opensmartroute.learning.autopilot` Source: [src/opensmartroute/learning/autopilot.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/learning/autopilot.py) Self-operation: the routing SLM runs its own improvement loop inside the live process. | Name | Kind | Summary | |---|---|---| | `Autopilot` | class | Runs a :class:`SelfImprover` on a schedule and on drift, inside a live process. | | `DriftMonitor` | class | Page-Hinkley over outcomes: alarms when the served success rate (or quality) drops for real. | ## `opensmartroute.learning.coldstart` Source: [src/opensmartroute/learning/coldstart.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/learning/coldstart.py) Cold start for new targets and self-improving target descriptions. | Name | Kind | Summary | |---|---|---| | `ExampleMiner` | class | Promote prompts a target handled well into that target's ``examples``. | | `RequestMemory` | class | Bounded LRU of request texts keyed by request id. | | `SimilarityFallback` | class | For targets with no observations yet, rank by request<->description similarity. | | `nearest_targets` | function `(new: RouteTarget, pool: Iterable[RouteTarget], k: int=3, embedder: Embedder \| None=None)` | The ``k`` most similar existing targets (cosine + same-kind and domain-overlap bonuses) for warm starts. | | `outcome_counts` | function `(feedback: FeedbackStore)` | Number of recorded outcomes per target id. | | `target_document` | function `(t: RouteTarget)` | Text used to embed a target: name, description, domains, actions, tags and up to 12 examples. | | `target_embedding` | function `(t: RouteTarget, embedder: Embedder \| None=None)` | Unit-norm centroid of the target document and its examples (hashing embedder by default). | | `warm_start` | function `(new: RouteTarget, registry: TargetRegistry, strategies: list[Strategy], k: int=3, shrink: float=0.5, embedder: Embedder \| None=None)` | Seed every learner with shrunk knowledge from the new target's nearest neighbours. | ## `opensmartroute.learning.contrastive` Source: [src/opensmartroute/learning/contrastive.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/learning/contrastive.py) Contrastive and reward-distilled router training (RouterDC, NeurIPS 2024; Zooter 2311.08692). | Name | Kind | Summary | |---|---|---| | `ContrastiveRouter` | class | Dual-encoder router trained with a contrastive or a distillation objective. | | `ContrastiveStrategy` | class | Scores candidates with a trained :class:`ContrastiveRouter`. | | `acceptable_set` | function `(row: EvalRow, slack: float=0.05)` | Targets that are 'fine' for a row: ``expected`` + ``acceptable`` when labelled, otherwise every. | | `soft_labels` | function `(scores: dict[str, float], temperature: float=0.1)` | Zooter soft labels: ``softmax(score / temperature)`` over the scored targets. | ## `opensmartroute.learning.credit` Source: [src/opensmartroute/learning/credit.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/learning/credit.py) Delayed, task-level credit assignment for agentic trajectories. | Name | Kind | Summary | |---|---|---| | `TaskCredit` | class | Buffers per-step outcomes of a task and redistributes the final reward (uniform / discounted / last / blend). | | `TaskPins` | class | Admission-time pinning: ``task_id -> target_id`` while the target keeps succeeding. | ## `opensmartroute.learning.embed` Source: [src/opensmartroute/learning/embed.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/learning/embed.py) Pretrained transformer embeddings as frozen features for the routing SLM. | Name | Kind | Summary | |---|---|---| | `EmbeddingFeaturizer` | class | Hashed features plus a frozen dense embedding, indexed above the hashed space (``dim + i``). | | `load_embedder` | function `(name: str)` | Build the embedder a model file names: ``sentence-transformers/...`` (or any Hugging Face id) via. | ## `opensmartroute.learning.handoff` Source: [src/opensmartroute/learning/handoff.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/learning/handoff.py) Permanent-handoff policy from censored teacher signals (TACIT-Switch 2608.27911). | Name | Kind | Summary | |---|---|---| | `HandoffPolicy` | class | Permanent handoff of a task to ``fallback_target`` once the failure risk is too high. | | `MixtureCureModel` | class | Weibull mixture-cure model on cumulative risk with censoring. | | `Trajectory` | class | Per-task state tracked by :class:`HandoffPolicy`: cumulative risk, steps, failure and hand-off flags. | ## `opensmartroute.learning.multiturn` Source: [src/opensmartroute/learning/multiturn.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/learning/multiturn.py) Multi-turn routing with history-target joint embeddings (MTRouter 2604.23530). | Name | Kind | Summary | |---|---|---| | `HistoryTargetModel` | class | Logistic model over ``h * e_t`` with a shared weight vector and per-target biases. | | `HistoryTargetStrategy` | class | Scores each target by the learned success probability given the conversation so far. | | `history_vector` | function `(request: RouteRequest, embedder: Embedder, turns: int=6, decay: float=0.7)` | Recency-weighted joint embedding of the last ``turns`` messages and the current text. | ## `opensmartroute.learning.personal` Source: [src/opensmartroute/learning/personal.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/learning/personal.py) Few-shot personalisation (GMTRouter 2511.08590; SkillFeed 2608.28241). | Name | Kind | Summary | |---|---|---| | `SkillAffinity` | class | Profile-conditioned skill relevance: Beta posterior per (profile bucket, skill). | | `UserAdaptiveStrategy` | class | Per-user Beta posteriors per target, shrunk toward similar users and the global posterior. | | `profile_bucket` | function `(profile: dict[str, Any], keys: tuple[str, ...]=('tier', 'expertise', 'role', 'language'))` | Coarse profile bucket used to pool users with the same declared attributes. | | `profile_vector` | function `(profile: dict[str, Any], dim: int=_PROFILE_DIM)` | Hashed one-hot encoding of ``key=value`` pairs (lists expand to one pair per element). | ## `opensmartroute.learning.policy_gradient` Source: [src/opensmartroute/learning/policy_gradient.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/learning/policy_gradient.py) End-to-end policy-gradient routing (Router-R1 2506.09033; RLCascadeRouter 2608.15817). | Name | Kind | Summary | |---|---|---| | `PolicyGradientStrategy` | class | REINFORCE-trained softmax routing policy over hashed request features. | | `RegretReport` | class | Decision regret versus prediction error on a scored dataset. | | `decision_regret` | function `(rows: Sequence[EvalRow], targets: Sequence[RouteTarget], choose: Callable[[EvalRow, Sequence[RouteTarget]], str], predict: Callable[[EvalRow, RouteTarget], float] \| None=None)` | Mean decision regret of ``choose`` against the per-row oracle, next to the prediction error of. | | `decision_reward` | function `(outcome: Outcome, target: RouteTarget \| None, objective: Objective, *, cost_scale: float=0.01, latency_scale: float=2000.0)` | Scalar decision reward of an outcome under ``objective`` (quality minus normalised cost/latency). | ## `opensmartroute.learning.self_improve` Source: [src/opensmartroute/learning/self_improve.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/learning/self_improve.py) Self-improvement loop: refresh the catalogue, gather evidence, train a challenger, promote it only if better. | Name | Kind | Summary | |---|---|---| | `ImprovementReport` | class | What one cycle did: evidence gathered, catalogue changes, champion vs challenger, and the verdict. | | `SelfImprover` | class | Closed loop that keeps a :class:`RouterSLM` current with the model market and its own traffic. | ## `opensmartroute.learning.slm` Source: [src/opensmartroute/learning/slm.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/learning/slm.py) The OpenSmartRoute routing SLM: a small, self-contained model that picks the target for a prompt. | Name | Kind | Summary | |---|---|---| | `SLM_FORMAT` | constant | on-disk format version of ``RouterSLM.state()``. | | `RouterSLM` | class | Small routing model: dual encoder + target catalogue snapshot + calibration, in one JSON file. | | `SLMReport` | class | How an SLM did on a set of rows: accuracy, realised quality, cost, calibration and the training loss. | | `SLMStrategy` | class | Ensemble member backed by a :class:`RouterSLM`; scores are its probabilities, and it keeps learning online. | | `distill_router` | function `(router: Router, texts: Iterable[str], *, targets: Sequence[RouteTarget] \| None=None, settings: Settings \| None=None, seed: int=0)` | Compress the full router into an SLM: route every text, take the ensemble's ranked utilities as soft. | ## `opensmartroute.math` Source: [src/opensmartroute/math/__init__.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/math/__init__.py) Mathematical toolkit behind OpenSmartRoute's decisions. | Name | Kind | Summary | |---|---|---| | `EWMA` | re-export of [`opensmartroute.math.estimators.EWMA`](#opensmartroutemathestimators) | Exponentially weighted moving mean and variance with a z-score helper. | | `Bandit` | re-export of [`opensmartroute.math.bandits.Bandit`](#opensmartroutemathbandits) | Common interface. ``context`` is optional; non-contextual learners ignore it. | | `BradleyTerry` | re-export of [`opensmartroute.math.preference.BradleyTerry`](#opensmartroutemathpreference) | Online Bradley-Terry: per-context target strengths from pairwise wins, with L2 and forgetting. | | `ConformalCalibrator` | re-export of [`opensmartroute.math.calibration.ConformalCalibrator`](#opensmartroutemathcalibration) | Split conformal prediction over routing candidates. | | `CostAwareBandit` | re-export of [`opensmartroute.math.bandits.CostAwareBandit`](#opensmartroutemathbandits) | Lagrangian budget wrapper (C2MAB-V flavour). | | `DelayedFeedback` | re-export of [`opensmartroute.math.bandits.DelayedFeedback`](#opensmartroutemathbandits) | Delayed-feedback wrapper (Joulani, György & Szepesvári, 2013). | | `DirichletProbe` | re-export of [`opensmartroute.math.dirichlet.DirichletProbe`](#opensmartroutemathdirichlet) | Linear Dirichlet head over a fixed target list. | | `Elo` | re-export of [`opensmartroute.math.preference.Elo`](#opensmartroutemathpreference) | Elo rating — a fixed-step Bradley–Terry with K-factor; handy for fast warm-up. | | `EnergyModel` | re-export of [`opensmartroute.math.energy.EnergyModel`](#opensmartroutemathenergy) | Per-target linear energy model ``wh = e0 + e_in * prompt + e_out * output`` fit by ridge. | | `EpsilonGreedy` | re-export of [`opensmartroute.math.bandits.EpsilonGreedy`](#opensmartroutemathbandits) | Explore uniformly with probability ``epsilon``, otherwise exploit the empirical means. | | `HardwareProfile` | re-export of [`opensmartroute.math.energy.HardwareProfile`](#opensmartroutemathenergy) | Static device characterisation used before measurements exist. | | `IRTModel` | re-export of [`opensmartroute.math.irt.IRTModel`](#opensmartroutemathirt) | Online 2PL IRT: target ability, item difficulty / discrimination via SGD, with forgetting and merge support. | | `IsotonicCalibrator` | re-export of [`opensmartroute.math.calibration.IsotonicCalibrator`](#opensmartroutemathcalibration) | Monotone non-decreasing map score -> P(correct), fitted with PAV. | | `LinUCB` | re-export of [`opensmartroute.math.bandits.LinUCB`](#opensmartroutemathbandits) | Disjoint LinUCB (Li et al., WWW 2010) with Sherman–Morrison updates and optional. | | `MarkovChain` | re-export of [`opensmartroute.math.markov.MarkovChain`](#opensmartroutemathmarkov) | Dirichlet-smoothed transition counts. ``decay`` < 1 multiplies a row's counts by ``decay``. | | `MultiKnapsackBandit` | re-export of [`opensmartroute.math.bandits.MultiKnapsackBandit`](#opensmartroutemathbandits) | Bandits with several knapsack constraints (Badanidiyuru et al., 2013) with the. | | `PageHinkley` | re-export of [`opensmartroute.math.estimators.PageHinkley`](#opensmartroutemathestimators) | Detects a *decrease* in the monitored mean (e.g. quality dropping). | | `RoutingMDP` | re-export of [`opensmartroute.math.markov.RoutingMDP`](#opensmartroutemathmarkov) | Finite-horizon / discounted MDP over conversation states and route actions. | | `TemperatureScaler` | re-export of [`opensmartroute.math.calibration.TemperatureScaler`](#opensmartroutemathcalibration) | Fits :math:`\tau` for ``confidence = softmax(u/\tau)[argmax]`` by 1-D golden-section. | | `ThompsonBeta` | re-export of [`opensmartroute.math.bandits.ThompsonBeta`](#opensmartroutemathbandits) | Beta–Bernoulli Thompson sampling. | | `UCB1` | re-export of [`opensmartroute.math.bandits.UCB1`](#opensmartroutemathbandits) | UCB1: :math:`\hat\mu_a + c\sqrt{\frac{2\ln t}{n_a}}`; untried arms get +inf. | | `Welford` | re-export of [`opensmartroute.math.estimators.Welford`](#opensmartroutemathestimators) | Numerically stable running mean / variance (Welford's algorithm). | | `WindowDrift` | re-export of [`opensmartroute.math.estimators.WindowDrift`](#opensmartroutemathestimators) | ADWIN-lite: compare the first and second half of a sliding window with a. | | `bayesian_average` | re-export of [`opensmartroute.math.estimators.bayesian_average`](#opensmartroutemathestimators) | Shrink a small-sample mean toward a prior: (n·m + k·μ₀)/(n + k). | | `brier_score` | re-export of [`opensmartroute.math.calibration.brier_score`](#opensmartroutemathcalibration) | Mean squared error between confidence and the 0/1 correctness label. | | `conformal_quantile` | re-export of [`opensmartroute.math.calibration.conformal_quantile`](#opensmartroutemathcalibration) | Finite-sample corrected :math:`\lceil (n+1)(1-\alpha)\rceil / n` empirical quantile. | | `digamma` | re-export of [`opensmartroute.math.dirichlet.digamma`](#opensmartroutemathdirichlet) | Digamma :math:`\psi(x)` for x > 0 via recurrence to x >= 6 and the asymptotic series. | | `dominates` | re-export of [`opensmartroute.math.decision.dominates`](#opensmartroutemathdecision) | a dominates b if it is ≥ quality, ≤ cost, ≤ latency and strictly better in one. | | `entropy` | re-export of [`opensmartroute.math.estimators.entropy`](#opensmartroutemathestimators) | Shannon entropy (nats) of a probability vector. | | `erlang_c` | re-export of [`opensmartroute.math.decision.erlang_c`](#opensmartroutemathdecision) | P(an arriving request must wait) for M/M/c. Returns 1.0 if unstable. | | `expected_calibration_error` | re-export of [`opensmartroute.math.estimators.expected_calibration_error`](#opensmartroutemathestimators) | ECE: how well does router confidence predict routing correctness?. | | `expected_wait` | re-export of [`opensmartroute.math.decision.expected_wait`](#opensmartroutemathdecision) | Mean time in queue (Wq) for M/M/c, in the same time unit as the rates. | | `gini` | re-export of [`opensmartroute.math.estimators.gini`](#opensmartroutemathestimators) | Gini impurity ``1 - sum(p^2)``; 0 = certain. | | `hardware_profile` | re-export of [`opensmartroute.math.energy.hardware_profile`](#opensmartroutemathenergy) | Built-in profile by name (``a100-80gb``, ``h100-sxm``, ``l4``, ``rtx-4090``, ``cpu-16c``, ``npu-edge``). | | `item_key` | re-export of [`opensmartroute.math.irt.item_key`](#opensmartroutemathirt) | Default item id: domain × difficulty bucket, e.g. ``legal/3``. | | `kingman_wait` | re-export of [`opensmartroute.math.decision.kingman_wait`](#opensmartroutemathdecision) | Kingman's G/G/1 approximation. | | `littles_law` | re-export of [`opensmartroute.math.decision.littles_law`](#opensmartroutemathdecision) | L = λ·W — average number of in-flight requests. | | `normalized_entropy` | re-export of [`opensmartroute.math.estimators.normalized_entropy`](#opensmartroutemathestimators) | 0 = certain, 1 = uniform. Useful as an 'ask the LLM judge' trigger. | | `pareto_front` | re-export of [`opensmartroute.math.decision.pareto_front`](#opensmartroutemathdecision) | Keys of the non-dominated (quality, cost, latency) points. | | `reliability_diagram` | re-export of [`opensmartroute.math.calibration.reliability_diagram`](#opensmartroutemathcalibration) | Per-bin (mean confidence, empirical accuracy, count) — plot or print it. | | `servers_for_sla` | re-export of [`opensmartroute.math.decision.servers_for_sla`](#opensmartroutemathdecision) | Smallest c such that E[Wq] ≤ max_wait and P(wait) ≤ max_p_wait. | | `sigmoid` | re-export of [`opensmartroute.math.irt.sigmoid`](#opensmartroutemathirt) | Overflow-safe logistic function. | | `softmax` | re-export of [`opensmartroute.math.estimators.softmax`](#opensmartroutemathestimators) | Numerically stable softmax; lower ``temperature`` sharpens the distribution. | | `topsis` | re-export of [`opensmartroute.math.decision.topsis`](#opensmartroutemathdecision) | TOPSIS closeness coefficient in [0,1]; quality is a benefit, cost/latency are costs. | | `weighted_sum` | re-export of [`opensmartroute.math.decision.weighted_sum`](#opensmartroutemathdecision) | Scalarise ``w_q * quality - w_c * norm(cost) - w_l * norm(latency)`` with min-max normalised cost / latency. | | `wilson_interval` | re-export of [`opensmartroute.math.estimators.wilson_interval`](#opensmartroutemathestimators) | Wilson score interval for a binomial proportion (robust at small n). | ## `opensmartroute.math.bandits` Source: [src/opensmartroute/math/bandits.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/math/bandits.py) Multi-armed and contextual bandits for online routing decisions. | Name | Kind | Summary | |---|---|---| | `Bandit` | class | Common interface. ``context`` is optional; non-contextual learners ignore it. | | `ThompsonBeta` | class | Beta–Bernoulli Thompson sampling. | | `UCB1` | class | UCB1: :math:`\hat\mu_a + c\sqrt{\frac{2\ln t}{n_a}}`; untried arms get +inf. | | `LinUCB` | class | Disjoint LinUCB (Li et al., WWW 2010) with Sherman–Morrison updates and optional. | | `EpsilonGreedy` | class | Explore uniformly with probability ``epsilon``, otherwise exploit the empirical means. | | `CostAwareBandit` | class | Lagrangian budget wrapper (C2MAB-V flavour). | | `MultiKnapsackBandit` | class | Bandits with several knapsack constraints (Badanidiyuru et al., 2013) with the. | | `DelayedFeedback` | class | Delayed-feedback wrapper (Joulani, György & Szepesvári, 2013). | ## `opensmartroute.math.calibration` Source: [src/opensmartroute/math/calibration.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/math/calibration.py) Calibration and distribution-free risk control for routing confidence. | Name | Kind | Summary | |---|---|---| | `ConformalCalibrator` | class | Split conformal prediction over routing candidates. | | `IsotonicCalibrator` | class | Monotone non-decreasing map score -> P(correct), fitted with PAV. | | `TemperatureScaler` | class | Fits :math:`\tau` for ``confidence = softmax(u/\tau)[argmax]`` by 1-D golden-section. | | `brier_score` | function `(confidences: list[float], hits: list[bool])` | Mean squared error between confidence and the 0/1 correctness label. | | `conformal_quantile` | function `(scores: list[float], alpha: float)` | Finite-sample corrected :math:`\lceil (n+1)(1-\alpha)\rceil / n` empirical quantile. | | `reliability_diagram` | function `(confidences: list[float], hits: list[bool], bins: int=10)` | Per-bin (mean confidence, empirical accuracy, count) — plot or print it. | ## `opensmartroute.math.decision` Source: [src/opensmartroute/math/decision.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/math/decision.py) Multi-objective decision helpers and queueing theory for real-time routing. | Name | Kind | Summary | |---|---|---| | `dominates` | function `(a: Point, b: Point)` | a dominates b if it is ≥ quality, ≤ cost, ≤ latency and strictly better in one. | | `pareto_front` | function `(points: dict[str, Point])` | Keys of the non-dominated (quality, cost, latency) points. | | `weighted_sum` | function `(points: dict[str, Point], w_q: float, w_c: float, w_l: float)` | Scalarise ``w_q * quality - w_c * norm(cost) - w_l * norm(latency)`` with min-max normalised cost / latency. | | `topsis` | function `(points: dict[str, Point], weights: Sequence[float]=(0.6, 0.25, 0.15))` | TOPSIS closeness coefficient in [0,1]; quality is a benefit, cost/latency are costs. | | `erlang_c` | function `(arrival_rate: float, service_rate: float, servers: int)` | P(an arriving request must wait) for M/M/c. Returns 1.0 if unstable. | | `expected_wait` | function `(arrival_rate: float, service_rate: float, servers: int)` | Mean time in queue (Wq) for M/M/c, in the same time unit as the rates. | | `servers_for_sla` | function `(arrival_rate: float, service_rate: float, max_wait: float, max_p_wait: float=0.2)` | Smallest c such that E[Wq] ≤ max_wait and P(wait) ≤ max_p_wait. | | `littles_law` | function `(arrival_rate: float, mean_time_in_system: float)` | L = λ·W — average number of in-flight requests. | | `kingman_wait` | function `(utilization: float, ca2: float, cs2: float, mean_service: float)` | Kingman's G/G/1 approximation. | ## `opensmartroute.math.dirichlet` Source: [src/opensmartroute/math/dirichlet.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/math/dirichlet.py) Dirichlet probe over host hidden states (ProbeDirichlet, RouterXBench 2602.11877). | Name | Kind | Summary | |---|---|---| | `DirichletProbe` | class | Linear Dirichlet head over a fixed target list. | | `digamma` | function `(x: float)` | Digamma :math:`\psi(x)` for x > 0 via recurrence to x >= 6 and the asymptotic series. | ## `opensmartroute.math.energy` Source: [src/opensmartroute/math/energy.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/math/energy.py) Hardware-aware energy characterisation (HW-Router 2608.14575; 2608.28044). | Name | Kind | Summary | |---|---|---| | `EnergyModel` | class | Per-target linear energy model ``wh = e0 + e_in * prompt + e_out * output`` fit by ridge. | | `HardwareProfile` | class | Static device characterisation used before measurements exist. | | `hardware_profile` | function `(name: str)` | Built-in profile by name (``a100-80gb``, ``h100-sxm``, ``l4``, ``rtx-4090``, ``cpu-16c``, ``npu-edge``). | ## `opensmartroute.math.estimators` Source: [src/opensmartroute/math/estimators.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/math/estimators.py) Streaming estimators, drift detection and calibration. | Name | Kind | Summary | |---|---|---| | `EWMA` | class | Exponentially weighted moving mean and variance with a z-score helper. | | `Welford` | class | Numerically stable running mean / variance (Welford's algorithm). | | `PageHinkley` | class | Detects a *decrease* in the monitored mean (e.g. quality dropping). | | `WindowDrift` | class | ADWIN-lite: compare the first and second half of a sliding window with a. | | `wilson_interval` | function `(successes: float, n: int, z: float=1.96)` | Wilson score interval for a binomial proportion (robust at small n). | | `bayesian_average` | function `(mean: float, n: int, prior_mean: float, prior_n: float=5.0)` | Shrink a small-sample mean toward a prior: (n·m + k·μ₀)/(n + k). | | `softmax` | function `(xs: list[float], temperature: float=1.0)` | Numerically stable softmax; lower ``temperature`` sharpens the distribution. | | `entropy` | function `(ps: list[float])` | Shannon entropy (nats) of a probability vector. | | `normalized_entropy` | function `(ps: list[float])` | 0 = certain, 1 = uniform. Useful as an 'ask the LLM judge' trigger. | | `gini` | function `(ps: list[float])` | Gini impurity ``1 - sum(p^2)``; 0 = certain. | | `expected_calibration_error` | function `(confidences: list[float], hits: list[bool], bins: int=10)` | ECE: how well does router confidence predict routing correctness?. | ## `opensmartroute.math.irt` Source: [src/opensmartroute/math/irt.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/math/irt.py) Item Response Theory for routing (IRT-Router, ACL 2025). | Name | Kind | Summary | |---|---|---| | `sigmoid` | function `(z: float)` | Overflow-safe logistic function. | | `IRTModel` | class | Online 2PL IRT: target ability, item difficulty / discrimination via SGD, with forgetting and merge support. | | `item_key` | function `(domain: str, complexity: float, buckets: int=4)` | Default item id: domain × difficulty bucket, e.g. ``legal/3``. | ## `opensmartroute.math.markov` Source: [src/opensmartroute/math/markov.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/math/markov.py) Markov chains and MDPs for conversational / multi-step routing. | Name | Kind | Summary | |---|---|---| | `MarkovChain` | class | Dirichlet-smoothed transition counts. ``decay`` < 1 multiplies a row's counts by ``decay``. | | `RoutingMDP` | class | Finite-horizon / discounted MDP over conversation states and route actions. | ## `opensmartroute.math.preference` Source: [src/opensmartroute/math/preference.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/math/preference.py) Bradley–Terry pairwise preference model (RouteLLM, Prompt-to-Leaderboard). | Name | Kind | Summary | |---|---|---| | `BradleyTerry` | class | Online Bradley-Terry: per-context target strengths from pairwise wins, with L2 and forgetting. | | `Elo` | class | Elo rating — a fixed-step Bradley–Terry with K-factor; handy for fast warm-up. | ## `opensmartroute.mcp_server` Source: [src/opensmartroute/mcp_server.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/mcp_server.py) Model Context Protocol server: the router as a set of tools for any IDE or agent. | Name | Kind | Summary | |---|---|---| | `PROTOCOL_VERSION` | constant | : Protocol revision this server speaks by default. | | `SUPPORTED_PROTOCOL_VERSIONS` | constant | : Revisions accepted from clients (the client's choice is echoed when it is one of these). | | `MCPServer` | class | Serve a :class:`Router` (or an enterprise router wrapping one) over MCP. | | `RemoteMCP` | class | Forward JSON-RPC messages to an HTTP MCP endpoint (``POST /mcp``) - the bridge's counterpart of. | | `ToolError` | class | Raised inside a tool: reported to the client as a tool result with ``isError`` (not a protocol error). | | `bridge_stdio` | function `(url: str, api_key: str \| None=None)` | Expose a remote HTTP MCP endpoint as a local stdio server (for clients that only spawn processes). | | `decision_dict` | function `(d: RouteDecision)` | ``RouteDecision.to_dict()`` plus signals and the ranked candidates - what IDE clients want to show. | | `serve_stdio` | function `(server: Any, stdin: IO[bytes] \| None=None, stdout: IO[str] \| None=None)` | Run ``server`` (anything with ``handle_json``) over newline-delimited JSON on stdin/stdout until EOF. | ## `opensmartroute.observability` Source: [src/opensmartroute/observability.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/observability.py) Tracing and observability: every step of routing, execution and learning as a structured event. | Name | Kind | Summary | |---|---|---| | `EVENT_NAMES` | constant | Router.route. | | `METRIC_PREFIX` | constant | Prometheus metric name prefix (``osr_``). | | `SPAN_NAMES` | constant | : Every span name the SDK opens (an :class:`Event` with ``kind="span"``). | | `TRACEPARENT_KEY` | constant | request.context key (and HTTP header) carrying an inbound W3C trace context. | | `TRACE_ID_HEADER` | constant | response header ``osr serve`` sets to the trace id of the request. | | `Event` | class | One captured span or event: flat, JSON-friendly, never carries request text. | | `EventSink` | class | Receiver port: override :meth:`emit`; live bridges may also implement :meth:`span_start` / :meth:`span_end`. | | `FileSink` | class | Append-only JSONL file, one event per line. | | `LoggingSink` | class | One JSON line per event on the ``opensmartroute.events`` logger (level follows the event). | | `MemorySink` | class | Thread-safe ring buffer of the newest ``max_events`` events; queryable by request, trace, name or level. | | `MetricsSink` | class | Counters and latency percentiles derived from events; :meth:`prometheus` renders the exposition text. | | `Span` | class | An open unit of work; a context manager that records duration, status and nested events. | | `Tracer` | class | Opens spans, records events and fans them out to sinks; ``sample_rate`` < 1 traces a share of requests. | | `configure_tracing` | function `(*sinks: EventSink, sample_rate: float \| None=None, settings: Settings \| None=None)` | Install sinks on the process-wide tracer. With no sinks, use the ones named by settings. | | `current_tracer` | function `()` | The tracer of the innermost open span, else the one bound with :func:`use_tracer`, else the global one. | | `get_tracer` | function `()` | The process-wide tracer (disabled until :func:`configure_tracing` adds sinks). | | `text_digest` | function `(text: str)` | Loggable stand-in for request text: ``{"text_sha256": <16 hex>, "text_len": n}``. | | `use_tracer` | class | ``with use_tracer(t):`` makes ``t`` the tracer that :func:`current_tracer` returns in this context. | ## `opensmartroute.ocm` Source: [src/opensmartroute/ocm.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/ocm.py) Open Capability Manifest (OCM) - a vendor-neutral description of any routable capability. | Name | Kind | Summary | |---|---|---| | `OCM_KINDS` | constant | : Manifest kinds (identical to :class:`~opensmartroute.TargetKind` values). | | `OCM_PROTOCOLS` | constant | : Endpoint protocols a manifest may declare; ``http`` and ``a2a`` can be bound to executors directly. | | `OCM_VERSION` | constant | : Current manifest version (the ``ocm`` field). | | `bind_endpoint` | function `(target: RouteTarget, region: str \| None=None, **kw: Any)` | Attach an executor for the selected endpoint (``http`` -> :func:`~opensmartroute.adapters.http_handler`,. | | `capability_from_target` | function `(target: RouteTarget)` | Reverse mapping: publish a target as an OCM v1 manifest (``dict``; dump as YAML or JSON). | | `dump_capability` | function `(doc: dict[str, Any])` | Serialise a manifest to YAML when PyYAML is available, else to pretty JSON (both are valid OCM). | | `is_capability` | function `(doc: Any)` | True when ``doc`` looks like an OCM manifest (has an ``ocm`` version field). | | `load_capabilities` | function `(paths: Iterable[str \| Path] \| str \| Path, *, strict: bool=True)` | Load manifests from files and/or directories (``**/capability.{yaml,yml,json}``). | | `load_capability` | function `(path: str \| Path, *, strict: bool=True)` | Load one ``capability.yaml`` / ``.json`` manifest as a target. | | `select_endpoint` | function `(target: RouteTarget, region: str \| None=None)` | The manifest endpoint to use: a region-matching one first, then the first region-less one. | | `target_from_capability` | function `(doc: dict[str, Any], *, strict: bool=True)` | Convert a manifest into a :class:`~opensmartroute.RouteTarget` (validating first when ``strict``). | | `validate_capability` | function `(doc: Any)` | Validate a manifest; returns a list of human-readable problems (empty = valid). | ## `opensmartroute.policy` Source: [src/opensmartroute/policy/__init__.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/policy/__init__.py) Policy layer: hard constraints that are never traded off against utility. | Name | Kind | Summary | |---|---|---| | `DataBoundaryRule` | class | The target's boundary must be at least as strict as the request's (public < private < on_prem). | | `JailbreakRule` | class | Risky prompts may only reach humans or targets tagged as safe. | | `LanguageRule` | class | The detected language must be declared by the target, unless it declares the wildcard language. | | `Policy` | class | Ordered chain of :data:`PolicyRule`; returns the first rejection reason or ``None``. | | `PolicyRule` | constant | A hard-constraint check: return a rejection reason, or None to let the target through. | | `allow_list` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | When the request names ``allow_targets``, reject everything else. | | `allowed_kinds` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | Reject kinds outside the request's ``allowed_kinds``. | | `context_window` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | Reject when the estimated input exceeds the target's ``context_window``. | | `cost_budget` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | Reject targets whose unit cost exceeds the request's ``max_cost_per_1k``. | | `default_rules` | function `(settings: PolicySettings \| None=None)` | The built-in chain, in evaluation order (cheap identity checks first). | | `deny_list` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | Reject targets listed in the request's ``deny_targets``. | | `enabled` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | Reject disabled targets. | | `input_tokens` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | Reject when the estimated input exceeds the target's ``max_tokens_in``. | | `latency_slo` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | Reject targets whose declared latency exceeds the request's ``max_latency_ms``. | | `modalities` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | Every modality detected in the request must be supported by the target. | | `pii` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | Requests flagged as containing PII may only reach targets with ``pii_allowed``. | | `region` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | The request's ``region`` must be one the target serves (targets with no regions serve all). | | `rule_name` | function `(rule: PolicyRule)` | Display name of a rule: its ``name`` attribute, else ``__name__``, else the class name. | | `tenant` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | The request's ``tenant`` must be permitted by targets that restrict tenants. | | `tools` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | When tools are required (constraint or ``context['tools']``), LLM targets must ``supports_tools``. | ## `opensmartroute.realtime` Source: [src/opensmartroute/realtime/__init__.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/realtime/__init__.py) Real-time operational controls: health, circuit breaking, rate & budget limits. | Name | Kind | Summary | |---|---|---| | `BreakerState` | class | Circuit-breaker states: closed (healthy), open (tripped), half_open (probing recovery). | | `CircuitBreaker` | class | Classic three-state breaker with a sliding failure window. | | `TokenBucket` | class | Rate limiter: ``rate`` tokens/sec, burst up to ``capacity``. Thread-safe. | | `Budget` | class | Rolling spend cap (e.g. USD per hour) per target or tenant. | | `LatencyWindow` | class | Rolling window of the last ``size`` latencies with nearest-rank percentiles (p50 / p90 / p99). | | `TargetHealth` | class | Live health of one target: breaker, latency EWMA + percentiles, success counts, optional rate limit / budget. | | `HealthRegistry` | class | Tracks health per target id. Shared by policy + strategy + executor. | | `HealthPolicy` | class | Policy that additionally rejects targets whose breaker is open, whose rate. | | `HealthStrategy` | class | Soft signal: observed reliability x latency-SLO fit from live health data. | ## `opensmartroute.retrieval` Source: [src/opensmartroute/retrieval.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/retrieval.py) Retrieval-based candidate narrowing for very large target pools (ToolRet / Skill-RAG). | Name | Kind | Summary | |---|---|---| | `BM25Index` | class | In-memory BM25 (Okapi) index over target documents. | | `DenseIndex` | class | Cosine index. Default = sparse signed-hashing features with an inverted index (pure. | | `NarrowingStats` | class | Counters kept by the retrieval middleware: calls, how often it narrowed, last pool and kept sizes. | | `RetrievalResult` | class | Narrowed candidate ids with fused scores and how many came from the lexical / dense legs. | | `Retriever` | class | Hybrid lexical + dense narrowing that tracks a registry (or an explicit pool). | | `execute_tool_target` | function `(registry: TargetRegistry, allow: Callable[[RouteTarget, dict[str, Any]], bool] \| None=None, target_id: str='execute_tool')` | A *tool that runs a tool by id* (with an optional allow-policy hook). | | `narrow_signals_hint` | function `(signals: Signals)` | Extra lexical hints from signals (domains / actions) appended to the retrieval query. | | `rrf` | function `(rankings: list[list[tuple[str, float]]], k: int=60, weights: list[float] \| None=None)` | Reciprocal Rank Fusion over several ranked lists of ``(id, score)``. | | `select_skill_set` | function `(request: RouteRequest \| str, pool: list[RouteTarget], k: int=3, embedder: Embedder \| None=None, redundancy: float=0.7, min_gain: float=0.02, relevance: dict[str, float] \| None=None)` | Greedy facility-location selection of complementary targets. | | `target_text` | function `(t: RouteTarget)` | Lexical document for a target: id, name, description, capabilities and up to 8 examples. | | `tokenize` | function `(text: str)` | Lower-case alphanumeric tokens. | | `tool_search_target` | function `(retriever: Retriever, registry: TargetRegistry, kinds: tuple[str, ...]=('tool', 'skill'), k: int=8, target_id: str='tool_search')` | A *tool that finds tools*: returns compact descriptors for the ``k`` best matches. | ## `opensmartroute.router` Source: [src/opensmartroute/router.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/router.py) The Router: signals -> policy -> strategies -> ensemble -> decision (+plan). | Name | Kind | Summary | |---|---|---| | `Router` | class | Signals -> policy -> strategies -> ensemble -> decision (+ optional plan). | | `NoRouteError` | re-export of [`opensmartroute.errors.NoRouteError`](#opensmartrouteerrors) | No target satisfied the hard constraints. | | `DEFAULT_WEIGHTS` | constant | : Library-default ensemble weights (``Settings().weights``); kept for backwards compatibility. | ## `opensmartroute.sdk` Source: [src/opensmartroute/sdk.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/sdk.py) OpenSmartRoute SDK: decorator-driven registration of routing components. | Name | Kind | Summary | |---|---|---| | `ComponentKind` | class | The component families a :class:`ComponentRegistry` can hold. | | `ComponentRegistry` | class | Blueprints for every routing component, with decorators that register into it. | | `FunctionMiddleware` | class | Adapts ``fn(request, next_) -> RouteDecision``. | | `FunctionSignal` | class | Adapts ``fn(request, signals) -> None \| {field: value}``. | | `FunctionStrategy` | class | Adapts ``fn(request, signals, candidates) -> {target_id: score \| StrategyScore}``. | | `Registration` | class | A registered blueprint. ``factory()`` returns a fresh component instance. | | `agent` | constant | @agent(id, ...): shorthand for an ``agent`` target. | | `components` | constant | : The process-wide registry that the top-level decorators (``opensmartroute.strategy`` ...) bind to. | | `middleware` | constant | @middleware: register a Middleware class or ``fn(request, next_route)``. | | `policy_rule` | constant | @policy_rule: register ``fn(target, request, signals) -> reason \| None``. | | `signal` | constant | @signal: register a SignalExtractor class or ``fn(request, signals) -> mapping``. | | `skill` | constant | @skill(id, ...): shorthand for a ``skill`` target. | | `strategy` | constant | @strategy(weight=, name=): register a Strategy class or ``fn(request, signals, candidates)``. | | `target` | constant | @target(id, kind, ...): the decorated callable becomes a RouteTarget handler. | | `telemetry` | constant | @telemetry: register a Telemetry sink class or factory. | | `tool` | constant | @tool(id, ...): shorthand for a ``tool`` target. | ## `opensmartroute.security` Source: [src/opensmartroute/security/__init__.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/security/__init__.py) Security controls for the routing control plane. | Name | Kind | Summary | |---|---|---| | `GadgetDetector` | re-export of [`opensmartroute.security.gadget.GadgetDetector`](#opensmartroutesecuritygadget) | Learned tail classifier. ``score(text)`` in [0, 1]; ``split(text)`` finds the gadget tail. | | `GuardMiddleware` | class | Applies :class:`InputGuard` (+ optional redaction) before routing. | | `GuardReport` | class | Outcome of :meth:`InputGuard.inspect`: pass / fail, reasons, gadget suspicion score and the cleaned text. | | `InputGuard` | class | Validates and normalises request text. Cheap: O(len(text)). | | `OriginPolicy` | re-export of [`opensmartroute.security.provenance.OriginPolicy`](#opensmartroutesecurityprovenance) | Decides which tool arguments must be user-originated and verifies them. | | `OriginRule` | re-export of [`opensmartroute.security.provenance.OriginRule`](#opensmartroutesecurityprovenance) | Per-target override. ``sensitive=None`` = infer from parameter names. | | `OriginViolation` | re-export of [`opensmartroute.security.provenance.OriginViolation`](#opensmartroutesecurityprovenance) | Raised by :class:`OriginPolicy` when a sensitive argument of a state-changing tool comes from untrusted text. | | `Redactor` | class | Replace PII with typed placeholders, e.g. ````. Reversible per-request. | | `ResourceLimitExceeded` | re-export of [`opensmartroute.security.limits.ResourceLimitExceeded`](#opensmartroutesecuritylimits) | A task went over one of its :class:`ResourceLimits` (steps, tool calls, depth, tokens, cost, wall time). | | `ResourceLimitMiddleware` | re-export of [`opensmartroute.security.limits.ResourceLimitMiddleware`](#opensmartroutesecuritylimits) | Charge one step (+ the request's token estimate) per ``route()`` call. | | `ResourceLimiter` | re-export of [`opensmartroute.security.limits.ResourceLimiter`](#opensmartroutesecuritylimits) | Thread-safe per-task accounting against :class:`ResourceLimits`. | | `ResourceLimits` | re-export of [`opensmartroute.security.limits.ResourceLimits`](#opensmartroutesecuritylimits) | Per-task ceilings that stop runaway agents; enforced by :class:`ResourceLimiter`. | | `SafetyCase` | re-export of [`opensmartroute.security.safety.SafetyCase`](#opensmartroutesecuritysafety) | One red-team scenario: an adversarial request, its clean baseline and the routing invariants to check. | | `apply_limits` | re-export of [`opensmartroute.security.limits.apply_limits`](#opensmartroutesecuritylimits) | Wrap handlers of the given kinds. Returns the number wrapped. | | `apply_origin_policy` | re-export of [`opensmartroute.security.provenance.apply_origin_policy`](#opensmartroutesecurityprovenance) | Wrap every state-changing target in ``registry``. Returns the number wrapped. | | `description_risk` | re-export of [`opensmartroute.security.injection.description_risk`](#opensmartroutesecurityinjection) | Risk of a *tool/agent description*: max of injection lexicon and learned gadget score. | | `guard_handler` | re-export of [`opensmartroute.security.provenance.guard_handler`](#opensmartroutesecurityprovenance) | Return a copy of ``target`` whose handler enforces ``policy`` before invoking the tool. | | `injection_risk` | re-export of [`opensmartroute.security.injection.injection_risk`](#opensmartroutesecurityinjection) | Shortcut for ``inspect_injection(text).score``. | | `inspect_injection` | re-export of [`opensmartroute.security.injection.inspect_injection`](#opensmartroutesecurityinjection) | Score instruction-injection likelihood with the matched lexicon entries for the trace. | | `limit_handler` | re-export of [`opensmartroute.security.limits.limit_handler`](#opensmartroutesecuritylimits) | Copy of ``target`` whose handler charges one tool call (and nesting depth) per invocation. | | `load_secret` | function `(name: str, *, file_env_suffix: str='_FILE', default: str \| None=None)` | 12-factor secret loading: ``NAME`` env var, else the file at ``NAME_FILE`` (k8s/Docker secrets). | | `mark_untrusted` | re-export of [`opensmartroute.security.provenance.mark_untrusted`](#opensmartroutesecurityprovenance) | Record content that entered the request from a non-user source. | | `run_safety_suite` | re-export of [`opensmartroute.security.safety.run_safety_suite`](#opensmartroutesecuritysafety) | Run every case through ``route`` and return pass/fail details grouped by category. | | `sanitize_for_prompt` | function `(text: str, max_len: int=4000)` | Make user text safe to embed in an LLM-judge prompt. | | `shannon_entropy` | function `(s: str)` | Character-level Shannon entropy in bits (high values flag encoded / random payloads). | | `strip_steering` | re-export of [`opensmartroute.security.injection.strip_steering`](#opensmartroutesecurityinjection) | Remove sentences that instruct the *router* (self-declared complexity, "use the best. | | `synthesize_gadget_corpus` | re-export of [`opensmartroute.security.gadget.synthesize_gadget_corpus`](#opensmartroutesecuritygadget) | Labelled ``(text, label)`` rows: clean prompts + prompts with an appended gadget. | ## `opensmartroute.security.gadget` Source: [src/opensmartroute/security/gadget.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/security/gadget.py) Learned confounder-gadget detector (Rerouting LLM Routers, Shafran et al. 2025). | Name | Kind | Summary | |---|---|---| | `CLEAN` | constant | class labels used by the detector and the synthetic corpus. | | `GADGET` | constant | class labels used by the detector and the synthetic corpus. | | `GadgetDetector` | class | Learned tail classifier. ``score(text)`` in [0, 1]; ``split(text)`` finds the gadget tail. | | `synthesize_gadget_corpus` | function `(per_template: int=2, n_gadgets: int=260, seed: int=0)` | Labelled ``(text, label)`` rows: clean prompts + prompts with an appended gadget. | | `synthesize_gadgets` | function `(n: int, seed: int=0)` | Adversarial suffixes drawn from the gadget families described in the paper. | | `tail_features` | function `(tokens: Sequence[str])` | Hand-crafted statistics of a token window (all in [0, 1]) – complements hashed n-grams. | ## `opensmartroute.security.injection` Source: [src/opensmartroute/security/injection.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/security/injection.py) Instruction-injection detection for *text that is not the user's request*. | Name | Kind | Summary | |---|---|---| | `InjectionReport` | class | Instruction-injection score in [0, 1] plus the lexicon hits that produced it. | | `description_risk` | function `(text: str, gadget_score: float \| None=None)` | Risk of a *tool/agent description*: max of injection lexicon and learned gadget score. | | `injection_risk` | function `(text: str)` | Shortcut for ``inspect_injection(text).score``. | | `inspect_injection` | function `(text: str)` | Score instruction-injection likelihood with the matched lexicon entries for the trace. | ## `opensmartroute.security.limits` Source: [src/opensmartroute/security/limits.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/security/limits.py) Resource-amplification limits ("Beyond Max Tokens"). | Name | Kind | Summary | |---|---|---| | `ResourceLimitExceeded` | class | A task went over one of its :class:`ResourceLimits` (steps, tool calls, depth, tokens, cost, wall time). | | `ResourceLimitMiddleware` | class | Charge one step (+ the request's token estimate) per ``route()`` call. | | `ResourceLimiter` | class | Thread-safe per-task accounting against :class:`ResourceLimits`. | | `ResourceLimits` | class | Per-task ceilings that stop runaway agents; enforced by :class:`ResourceLimiter`. | | `TaskUsage` | class | Running consumption of one task (steps, tool calls, depth, tokens, cost, timestamps). | | `apply_limits` | function `(registry: TargetRegistry, limiter: ResourceLimiter, kinds: tuple[str, ...]=('tool', 'agent', 'workflow'))` | Wrap handlers of the given kinds. Returns the number wrapped. | | `limit_handler` | function `(target: RouteTarget, limiter: ResourceLimiter)` | Copy of ``target`` whose handler charges one tool call (and nesting depth) per invocation. | | `task_id_of` | function `(request: RouteRequest, key: str='task_id')` | ``context[key]`` when present, otherwise the request id (each request is its own task). | ## `opensmartroute.security.provenance` Source: [src/opensmartroute/security/provenance.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/security/provenance.py) Origin (provenance) policy for tool parameters – ROPE-style control-flow integrity. | Name | Kind | Summary | |---|---|---| | `OriginFinding` | class | Where one tool argument's value came from (user, context, untrusted content or unknown). | | `OriginPolicy` | class | Decides which tool arguments must be user-originated and verifies them. | | `OriginReport` | class | Provenance check of a tool call: all findings plus the sensitive parameters with a disallowed origin. | | `OriginRule` | class | Per-target override. ``sensitive=None`` = infer from parameter names. | | `OriginViolation` | class | Raised by :class:`OriginPolicy` when a sensitive argument of a state-changing tool comes from untrusted text. | | `SENSITIVE_PARAM_HINTS` | constant | Parameter-name fragments treated as sensitive when a target has no explicit OriginRule. | | `STATE_CHANGING_ACTIONS` | constant | Verbs in a tool's id / description / actions that mark it as state-changing (side effects). | | `apply_origin_policy` | function `(registry: TargetRegistry, policy: OriginPolicy)` | Wrap every state-changing target in ``registry``. Returns the number wrapped. | | `guard_handler` | function `(target: RouteTarget, policy: OriginPolicy, arguments_key: str='tool_arguments')` | Return a copy of ``target`` whose handler enforces ``policy`` before invoking the tool. | | `mark_untrusted` | function `(request: RouteRequest, text: str, source: str='retrieved')` | Record content that entered the request from a non-user source. | | `untrusted_texts` | function `(request: RouteRequest)` | ``(source, text)`` pairs recorded by :func:`mark_untrusted` on this request. | | `user_texts` | function `(request: RouteRequest)` | Everything the user actually typed: current text, prior user turns and the pre-guard original text. | ## `opensmartroute.security.safety` Source: [src/opensmartroute/security/safety.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/security/safety.py) Safety-routing regression suite ("When Safety Routing Breaks"). | Name | Kind | Summary | |---|---|---| | `SafetyCase` | class | One red-team scenario: an adversarial request, its clean baseline and the routing invariants to check. | | `SafetyResult` | class | Verdict for one :class:`SafetyCase`: chosen vs baseline target and the invariants that failed. | | `default_cases` | function `(seed: int=0)` | Deployment-agnostic red-team cases; extend with catalogue-specific ones. | | `run_safety_suite` | function `(route: RouteCall, cases: Sequence[SafetyCase] \| None=None, *, seed: int=0)` | Run every case through ``route`` and return pass/fail details grouped by category. | ## `opensmartroute.server` Source: [src/opensmartroute/server.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/server.py) Optional FastAPI server exposing the router over HTTP. | Name | Kind | Summary | |---|---|---| | `AUTO_SLUG` | constant | : Slug prefix of the router's own pseudo-models on the OpenAI-compatible proxy (``osr/auto``). | | `DEFAULT_PROXY_ALIASES` | constant | : Model names that mean "let the router choose" on the OpenAI-compatible proxy. | | `AUTO_VARIANTS` | constant | : ``osr/auto:`` presets: objective weights, hard constraints and allowed kinds. | | `APP_HEADER` | constant | : Request header naming the calling application (OpenRouter-style attribution); echoed in metadata. | | `TARGET_HEADER` | constant | : Response headers carrying the decision (target id, request id) beside the OpenAI-shaped body. | | `REQUEST_ID_HEADER` | constant | : Response headers carrying the decision (target id, request id) beside the OpenAI-shaped body. | | `OPEN_PATHS` | constant | : Paths that never require a token when access control is on (probes, metrics, token check, OpenAPI pages). | | `UNTRACED_PATHS` | constant | : Paths that never open an ``http.request`` span (probes and the observability endpoints themselves). | | `bearer_token` | function `(headers: Any)` | The access token of a request: ``X-API-Key`` first, else ``Authorization: Bearer ``. | | `token_accepted` | function `(token: str \| None, tokens: Iterable[str])` | Constant-time membership test of ``token`` in the configured access tokens. | | `create_app` | function `(router: Any, proxy_aliases: frozenset[str]=DEFAULT_PROXY_ALIASES, autopilot: Any=None, auth_tokens: Iterable[str]=())` | FastAPI app: ``/route``, ``/feedback``, ``/targets``, ``/stats``, ``/healthz`` and the OpenAI-compatible ``/v1``. | | `resolve_model` | function `(model: str, aliases: frozenset[str]=DEFAULT_PROXY_ALIASES)` | Split a proxy ``model`` into ``(pinned target id \| None, route options)``. | | `proxy_request` | function `(body: dict[str, Any], aliases: frozenset[str]=frozenset({'auto'}), app: str \| None=None)` | Turn an OpenAI chat-completion body into a :class:`RouteRequest`. | | `proxy_response` | function `(decision: RouteDecision, result: Any, request: RouteRequest \| None=None)` | Shape an execution result as an OpenAI ``chat.completion`` object (``model`` = chosen target id). | ## `opensmartroute.settings` Source: [src/opensmartroute/settings.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/settings.py) Typed, environment-overridable settings: the only home for OpenSmartRoute's tunable constants. | Name | Kind | Summary | |---|---|---| | `BanditSettings` | class | Thompson-sampling strategy (:class:`~opensmartroute.BanditStrategy`). | | `CapabilitySettings` | class | Capability-fit strategy (:class:`~opensmartroute.CapabilityStrategy`). | | `ObservabilitySettings` | class | Tracing and event capture (:mod:`opensmartroute.observability`); read by ``Tracer.from_settings``. | | `PolicySettings` | class | Hard-constraint thresholds (read by :class:`~opensmartroute.Policy`). | | `RoutingSettings` | class | Ensemble, confidence and plan composition (read by :class:`~opensmartroute.Router`). | | `RulesSettings` | class | Declarative rules strategy (:class:`~opensmartroute.RulesStrategy`). | | `SLMSettings` | class | Routing SLM, dataset collection and the self-improvement loop (:mod:`opensmartroute.learning.slm`). | | `ServerSettings` | class | ``osr serve`` (:mod:`opensmartroute.server`): access control of the self-hosted HTTP API. | | `Settings` | class | All tunables, grouped by consumer. Immutable; derive variants with :meth:`replace`. | | `WeightSettings` | class | Ensemble weight per strategy name (looked up as ``weights[strategy.name]``). | | `configure` | function `(settings: Settings \| None=None, **groups: Any)` | Install process-wide settings. ``configure()`` with no arguments re-reads the environment;. | | `get_settings` | function `()` | The process-wide :class:`Settings` (environment overlay applied once, lazily). | | `resolve` | function `(settings: Settings \| None)` | ``settings`` if given, else the process-wide settings (component constructors use this). | ## `opensmartroute.signals` Source: [src/opensmartroute/signals/__init__.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/signals/__init__.py) Signal extraction: cheap, deterministic features computed from the request. | Name | Kind | Summary | |---|---|---| | `ACTION_LEXICON` | constant | Action label -> trigger phrases; the keys are the ontology action names (summarize, translate, ... escalate). | | `DEFAULT_EXTRACTORS` | constant | after domain detection (uses domain bonus). | | `DEFAULT_ONTOLOGY` | re-export of [`opensmartroute.signals.ontology.DEFAULT_ONTOLOGY`](#opensmartroutesignalsontology) | the built-in TASK_TYPES ontology used by TaskTypeSignal. | | `DOMAIN_LEXICON` | constant | Domain label -> trigger phrases; the keys are the ontology domain names targets and skills may declare. | | `EVENT_LEXICON` | re-export of [`opensmartroute.signals.events.EVENT_LEXICON`](#opensmartroutesignalsevents) | Event name -> (domains, actions). Keys are matched exactly, then by ``subject.*`` prefix. | | `EVENT_SUBJECTS` | re-export of [`opensmartroute.signals.events.EVENT_SUBJECTS`](#opensmartroutesignalsevents) | Generic fallbacks when the exact event / prefix is unknown: subject -> domain, verb -> action. | | `EVENT_VERBS` | re-export of [`opensmartroute.signals.events.EVENT_VERBS`](#opensmartroutesignalsevents) | event verb -> action when the exact event / prefix is unknown. | | `TASK_PRIORS` | re-export of [`opensmartroute.signals.models.TASK_PRIORS`](#opensmartroutesignalsmodels) | name: (difficulty, reasoning_need, expected_output_tokens). | | `TASK_TYPES` | re-export of [`opensmartroute.signals.ontology.TASK_TYPES`](#opensmartroutesignalsontology) | ---------------------------------------------------------------- transformation. | | `ComplexitySignal` | class | Hybrid-LLM-style difficulty estimate in [0, 1]. | | `DomainActionSignal` | class | Arch-Router-style domain / action detection via lexicon matching. | | `DraftResponseSignal` | re-export of [`opensmartroute.signals.uncertainty.DraftResponseSignal`](#opensmartroutesignalsuncertainty) | Query-response mixed representation: run a cheap drafter and expose draft features. | | `EventInfo` | re-export of [`opensmartroute.signals.events.EventInfo`](#opensmartroutesignalsevents) | Parsed event: raw name, subject, verb and the domains / actions it implies. | | `EventSignal` | re-export of [`opensmartroute.signals.events.EventSignal`](#opensmartroutesignalsevents) | Domains / actions from ``context["event"]`` and ``context["intent"]`` (event-driven requests). | | `EventTrigger` | re-export of [`opensmartroute.signals.uncertainty.EventTrigger`](#opensmartroutesignalsuncertainty) | Event-triggered invocation: evaluate uncertainty features against rules and return the. | | `HashedClassifier` | re-export of [`opensmartroute.signals.models.HashedClassifier`](#opensmartroutesignalsmodels) | Multinomial logistic regression over hashed features (sparse weights per class). | | `HashedFeaturizer` | re-export of [`opensmartroute.signals.models.HashedFeaturizer`](#opensmartroutesignalsmodels) | Hashing-trick sparse features: words, bigrams, character n-grams and a few numeric text statistics. | | `HashedRegressor` | re-export of [`opensmartroute.signals.models.HashedRegressor`](#opensmartroutesignalsmodels) | Squared-loss linear regressor on hashed features; output clipped to [lo, hi]. | | `HistorySignal` | class | Conversation-state features from ``request.history`` (RCRouter-style):. | | `LanguageSignal` | class | ``language`` from script / stop-word hints (``LANGUAGE_HINTS``); defaults to ``en``. | | `LearnedDifficultySignal` | class | Blend the heuristic complexity with a trained regressor (Hybrid-LLM style difficulty). | | `LengthSignal` | class | ``token_estimate`` from text + history length (about 4 characters per token). | | `ModalitySignal` | class | ``modalities`` from context keys (``images`` / ``image_url``, ``audio``, ``files``) on top of ``text``. | | `OutputLengthSignal` | class | Expected output tokens (drives cost estimates and effort/max_tokens choices). | | `ProfileSignal` | class | User-profile features (``request.profile``): tier, expertise, preferences. | | `ReasoningNeedSignal` | class | How much a request benefits from extended thinking (ThinkSwitcher / Sketch-of-Thought). | | `SensitivitySignal` | class | PII and prompt-injection / jailbreak heuristics (vLLM-semantic-router-style). | | `SignalExtractor` | class | Fills in part of a :class:`Signals` object. | | `SignalModelBundle` | re-export of [`opensmartroute.signals.models.SignalModelBundle`](#opensmartroutesignalsmodels) | All learned signal models together, persisted as one JSON document. | | `TaskOntology` | re-export of [`opensmartroute.signals.ontology.TaskOntology`](#opensmartroutesignalsontology) | Lookup helpers over :data:`TASK_TYPES`. | | `TaskType` | re-export of [`opensmartroute.signals.ontology.TaskType`](#opensmartroutesignalsontology) | One leaf of the task ontology: family, description, the actions it maps to and template phrasings. | | `TaskTypeSignal` | class | Ontology task type. Uses the learned classifier when one is loaded, else maps the. | | `TriggerRule` | re-export of [`opensmartroute.signals.uncertainty.TriggerRule`](#opensmartroutesignalsuncertainty) | ``feature >= threshold`` (or ``<=`` when ``below=True``) fires ``action``. | | `UncertaintyGate` | re-export of [`opensmartroute.signals.uncertainty.UncertaintyGate`](#opensmartroutesignalsuncertainty) | A cascade quality gate built from response uncertainty. | | `VerbalisedDifficultySignal` | re-export of [`opensmartroute.signals.uncertainty.VerbalisedDifficultySignal`](#opensmartroutesignalsuncertainty) | Blend a verbalised difficulty into ``complexity`` and ``reasoning_need``. | | `WorkflowSignal` | re-export of [`opensmartroute.signals.events.WorkflowSignal`](#opensmartroutesignalsevents) | Detect a workflow trigger (``context["workflow"]`` or "run the workflow\|process\|..."). | | `extract_signals` | function `(request: RouteRequest, extractors: list[SignalExtractor] \| None=None)` | Run the extractor chain (``DEFAULT_EXTRACTORS`` unless given) and return the populated :class:`Signals`. | | `learned_extractors` | function `(bundle: SignalModelBundle)` | Extractor pipeline with the trained :class:`SignalModelBundle` plugged in. | | `load_training_rows` | re-export of [`opensmartroute.signals.models.load_training_rows`](#opensmartroutesignalsmodels) | Read JSONL training rows. Accepts eval-dataset rows too (``prompt``/``text``,. | | `parse_difficulty` | re-export of [`opensmartroute.signals.uncertainty.parse_difficulty`](#opensmartroutesignalsuncertainty) | Map a verbalised difficulty (number, ``"7/10"``, ``"hard"``) to [0, 1]; ``None`` if unparseable. | | `parse_event` | re-export of [`opensmartroute.signals.events.parse_event`](#opensmartroutesignalsevents) | Map an event name (``subject.verb``, ``subject:verb`` or ``subject_verb``) onto domains / actions. | | `response_uncertainty` | re-export of [`opensmartroute.signals.uncertainty.response_uncertainty`](#opensmartroutesignalsuncertainty) | Post-hoc uncertainty features for a set of sampled answers. | | `semantic_entropy` | re-export of [`opensmartroute.signals.uncertainty.semantic_entropy`](#opensmartroutesignalsuncertainty) | Entropy (nats) over meaning clusters of sampled answers; 0 when every sample agrees. | | `synthesize_dataset` | re-export of [`opensmartroute.signals.models.synthesize_dataset`](#opensmartroutesignalsmodels) | Generate labelled prompts from ontology templates x topics with light noise. | ## `opensmartroute.signals.events` Source: [src/opensmartroute/signals/events.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/signals/events.py) Event- and workflow-driven signals for agentic inputs. | Name | Kind | Summary | |---|---|---| | `EVENT_LEXICON` | constant | Event name -> (domains, actions). Keys are matched exactly, then by ``subject.*`` prefix. | | `EVENT_SUBJECTS` | constant | Generic fallbacks when the exact event / prefix is unknown: subject -> domain, verb -> action. | | `EVENT_VERBS` | constant | event verb -> action when the exact event / prefix is unknown. | | `EventInfo` | class | Parsed event: raw name, subject, verb and the domains / actions it implies. | | `EventSignal` | class | Domains / actions from ``context["event"]`` and ``context["intent"]`` (event-driven requests). | | `WorkflowSignal` | class | Detect a workflow trigger (``context["workflow"]`` or "run the workflow\|process\|..."). | | `parse_event` | function `(name: str)` | Map an event name (``subject.verb``, ``subject:verb`` or ``subject_verb``) onto domains / actions. | ## `opensmartroute.signals.models` Source: [src/opensmartroute/signals/models.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/signals/models.py) Learned signal models: hashed n-gram linear models with no dependencies. | Name | Kind | Summary | |---|---|---| | `TASK_PRIORS` | constant | name: (difficulty, reasoning_need, expected_output_tokens). | | `HashedClassifier` | class | Multinomial logistic regression over hashed features (sparse weights per class). | | `HashedFeaturizer` | class | Hashing-trick sparse features: words, bigrams, character n-grams and a few numeric text statistics. | | `HashedRegressor` | class | Squared-loss linear regressor on hashed features; output clipped to [lo, hi]. | | `SignalModelBundle` | class | All learned signal models together, persisted as one JSON document. | | `load_training_rows` | function `(path: str \| Path)` | Read JSONL training rows. Accepts eval-dataset rows too (``prompt``/``text``,. | | `synthesize_dataset` | function `(ontology: TaskOntology=DEFAULT_ONTOLOGY, topics: Sequence[str] \| None=None, per_template: int=6, seed: int=0)` | Generate labelled prompts from ontology templates x topics with light noise. | ## `opensmartroute.signals.ontology` Source: [src/opensmartroute/signals/ontology.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/signals/ontology.py) Task ontology: families -> types -> subtypes, with an orthogonal *domain* axis. | Name | Kind | Summary | |---|---|---| | `DEFAULT_ONTOLOGY` | constant | the built-in TASK_TYPES ontology used by TaskTypeSignal. | | `TASK_TYPES` | constant | ---------------------------------------------------------------- transformation. | | `TaskOntology` | class | Lookup helpers over :data:`TASK_TYPES`. | | `TaskType` | class | One leaf of the task ontology: family, description, the actions it maps to and template phrasings. | ## `opensmartroute.signals.uncertainty` Source: [src/opensmartroute/signals/uncertainty.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/signals/uncertainty.py) Uncertainty signals that come from *outside the query text*. | Name | Kind | Summary | |---|---|---| | `HEDGES` | constant | Hedging phrases counted in a draft response (uncertainty evidence). | | `SELF_CORRECTIONS` | constant | Self-correction markers counted in a draft response. | | `DraftResponseSignal` | class | Query-response mixed representation: run a cheap drafter and expose draft features. | | `EventTrigger` | class | Event-triggered invocation: evaluate uncertainty features against rules and return the. | | `TriggerRule` | class | ``feature >= threshold`` (or ``<=`` when ``below=True``) fires ``action``. | | `UncertaintyGate` | class | A cascade quality gate built from response uncertainty. | | `VerbalisedDifficultySignal` | class | Blend a verbalised difficulty into ``complexity`` and ``reasoning_need``. | | `draft_features` | function `(text: str, draft: str, expected_tokens: int=0)` | Features of a cheap draft answer relative to the query: hedging, refusal, self-correction,. | | `parse_difficulty` | function `(value: Any)` | Map a verbalised difficulty (number, ``"7/10"``, ``"hard"``) to [0, 1]; ``None`` if unparseable. | | `response_uncertainty` | function `(samples: list[str], embedder: Embedder \| None=None, threshold: float=0.85, p_true: float \| None=None)` | Post-hoc uncertainty features for a set of sampled answers. | | `semantic_clusters` | function `(samples: Iterable[str], embedder: Embedder \| None=None, threshold: float=0.85)` | Greedy single-link clustering of answers into meaning classes; returns a cluster id per sample. | | `semantic_entropy` | function `(samples: Iterable[str], embedder: Embedder \| None=None, threshold: float=0.85)` | Entropy (nats) over meaning clusters of sampled answers; 0 when every sample agrees. | ## `opensmartroute.stack` Source: [src/opensmartroute/stack.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/stack.py) Declarative *stacks*: one document that describes a whole routing setup. | Name | Kind | Summary | |---|---|---| | `REGISTRY_SCHEME` | constant | : Scheme of marketplace references inside ``imports`` (``registry://[@]``). | | `STACK_VERSION` | constant | : Current stack document version (the ``osr`` field). | | `Resolver` | constant | : Resolver for ``registry://`` imports: takes the reference (without the scheme) and returns the stack document. | | `Stack` | class | A resolved stack: every import merged, ready to build a router. | | `StackChange` | class | One line of a plan: ``add`` / ``change`` / ``remove`` of a target, rule, setting or objective weight. | | `StackPlan` | class | What applying ``desired`` would change compared with ``current`` (``osr stack plan``). | | `dump_stack` | function `(doc: dict[str, Any], fmt: str \| None=None)` | Serialise a stack document: ``fmt`` = ``"yaml"`` \| ``"json"``; default YAML when PyYAML is installed. | | `is_stack` | function `(doc: Any)` | True when ``doc`` looks like a stack document (``kind: stack`` or an ``osr`` version with stack sections). | | `load_stack` | function `(source: str \| Path \| dict[str, Any], *, resolver: Resolver \| None=None, settings: Settings \| None=None, _seen: frozenset[str]=frozenset())` | Load and resolve a stack (file path, ``registry://`` reference or inline mapping), imports first. | | `plan_stack` | function `(desired: Stack, current: Stack \| None=None)` | Diff ``desired`` against ``current`` (``None`` = nothing deployed: everything is an ``add``). | | `starter_stack` | function `(name: str='starter', description: str='')` | A minimal, valid stack document to start from (what the marketplace publish wizard pre-fills). | | `validate_stack` | function `(doc: Any)` | Validate a stack document; returns human-readable problems (empty list = valid). Imports are not resolved. | ## `opensmartroute.strategies` Source: [src/opensmartroute/strategies/__init__.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/__init__.py) | Name | Kind | Summary | |---|---|---| | `DEFAULT_RULES` | re-export of [`opensmartroute.strategies.protocol.DEFAULT_RULES`](#opensmartroutestrategiesprotocol) | Built-in risk / budget ladder: handoff > debate > aggregate > cascade > single. | | `AggregateResult` | re-export of [`opensmartroute.strategies.aggregate.AggregateResult`](#opensmartroutestrategiesaggregate) | Outcome of :class:`MixtureOfAgents`: the final response, participants, winner and agreement ratio. | | `AnnotatorPool` | re-export of [`opensmartroute.strategies.human.AnnotatorPool`](#opensmartroutestrategieshuman) | Skill estimation from agreement (Dawid-Skene EM) and quorum selection. Labels usually arrive from. | | `AnnotatorSkill` | re-export of [`opensmartroute.strategies.human.AnnotatorSkill`](#opensmartroutestrategieshuman) | Per-domain Beta accuracy estimate, cost and latency of one human annotator. | | `AuctionResult` | re-export of [`opensmartroute.strategies.auction.AuctionResult`](#opensmartroutestrategiesauction) | Winner, second-price payment and the per-bidder surplus / corrected claims of one auction. | | `AuctionStrategy` | re-export of [`opensmartroute.strategies.auction.AuctionStrategy`](#opensmartroutestrategiesauction) | Error-aware reverse auction: bias-corrected bids, highest surplus wins, second-price payment. | | `BanditStrategy` | re-export of [`opensmartroute.strategies.bandit.BanditStrategy`](#opensmartroutestrategiesbandit) | Thompson-sampling Beta bandit per (context, target); context = dominant domain (+ plan role). | | `BeliefTracker` | re-export of [`opensmartroute.strategies.cascade.BeliefTracker`](#opensmartroutestrategiescascade) | AutoMix-style POMDP belief over the hidden state "current answer is correct". | | `BudgetVariant` | re-export of [`opensmartroute.strategies.elastic.BudgetVariant`](#opensmartroutestrategieselastic) | One budget of an elastic model. ``quality`` and ``cost_scale`` are relative to the parent. | | `CacheEntry` | re-export of [`opensmartroute.strategies.semantic_cache.CacheEntry`](#opensmartroutestrategiessemantic_cache) | A cached response keyed by the embedding of its prompt, with source target, quality, timestamp and hits. | | `CacheHit` | re-export of [`opensmartroute.strategies.semantic_cache.CacheHit`](#opensmartroutestrategiessemantic_cache) | The matched :class:`CacheEntry` and its similarity to the query. | | `CapabilityStrategy` | re-export of [`opensmartroute.strategies.capability.CapabilityStrategy`](#opensmartroutestrategiescapability) | Declarative fit: domain / action overlap, complexity band, language, modality and quality prior. | | `Cascade` | re-export of [`opensmartroute.strategies.cascade.Cascade`](#opensmartroutestrategiescascade) | Execute ranked targets in planner order (cheapest / MDP / POMDP), stopping when the quality gate passes. | | `CascadePlanner` | re-export of [`opensmartroute.strategies.cascade.CascadePlanner`](#opensmartroutestrategiescascade) | Finite-horizon MDP over an ordered cascade with a stop action after each step. | | `CascadeResult` | re-export of [`opensmartroute.strategies.cascade.CascadeResult`](#opensmartroutestrategiescascade) | Final response of a cascade with every step, the planned order and the planner's expected value. | | `CascadeStep` | re-export of [`opensmartroute.strategies.cascade.CascadeStep`](#opensmartroutestrategiescascade) | One executed rung of a cascade: gate quality, latency, cost, acceptance and POMDP belief. | | `DeferStrategy` | re-export of [`opensmartroute.strategies.defer.DeferStrategy`](#opensmartroutestrategiesdefer) | Learning-to-defer: scores human targets by risk, PII, escalation intent, frustration and model uncertainty. | | `EdgeCloudStrategy` | re-export of [`opensmartroute.strategies.edge.EdgeCloudStrategy`](#opensmartroutestrategiesedge) | Edge vs cloud tier choice: learned edge competence per complexity bucket vs upload / decode penalties. | | `EffortStrategy` | re-export of [`opensmartroute.strategies.defer.EffortStrategy`](#opensmartroutestrategiesdefer) | Match a target's reasoning ``effort_level`` to ``signals.reasoning_need``; penalise over- and under-thinking. | | `EscalationDecision` | re-export of [`opensmartroute.strategies.escalation.EscalationDecision`](#opensmartroutestrategiesescalation) | Verdict after a streamed chunk: continue / escalate / done with the competence estimate and reason. | | `EscalationResult` | re-export of [`opensmartroute.strategies.modality.EscalationResult`](#opensmartroutestrategiesmodality) | Outcome of :class:`ModalityEscalation`: which target answered, whether it escalated, confidence, cost. | | `HiddenStateStrategy` | re-export of [`opensmartroute.strategies.probe.HiddenStateStrategy`](#opensmartroutestrategiesprobe) | Dirichlet probe over a dense request representation; confidence drops with epistemic uncertainty. | | `HumanRoutingStrategy` | re-export of [`opensmartroute.strategies.human.HumanRoutingStrategy`](#opensmartroutestrategieshuman) | Scores HUMAN candidates by estimated accuracy in the request's domain. | | `ImportanceGate` | re-export of [`opensmartroute.strategies.memory.ImportanceGate`](#opensmartroutestrategiesmemory) | Hashed logistic gate: P(item will be used later \| text, hint). | | `LLMJudgeStrategy` | re-export of [`opensmartroute.strategies.llm_judge.LLMJudgeStrategy`](#opensmartroutestrategiesllm_judge) | LLM-as-router with optional **score calibration**. | | `MemoryItem` | re-export of [`opensmartroute.strategies.memory.MemoryItem`](#opensmartroutestrategiesmemory) | One stored memory: text, size, turn written, learned importance gate, tier and usage counters. | | `MemoryRouter` | re-export of [`opensmartroute.strategies.memory.MemoryRouter`](#opensmartroutestrategiesmemory) | Routes memory writes to tiers under budgets and recalls the most valuable items per token. | | `MemoryTier` | re-export of [`opensmartroute.strategies.memory.MemoryTier`](#opensmartroutestrategiesmemory) | A storage tier: token capacity, read / write cost per token, latency and the minimum item value it accepts. | | `MixtureOfAgents` | re-export of [`opensmartroute.strategies.aggregate.MixtureOfAgents`](#opensmartroutestrategiesaggregate) | Route-or-aggregate switch: below a confidence threshold call the top-k alternatives and aggregate. | | `ModalityEscalation` | re-export of [`opensmartroute.strategies.modality.ModalityEscalation`](#opensmartroutestrategiesmodality) | Try the text-only target on the surrogate first; escalate to the multimodal target on low confidence. | | `ModalityStrategy` | re-export of [`opensmartroute.strategies.modality.ModalityStrategy`](#opensmartroutestrategiesmodality) | Coverage of the request's modalities by each candidate, with surrogate discounts. | | `MultiRoundExecutor` | re-export of [`opensmartroute.strategies.progress.MultiRoundExecutor`](#opensmartroutestrategiesprogress) | Router-R1 style: keep routing/executing until the judge accepts or rounds run out. | | `ProgressRouter` | re-export of [`opensmartroute.strategies.progress.ProgressRouter`](#opensmartroutestrategiesprogress) | Route each step of a task with trajectory context. | | `ProtocolChoice` | re-export of [`opensmartroute.strategies.protocol.ProtocolChoice`](#opensmartroutestrategiesprotocol) | The protocol picked for a request, the risk that drove it, the reason and the matching rule. | | `ProtocolPolicy` | re-export of [`opensmartroute.strategies.protocol.ProtocolPolicy`](#opensmartroutestrategiesprotocol) | Ordered rule table from (risk, budget, task type) to a protocol, with per-protocol ledgers. | | `ProtocolRule` | re-export of [`opensmartroute.strategies.protocol.ProtocolRule`](#opensmartroutestrategiesprotocol) | First matching rule wins. ``None`` bounds are open. | | `QuorumPlan` | re-export of [`opensmartroute.strategies.human.QuorumPlan`](#opensmartroutestrategieshuman) | A chosen set of annotators with the quorum's majority accuracy, total cost and latency. | | `RecallResult` | re-export of [`opensmartroute.strategies.memory.RecallResult`](#opensmartroutestrategiesmemory) | Items recalled for a turn with their total tokens, read cost and latency. | | `Round` | re-export of [`opensmartroute.strategies.progress.Round`](#opensmartroutestrategiesprogress) | One route -> execute -> judge iteration of :class:`MultiRoundExecutor`. | | `RoundResult` | re-export of [`opensmartroute.strategies.progress.RoundResult`](#opensmartroutestrategiesprogress) | Final response of a multi-round run with every round and the shared task id. | | `Rule` | re-export of [`opensmartroute.strategies.rules.Rule`](#opensmartroutestrategiesrules) | If all `when` conditions match, boost `prefer` targets and penalise `avoid`. | | `RulesStrategy` | re-export of [`opensmartroute.strategies.rules.RulesStrategy`](#opensmartroutestrategiesrules) | Applies declarative :class:`Rule` preferences (prefer / avoid / pin) when their ``when`` conditions match. | | `SelfEscalation` | re-export of [`opensmartroute.strategies.escalation.SelfEscalation`](#opensmartroutestrategiesescalation) | Streaming competence monitor with Bayesian optimal stopping. | | `SemanticCache` | re-export of [`opensmartroute.strategies.semantic_cache.SemanticCache`](#opensmartroutestrategiessemantic_cache) | Embedding-keyed LRU cache with TTL and a similarity threshold. Safe to share across threads:. | | `SemanticCacheStrategy` | re-export of [`opensmartroute.strategies.semantic_cache.SemanticCacheStrategy`](#opensmartroutestrategiessemantic_cache) | Scores the cache target by calibrated hit quality; leaves real targets to the other strategies. | | `SessionAffinityStrategy` | re-export of [`opensmartroute.strategies.session.SessionAffinityStrategy`](#opensmartroutestrategiessession) | Prefer the target already serving ``context["session_id"]`` unless the intent shifted or it failed. | | `SessionState` | re-export of [`opensmartroute.strategies.session.SessionState`](#opensmartroutestrategiessession) | What the strategy remembers about one session. | | `SimilarityStrategy` | re-export of [`opensmartroute.strategies.similarity.SimilarityStrategy`](#opensmartroutestrategiessimilarity) | Embed the request and each target's examples / description; score by best and top-k mean similarity. | | `SpeculativeCascade` | re-export of [`opensmartroute.strategies.speculative.SpeculativeCascade`](#opensmartroutestrategiesspeculative) | Two-target cascade that overlaps the draft and the strong call when it pays. | | `SpeculativeResult` | re-export of [`opensmartroute.strategies.speculative.SpeculativeResult`](#opensmartroutestrategiesspeculative) | Outcome of a speculative run: chosen mode, whether the draft was accepted, latency and cost. | | `Strategy` | re-export of [`opensmartroute.strategies.base.Strategy`](#opensmartroutestrategiesbase) | Scores each candidate target in [0, 1] and explains why. | | `StreamResult` | re-export of [`opensmartroute.strategies.escalation.StreamResult`](#opensmartroutestrategiesescalation) | Text consumed by :func:`wrap_stream`, whether it escalated and the final decision. | | `TaskProgress` | re-export of [`opensmartroute.strategies.progress.TaskProgress`](#opensmartroutestrategiesprogress) | Where a multi-step task stands: step index, budget spent, failures in a row, last target, history. | | `TaskTableStrategy` | re-export of [`opensmartroute.strategies.task_table.TaskTableStrategy`](#opensmartroutestrategiestask_table) | Static ``task_type -> target -> quality`` table with family and prior fallbacks; learns from outcomes. | | `TokenBudgetStrategy` | re-export of [`opensmartroute.strategies.elastic.TokenBudgetStrategy`](#opensmartroutestrategieselastic) | Score budgeted siblings by fit between the budget and the tokens the answer needs. | | `decide_effort` | re-export of [`opensmartroute.strategies.defer.decide_effort`](#opensmartroutestrategiesdefer) | Pick the effort level whose numeric value is closest to the reasoning need. | | `default_bid` | re-export of [`opensmartroute.strategies.auction.default_bid`](#opensmartroutestrategiesauction) | Catalogue-derived bid: quality prior as the claim, unit cost per 1k tokens x tokens as the price. | | `default_strategies` | function `(seed: int \| None=None, settings: Settings \| None=None, state_dir: str \| Path \| None=None)` | The zero-configuration ensemble: capability fit + example similarity + Thompson bandit. | | `expand_elastic` | re-export of [`opensmartroute.strategies.elastic.expand_elastic`](#opensmartroutestrategieselastic) | Create one sibling target per budget variant, sharing the parent's family, handler and metadata. | | `expected_mode_costs` | re-export of [`opensmartroute.strategies.speculative.expected_mode_costs`](#opensmartroutestrategiesspeculative) | Expected weighted cost (``objective.cost`` x USD + ``objective.latency`` x seconds) per mode. | | `failure_risk` | re-export of [`opensmartroute.strategies.protocol.failure_risk`](#opensmartroutestrategiesprotocol) | Risk in [0, 1] that a single call fails: low confidence, high complexity and reasoning need,. | | `hashing_embedder` | re-export of [`opensmartroute.strategies.similarity.hashing_embedder`](#opensmartroutestrategiessimilarity) | Word + bigram hashing embedder. Deterministic, zero deps, decent for routing. | | `majority_vote` | re-export of [`opensmartroute.strategies.aggregate.majority_vote`](#opensmartroutestrategiesaggregate) | Largest meaning cluster wins; returns ``(answer, winner_target_id, agreement)``. | | `quorum_accuracy` | re-export of [`opensmartroute.strategies.human.quorum_accuracy`](#opensmartroutestrategieshuman) | P(weighted majority is correct) for independent annotators with the given accuracies. | | `request_modalities` | re-export of [`opensmartroute.strategies.modality.request_modalities`](#opensmartroutestrategiesmodality) | Modalities a request carries and which of them have a text surrogate in ``context``. | | `wrap_stream` | re-export of [`opensmartroute.strategies.escalation.wrap_stream`](#opensmartroutestrategiesescalation) | Consume ``chunks`` until the monitor escalates or the stream ends. | ## `opensmartroute.strategies.aggregate` Source: [src/opensmartroute/strategies/aggregate.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/aggregate.py) Routing / aggregation switch (Mixture-of-Agents, Wang et al. 2024; JiSi 2601.01330). | Name | Kind | Summary | |---|---|---| | `AggregateResult` | class | Outcome of :class:`MixtureOfAgents`: the final response, participants, winner and agreement ratio. | | `Aggregator` | constant | (request, [(target_id, answer)]) -> answer. | | `MixtureOfAgents` | class | Route-or-aggregate switch: below a confidence threshold call the top-k alternatives and aggregate. | | `Participant` | class | One target's contribution to an aggregate: response, cost, latency, error and whether it agreed. | | `majority_vote` | function `(request: RouteRequest, answers: list[tuple[str, Any]], embedder: Embedder \| None=None, threshold: float=0.85)` | Largest meaning cluster wins; returns ``(answer, winner_target_id, agreement)``. | ## `opensmartroute.strategies.auction` Source: [src/opensmartroute/strategies/auction.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/auction.py) Error-aware reverse auction across providers (EA-RAM 2608.12719). | Name | Kind | Summary | |---|---|---| | `AuctionResult` | class | Winner, second-price payment and the per-bidder surplus / corrected claims of one auction. | | `AuctionStrategy` | class | Error-aware reverse auction: bias-corrected bids, highest surplus wins, second-price payment. | | `BidFn` | constant | -> (claimed P(success), price). | | `BidderRecord` | class | Calibration ledger of one bidder: signed claim bias and a Beta record of realised success. | | `default_bid` | function `(target: RouteTarget, request: RouteRequest, signals: Signals)` | Catalogue-derived bid: quality prior as the claim, unit cost per 1k tokens x tokens as the price. | ## `opensmartroute.strategies.bandit` Source: [src/opensmartroute/strategies/bandit.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/bandit.py) Online-learning strategy: contextual Thompson-sampling bandit. | Name | Kind | Summary | |---|---|---| | `BanditStrategy` | class | Thompson-sampling Beta bandit per (context, target); context = dominant domain (+ plan role). | ## `opensmartroute.strategies.base` Source: [src/opensmartroute/strategies/base.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/base.py) Strategy interface. | Name | Kind | Summary | |---|---|---| | `Strategy` | class | Scores each candidate target in [0, 1] and explains why. | | `role_of` | function `(signals: Signals)` | Plan-slot role (``persona`` / ``skill`` / ``llm``) the router is currently filling, or. | | `memory_key` | function `(request_id: str, role: str \| None)` | Key for per-request strategy memory: the same request is scored once per plan role, and. | ## `opensmartroute.strategies.capability` Source: [src/opensmartroute/strategies/capability.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/capability.py) Capability-fit strategy: match request signals to a target's declared capabilities. | Name | Kind | Summary | |---|---|---| | `CapabilityStrategy` | class | Declarative fit: domain / action overlap, complexity band, language, modality and quality prior. | ## `opensmartroute.strategies.cascade` Source: [src/opensmartroute/strategies/cascade.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/cascade.py) Cascade execution (FrugalGPT / Router-R1 multi-round / AutoMix POMDP). | Name | Kind | Summary | |---|---|---| | `CascadeStep` | class | One executed rung of a cascade: gate quality, latency, cost, acceptance and POMDP belief. | | `CascadeResult` | class | Final response of a cascade with every step, the planned order and the planner's expected value. | | `CascadePlanner` | class | Finite-horizon MDP over an ordered cascade with a stop action after each step. | | `BeliefTracker` | class | AutoMix-style POMDP belief over the hidden state "current answer is correct". | | `Cascade` | class | Execute ranked targets in planner order (cheapest / MDP / POMDP), stopping when the quality gate passes. | ## `opensmartroute.strategies.defer` Source: [src/opensmartroute/strategies/defer.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/defer.py) Defer-to-human and effort ("think or not") strategies. | Name | Kind | Summary | |---|---|---| | `DeferStrategy` | class | Learning-to-defer: scores human targets by risk, PII, escalation intent, frustration and model uncertainty. | | `EffortStrategy` | class | Match a target's reasoning ``effort_level`` to ``signals.reasoning_need``; penalise over- and under-thinking. | | `decide_effort` | function `(signals: Signals, levels: tuple[str, ...]=('none', 'low', 'medium', 'high'))` | Pick the effort level whose numeric value is closest to the reasoning need. | ## `opensmartroute.strategies.edge` Source: [src/opensmartroute/strategies/edge.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/edge.py) Edge-cloud token-aware routing (Pro-Router 2608.28726; RelayLLM 2601.05167). | Name | Kind | Summary | |---|---|---| | `EdgeCloudStrategy` | class | Edge vs cloud tier choice: learned edge competence per complexity bucket vs upload / decode penalties. | ## `opensmartroute.strategies.elastic` Source: [src/opensmartroute/strategies/elastic.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/elastic.py) Token-budget-aware routing to elastic / many-in-one models (Nemotron Elastic 2511.16664; Star 2605.07182). | Name | Kind | Summary | |---|---|---| | `BudgetVariant` | class | One budget of an elastic model. ``quality`` and ``cost_scale`` are relative to the parent. | | `TokenBudgetStrategy` | class | Score budgeted siblings by fit between the budget and the tokens the answer needs. | | `expand_elastic` | function `(parent: RouteTarget, variants: list[BudgetVariant \| dict[str, Any]])` | Create one sibling target per budget variant, sharing the parent's family, handler and metadata. | ## `opensmartroute.strategies.escalation` Source: [src/opensmartroute/strategies/escalation.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/escalation.py) Bayesian self-escalation during generation (2608.24087). | Name | Kind | Summary | |---|---|---| | `REFUSALS` | constant | Refusal phrases that lower the streaming competence posterior. | | `EscalationDecision` | class | Verdict after a streamed chunk: continue / escalate / done with the competence estimate and reason. | | `SelfEscalation` | class | Streaming competence monitor with Bayesian optimal stopping. | | `StreamResult` | class | Text consumed by :func:`wrap_stream`, whether it escalated and the final decision. | | `wrap_stream` | function `(chunks: Iterable[str], monitor: SelfEscalation \| None=None)` | Consume ``chunks`` until the monitor escalates or the stream ends. | ## `opensmartroute.strategies.human` Source: [src/opensmartroute/strategies/human.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/human.py) Routing among human annotators and experts (QUORUM 2608.27974; Dawid-Skene 1979). | Name | Kind | Summary | |---|---|---| | `AnnotatorPool` | class | Skill estimation from agreement (Dawid-Skene EM) and quorum selection. Labels usually arrive from. | | `AnnotatorSkill` | class | Per-domain Beta accuracy estimate, cost and latency of one human annotator. | | `HumanRoutingStrategy` | class | Scores HUMAN candidates by estimated accuracy in the request's domain. | | `QuorumPlan` | class | A chosen set of annotators with the quorum's majority accuracy, total cost and latency. | | `quorum_accuracy` | function `(accuracies: Sequence[float])` | P(weighted majority is correct) for independent annotators with the given accuracies. | ## `opensmartroute.strategies.llm_judge` Source: [src/opensmartroute/strategies/llm_judge.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/llm_judge.py) Generative routing: let an LLM act as the router (Router-R1 / LLM-as-judge). | Name | Kind | Summary | |---|---|---| | `PROMPT` | constant | Prompt template for the judge; formatted with complexity, domains, actions, the catalogue and the request. | | `LLMJudgeStrategy` | class | LLM-as-router with optional **score calibration**. | ## `opensmartroute.strategies.memory` Source: [src/opensmartroute/strategies/memory.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/memory.py) Memory-tier routing for agents (BudgetMem 2602.06025; Gated-Memory Routing 2609.00237). | Name | Kind | Summary | |---|---|---| | `ImportanceGate` | class | Hashed logistic gate: P(item will be used later \| text, hint). | | `MemoryItem` | class | One stored memory: text, size, turn written, learned importance gate, tier and usage counters. | | `MemoryRouter` | class | Routes memory writes to tiers under budgets and recalls the most valuable items per token. | | `MemoryTier` | class | A storage tier: token capacity, read / write cost per token, latency and the minimum item value it accepts. | | `RecallResult` | class | Items recalled for a turn with their total tokens, read cost and latency. | ## `opensmartroute.strategies.modality` Source: [src/opensmartroute/strategies/modality.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/modality.py) Multimodal routing and modality escalation (LatentRouter 2605.11301; modality escalation. | Name | Kind | Summary | |---|---|---| | `EscalationResult` | class | Outcome of :class:`ModalityEscalation`: which target answered, whether it escalated, confidence, cost. | | `ModalityEscalation` | class | Try the text-only target on the surrogate first; escalate to the multimodal target on low confidence. | | `ModalityStrategy` | class | Coverage of the request's modalities by each candidate, with surrogate discounts. | | `request_modalities` | function `(request: RouteRequest)` | Modalities a request carries and which of them have a text surrogate in ``context``. | ## `opensmartroute.strategies.probe` Source: [src/opensmartroute/strategies/probe.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/probe.py) Hidden-state routing with a Dirichlet probe (ProbeDirichlet, RouterXBench 2602.11877). | Name | Kind | Summary | |---|---|---| | `HiddenStateStrategy` | class | Dirichlet probe over a dense request representation; confidence drops with epistemic uncertainty. | | `StateFn` | constant | request -> dense feature / hidden-state vector. | ## `opensmartroute.strategies.progress` Source: [src/opensmartroute/strategies/progress.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/progress.py) Agentic trajectories: per-step routing and multi-round execution. | Name | Kind | Summary | |---|---|---| | `MultiRoundExecutor` | class | Router-R1 style: keep routing/executing until the judge accepts or rounds run out. | | `ProgressRouter` | class | Route each step of a task with trajectory context. | | `Round` | class | One route -> execute -> judge iteration of :class:`MultiRoundExecutor`. | | `RoundResult` | class | Final response of a multi-round run with every round and the shared task id. | | `TaskProgress` | class | Where a multi-step task stands: step index, budget spent, failures in a row, last target, history. | ## `opensmartroute.strategies.protocol` Source: [src/opensmartroute/strategies/protocol.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/protocol.py) Collaboration-protocol selection (2608.14927). | Name | Kind | Summary | |---|---|---| | `DEFAULT_RULES` | constant | Built-in risk / budget ladder: handoff > debate > aggregate > cascade > single. | | `PROTOCOLS` | constant | in escalation order. | | `Protocol` | constant | execution protocols. | | `ProtocolChoice` | class | The protocol picked for a request, the risk that drove it, the reason and the matching rule. | | `ProtocolPolicy` | class | Ordered rule table from (risk, budget, task type) to a protocol, with per-protocol ledgers. | | `ProtocolRule` | class | First matching rule wins. ``None`` bounds are open. | | `failure_risk` | function `(decision: RouteDecision \| None, signals: Signals \| None=None, uncertainty: float \| None=None)` | Risk in [0, 1] that a single call fails: low confidence, high complexity and reasoning need,. | ## `opensmartroute.strategies.rules` Source: [src/opensmartroute/strategies/rules.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/rules.py) Declarative rule-based routing (Arch-Router-style domain/action preferences). | Name | Kind | Summary | |---|---|---| | `Rule` | class | If all `when` conditions match, boost `prefer` targets and penalise `avoid`. | | `RulesStrategy` | class | Applies declarative :class:`Rule` preferences (prefer / avoid / pin) when their ``when`` conditions match. | ## `opensmartroute.strategies.semantic_cache` Source: [src/opensmartroute/strategies/semantic_cache.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/semantic_cache.py) Semantic caching as a routing target (GPTCache; vLLM semantic router 2603.04444). | Name | Kind | Summary | |---|---|---| | `CacheEntry` | class | A cached response keyed by the embedding of its prompt, with source target, quality, timestamp and hits. | | `CacheHit` | class | The matched :class:`CacheEntry` and its similarity to the query. | | `SemanticCache` | class | Embedding-keyed LRU cache with TTL and a similarity threshold. Safe to share across threads:. | | `SemanticCacheStrategy` | class | Scores the cache target by calibrated hit quality; leaves real targets to the other strategies. | ## `opensmartroute.strategies.session` Source: [src/opensmartroute/strategies/session.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/session.py) Session affinity: keep a conversation with the target that is already serving it. | Name | Kind | Summary | |---|---|---| | `SessionAffinityStrategy` | class | Prefer the target already serving ``context["session_id"]`` unless the intent shifted or it failed. | | `SessionState` | class | What the strategy remembers about one session. | ## `opensmartroute.strategies.similarity` Source: [src/opensmartroute/strategies/similarity.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/similarity.py) Similarity-based routing (UniRoute / GraphRouter flavour). | Name | Kind | Summary | |---|---|---| | `hashing_embedder` | function `(dim: int=512)` | Word + bigram hashing embedder. Deterministic, zero deps, decent for routing. | | `cosine` | function `(a: list[float], b: list[float])` | Cosine similarity; zero vectors are treated as unit norm (no division by zero). | | `SimilarityStrategy` | class | Embed the request and each target's examples / description; score by best and top-k mean similarity. | ## `opensmartroute.strategies.speculative` Source: [src/opensmartroute/strategies/speculative.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/speculative.py) Speculative (draft-based) cascades (speculative cascades, Narasimhan et al. 2024; Differential. | Name | Kind | Summary | |---|---|---| | `SpeculativeCascade` | class | Two-target cascade that overlaps the draft and the strong call when it pays. | | `SpeculativeResult` | class | Outcome of a speculative run: chosen mode, whether the draft was accepted, latency and cost. | | `expected_mode_costs` | function `(p_accept: float, draft: RouteTarget, strong: RouteTarget, objective: Objective, *, tokens: int=500, cancellable: bool=False, verify_ms: float=0.0)` | Expected weighted cost (``objective.cost`` x USD + ``objective.latency`` x seconds) per mode. | ## `opensmartroute.strategies.task_table` Source: [src/opensmartroute/strategies/task_table.py](https://github.com/isathish/OpenSmartRoute/blob/main/src/opensmartroute/strategies/task_table.py) Static task table strategy (SCX Router, 2609.02292). | Name | Kind | Summary | |---|---|---| | `TaskTableStrategy` | class | Static ``task_type -> target -> quality`` table with family and prior fallbacks; learns from outcomes. | --- # Reference deployments OpenSmartRoute runs in three shapes. All three use the same `Router`; only where it lives differs. | Shape | When | How | |---|---|---| | **Library** | one Python service, in-process decisions (~0.3 ms / route) | `pip install opensmartroute` and call `Router.route()`; see [docs/SDK.md](https://opensmartroute.ai/docs/SDK.md) | | **Sidecar** | polyglot services, per-pod learner state | run the image next to your app: `docker run -p 8000:8000 -v ./examples:/config ghcr.io/isathish/opensmartroute:1.0.0` and call `POST /route` | | **Control plane** | many callers, shared catalogue, shared learning | the Helm chart in [helm/opensmartroute](https://github.com/isathish/OpenSmartRoute/blob/main/deploy/helm/opensmartroute) behind a `Service`/`Ingress`; state in Redis/SQL via `RouterBuilder.with_state_store` | ## Image [Dockerfile](https://github.com/isathish/OpenSmartRoute/blob/main/deploy/Dockerfile) builds a non-root, read-only-rootfs image with the `yaml` and `server` extras: ```sh docker build -t opensmartroute:1.0.0 -f deploy/Dockerfile . docker run --rm -p 8000:8000 -v "$PWD/examples:/config:ro" opensmartroute:1.0.0 curl -s localhost:8000/route -H 'content-type: application/json' -d '{"text":"summarise this contract"}' ``` Configuration is by environment (`OSR_TARGETS`, `OSR_RULES`, `OSR_MODELS`, `OSR_SLM`, `OSR_STATE`, `OSR_HOST`, `OSR_PORT`) so the same image serves both sidecar and control-plane roles; [entrypoint.sh](https://github.com/isathish/OpenSmartRoute/blob/main/deploy/entrypoint.sh) translates them into `osr serve` arguments. `OSR_SLM` points at a routing SLM file from `osr slm train`; it joins the strategy ensemble and is reported as the `slm` strategy in every decision trace. The OpenAI-compatible proxy (`/v1/chat/completions`) executes LLM targets when the container knows an upstream: set `OSR_LLM_BASE_URL` (OpenAI, Azure OpenAI, vLLM, Ollama, OpenRouter, LiteLLM ...) and `OSR_LLM_API_KEY` (or `OSR_LLM_API_KEY_ENV` naming another variable / `_FILE` mount). Each LLM target's `metadata.model` is the upstream model name; `OSR_LLM_MODEL` is the default for targets without one. Without a base URL the proxy still routes but answers 502 for LLM targets. Access control: the image starts open (every endpoint reachable) so it can act as a sidecar on a private network. To require a token set `OSR_SERVER_AUTH_TOKENS=,` (or mount the list and point `OSR_SERVER_AUTH_TOKENS_FILE` at it) and, for a hard guarantee, `OSR_SERVER_REQUIRE_AUTH=1` so the container refuses to start without one. Mint tokens with `osr token generate`; clients send `Authorization: Bearer ` or `X-API-Key: `, and `osr login --url https:// --token ` stores one for the CLI (`osr whoami` then shows `edition self-hosted`). `/healthz`, `/readyz`, `/metrics`, `/whoami` and the OpenAPI documents stay open for probes and discovery. Self-operation: `OSR_AUTOPILOT=1` (with `OSR_SLM` and a writable `OSR_STATE`) runs the SLM's improvement loop inside the container - every `OSR_SLM_AUTOPILOT_INTERVAL_S` seconds (default one hour) and early when the drift monitor sees the served success rate drop, it gathers evidence (feedback joined to prompts, cached datasets, seeds), trains a challenger, and hot-swaps it into the ensemble when it beats the champion on the holdout. The promoted model is written to `$OSR_STATE/autopilot/slm.json` (or `OSR_CACHE_DIR`), which the entrypoint prefers over `OSR_SLM` on the next start. `OSR_AUTOPILOT_ARGS` passes extra flags (`--offline`, `--source NAME`, `--tier TIER=TARGET`, `--search QUERY`); `OSR_CATALOGUE` names a catalogue JSON to refresh. `GET /stats` shows the `autopilot` status and `POST /autopilot/cycle` schedules a cycle now. Observability: every request is traced (spans and events per stage, see [docs/OBSERVABILITY.md](https://opensmartroute.ai/docs/OBSERVABILITY.md)). `GET /healthz` is the liveness probe and `GET /readyz` the readiness probe (503 until the router can score). `GET /metrics` serves the Prometheus exposition (`osr_route_decisions_total{target}`, `osr_route_latency_ms_bucket`, `osr_route_policy_rejections_total`, `osr_route_errors_total`, `osr_outcome_*`, `osr_decision_cache_*`, a `targets` gauge, plus per-stage `osr_span_duration_ms`), `GET /trace/{request_id}` the full trace of one request (each response carries `X-OSR-Trace-Id`; send `traceparent` to join your own trace) and `GET /events?name=route.*` the recent buffer. Tune with `OSR_OBSERVABILITY_MEMORY_EVENTS` (buffer size, `0` off), `OSR_OBSERVABILITY_LOG_EVENTS=true` (JSON lines on stdout), `OSR_OBSERVABILITY_EVENTS_FILE` (JSONL under the state volume), `OSR_OBSERVABILITY_OTEL=true` (OpenTelemetry, needs the `otel` extra and the usual `OTEL_EXPORTER_*` variables) and `OSR_OBSERVABILITY_SAMPLE_RATE` under heavy load. The hosted platform image ([platform/README.md](https://github.com/isathish/OpenSmartRoute/blob/main/platform/README.md)) reads the same variables and adds the workspace-scoped `GET /api/v1/trace/{request_id}`, `GET /api/v1/events` and the public `GET /api/v1/status`. ## Helm ```sh helm install router deploy/helm/opensmartroute \ --set-file config.targets=examples/targets.yaml \ --set-file config.rules=examples/rules.yaml ``` Key values (see [values.yaml](https://github.com/isathish/OpenSmartRoute/blob/main/deploy/helm/opensmartroute/values.yaml) for all): - `config.targets` / `config.rules` / `config.models` -> rendered into a ConfigMap at `/config`; the pod restarts on change (config checksum annotation). - `autopilot.enabled` / `autopilot.intervalSeconds` / `autopilot.args` -> self-operation as above (needs `config.slm` and `persistence.enabled`). The autopilot promotes models per pod: with `replicaCount` above 1 each replica trains and serves its own champion, so run one replica (or one autopilot pod behind a shared `StateStore`) when the loop is on. - `auth.tokens` / `auth.existingSecret` / `auth.required` -> access tokens for the HTTP API (a chart-managed Secret, or your own with a `tokens` key), and whether the pod may start without one. - `persistence.enabled` -> keep bandit posteriors and the feedback log on a PVC (single replica or RWX class). For several replicas leave it off and use the Redis/SQL `StateStore`. - `autoscaling`, `podDisruptionBudget`, `networkPolicy`, `ingress` -> standard knobs, all off or conservative by default. - Security: `runAsNonRoot`, `readOnlyRootFilesystem`, all capabilities dropped, `automountServiceAccountToken: false`. Secrets (MCP manifest signing key `OSR_MCP_KEY`, encrypted-state key) are injected with `env`/`envFrom` `secretKeyRef`s - never in `values.yaml`. Endpoints: `GET /healthz`, `GET /whoami`, `GET /targets`, `GET /stats`, `POST /route`, `POST /feedback`, `GET /v1/models`, `POST /v1/chat/completions` (OpenAI-compatible proxy, `model: "auto"`). --- # Architecture OpenSmartRoute is a **decision layer**, not a gateway. Given a request and a catalogue of targets it returns *which target should handle this, how confident we are, and why* — and it learns from what happened next. Executing the target is optional and pluggable. ```mermaid flowchart LR subgraph In REQ[RouteRequest] end subgraph Middleware G[Guard] --> T[Tenant] --> C[Cache] end subgraph Core S[Signals] --> P[Policy] --> ST[Strategies] --> E[Ensemble] --> D[Decision + Trace] end subgraph Learn O[Outcome] --> AL[AutoLearner] --> ST O --> H[Health] --> P end REQ --> G C --> S D --> X[Executor / handler] X --> O CAT[(TargetRegistry)] --> P ``` ## Layers | Layer | Package | Responsibility | Depends on | |---|---|---|---| | **Core model** | `core` | `RouteTarget`, `RouteRequest`, `RouteDecision`, `RouteTrace`, `Outcome`, `TargetRegistry` | stdlib | | **Signals** | `signals` | Deterministic features of a request: complexity, domain/action, language, modality, PII, jailbreak, token estimate; opt-in verbalised difficulty, draft-response and response-uncertainty features (`signals.uncertainty`) | core | | **Policy** | `policy` | Hard constraints → admissible set + rejection reasons. Never traded off. | core, signals | | **Strategies** | `strategies` | Soft scorers ∈ [0,1] with rationale: rules, capability fit, similarity, Thompson bandit, LLM judge, cascade executor; opt-in hidden-state probe, token budgets, edge/cloud tiers, auctions; post-answer controls (self-escalation, mixture-of-agents, protocol selection) | core, signals | | **Learning** | `learning`, `math` | Strategies that update from outcomes (IRT, Bradley–Terry, LinUCB, Markov/MDP, multi-turn history embeddings, per-user shrinkage) and the pure-math they rest on (bandits, calibration, Dirichlet probe, energy model, mixture-cure hand-off); `AutoLearner` fan-out, drift, atomic persistence; the routing SLM (`RouterSLM`, `SLMStrategy`, `distill_router`) and its champion/challenger `SelfImprover` | strategies | | **Discovery** | `discovery`, `retrieval` | Retrieve-then-rank narrowing, schema-aware tool matching, cache-preserving tool order, skill graphs and submodular skill sets for large catalogues | core, strategies | | **Router** | `router` | Orchestrates signals → policy → strategies → confidence-weighted ensemble → decision; optional judge escalation; optional plan (persona → skill → model) | all above | | **Real-time** | `realtime` | Circuit breaker, token bucket, rolling budget, `HealthRegistry`; `HealthPolicy` (hard) and `HealthStrategy` (soft) | policy, strategies, math | | **Enterprise** | `enterprise` | `RouterBuilder`, `EnterpriseRouter`, middleware chain, telemetry/state/audit ports and in-process adapters | router, realtime, learning | | **Security** | `security` | `InputGuard` (size, normalisation, confounder gadgets), `Redactor`, `GuardMiddleware`, prompt sanitiser, secret loading | enterprise | | **Adapters** | `adapters` | OpenAI-compatible HTTP client + glue (judge, embedder, handler); agent harnesses; MCP tools and server recommendation, A2A cards, SKILL.md and persona loaders, semantic-router import; live model catalogue (OpenRouter prices, Hugging Face cards) and web-search knowledge (https-only, byte-capped, risk-scored); optional sentence-transformers and OpenTelemetry | strategies, enterprise | | **Config** | `config` | JSON/YAML loaders for targets and rules with validation errors | core, strategies | | **Surface** | `aio`, `server`, `cli`, `eval` | Async façade, FastAPI app, `osr` CLI, RouterBench-style evaluation, dataset collection from the Hugging Face Hub (`eval.collect`) | router, enterprise | Dependency direction is strictly downward in this table. `math` depends on nothing but stdlib and is usable on its own. ## The decision pipeline, precisely 1. **Signals** — `extract_signals(request)`; ~0.1 ms; no I/O. 2. **Candidate pool** — `registry.all(primary_only=True)`, optionally filtered by `kinds`. 3. **Policy** — `Policy.filter(pool)` → `(admissible, rejections)`. `HealthPolicy` adds breaker/rate/budget checks (non-consuming). Empty admissible set → `NoRouteError` with the reasons. 4. **Pinned rules** — a matching `Rule(pin=True)` narrows the admissible set (Arch-Router semantics). 5. **Scoring** — every strategy returns `{target_id: StrategyScore(score, rationale, confidence)}`. 6. **Ensemble** — quality estimate $\hat q = \sum_k w_k\kappa_k s_k / \sum_k w_k\kappa_k$ where $\kappa$ is the strategy's self-confidence (learners report $\kappa=\min(1,n/n_0)$, so they are silent until they have evidence). Utility $U=\,w_q\hat q - w_c\,\tilde c - w_\ell\,\tilde\ell$ with log-min-max normalised cost/latency and a hard quality floor. 7. **Confidence** — softmax margin over utilities ($\tau=0.1$). If below `escalate_llm_judge_below` and a judge is configured, re-score once with the judge included. 8. **Decision** — top target, ranked alternatives, full trace. Optionally a `RoutePlan` where each slot (persona, skill, llm) is its own sub-routing over that kind; a slot is left empty when its best candidate's quality estimate is below `slot_quality_floor`, so irrelevant skills are never attached. 9. **Execute** (optional, `execution.py`) — `Router.run()` walks the plan: the `skill` slot's handler runs as a pre-processor (may rewrite the request or attach `skill_output`), persona/skill/primary `instructions` are composed into `context["system"]`, then the primary handler is called (`chat_handler`, a skill function, or an agent harness via `CallableHarness`/`HTTPHarness`/`SubprocessHarness`). One `Outcome` per participant (`role=None|"persona"|"skill"`, shared `task_id`) is recorded — success, latency, cost from tokens × unit cost. 10. **Learn** — `Outcome` → every strategy's `update()`, drift detectors, health registry, feedback store. ## Extension points | To add… | Implement | Register via | |---|---|---| | a target | `RouteTarget(...)` or a YAML entry | `registry.add()` / `load_targets()` | | a signal | `SignalExtractor.extract()` | `Router(extractors=[...])` | | a hard rule | `Policy.check()` → reason or `None` | `RouterBuilder.with_policy()` | | a scorer | `Strategy.score()` (+ `update()` if it learns) | `with_strategy(s, weight)` | | pre/post processing | `Middleware.__call__(request, next_)` | `with_middleware()` | | metrics/tracing | `Telemetry.on_decision/on_outcome/on_error` | `with_telemetry()` | | span/event capture | `EventSink.emit(event)` (+ `span_start/span_end` for live bridges) | `with_tracing(sink)` / `configure_tracing(sink)` | | persistence | `StateStore.get/put/delete` | pass to learners / caches | | a provider | `RouteTarget.handler = chat_handler(client, model)` | `adapters.OpenAICompatClient` | ## Concurrency model - `Router.route()` is pure with respect to router state (no writes) and safe to call from many threads. - Learners, health registry, caches and stores take their own locks on write. - `AsyncRouter` runs routing in the default executor and awaits coroutine handlers. - Multi-replica: learner updates are commutative sums (Beta counts, IRT gradients are small and order-insensitive in practice, BT strengths, LinUCB `A`/`b`, Markov counts), so replicas can be folded together with `merge()` / `learning.merge_learners()`, or readers can adopt a writer's snapshot through a `StateStore` with `AutoLearner.refresh()`. Corrupt snapshots are quarantined, never fatal (`AutoLearner.quarantined`). ## Performance envelope Measured with `python scripts/bench.py --n 200 --scale` (CPython 3.11, x86-64 Windows laptop, pure Python, no C extensions). Numbers are wall-clock per `route()` including signal extraction, policy, every strategy and the ensemble; re-run the script on your own hardware before sizing. **16-target example catalogue** (`examples/targets.yaml`, `examples/eval_dataset.jsonl`): | Configuration | p50 | p95 | p99 | |---|---|---|---| | rules + capability + similarity + bandit | 5.8 ms | 6.2 ms | 6.5 ms | | + IRT + preference + LinUCB + Markov + health | 8.6 ms | 12.9 ms | 17.8 ms | **Scaling with catalogue size** (synthetic LLM catalogue, defaults + auto-learning). Cost is linear in admissible targets x strategies until retrieval narrowing (BM25 + hashed dense, reciprocal-rank fusion) caps the candidate pool; the default `Settings.routing` only narrows above 500 admissible targets (to 50), `RouterBuilder.with_retrieval(narrow_above=32, narrow_to=24)` turns it on earlier: | Targets | Narrowing | p50 | p95 | p99 | |---|---|---|---|---| | 16 | default (none) | 9.2 ms | 9.5 ms | 10.0 ms | | 16 | top-24 above 32 (none triggered) | 9.2 ms | 9.6 ms | 10.5 ms | | 64 | default (none) | 35.7 ms | 36.6 ms | 37.1 ms | | 64 | top-24 above 32 | 14.5 ms | 15.1 ms | 15.6 ms | | 256 | default (none) | 142 ms | 145 ms | 158 ms | | 256 | top-24 above 32 | 16.7 ms | 17.6 ms | 18.5 ms | | 1024 | default (auto top-50) | 38.7 ms | 43.5 ms | 49.1 ms | | 1024 | top-24 above 32 | 24.1 ms | 28.2 ms | 32.9 ms | Rule of thumb: keep the scored pool at or below ~50 targets and routing stays under 25 ms p99 in pure Python regardless of catalogue size; the retrieval stage itself is ~10 ms at 1k targets and grows roughly linearly in the catalogue. The LLM judge, when triggered, adds one provider round-trip; keep `escalate_llm_judge_below` low (<= 0.5) so it fires only on genuinely ambiguous requests. ## Deployment shapes 1. **Library** — import in-process. Lowest latency. State on local disk or in-memory. 2. **Sidecar** — `osr serve` per application (container image from `deploy/Dockerfile`); shared `StateStore` for global learning. 3. **Control plane** — one HA deployment (Helm chart in `deploy/helm/opensmartroute`); a gateway (LiteLLM, Envoy, aisix) calls `/route`, executes the chosen target, and posts `/feedback`. Multi-replica learning: run one *writer* replica that learns and saves through a `StateStore`, and have readers call `AutoLearner.refresh()` on a timer; or let every replica learn independently and fold them together periodically with `learning.merge_learners()`. See [deploy/README.md](https://opensmartroute.ai/docs/deploy.md). ## Non-goals Model serving, protocol translation between providers, prompt versioning, chat UI. See [ROADMAP.md](https://opensmartroute.ai/docs/ROADMAP.md#out-of-scope). --- # Mathematical Foundations Every routing decision in OpenSmartRoute is the output of explicit, inspectable formulas. This document lists them, the research they come from, and where they live in the code. ## 1. The routing problem Given a request $x$ with signal vector $\phi(x)$ and candidate set $\mathcal{T}$ (after policy filtering), choose $$ t^\* = \arg\max_{t \in \mathcal{T}} \; U(x,t), \qquad U(x,t) = w_q\,\hat q(x,t) - w_c\,\tilde c_t - w_l\,\tilde \ell_t $$ subject to $\hat q(x,t) \ge q_{\min}$ (quality floor, Hybrid-LLM). Costs and latencies are **log-min-max normalised** so one outlier (a \$0.50/1k human queue) doesn't flatten the rest: $$ \tilde c_t = \frac{\log(1 + s\,c_t) - \min_k \log(1 + s\,c_k)}{\max_k \log(1 + s\,c_k) - \min_k \log(1 + s\,c_k)} $$ Code: `router.py::_norm`, `Router._rank`. ## 2. Ensemble quality estimate Each strategy $k$ returns a score $s_k(t) \in [0,1]$ and a self-confidence $\kappa_k(t)$. The ensemble is a confidence-weighted mean: $$ \hat q(x,t) = \frac{\sum_k w_k\,\kappa_k(t)\,s_k(t)}{\sum_k w_k\,\kappa_k(t)} $$ Learners report $\kappa = \min(1, n/n_0)$ so they are silent until they have data — this is what lets rules and declared capabilities dominate at cold start and learned models take over as evidence accumulates. ## 3. Decision confidence $$ \text{conf} = \frac{e^{U_1/\tau}}{\sum_j e^{U_j/\tau}}, \quad \tau = 0.1 $$ Low confidence (or high normalised entropy $H/\log|\mathcal T|$) triggers the LLM-judge escalation (Router-R1 pattern). Calibration is measured with **ECE** (`math.estimators.expected_calibration_error`). ## 4. Online learning (auto-learn) ### 4.1 Thompson sampling (Beta–Bernoulli) — `math.bandits.ThompsonBeta` $\theta_t \sim \mathrm{Beta}(\alpha_t,\beta_t)$; on reward $r\in[0,1]$: $\alpha_t \mathrel{+}= r$, $\beta_t \mathrel{+}= 1-r$. Optional forgetting $\alpha \leftarrow \alpha_0 + \gamma(\alpha-\alpha_0)$ for non-stationarity. ### 4.2 UCB1 — `math.bandits.UCB1` $\hat\mu_t + c\sqrt{2\ln n / n_t}$ (Auer, Cesa-Bianchi & Fischer 2002). ### 4.3 LinUCB (contextual) — `math.bandits.LinUCB`, `learning.LinUCBStrategy` Per arm $A_t = I + \sum x x^\top$, $b_t = \sum r x$, $\hat\theta_t = A_t^{-1} b_t$, score $x^\top \hat\theta_t + \alpha\sqrt{x^\top A_t^{-1} x}$ (Li et al., WWW 2010). Context $x$ = `learning.signal_vector` (complexity, length, PII, jailbreak, tools, language, multimodal, domain one-hots). This is the MixLLM formulation. ### 4.4 Cost-aware bandit (Lagrangian) — `math.bandits.CostAwareBandit` Score $= \text{inner}(t) - \lambda c_t$ with dual ascent $\lambda \leftarrow \max(0, \lambda + \eta(\bar c - B))$ so average spend tracks budget $B$ (C2MAB-V, Dai et al. 2024). ### 4.5 Item Response Theory (2PL) — `math.irt.IRTModel`, `learning.IRTStrategy` $$P(\text{success}\mid t,i) = \sigma\big(a_i(\theta_t - b_i)\big)$$ Target ability $\theta_t$, item (domain × difficulty bucket) difficulty $b_i$, discrimination $a_i$; fitted by online SGA on Bernoulli log-likelihood with L2. Interpretable and cold-start friendly (IRT-Router, Song et al., ACL 2025). ### 4.6 Bradley–Terry / Elo — `math.preference`, `learning.PreferenceStrategy` $$P(i \succ j) = \sigma(\xi_i - \xi_j)$$ Learned per domain → a *prompt-specific leaderboard* (Prompt-to-Leaderboard, Frick et al. 2025; RouteLLM preference data). Selection probability via Luce: $P(t) = e^{\xi_t}/\sum_k e^{\xi_k}$. ### 4.7 Markov chain & MDP — `math.markov`, `learning.MarkovStrategy` Conversation state $s$ = dominant domain. Transition matrix with Dirichlet smoothing $$P_{ij} = \frac{n_{ij} + \alpha}{\sum_k n_{ik} + \alpha|S|}$$ $k$-step prediction $\pi P^k$, first-passage probability to *escalate*, stationary distribution by power iteration, next-state entropy. Routing MDP (value iteration): $$V(s) = \max_a\Big[R(s,a) + \gamma\sum_{s'}P(s'|s)V(s')\Big]$$ `MarkovStrategy` scores $(1-\beta)\,Q(s,a) + \beta\sum_{s'}P(s'|s)\,Q(s',a)$ — prefer targets that are also good for where the conversation is *going*. ### 4.8 Drift detection — `math.estimators` - **Page–Hinkley**: $m_T=\sum_{t\le T}(x_t-\bar x_t-\delta)$, alarm if $\max_t m_t - m_T > \lambda$. - **ADWIN-lite**: split window, Hoeffding bound $\epsilon=\sqrt{\tfrac{1}{2m}\ln\tfrac{4}{\delta}}$. `AutoLearner` flags drifted targets; `HealthStrategy`/`HealthPolicy` can demote them. ### 4.9 Uncertainty on success rates **Wilson interval** lower bound is used as the reliability estimate so a target with 2/2 successes is *not* considered 100 % reliable. ## 5. Multi-objective decisions — `math.decision` - **Pareto front** over (quality↑, cost↓, latency↓). - **TOPSIS** closeness $C = d^-/(d^+ + d^-)$ to the ideal point. - **Weighted sum** (the default scalarisation). ## 6. Real-time capacity — `math.decision` - **Erlang-C**: $P(\text{wait}) = \dfrac{\frac{A^c}{c!}\frac{1}{1-\rho}}{\sum_{k