OpenSmartRoute

An open, intelligent route to the right decision, solution, or destination.

OpenSmartRoute is an open-source AI decision control plane. For each request it picks the best LLM, agent harness, skill, persona, tool, workflow or human, honours hard constraints (privacy, region, budget, latency), explains the choice, executes the resulting plan and learns from every outcome. The core is pure Python with zero runtime dependencies.

flowchart LR
    R([request]) --> G["guard<br/>redact PII"]
    G --> S["signals<br/>&lt; 1 ms"]
    S --> P["policy<br/>hard constraints"]
    P --> ST["strategies + ensemble utility<br/>rules, capability, similarity, bandit, LLM judge"]
    ST --> D["decision<br/>trace + plan"]
    D --> X["execute<br/>compact prompt, persona -> skill -> model + tools / agent / tool / human"]
    X -. outcomes .-> ST

Why

• Everything is a route target. One RouteTarget contract for models, agent harnesses, SKILL.md packages, personas, MCP tools, workflows and human queues; one policy layer; one learning loop.
• Plans, not just picks. route(plan=True) composes persona, skill, model and the tools the model may call; run() executes the plan - tool calls the model makes to offered tools included - and records an Outcome per participant. An instructions-only skill or persona runs on the plan's model with its body disclosed in the system prompt.
• Fewer tokens before the call. estimate() quotes tokens, cost, latency and quality per candidate (calibrated against billed tokens), and every model call is compacted first - whitespace, repeated log lines and stale history go, code and words stay - with the saving reported as tokens_saved.
• Keeps itself current. A running router follows the marketplace (new skills and tools land in the catalogue on a schedule), the self-improvement loop reads vendor blogs and release feeds for new models, and IDEs reach the marketplace through the local MCP server.
• Constraints are never traded off. PII, data boundary, region, tenant, cost and latency SLOs are filtered before any score is computed, with the rejection reason in the trace. Redacted PII is restored only for targets allowed to hold it; every other target, and every log, sees placeholders.
• Learns in production. Thompson bandits, Item Response Theory, Bradley-Terry preferences, LinUCB, Markov lookahead, task-level credit assignment, drift detection and forgetting.
• Honest evaluation. Baselines, oracle, label-noise floor, paraphrase robustness, calibration (ECE, Brier, conformal sets), off-policy estimators, public benchmark presets.
• Secure by design. Learned and heuristic guards against rerouting gadgets and prompt injection, PII redaction, resource limits, signed MCP manifests, encrypted state, hash-chained audit.
• Enterprise-ready. Builder, middleware, telemetry / state / audit ports, Redis and SQL stores, shadow and A/B routing with SPRT, tenant fair share, async facade, container image and Helm chart.

Install

The osr command line and the Python package ship together. On Linux and macOS:

curl -LsSf https://opensmartroute.ai/install.sh | sh

On Windows (PowerShell):

powershell -ExecutionPolicy ByPass -c "irm https://opensmartroute.ai/install.ps1 | iex"

The installer puts osr in an isolated environment (uv, pipx or a private venv - whichever is available, never the system Python) and adds it to your PATH. OSR_VERSION=2.1.0 pins a release, OSR_EXTRAS=all installs every optional dependency, OSR_INSTALLER=uv|pipx|venv forces a backend and OSR_NO_MODIFY_PATH=1 leaves your shell configuration alone. The scripts are install.sh and install.ps1 in this repository and in the dist artifact of every release pipeline run.

If you already manage Python tools yourself:

uv tool install 'opensmartroute[yaml,server]'   # or: pipx install 'opensmartroute[yaml,server]'
pip install opensmartroute                       # library only, zero runtime dependencies
pip install 'opensmartroute[yaml]'               # + YAML catalogues and rules
pip install 'opensmartroute[server]'             # + FastAPI server and OpenAI-compatible proxy
pip install 'opensmartroute[embeddings]'         # + sentence-transformers similarity
pip install 'opensmartroute[fast]'               # + numpy-accelerated SLM training (pure Python otherwise)
pip install 'opensmartroute[tiktoken]'           # + exact OpenAI tokenisation for estimates
pip install 'opensmartroute[otel]'               # + OpenTelemetry telemetry
pip install 'opensmartroute[crypto]'             # + AES-GCM encrypted state
pip install 'opensmartroute[tensorflow]'         # + Keras token-count and routing-head models (large; not in `all`)
pip install 'opensmartroute[all]'                # everything above except tensorflow (+ the dev tools)

osr doctor reports what the installation has: interpreter, package, extras, the bundled routing model, the sign-in and (with --targets) whether a catalogue loads.

Container image: crosrua2c6onyidguw.azurecr.io/opensmartroute:<version> (Deploy with Docker and Helm). From a checkout: pip install -e '.[dev]'.

Then sign in. The hosted platform (community or enterprise edition) uses a browser hand-shake; a self-hosted osr serve accepts a token you generate yourself:

osr login                                              # opens https://opensmartroute.ai/cli/authorize
osr login --url https://osr.example.com                # your own platform deployment
osr login --url http://router:8000 --token osr_local_...   # self-hosted server (see `osr serve --generate-token`)
osr whoami                                             # workspace, plan, edition, key

Credentials live in ~/.config/opensmartroute/credentials.json (%APPDATA%\opensmartroute on Windows), one profile per --profile; OSR_API_URL / OSR_API_KEY override them in CI.

Quick start

Three lines, nothing configured - a router in your process over the built-in starter catalogue (the one osr route uses without --targets):

from opensmartroute import Router

router = Router.starter()
print(router.route("Prove that sqrt(2) is irrational, step by step.").target.id)   # llm-frontier

Or the hosted platform (or your own osr serve) over HTTP, with the key osr login saved or OSR_API_KEY:

from opensmartroute import OpenSmartRoute

osr = OpenSmartRoute()
d = osr.route("Refund order #123, the customer is upset", objective={"cost": 0.4, "quality": 0.6})
print(d.target_id, d.confidence, d.explanation)
osr.feedback(d, success=True, quality=0.9)          # the learners improve on your traffic
print(osr.ask("What is the capital of France?"))   # OpenAI-compatible chat, the router picks the model

Timeouts, retries with Retry-After, typed errors that never carry the key, and no key over plain http:// to a public host - see SDK guide. Your own catalogue, in full:

from opensmartroute import (Router, TargetRegistry, RouteTarget, TargetKind, Capabilities,
                            TargetConstraints, Outcome)

registry = TargetRegistry([
    RouteTarget("llm-small", TargetKind.LLM,
                capabilities=Capabilities(max_complexity=0.45),
                cost={"usd_per_1k_tokens": 0.0002}, latency_ms=300, quality_prior=0.55,
                examples=["Hi, how are you?", "What is the capital of France?"]),
    RouteTarget("llm-frontier", TargetKind.LLM,
                capabilities=Capabilities(min_complexity=0.5, domains=["math", "coding"]),
                cost={"usd_per_1k_tokens": 0.015}, latency_ms=2500, quality_prior=0.93,
                examples=["Prove the theorem step by step."]),
    RouteTarget("llm-onprem", TargetKind.LLM,          # the only target allowed to see private data
                constraints=TargetConstraints(regions=["eu"], data_boundary="private", pii_allowed=True),
                cost={"usd_per_1k_tokens": 0.004}, latency_ms=900, quality_prior=0.7),
    RouteTarget("human", TargetKind.HUMAN,
                capabilities=Capabilities(actions=["escalate"], tags=["safety"]),
                cost={"usd_per_1k_tokens": 0.5}, latency_ms=300_000),
])

router = Router(registry)
d = router.route("Prove that sqrt(2) is irrational, step by step.")
print(d.target.id, f"{d.confidence:.2f}")     # llm-frontier, with the calibrated confidence
print(d.trace.explain())                       # per-strategy scores and rationales

router.learn(Outcome(request_id=d.request_id, target_id=d.target.id, success=True,
                     quality=0.9, cost_usd=0.002, latency_ms=1800, domains=d.trace.signals.domains))

Hard constraints and a per-request objective - the policy filters before anything is scored, and the trace says who was rejected and why:

from opensmartroute import RouteRequest, RequestConstraints, Objective

req = RouteRequest("Summarize this patient intake note.",
                   constraints=RequestConstraints(region="eu", data_boundary="private", max_cost_per_1k=0.005),
                   objective=Objective(quality=1.0, cost=0.5, latency=0.1, quality_floor=0.6))
d = router.route(req)
print(d.target.id)                 # llm-onprem
print(d.trace.policy_rejections)   # {'llm-small': 'data boundary public < required private', ...}

When no target is admissible the router raises NoRouteError (details["rejections"] carries the same map) instead of guessing.

Catalogue and rules from YAML, evaluated and served from the command line:

osr -t examples/targets.yaml -r examples/rules.yaml route "I want a refund for order #123" --plan
osr -t examples/targets.yaml -r examples/rules.yaml eval examples/eval_dataset.jsonl --frontier
export OSR_LLM_BASE_URL=https://api.openai.com/v1 OSR_LLM_API_KEY=sk-...   # any OpenAI-compatible endpoint
osr -t examples/targets.yaml -r examples/rules.yaml serve        # http://127.0.0.1:8000/docs

serve routes without a provider; OSR_LLM_BASE_URL (OpenAI, Azure, vLLM, Ollama, OpenRouter, ...) is what lets /v1/chat/completions and router.run() actually run the chosen model.

Wiring real providers, executing plans with router.run(), agent harnesses, SKILL.md and MCP catalogues, the decorator SDK and the enterprise builder are covered step by step in User guide.

What it routes

Catalogues load from YAML or JSON, MCP tools/list payloads, A2A agent cards, SKILL.md and persona directories and vLLM semantic-router configurations; the router plugs into LangGraph and Agent Framework graphs as a node and into any chat loop as an OpenAI tool.

How it decides

1. Signals (under 1 ms): task type, domains, complexity, reasoning need, PII, language, modality, history; optional verbalised difficulty, cheap-draft features and hidden-state probes.
2. Policy: hard constraints filter targets before any scoring.
3. Strategies: rules, capability fit, example similarity, task table, Thompson and LinUCB bandits, IRT, Bradley-Terry, Markov lookahead, multi-turn history embeddings, learning-to-defer, edge/cloud tiers, token budgets, auctions, user adaptation and an LLM judge consulted only below a confidence threshold; optionally a routing SLM (RouterSLM) distilled from the whole ensemble that keeps improving on its own from outcomes, public routing datasets, a live model catalogue and the article feeds you point it at - a trained one ships inside the package (--slm bundled, RouterBuilder.with_slm()), so an on-prem install starts warm (guide).
4. Utility: confidence-weighted ensemble, then w_q * quality - w_c * norm(cost) - w_l * norm(latency) with a hard quality floor; temperature scaling and conformal candidate sets calibrate the confidence; the router abstains when nothing is safe enough.
5. After the answer: cascades and self-escalation stop or reroute on response-side uncertainty (semantic entropy, P(True), streaming competence posterior); mixture-of-agents aggregation and permanent hand-off policies cover agentic trajectories.

Formulas and citations: Mathematics, Research.

Production

from opensmartroute import RouteRequest, RequestConstraints
from opensmartroute.enterprise import RouterBuilder, MetricsTelemetry, FileAuditSink, TenantMiddleware
from opensmartroute.security import GuardMiddleware

app = (RouterBuilder(registry)
       .with_defaults().with_auto_learning(state_dir=".osr-state")
       .with_health(latency_slo_ms=3000)
       .with_middleware(GuardMiddleware(redact=True),
                        TenantMiddleware({"acme": {"deny_targets": ["llm-frontier"]},
                                          "globex": {"data_boundary": "on_prem"}}))
       .with_telemetry(MetricsTelemetry()).with_audit(FileAuditSink("audit.jsonl"))
       .build())

req = RouteRequest("Prove that sqrt(2) is irrational.", constraints=RequestConstraints(tenant="acme"))
d = app.route(req)                 # guard -> tenant -> policy -> strategies; audited and measured
print(d.target.id, app.health_snapshot())
# app.run(req) executes the plan and records the Outcome once targets carry handlers
# (`attach_chat_handlers`, agent harnesses, MCP tools - docs/GUIDE.md); without one it raises ExecutionError.

Routing overhead is 4.6 ms p50 with the default strategies and 7.4 ms with every learner enabled on the 27-target examples/targets.yaml with its rules (the repository's benchmark, 300 requests, CPython 3.11 on a Windows laptop, re-measured 2026-10-09; the continuous build fails when p95 passes 50 ms); retrieve-then-rank keeps it flat for thousands of tools (measurements). osr serve (or create_app(app) with the router above) exposes a FastAPI app and an OpenAI-compatible proxy (/v1/chat/completions with streaming and tool calling, /v1/responses, /v1/embeddings, /v1/models), the Anthropic Messages API (/v1/messages) and an MCP endpoint (/mcp): point the official OpenAI or Anthropic SDK at it with model="osr/auto" and keep the rest of the code; guard and tenant violations return 400, no admissible target 422, an unreachable provider 503 and a failing target falls back to the next candidate. Circuit breakers open on provider failures and recover on their own; learner state survives restarts from state_dir or a Redis / SQL store. Deployment references: Deploy with Docker and Helm, Enterprise builder.

Hosted platform

platform/ packages the router as a service from two container images: osr-platform-api (FastAPI: self-serve signup with hashed API keys, plans and quotas, usage metering, the metered /api/v1/route family with the full trace, per-tenant constraints and workspace policy, the audit trail, per-request traces and a live event stream, and the OpenAI-compatible /v1/chat/completions proxy - one process, or one container per domain behind a gateway) and osr-platform-web (Next.js: landing page, the rendered documentation, pricing, a live playground and the account dashboard with activity, events, governance and health pages; it proxies /api and /v1 so the browser only talks to one origin). OSR_PLATFORM_EDITION=community runs the core Router; enterprise runs the builder above with auto-learning, health, guard, metrics, a hash-chained audit log and per-tenant constraints. The Azure solution template deploys everything to Azure Container Apps with Azure OpenAI (Install the platform); the reference deployment is https://opensmartroute.ai. The end-user guide is Platform guide; the REST API reference is generated from the platform's OpenAPI document.

curl -s -XPOST $OSR/api/v1/signup -H 'content-type: application/json' -d '{"email":"you@example.com"}'
curl -s $OSR/api/v1/route -H "authorization: Bearer $KEY" -H 'content-type: application/json' \
     -d '{"text":"Prove that sqrt(2) is irrational."}'

From TypeScript or JavaScript, the client on npm - npm install @opensmartroute/sdk (platform/sdk-ts) - wraps the same endpoints and the OpenAI-compatible /v1 surface with streaming, zero runtime dependencies, Node 20+ / Deno / Bun / edge / browser:

import { OpenSmartRoute } from "@opensmartroute/sdk";
const osr = new OpenSmartRoute({ baseUrl: "https://opensmartroute.ai", apiKey: process.env.OSR_API_KEY! });
const decision = await osr.route({ text: "Prove that sqrt(2) is irrational.", top_k: 3 });

Security

Routing is a control plane; its integrity is a security property. InputGuard combines a heuristic and a learned detector for confounder gadgets that reroute queries, an injection-risk scorer, PII redaction and prompt sanitisation for the LLM judge. Per-task resource limits, an origin policy for sensitive tool parameters, signed MCP manifests, AES-GCM state, content-free logs and hash-chained audit complete the model. CI runs ruff -S, mypy, bandit, pip-audit and the osr safety red-team suite. Threat model: Security model. Reporting: Reporting vulnerabilities.

Roadmap

Every item in the research track is implemented on main with tests; a version is Complete when its measured exit criterion is published (8/8 met) and Shipped when it is in a tagged release - the current release is 2.1.0 (Changelog). 1.0 means what the repository can prove: frozen public API with a deprecation path, published performance envelope, reference deployments, security controls with a red-team suite in CI and a published review pack (External review pack), the public leaderboard run (examples/leaderboard/results, produced by the recipe in examples/leaderboard) and consecutive releases without a breaking change (0.4.0 through 2.1.0, additions only). Adoption evidence - an independent security report, accepted leaderboard listings, two production users (ADOPTERS.md) - is tracked, never claimed, and the release tooling refuses a release while a readiness row is open. Details in docs/ROADMAP.md and Changelog.

License

Apache-2.0