Python SDK quickstart

Route between models, agents, skills, tools and humans inside your own process in about five minutes. The core package has zero runtime dependencies; everything here runs offline. The user guide continues where this page stops - real providers, plans, learning, the enterprise builder.

1. Install

pip install opensmartroute

2. Describe your targets

A target is anything a request can be sent to. Declare what each one is good at, what it costs and how fast it is; the router does the rest.

from opensmartroute import Router, TargetRegistry, RouteTarget, TargetKind, Capabilities, Outcome, TargetConstraints

registry = TargetRegistry(
    [
        RouteTarget(
            "llm-small",
            TargetKind.LLM,
            capabilities=Capabilities(max_complexity=0.45),
            cost={"usd_per_1k_tokens": 0.0002},
            latency_ms=300,
            quality_prior=0.55,
            examples=["Hi, how are you?", "What is the capital of France?"],
        ),
        RouteTarget(
            "llm-frontier",
            TargetKind.LLM,
            capabilities=Capabilities(min_complexity=0.5, domains=["math", "coding"]),
            cost={"usd_per_1k_tokens": 0.015},
            latency_ms=2500,
            quality_prior=0.93,
            examples=["Prove the theorem step by step."],
        ),
        RouteTarget(
            "llm-onprem",
            TargetKind.LLM,
            capabilities=Capabilities(domains=["general"]),
            constraints=TargetConstraints(regions=["eu"], data_boundary="private", pii_allowed=True),
            cost={"usd_per_1k_tokens": 0.001},
            latency_ms=1200,
            quality_prior=0.7,
        ),
        RouteTarget(
            "human",
            TargetKind.HUMAN,
            capabilities=Capabilities(actions=["escalate"], tags=["safety"]),
            cost={"usd_per_1k_tokens": 0.5},
            latency_ms=300_000,
        ),
    ]
)

Catalogues also load from YAML or JSON files, MCP tools/list payloads, A2A agent cards, SKILL.md and persona directories - see the user guide. With a targets.yaml (and optionally a rules.yaml) on disk - the repository ships a pair under examples/ - the whole setup is one line (YAML files need pip install 'opensmartroute[yaml]', JSON needs nothing):

from opensmartroute import Router

router = Router.from_files("targets.yaml", "rules.yaml", state_dir=".osr-state")  # state_dir keeps what it learns

3. Route

router = Router(registry)
d = router.route("Prove that sqrt(2) is irrational, step by step.")
print(d.target.id, f"{d.confidence:.2f}")  # llm-frontier 0.89
print(d.trace.explain())  # per-strategy scores and rationales

Every decision carries a trace: which signals were extracted, which targets policy rejected and why, how each strategy scored the rest. Nothing is a black box.

Every route() call runs the same five stages - the trace records each one:

flowchart TB
    R["router.route(request)"] --> SIG["Signals: complexity, domains, intent, PII"]
    SIG --> POL{"Policy: hard constraints + rules"}
    POL -->|rejected| REJ["trace.policy_rejections"]
    POL -->|admissible| STR["Strategies score each candidate"]
    STR --> ENS["Ensemble to utility, weighted by the objective"]
    ENS --> DEC["Decision: target, confidence, alternatives, trace"]

4. Constraints and objectives

Hard constraints are filtered before any scoring - they are never traded off against quality. The objective sets the trade-off between quality, cost and latency per request. A note with private patient data only clears the on-prem target; the public models are excluded before scoring begins.

from opensmartroute import RouteRequest, RequestConstraints, Objective

req = RouteRequest(
    "Summarize this patient intake note.",
    constraints=RequestConstraints(region="eu", data_boundary="private", max_cost_per_1k=0.005),
    objective=Objective(quality=1.0, cost=0.5, latency=0.1),
)
d = router.route(req)
print(d.target.id)  # llm-onprem
print(d.trace.policy_rejections)  # {'llm-small': 'data boundary public < required private', ...}

5. Close the loop

Report how the routed answer went and the router's learners (bandits, IRT, Bradley-Terry) shift future decisions toward what actually works for your traffic:

router.learn(
    Outcome(
        request_id=d.request_id,
        target_id=d.target.id,
        success=True,
        quality=0.9,
        cost_usd=0.002,
        latency_ms=1800,
        domains=d.trace.signals.domains,
    )
)

Attach handlers to targets and router.run(req) executes the chosen target or plan and records the outcomes itself - wiring real providers is the user guide's fourth chapter.

6. The same thing from the command line

Without --targets the route, estimate, serve, targets and mcp commands use a built-in starter catalogue (llm-small, llm-frontier, llm-onprem, with two rules), so the first command works straight after the install; osr stack init --out stack.json writes that catalogue to disk to edit, and -t stack.json -r stack.json routes through your copy. The repository's examples/ folder holds the richer YAML pair used below.

osr route "Prove that sqrt(2) is irrational"                     # built-in starter catalogue, no files needed
osr -t examples/targets.yaml -r examples/rules.yaml route "I want a refund for order #123" --plan
osr -t examples/targets.yaml -r examples/rules.yaml estimate "Summarise this contract" # tokens, cost, quality per candidate
osr -t examples/targets.yaml -r examples/rules.yaml eval examples/eval_dataset.jsonl --frontier
export OSR_LLM_BASE_URL=https://api.openai.com/v1 OSR_LLM_API_KEY=sk-...   # the provider `serve` executes on
osr -t examples/targets.yaml -r examples/rules.yaml serve        # http://127.0.0.1:8000/docs
osr -t examples/targets.yaml mcp                                 # MCP server for your IDE, marketplace tools included

Routing needs no provider. Execution does: OSR_LLM_BASE_URL names any OpenAI-compatible endpoint (OpenAI, Azure OpenAI, vLLM, Ollama, OpenRouter, ...) and OSR_LLM_API_KEY its key; without them the served /v1/chat/completions answers 503 because no target has a handler, and routing still works.

osr serve exposes the same /route, /feedback and /mcp endpoints as the hosted platform and the same /v1 surface: OpenAI chat/completions (streaming, tool calling), responses, embeddings, moderations, audio/speech, audio/transcriptions, images/generations and the Anthropic messages API - point the official OpenAI or Anthropic SDK at it with model="osr/auto" (or "auto"), an MCP client at /mcp, or the TypeScript client (npm install @opensmartroute/sdk, apiPrefix: ""). Every model call is compacted first (repeated lines, stray whitespace and stale history go; the saving is reported as tokens_saved), --marketplace-refresh 86400 keeps the served catalogue in step with the marketplace, and --guard --redact-pii --output-guard redact --health --audit audit.jsonl turns on the hardening the deploy guide describes.

FAQ

Where next

• User guide - targets, rules, providers, plans, learning, the CLI, latency numbers.
• SDK guide - decorators, settings, error handling, threading, extension points.
• Enterprise builder - middleware, telemetry, stores, shadow and A/B rollout.
• Deploy with Docker and Helm - run osr serve on your own infrastructure.
• TypeScript SDK - call osr serve or the platform from Node, Deno, Bun, edge and browsers.
• Platform quickstart - the hosted version of everything above.