osr-routing-catalogue

Routing is signals -> policy -> strategies -> ensemble -> decision (+plan). The catalogue is data: add a target and it is routable immediately, no retraining.

Target schema (targets.yaml / .json, or TargetRegistry.from_dicts)

Top level is targets: [...] (a bare list also works). Keys are exactly the RouteTarget fields - unknown keys raise ConfigurationError.

targets:
  - id: llm-small                      # required, unique; used in rules, outcomes, plans
    kind: llm                          # llm | agent | skill | persona | tool | workflow | human | destination
    name: Small fast model
    description: Cheap, fast generalist for simple chat, classification and extraction.
    capabilities:
      domains: [general, customer_support]      # match signals.domains
      actions: [qa, classify, extract, summarize]
      languages: [en, es]                       # default [en]; "*" = any
      modalities: [text]                        # default [text]
      min_complexity: 0.0                       # request complexity band this target fits
      max_complexity: 0.45
      context_window: 128000                    # optional; policy rejects oversize inputs
      supports_tools: true
      tags: []
    constraints:
      regions: [us, eu]                         # empty = anywhere
      data_boundary: public                     # public | private | on_prem
      pii_allowed: true
      max_tokens_in: null
      tenants: []                               # empty = every tenant
    cost: { usd_per_1k_tokens: 0.0002 }         # also wh_per_1k_tokens, gco2_per_1k_tokens
    latency_ms: 300
    quality_prior: 0.55                         # 0..1 belief before any outcomes
    examples: ["Hi, how are you today?"]        # anchor the similarity strategy
    instructions: ""                            # system prompt fragment, disclosed only when selected
    primary: true                               # false = slot-only (persona/skill layered on a model)
    family: ""                                  # siblings share breaker/budget state
    effort: ""                                  # reasoning-effort variant label
    enabled: true
    metadata: {}                                # free-form; keys read by optional strategies below

metadata keys that optional strategies read (all opt-in, none required):

• rate_limit, budget, breaker - live limits applied when health controls are on (RouterBuilder.with_health(), enterprise.harden(health=True), osr serve --health): rate_limit: 600 (requests per minute) or {per_minute | per_second, burst}; budget: 5 (USD per hour) or {usd, per: minute|hour|day|week} / {usd, window_s}; breaker: {failure_threshold, window_s, recovery_s, half_open_max_calls}. An exhausted bucket or budget rejects the target in the policy (rate limited / budget exhausted in the trace) and the request goes to the next best candidate; /stats reports budget_remaining per target.
• weight with a shared top-level family - interchangeable deployments of one model (a region each, a key each): RouterBuilder.with_spread() (strategies.SpreadStrategy) draws one sibling per request in proportion to weight (default 1), so load spreads in the declared ratio; breakers, rate_limit and budget still apply per sibling and drop a saturated one out of the draw.
• tier: edge | cloud - strategies.EdgeCloudStrategy (edge decode time vs SLO, cloud upload penalty, cloud excluded for private / on_prem requests).
• input_schema: {properties: {...}, required: [...]} on kind: tool - discovery.SchemaAwareStrategy scores how many required parameters the request text can fill (MCP imports set it automatically).
• requires, conflicts, composes (lists of skill ids) on kind: skill - discovery.SkillGraph; SKILL.md packages declare them as osr-requires / osr-conflicts / osr-composes frontmatter.
• Elastic models: do not hand-write <id>@<budget> siblings; call strategies.expand_elastic(parent, [BudgetVariant(tokens, quality=, cost_scale=)]) and add the result; it sets metadata.token_budget, which strategies.TokenBudgetStrategy scores against the tokens the answer needs.

Guidance:

• Give every target 2-5 examples phrased like real requests; similarity is the cheapest strong signal.
• domains and actions use the signal ontology names (signals.DOMAIN_LEXICON / ACTION_LEXICON). Actions: summarize, translate, classify, extract, ocr (scanned, handwritten or photographed documents, the text in a screenshot), code_generation, code_review, reasoning, planning, qa, search (look it up on the live web, latest news, current price, cite sources), generation (writing and editing help - rewrite, proofread, shorten, reply to), image, speech (read aloud, text to speech), transcribe (voice memo, dictation), embed (embeddings, semantic search), browse (open the page, log in, fill in the form, click through, download), action (send, deploy, create a ticket, notify, rename, move, paste, type into the window in front), escalate (only asking for a human or a manager; "bullet points for my manager" is a summary). Domains add observability (logs, traces, latency, alerts, incidents, root cause), science and education to the business domains. A human target scores as a last resort: without an escalate ask it is halved (capability.human_fallback_penalty), so it never wins a plain question on its quality prior.
• general in domains means "any domain": such a target gets the mild generalist fit on a medical or legal text and is never penalised as off-domain (a translator declared for general must not lose every specialised request to a specialist that happens to share the domain). Omit general when the target should really only see its listed domains. A declared actions list that matches nothing the request asks for is penalised (off_scope_penalty), so declare what the target does, not everything it could do.
• Set min_complexity/max_complexity bands so the capability strategy can penalise overkill and too-hard routes; overlap bands slightly. A kind: tool a little past its band is only decayed; the tool out-of-scope penalty starts tool_band_tolerance (0.1) beyond max_complexity or when the request asks for an action the tool does not declare.
• quality_prior is a prior, not a score - bandits/IRT move it with Outcomes.
• Personas and skills are usually primary: false; they fill plan slots. Tools fill tool slots too: a model / agent plan offers up to tool_slots (2) tools whose ensemble quality clears tool_quality_floor (0.5) as functions the model may call - give them examples and domains so the right one is offered.
• Python handlers: registry.get("id").handler = fn where fn(request, **kw) -> response.

Rules (rules.yaml, Arch-Router style)

rules:
  - name: pii-stays-onprem
    when: { contains_pii: true }                  # domains, actions (any of), all_actions (every one),
    prefer: [llm-onprem, human-escalation]        #   language, min/max_complexity, contains_pii, needs_tools,
    avoid: [llm-frontier]                         #   event, workflow, context: {key: value}
    weight: 1.0
    pin: false                                    # true = restrict candidates to `prefer`

Rules push scores; they never override policy. There is no boost key - use weight. Pinned rules are tried in file order, so a narrow pin (all_actions: [image, extract] -> the OCR model) goes before a broad one (actions: [image] -> the image generator).

The same rule in Python: osr.rule("pii-stays-onprem", prefer=["llm-onprem"], avoid=["llm-frontier"], contains_pii=True) - the when keys become keywords and a mistyped one raises ConfigurationError; a predicate form @osr.rule(prefer=[...]) over fn(request, signals) -> bool covers what when cannot say (osr-decorator-sdk).

Hard constraints and objectives (per request)

from opensmartroute import RouteRequest, RequestConstraints, Objective
req = RouteRequest("Summarise this contract",
    constraints=RequestConstraints(region="eu", data_boundary="private", max_cost_per_1k=0.01,
                                   max_latency_ms=3000, allowed_kinds=["llm"], deny_targets=[],
                                   tenant="acme", require_tools=False),
    objective=Objective(quality=1.0, cost=0.3, latency=0.05, quality_floor=0.6, energy=0.0, carbon=0.0))

Constraints are filtered by Policy (rejections appear in decision.trace.policy_rejections); objective weights shape the utility quality - cost*norm(cost) - latency*norm(latency).

Request context keys that strategies read: user_id (learning.UserAdaptiveStrategy), last_target and last_failed (learning.HistoryTargetStrategy incumbent bonus / penalty), difficulty (signals.VerbalisedDifficultySignal), draft (signals.DraftResponseSignal), session_id (MarkovStrategy transitions). Pass prior turns in RouteRequest.history=[{"role": ..., "content": ...}] so multi-turn strategies see them.

Route, plan, learn

from opensmartroute import Router, Outcome, load_targets, load_rules
router = Router(load_targets("targets.yaml"), strategies=[load_rules("rules.yaml"), *default_strategies()])
d = router.route(req, plan=True)          # d.target, d.confidence, d.alternatives, d.plan.slots, d.trace
print(d.trace.explain())                  # per-strategy scores for every candidate
router.learn(Outcome(d.request_id, d.target.id, success=True, quality=0.9, cost_usd=0.002,
                     latency_ms=1200, domains=d.trace.signals.domains, complexity=d.trace.signals.complexity))
res = router.run(req)                     # route + execute plan; res.text, res.outcomes, res.system_prompt
res = router.run(req, fallbacks=True)     # a failing / empty target is learned as a failure and the next one answers

CLI equivalents: osr -t targets.yaml -r rules.yaml route "text" [--plan] [--json], osr -t targets.yaml targets, osr -t targets.yaml stats.

Debugging a surprising route

1. d.trace.explain() - check policy: N -> M candidates and the rejection reasons first.
2. Compare capability (domain/action/complexity fit) vs similarity (examples) columns; fix the catalogue (examples, complexity band) before touching weights.
3. Weights and thresholds live in Settings (osr settings); override with OSR_WEIGHTS_<NAME> or Router(weights={...}) only after the catalogue is right.