osr-routing-catalogue

Routing is signals -> policy -> strategies -> ensemble -> decision (+plan). The catalogue is data: add a target and it is routable immediately, no retraining.

Target schema (targets.yaml / .json, or TargetRegistry.from_dicts)#

Top level is targets: [...] (a bare list also works). Keys are exactly the RouteTarget fields - unknown keys raise ConfigurationError.

targets:
  - id: llm-small                      # required, unique; used in rules, outcomes, plans
    kind: llm                          # llm | agent | skill | persona | tool | workflow | human | destination
    name: Small fast model
    description: Cheap, fast generalist for simple chat, classification and extraction.
    capabilities:
      domains: [general, customer_support]      # match signals.domains
      actions: [qa, classify, extract, summarize]
      languages: [en, es]                       # default [en]; "*" = any
      modalities: [text]                        # default [text]
      min_complexity: 0.0                       # request complexity band this target fits
      max_complexity: 0.45
      context_window: 128000                    # optional; policy rejects oversize inputs
      supports_tools: true
      tags: []
    constraints:
      regions: [us, eu]                         # empty = anywhere
      data_boundary: public                     # public | private | on_prem
      pii_allowed: true
      max_tokens_in: null
      tenants: []                               # empty = every tenant
    cost: { usd_per_1k_tokens: 0.0002 }         # also wh_per_1k_tokens, gco2_per_1k_tokens
    latency_ms: 300
    quality_prior: 0.55                         # 0..1 belief before any outcomes
    examples: ["Hi, how are you today?"]        # anchor the similarity strategy
    instructions: ""                            # system prompt fragment, disclosed only when selected
    primary: true                               # false = slot-only (persona/skill layered on a model)
    family: ""                                  # siblings share breaker/budget state
    effort: ""                                  # reasoning-effort variant label
    enabled: true
    metadata: {}                                # free-form; keys read by optional strategies below

metadata keys that optional strategies read (all opt-in, none required):

• tier: edge | cloud - strategies.EdgeCloudStrategy (edge decode time vs SLO, cloud upload penalty, cloud excluded for private / on_prem requests).
• input_schema: {properties: {...}, required: [...]} on kind: tool - discovery.SchemaAwareStrategy scores how many required parameters the request text can fill (MCP imports set it automatically).
• requires, conflicts, composes (lists of skill ids) on kind: skill - discovery.SkillGraph; SKILL.md packages declare them as osr-requires / osr-conflicts / osr-composes frontmatter.
• Elastic models: do not hand-write <id>@<budget> siblings; call strategies.expand_elastic(parent, [BudgetVariant(tokens, quality=, cost_scale=)]) and add the result; it sets metadata.token_budget, which strategies.TokenBudgetStrategy scores against the tokens the answer needs.

Guidance:

• Give every target 2-5 examples phrased like real requests; similarity is the cheapest strong signal.
• Set min_complexity/max_complexity bands so the capability strategy can penalise overkill and too-hard routes; overlap bands slightly.
• quality_prior is a prior, not a score - bandits/IRT move it with Outcomes.
• Personas and skills are usually primary: false; they fill plan slots.
• Python handlers: registry.get("id").handler = fn where fn(request, **kw) -> response.

Rules (rules.yaml, Arch-Router style)#

rules:
  - name: pii-stays-onprem
    when: { contains_pii: true }                  # domains, actions, language, min/max_complexity,
    prefer: [llm-onprem, human-escalation]        #   contains_pii, needs_tools, context: {key: value}
    avoid: [llm-frontier]
    weight: 1.0
    pin: false                                    # true = restrict candidates to `prefer`

Rules push scores; they never override policy. There is no boost key - use weight.

Hard constraints and objectives (per request)#

from opensmartroute import RouteRequest, RequestConstraints, Objective
req = RouteRequest("Summarise this contract",
    constraints=RequestConstraints(region="eu", data_boundary="private", max_cost_per_1k=0.01,
                                   max_latency_ms=3000, allowed_kinds=["llm"], deny_targets=[],
                                   tenant="acme", require_tools=False),
    objective=Objective(quality=1.0, cost=0.3, latency=0.05, quality_floor=0.6, energy=0.0, carbon=0.0))

Constraints are filtered by Policy (rejections appear in decision.trace.policy_rejections); objective weights shape the utility quality - cost*norm(cost) - latency*norm(latency).

Request context keys that strategies read: user_id (learning.UserAdaptiveStrategy), last_target and last_failed (learning.HistoryTargetStrategy incumbent bonus / penalty), difficulty (signals.VerbalisedDifficultySignal), draft (signals.DraftResponseSignal), session_id (MarkovStrategy transitions). Pass prior turns in RouteRequest.history=[{"role": ..., "content": ...}] so multi-turn strategies see them.

Route, plan, learn#

from opensmartroute import Router, Outcome, load_targets, load_rules
router = Router(load_targets("targets.yaml"), strategies=[load_rules("rules.yaml"), *default_strategies()])
d = router.route(req, plan=True)          # d.target, d.confidence, d.alternatives, d.plan.slots, d.trace
print(d.trace.explain())                  # per-strategy scores for every candidate
router.learn(Outcome(d.request_id, d.target.id, success=True, quality=0.9, cost_usd=0.002,
                     latency_ms=1200, domains=d.trace.signals.domains, complexity=d.trace.signals.complexity))
res = router.run(req)                     # route + execute plan; res.text, res.outcomes, res.system_prompt

CLI equivalents: osr -t targets.yaml -r rules.yaml route "text" [--plan] [--json], osr -t targets.yaml targets, osr -t targets.yaml stats.

Debugging a surprising route#

1. d.trace.explain() - check policy: N -> M candidates and the rejection reasons first.
2. Compare capability (domain/action/complexity fit) vs similarity (examples) columns; fix the catalogue (examples, complexity band) before touching weights.
3. Weights and thresholds live in Settings (osr settings); override with OSR_WEIGHTS_<NAME> or Router(weights={...}) only after the catalogue is right.