--- name: osr-evaluation description: Evaluate and benchmark an OpenSmartRoute router - author JSONL evaluation datasets, run `osr eval` with cost/quality frontiers, baselines, robustness, calibration (ECE, Brier, conformal) and ablation reports, load RouterBench-style presets, run off-policy evaluation of logged decisions, and gate CI on minimum accuracy. Use when measuring routing accuracy, comparing strategies, or building a regression suite for routing quality. license: Apache-2.0 compatibility: OpenSmartRoute >= 0.4, Python >= 3.10 metadata: author: opensmartroute osr-domains: "data_analysis coding general" osr-tags: "opensmartroute evaluation benchmark calibration" osr-quality-prior: "0.85" osr-primary: "false" --- # OpenSmartRoute evaluation ## Dataset format (JSONL, one row per request) ```json {"text": "Hi there! How's it going?", "acceptable": ["llm-small"]} {"text": "Prove sqrt(2) is irrational", "expected": "llm-frontier", "context": {"tier": "pro"}} {"prompt": "Summarise this PDF", "scores": {"llm-small": 0.4, "llm-mid": 0.8}, "task_type": "summarize"} ``` Fields: `text` (or `prompt`), `expected` (single id), `acceptable` (list of ids), `context`, `profile`, `task_type`, `scores` `{id: quality}` (RouterBench style), `samples` `{id: [floats]}`, `tokens` `{id: tokens spent}` (effort / elastic siblings). Unknown keys are ignored. A row is correct when the decision is `expected` or in `acceptable`; with `scores`, `mean_quality` reports the quality of the chosen target. Agentic tasks (`--agentic`) use a different JSONL: one multi-step task per line with a per-agent, per-step success / latency table (`eval.agentic.AgentTask`): ```json {"task_id": "t1", "steps": ["look up the order", "reason through the refund conflict step by step"], "success": {"fast-agent": [0.97, 0.45], "strong-agent": [0.94, 0.90]}, "latency_ms": {"fast-agent": [400, 400], "strong-agent": [2000, 2000]}, "domain": "customer_support"} ``` Presets: `--preset routerbench|llmrouterbench|routereval|xroutebench|routerxbench` maps public benchmark files onto your catalogue (`eval.datasets.load_benchmark(path, preset, model_map, limit)`). ## CLI ``` osr -t targets.yaml -r rules.yaml eval data.jsonl [--limit N] [--json] --frontier 2-D cost/quality frontier + area under it --frontier3 quality/cost/latency Pareto frontier + hypervolume --baselines random / cheapest / best-prior / task-table / oracle (+ noise floor); --quality-floor 0.7 --robustness repeat flip-rate, paraphrase stability, profile-swap fairness (first 200 rows) --calibration ECE, Brier, reliability diagram, conformal set coverage / size; plus held-out ECE after fitting a temperature / isotonic calibrator on the even rows (>= 20 rows) --ablation leave-one-strategy-out and leave-one-extractor-out deltas --multi-round route-execute-judge-refine loop vs best single target (rows with scores); --threshold --max-rounds --effort effort routing vs always-highest-effort (rows with scores + tokens; targets declare effort) --agentic DATASET is agentic tasks; per-step routing vs best single agent on accuracy-latency; --retries N --min-accuracy X exit status 2 when accuracy < X (CI gate) osr -t targets.yaml ope logs.jsonl [--max-weight 20] # IPS / SNIPS / DR off-policy evaluation osr -t targets.yaml safety [--learned-guard] [--json] # red-team safety-routing suite osr collect --cache-dir data --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small # ten public sources (routellm-battles, arena-55k/100k/140k, ppe-human, webdev-arena, mt-bench-human, # reward-bench, ultrafeedback, routellm-gpt4), resumable after a 429; --source NAME picks, --refresh refetches osr -t targets.yaml slm train --out slm.json --cache-dir data --report # every *-tiers cache; numpy when installed # judge --report against the constant baseline: "always mid" is acceptable ~64 % of the time on the arena corpora OSR_SLM_ENCODER=attention osr ... slm train ... --report # compare the transformer-block encoder on the holdout osr slm eval slm.json data.jsonl --min-accuracy 0.8 # CI gate for the routing SLM itself ``` ## Python ```python from opensmartroute.eval import evaluate, load_dataset, cost_quality_frontier, area_under_frontier, calibration_report rows = load_dataset("data.jsonl") r = evaluate(router, rows) # EvalResult print(r.summary()) # n, accuracy, mean_cost_per_1k, mean_latency_ms, mean_confidence, # mean_quality, ece, brier, abstain_rate, set_coverage, mean_set_size r.confusion # {expected: {chosen: count}} r.errors # rows that missed, with the decision trace ``` Also: `eval.baselines.baseline_suite`, `eval.robustness.{repeat_flip_rate, paraphrase_robustness, profile_swap_fairness}`, `eval.frontier.hypervolume`, `eval.ope.off_policy_evaluate`. Roadmap exit criteria as runnable measurements: `eval.criteria` (`cold_start_ratio`, `effort_token_savings`, `multi_round_vs_best_single`, `match_at_1_at_scale`, `conformal_coverage`, `knapsack_never_exceeds_cap`, `ope_within_live_ci`, `run_all`) and `eval.agentic.task_routing_frontier`; `python scripts/exit_criteria.py [--full]` prints the table quoted in `docs/ROADMAP.md`. ## Interpreting results - Accuracy below baselines' `best-prior` means the catalogue (examples / complexity bands) is wrong, not the weights. Fix data first. - `ece > 0.1` -> confidence is miscalibrated: run `router.calibrate()` on logged outcomes or lower `OSR_ROUTING_SOFTMAX_TEMPERATURE`. `calibration_report()["held_out"]` shows what a fitted `TemperatureScaler` / `IsotonicCalibrator` reaches out of sample (on the public human-preference suites raw ECE 0.25-0.32 drops to 0.04-0.06 isotonic); ship the fitted calibrator rather than tuning weights. - `--ablation` deltas near zero identify strategies you can drop to cut latency. - `--robustness` flip-rate > 0 with identical inputs means a stochastic strategy (bandit) is dominating; seed it or lower its weight. - Use `--frontier` when the goal is "same quality, less cost": pick the objective weights on the knee. ## CI gate pattern ```yaml - run: osr -t examples/targets.yaml -r examples/rules.yaml eval examples/eval_dataset.jsonl --min-accuracy 0.9 --json ``` Keep the dataset in the repo, deterministic (`seed=0` strategies), and add a row for every routing bug you fix.