Published runs - opensmartroute 1.1.0

Produced with the recipe in Benchmarks and leaderboard recipe on 2026-09-10, osr 1.1.0, seed 0, --limit 1000 per suite (the 0.5.0 run of 2026-09-06 is quoted in the comparison column below). One JSON per suite: <suite>.json is the learned router (--slm slm.json --baselines --calibration --robustness), <suite>.declarative.json is the same catalogue without the SLM (--baselines only). The per-row arrays calibration.conformal_pairs and calibration.temperature_samples were removed from the committed files; every aggregate they feed (reliability, held_out) is kept.

Data

Catalogue: ../targets.yaml (llm-small / llm-mid / llm-frontier). SLM bundle: osr -t examples/leaderboard/targets.yaml slm train --out slm.json --cache-dir data --source arena-55k-tiers --source routellm-battles-tiers --report --seed 0 -> 6 051 rows, 10 % holdout (605 rows) accuracy 0.724, ECE 0.051, Brier 0.196; the constant-predictor baseline on the same holdout is 0.727, so the SLM's measured skill on pairwise tier labels is 0 - the report says so rather than hiding it. sha256 d1a101088b179811c72bef873ebace926ef855f52b03573bfe983d2470469fde (4.0 MB, not committed; the command is deterministic for the same cache). The 0.5.0 bundle was 8faf8466...cbbb1e (5 956 rows, 0.721).

Results

Cost and latency per 1k tokens for the learned router: MT-Bench $0.00370 / 978 ms, PPE $0.00460 / 1096 ms, WebDev $0.00331 / 933 ms (static task table: $0.003 / 900 ms on MT-Bench, $0.015 / 2500 ms on PPE and WebDev, since it degenerates to "always the best single tier").

Reading: the learned router beats the declarative router by +9 to +38 pp and random / cheapest everywhere, at a fifth to a third of the cost of "always frontier" on PPE and WebDev. Against 0.5.0 the accuracies moved by -0.9 / +1.1 / 0.0 pp - noise - while raw ECE fell from 0.28-0.33 to 0.16-0.24: the 1.1.0 SLM trains with a validation split, early stopping and a calibrator fitted on the validation rows (G-1 in the gap plan), so the shipped confidence is closer to honest before any post-hoc fitting. It still does not beat the best single tier on any suite (-3 to -10 pp): a pairwise battle between two named models is a weak label for a three-tier decision, and the label-noise floor cannot be measured on these rows (one sample per prompt). A calibrator fitted on the even rows and scored on the odd rows brings held-out ECE to 0.05-0.12 (isotonic). These are the numbers a RouterArena / LLMRouterBench / xRouteBench entry should quote; the gated multi-sample suites (step 5 of the recipe) are what would let the router be scored against the noise floor.

RouterBench (0-shot) - 11 real models, real task labels

routerbench.json, produced by ../routerbench.py on 2026-09-10 with osr 1.1.0: withmartian/routerbench routerbench_0shot.pkl (36,497 prompts from ~85 named evaluations, 11 models with per-response correctness and cost), split 50 / 50 by a hash of sample_id. The catalogue (quality prior = mean correctness, cost = mean $ per response) and the SLM are built from the train half only (18,258 rows; the SLM's 10 % report holdout: accuracy 0.785 against a constant-predictor baseline of 0.804); everything below is scored on the test half (18,239 rows). The static task table is fitted on the train half; its in-sample variant (fitted on the rows it is scored on, which is what osr eval --baselines did by default when this run was made - it now splits by --seed and fits on the train half) is shown for reference.

Reading, in order of what matters:

• The router does not beat the static task table on RouterBench. With real task labels the table is a strong, cheap-to-fit competitor: 0.830 on the train-fitted variant. The router reaches 0.810 when told that quality is all that matters (cost weight 0), with a higher realised quality than the table (0.786 vs 0.780 - it picks a better-than-gpt-4 model on some prompts) but 9 % higher spend. Per task it loses to the table on 78 of 85 evaluations at the default objective. The v0.4 criterion stays not met, now measured on the suite it was written for.
• What the router buys is the cost axis. At the default objective it spends 9x less than the table ($0.34 vs $2.98 per 1k calls) for -11 pp; at cost weight 0.1 it spends 30 % less for -2.5 pp. Between weights 0.1 and 0.3 the frontier drops off a cliff (0.805 -> 0.603): on this catalogue the cheap models are much worse, so "a little cheaper" is not available - the operator has to choose the side of the cliff.
• The label-noise floor cannot be measured here either. RouterBench 0-shot carries one score per (prompt, model); noise_floor() needs repeated samples and reports 0 by construction. The recipe's earlier claim that these rows carry several samples was wrong and has been removed.
• Calibration. Raw confidence with 11 candidates is badly under-confident (mean 0.17-0.26 at 0.72-0.81 accuracy, ECE 0.55); an isotonic calibrator fitted on the even rows brings the odd rows to ECE 0.010 (temperature scaling alone: 0.28). Ship a fitted calibrator with any catalogue this wide.
• The SLM's own skill is again nil against the constant predictor on its holdout (0.785 vs 0.804): the learned gain over the declarative catalogue (0.313 -> 0.716 / 0.810) comes from the priors, the cost term and the bandit / similarity ensemble, not from the SLM as a stand-alone classifier.

To reproduce: hf download withmartian/routerbench --repo-type dataset --local-dir <dir> (the dataset is no longer gated), then python examples/leaderboard/routerbench.py --pkl <dir>/routerbench_0shot.pkl --work <scratch> (needs pandas to read the pickle; about 15 minutes, no network after the download).

Listings

Where this run has been submitted. A submission links the exact commit, the SLM SHA-256 above and this page.