# Published runs - opensmartroute 0.5.0 Produced with the recipe in [../README.md](https://opensmartroute.ai/docs/leaderboard.md) on 2026-09-06, `osr 0.5.0`, seed 0, `--limit 1000` per suite. One JSON per suite: `.json` is the learned router (`--slm slm.json --baselines --calibration --robustness`), `.declarative.json` is the same catalogue without the SLM (`--baselines` only). The per-row arrays `calibration.conformal_pairs` and `calibration.temperature_samples` were removed from the committed files; every aggregate they feed (`reliability`, `held_out`) is kept. ## Data | Role | Cache name | Hugging Face dataset | Rows after `--tier` relabelling | |---|---|---|---| | train | `arena-55k-tiers` | `lmarena-ai/arena-human-preference-55k` | 1 986 | | train | `routellm-battles-tiers` | `routellm/gpt4_judge_battles` | 4 000 | | held out | `mt-bench-human-tiers` | `lmsys/mt_bench_human_judgments` (`human`) | 2 228 | | held out | `ppe-human-tiers` | `lmarena-ai/PPE-Human-Preference-V1` (`test`) | 2 360 | | held out | `webdev-arena-tiers` | `lmarena-ai/webdev-arena-preference-10k` (`test`) | 1 070 | Catalogue: [../targets.yaml](https://github.com/isathish/OpenSmartRoute/blob/main/examples/leaderboard/targets.yaml) (`llm-small` / `llm-mid` / `llm-frontier`). SLM bundle: `osr -t examples/leaderboard/targets.yaml slm train --out slm.json --cache-dir data --source arena-55k-tiers --source routellm-battles-tiers --report --seed 0` -> 5 956 rows, 10 % holdout accuracy 0.721, ECE 0.056, `sha256 8faf8466902b753a9f22382de24bfd0e1d5271c7fe5d6564dec8105310dcbb1e` (4.0 MB, not committed; the command is deterministic for the same cache). ## Results | Suite (n=1000) | Router (learned) | Declarative | Static task table | Random | Cheapest | Best prior | Paraphrase | Repeat flips | ECE raw | ECE held-out temp. | ECE held-out isotonic | |---|---|---|---|---|---|---|---|---|---|---|---| | MT-Bench human | **0.523** | 0.421 | 0.541 | 0.441 | 0.372 | 0.343 | 1.000 | 0.000 | 0.328 | 0.056 | 0.048 | | PPE human | **0.552** | 0.418 | 0.651 | 0.445 | 0.174 | 0.651 | 0.898 | 0.000 | 0.298 | 0.121 | 0.035 | | WebDev Arena | **0.573** | 0.192 | 0.670 | 0.391 | 0.000 | 0.670 | 0.883 | 0.000 | 0.283 | 0.032 | 0.047 | Cost and latency per 1k tokens for the learned router: MT-Bench $0.00363 / 968 ms, PPE $0.00390 / 1005 ms, WebDev $0.00324 / 925 ms (static task table: $0.003 / 900 ms on MT-Bench, $0.015 / 2500 ms on PPE and WebDev, since it degenerates to "always the best single tier"). Reading: the learned router beats the declarative router by +10 to +38 pp and random / cheapest everywhere, at a fifth of the cost of "always frontier" on PPE and WebDev. It does **not** beat the best single tier on any suite (-2 to -10 pp): a pairwise battle between two named models is a weak label for a three-tier decision, and the label-noise floor cannot be measured on these rows (one sample per prompt). Raw confidence is over-confident (ECE 0.28-0.33); a calibrator fitted on the even rows and scored on the odd rows brings held-out ECE to 0.03-0.05 (isotonic). These are the numbers a RouterArena / LLMRouterBench / xRouteBench entry should quote; the gated multi-sample suites (step 5 of the recipe) are what would let the router be scored against the noise floor. ## Listings Where this run has been submitted. `python scripts/release.py readiness` counts rows whose status is `accepted` for the v1.0 adoption item "Leaderboard listings accepted" and reports it as open until one listing is accepted. A submission links the exact commit, the SLM SHA-256 above and this page. | Leaderboard | Entry | Status | Notes | |---|---|---|---| | RouterArena | - | not submitted | submission = a PR to the RouterArena repository with `runs/*.json` | | LLMRouterBench | - | not submitted | needs the gated multi-sample suites (step 5) for the noise-floor comparison | | xRouteBench | - | not submitted | component ablation report from `osr eval --ablation` |