Reproducible leaderboard runs

Everything the published public-suite numbers are made of, as commands you can re-run. The same recipe is what an OpenSmartRoute entry on RouterArena, LLMRouterBench or xRouteBench is built from; the runs themselves are recorded, with their submission status, in Published benchmark runs.

Requirements: pip install "opensmartroute[yaml]", network access to the Hugging Face datasets-server (public suites; RouterBench is downloaded with hf download, see step 5).

1. Catalogue

targets.yaml: three tiers, llm-small / llm-mid / llm-frontier. Battle datasets name 2023-24 models; osr collect --tier maps each model to the tier it played in (eval.collect.ARENA_TIERS), so every row's label is "which tier was good enough for this prompt" - the RouteLLM framing.

2. Collect the public suites

osr collect --cache-dir data --limit 4000 \
    --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small

Writes one <source>-tiers.jsonl (+ .meta.json) per public source: arena-55k, arena-100k, arena-140k, routellm-battles, ppe-human, webdev-arena, mt-bench-human, reward-bench, ultrafeedback, routellm-gpt4. --source NAME restricts the run; --refresh ignores the cache.

3. Train the routing SLM on the training suites only

osr -t examples/leaderboard/targets.yaml slm train --out slm.json --cache-dir data \
    --source arena-55k-tiers --source routellm-battles-tiers --report --seed 0

--report holds out 10 % and prints its accuracy. The suites in step 4 are not in --source, so they are held out at the suite level, not just the row level.

4. Evaluate on the held-out suites

for suite in mt-bench-human ppe-human webdev-arena; do
  osr -t examples/leaderboard/targets.yaml --slm slm.json eval data/$suite-tiers.jsonl \
      --baselines --calibration --robustness --limit 1000 --json > results/$suite.json
done
# declarative router (no learned signals) for the ablation column
osr -t examples/leaderboard/targets.yaml eval data/mt-bench-human-tiers.jsonl --baselines --limit 1000

What each flag reports and where it lands in the table:

The published table used --limit 1000 per suite and seed 0 throughout; the SLM bundle and the Router are deterministic, so re-running gives the same numbers for the same cache.

5. RouterBench and the other named benchmarks

withmartian/routerbench is no longer gated: hf download withmartian/routerbench --repo-type dataset --local-dir <dir>. routerbench.py runs the honest protocol on it - a 50 / 50 split by sample_id, catalogue and SLM from the train half, router vs the baselines (task table fitted on the train half) on the test half, frontier and calibration - and writes results/routerbench.json; the reading is in Published benchmark runs. Other benchmark files in the same shape evaluate directly through the layout presets:

osr -t examples/leaderboard/targets.yaml --slm slm.json eval routerbench_0shot.jsonl \
    --preset routerbench --baselines --calibration --limit 5000

noise_floor() needs rows with repeated samples per model (samples); RouterBench 0-shot carries one score per (prompt, model), so the label-noise floor is not measurable from it - only suites that publish several draws per model can be scored against it.

6. Publish

Commit results/*.json, the exact osr --version, the dataset ids, splits and row counts, and the SHA-256 of slm.json together; that triple is the reproducible config a leaderboard entry links to.

The run published with 0.5.0 is in Published benchmark runs (one JSON per suite, learned and declarative); it is the entry submitted to RouterArena, LLMRouterBench and xRouteBench.

7. Live multi-cloud run (real models, real prices)

../multicloud_benchmark.py routes ../multicloud_prompts.jsonl across three price tiers on Azure OpenAI, Amazon Bedrock and Google Vertex AI, answers every prompt for real, has a judge grade the routed answer against an always-frontier baseline blind, and prices both from the providers' token usage. Reports land in ../results/multicloud/: the cold-start run (150 prompts, 8 models, one judge) measured 56 % lower cost than all-gpt-4.1 at 9.22 vs 9.35; with the routing rules in ../multicloud_rules.yaml, a warm start from that run's judged outcomes and two judge families (gpt-4.1 and gemini-2.5-pro, 73 % agreement) the second run measured 64 % lower cost at 9.44 vs 9.39. --matrix then had every model answer every prompt directly, judged the same way (matrix-20260910-093639.md): a single cheap model, gpt-oss-120b on Bedrock, scored 9.66 for $0.047 and beat that router (9.44, $0.087) - the hand-set quality priors were wrong for this traffic and the default Objective(cost=0.15) kept 18 % of prompts on gpt-4.1. --replay scores router configurations offline against the matrix cells (no calls, replay-20260910-152410.md): the objective weight is the lever, warm-starting the learners is not (it spread traffic and raised cost at the default weight). The predicted best setting, Objective(cost=1.0) with the rules and no learned state, was then run live (20260910-100323.md, final table matrix-20260910-100351.md): 89 % lower cost than all-gpt-4.1 ($0.026 vs $0.246) at 9.53 vs 9.27, 68 wins / 52 ties / 30 losses, p50 1.2 s vs 2.0 s - second only to gpt-oss-120b direct (9.66, $0.047) and 44 % cheaper than it. Routing does not reduce tokens (reasoning models bill their thinking) and prompt compaction saved nothing on these short prompts; the savings come from which model bills the tokens. Total spend of all runs about $6.60. Credentials are read from the environment (see the module docstring); --probe checks every model with one tiny call, --cap-usd bounds the spend and --rejudge re-scores a run's stored answers without re-running the models.