Research Foundations
Problem statement, literature by family and which module borrows which idea.
This document is the evidence base for OpenSmartRoute. It (1) states the routing problem formally, (2) surveys the literature by family, (3) compares the systems that exist today, (4) lists the gaps that motivate this project, and (5) maps every borrowed idea to the module that implements it. It is maintained as the design changes; see ROADMAP.md for what is still open.
1. Problem statement#
A request $x$ arrives with context $c$ (tenant, session, constraints). A catalogue $\mathcal{T}$ of targets is available — models, agents, skills, personas, tools, workflows, humans. Each target $t$ has a cost $c_t(x)$, latency $\ell_t(x)$ and an unknown quality $q_t(x)\in[0,1]$. Routing chooses
$$ t^* = \arg\max_{t\in\mathcal{A}(x)} ; \mathbb{E}\big[q_t(x)\big] - \lambda_c, c_t(x) - \lambda_\ell, \ell_t(x) $$
where $\mathcal{A}(x)\subseteq\mathcal{T}$ is the admissible set after hard constraints (data boundary, region, PII, budget, SLO). Three things make this hard:
- $q_t(x)$ is never observed before routing — it must be predicted from $x$ and from $t$'s history.
- $\mathcal{T}$ changes — targets are added, deprecated, and degrade; the router must be inductive.
- Feedback is delayed, partial and noisy — a thumbs-up, a retry, a handoff, or nothing.
The survey by Varangot-Reille et al. (2025) formalises this as a performance–cost optimisation and
classifies solutions as pre-generation (predict, then call one target) or post-generation
(call, verify, escalate). OpenSmartRoute does both: strategies are pre-generation; Cascade and the
planned multi-round executor are post-generation.
2. Literature by family#
2.1 Supervised routers (predict quality from the prompt)#
| Work | Core idea | Result | Limitation |
|---|---|---|---|
| FORC — Šakota, Peyrard & West, WSDM 2024 | A meta-model predicts each LM's performance on an input; assign under a cost budget or minimise cost at a target quality | matches largest LM at −63 % cost on 14 datasets | fixed LM pool; retrain when the pool changes |
| Hybrid LLM — Ding et al., ICLR 2024 | Predict query difficulty; route to small vs large with a tunable quality threshold | −40 % large-model calls at equal quality | binary pool |
| RouteLLM — Ong et al., 2024 | Train routers on human preference (Chatbot Arena) with data augmentation; four router types (BT, matrix factorisation, BERT, causal LLM) | >2× cost reduction; routers transfer to new strong/weak pairs | pairwise (two targets) |
| Zooter — Lu et al., 2023 | Distil reward-model scores on training queries into a routing function; tag-based label smoothing | beats best single model on 26 subsets | needs an offline reward model |
| RouterDC — Chen et al., NeurIPS 2024 | Encoder + learnable LLM embeddings trained with dual contrastive losses (sample–LLM, sample–sample) | +2.8 % ID / +1.9 % OOD over best single LLM | handles "several are fine" better than pointwise |
| Arch-Router — Tran et al., 2025 | 1.5 B model maps query → user-defined domain / action; routing preferences encoded declaratively; new models added without retraining | SOTA on preference alignment for conversational data | routing policy still hand-authored |
| Prompt-to-Leaderboard — Frick et al., 2025 | LLM outputs Bradley–Terry coefficients per prompt → prompt-specific leaderboard; optimal router follows | #1 on Chatbot Arena (Jan 2025); power-law scaling | needs preference data at scale |
2.2 Inductive / representation-based routers (generalise to unseen targets)#
| Work | Core idea | What we took |
|---|---|---|
| GraphRouter — Feng, Shen & You, ICLR 2025 | Heterogeneous graph (task, query, LLM); edge prediction of effect + cost; new LLM = new node | targets are described by declared capabilities, not learned IDs |
| UniRoute — Jitkrittum et al., 2025 | Represent an LLM by its prediction vector on a representative prompt set; cluster-based routing; excess-risk bound; 30+ unseen LLMs | examples on a target; nearest-example similarity |
| EmbedLLM — Zhuang et al., 2024 | Encoder–decoder learns compact LLM embeddings that forecast benchmark performance | roadmap v0.5: target embeddings from outcomes |
| IRT-Router — Song et al., ACL 2025 | Item Response Theory: target ability θ, query difficulty b, discrimination a; interpretable; semantic warm-up for cold start | math.irt, learning.IRTStrategy |
2.3 Online / bandit routers (learn from live feedback)#
| Work | Core idea | What we took |
|---|---|---|
| MixLLM — Wang et al., NAACL 2025 | Contextual bandit; tag-enhanced query embedding; per-LLM quality & cost predictors; meta-decision for quality/cost/latency; continual training; handles pool changes | LinUCBStrategy on the signal vector; Objective weights |
| C2MAB-V — Dai et al., 2024 | Combinatorial MAB with versatile reward models and cost constraints; regret & violation bounds | CostAwareBandit (Lagrangian budget) |
| Thompson sampling / UCB1 (Thompson 1933; Auer et al. 2002) | Posterior sampling / optimism for exploration | ThompsonBeta, UCB1 |
| LinUCB (Li et al., WWW 2010) | Linear payoff with confidence ellipsoid | LinUCB |
2.4 Cascades and multi-round routers (post-generation)#
| Work | Core idea | What we took |
|---|---|---|
| FrugalGPT — Chen, Zaharia & Zou, 2023 | Call cheap first; a scorer decides whether to escalate | Cascade with pluggable quality_gate |
| AutoMix — Aggarwal et al., NeurIPS 2024 | Few-shot self-verification + POMDP router over verification noise; −50 % cost | roadmap v0.6: POMDP cascade |
| Router-R1 — Zhang, Feng & You, NeurIPS 2025 | Router is itself an LLM; interleaves think / route actions across rounds; RL with format + outcome + cost reward; conditions on model descriptors (price, latency, examples) | LLMJudgeStrategy (conditioned on the same descriptors), confidence-gated escalation; roadmap v0.6 multi-round |
| LLM-Blender — Jiang, Ren & Lin, ACL 2023 | PairRanker over candidate outputs + GenFuser | aggregation step for the multi-round executor |
2.5 Multi-agent, tool and skill routing#
| Work | Core idea | What we took |
|---|---|---|
| MasRouter — Yue et al., 2025 | Multi-Agent System Routing: choose collaboration mode → agent roles → LLM per role via cascaded controllers; −52 % overhead | RoutePlan slots (persona → skill → model) filled by sub-routing |
| ToolLLM — Qin et al., 2023 | 16 k APIs; neural API retriever recommends tools per instruction; DFS decision tree | tools are targets with examples; retrieval = SimilarityStrategy |
| AnyTool — Du et al., 2024 | Hierarchical API retriever (category → tool → endpoint) + self-reflection; revised evaluation protocol | roadmap v0.7: MCP catalogue import with hierarchical narrowing |
| Agent Skills (Anthropic, 2025) / skills as first-class units | Skills are declared with a description and triggers | TargetKind.SKILL with capabilities.actions |
2.6 Sequential decision making for conversations#
Dialogue policy as an MDP dates to Levin, Pieraccini & Eckert (2000) and is the basis for
POMDP dialogue systems (Young et al., 2013). OpenSmartRoute applies the same lens to who answers
next: MarkovChain learns intent transitions, RoutingMDP runs value iteration with
reward = quality − λ·cost, and MarkovStrategy scores targets by their value on the predicted
next state, not only the current one.
2.7 Capacity and queueing#
Skills-based routing in contact centres is a mature field (Koole & Mandelbaum 2002). Its tools
carry over directly to pools of humans or long-running agents: Erlang-C for P(wait) and
mean wait in M/M/c, Little's law for in-flight sizing, and Kingman's G/G/1 approximation
for bursty single endpoints. Implemented in math.decision.
2.8 Adversarial robustness of routers#
Rerouting LLM Routers (Shafran, Schuster, Ristenpart & Shmatikov, 2025) defines LLM control-plane
integrity and shows query-independent confounder gadgets — short token sequences that push any
query to the strong model in white- and black-box settings, with low perplexity (so perplexity
filtering fails). Consequences for us: routing signals are computed on the cleaned head of a
prompt (security.InputGuard), the guard reports the suspicion in the trace, and the roadmap keeps
a learned detector open. See SECURITY.md.
The 2026 literature widens the control-plane threat model beyond gadgets:
| Work | Threat | Consequence for us |
|---|---|---|
| ROPE — Aug 2026 | Indirect prompt injection through tool results steering state-changing tool calls; deterministic origin check on sensitive parameters keeps attack success at 1.6–2.6 % with 82–100 % utility | origin policy for tool parameters inside executed RoutePlans (v0.9) |
| Verifiable manifest signing for MCP pipelines — Jan 2026 | Tampered or drifting tool manifests | signed-manifest verification at import |
| "Recognition Without Enforcement" — Aug 2026 | Models recognise unsafe tool requests but still execute them | enforcement lives in policy, not in the model |
| "Beyond Max Tokens" — Jan 2026 | Resource amplification through tool chains | per-task depth / token caps |
| "When Safety Routing Breaks" — Sep 2026 | Routing itself as the safety bypass | adversarial routing regression suite |
2.9 Benchmarks and evaluation methodology#
| Benchmark | Scale | Use here |
|---|---|---|
| RouterBench — Hu et al., 2024 | 405 k inference outcomes, 11 LLMs, 8 datasets; defines the quality–cost frontier and AIQ | eval.cost_quality_frontier, area_under_frontier; eval.collect lists the source (routerbench, gated: the Hub serves no data files - load a local copy with DatasetCollector.store) |
| RouterEval — Huang et al., 2025 | 200 M records, 8 500 LLMs, 12 evaluations; shows model-level scaling — a good router improves as the pool grows and can beat the best single model | motivates large catalogues; eval.collect source routereval (gated: needs a Hub token) |
| LLMRouterBench — Jan 2026 | 400 k instances, 21 datasets, 33 models, 10 baselines; many routers, including commercial ones, do not reliably beat simple baselines; large oracle gap from model-recall failures; larger ensembles show diminishing returns vs curation; latency-aware analysis | roadmap v0.4: mandatory baselines, curation report |
| LLMRouter / xRouteBench — Feng & You, Aug 2026 | Routing as a sequential decision with five components (context encoder, model encoder, scoring, decision rule, learning signal); 16+ routers; single/multi-turn, agentic and personalised tasks; learned routers +14.6 % over the best fixed model | component ablation report; personalised split |
| RouterXBench / ProbeDirichlet — Feb 2026 | Three axes: router ability, scenario alignment, cross-domain robustness; hidden-state Dirichlet router +16.7 / +18.9 % | cross-domain split; math.DirichletProbe + strategies.HiddenStateStrategy |
| RouterArena — 2026 | Public leaderboard with arena score, accuracy and cost per 1 k queries | v1.0 submission target |
| "How Much of the Routing Gap Is Real?" — Jul 2026 | 12–36 % of the router-to-oracle gap on open pools is single-draw label noise; multi-sample oracle ($k\ge 20$) and best-of-K recover the floor | --oracle-samples, reproducible-headroom figure |
| "Most of the LLM Routing Gap Is Task Type" — Aug 2026 | 21 methods within a fraction of a point; a static per-task-type table answered 262/294 at $3.33 vs 245 at $7.69 for the best single model; 5.4 % run-to-run flips | TaskTableStrategy; --repeats |
| "When is Routing Meaningful?" — Jul 2026 | Hierarchic social entropy for pool diversity; <10 behaviourally distinct agents suffice; kNN routers collapse under paraphrase, prompted routers stay stable | diversity / coreset report; paraphrase-robustness score |
| MixInstruct (LLM-Blender) | oracle pairwise comparisons | preference data format for FeedbackStore.export_preferences |
| SkillFeed-Bench, Task2MCP, ComplexMCP — 2026 | counterfactual profile-conditioned skill routing; MCP server recommendation; multi-server tool tasks | v0.7 discovery evaluation |
2.10 Risk control, calibration and abstention#
| Work | Core idea | What we take |
|---|---|---|
| RACER — Feb 2026 | α-VOR: output a set of models with distribution-free mis-routing risk ≤ α, plus abstention; nested sets from concentration bounds | RouteDecision.candidate_set, abstain → HUMAN |
| RouteNLP — ACL 2026 (industry) | Conformal prediction to set cascade thresholds; distillation–routing co-optimisation by clustering escalation failures; −58 % cost, p99 1847 → 387 ms | conformal escalate_below; escalation-failure clustering feeds osr train |
| CR² — May 2026 | Cost-aware, risk-controlled routing | same family |
| Temperature scaling (Guo et al. 2017), isotonic regression, ECE | Post-hoc probability calibration | fitted confidence instead of fixed $\tau$ |
| Semantic entropy (Kuhn et al. 2023; Farquhar et al. 2024), P(True) (Kadavath et al. 2022) | Response-side uncertainty | signals.semantic_entropy(), response_uncertainty(), UncertaintyGate, EventTrigger |
| Learning to defer (Madras et al. 2018; Mozannar & Sontag 2020; Verma et al. 2023) | Classifier and expert trained jointly with a consistent surrogate; the expert has its own cost and accuracy | TargetKind.HUMAN as a deferral option |
2.11 Multi-turn, task-level and step-level routing for agents#
| Work | Core idea | What we take |
|---|---|---|
| TRACE-Router — Jul 2026 | Agentic tasks have one delayed, task-level outcome; assign a backend once at admission with a contextual bandit, pin it, learn from the terminal reward (accuracy + latency); +7–8 pts on τ²-Bench, +7.1 pts / −36 % latency on Terminal-Bench | Outcome.task_id, admission-time routing, delayed-feedback bandits |
| MTRouter — ACL 2026 | History–model joint embeddings; turn-level utility from logged trajectories; beats GPT-5 on ScienceWorld at −58.7 % cost with fewer switches | learning.HistoryTargetStrategy: logistic model over h * e_t (recency-weighted hashed history, catalogue target embedding) with shared weights and per-target bias; incumbent bonus / failed-turn penalty from context["last_target"] |
| ProgRouter — EMNLP 2026 Findings | Multi-view task-progress scorer + dual-path progress predictor + meta-gating; step-wise routing under time & cost budgets | progress-guided step routing |
| RLCascadeRouter — Aug 2026 | Cascade routing as an MDP with stop / select actions trained on trajectory returns; quality-estimator-free; prediction loss ≠ decision loss; unseen models without retraining | cascade rewrite on RoutingMDP |
| Bayesian self-escalation — Aug 2026 | Intra-generation delegation as Bayesian optimal stopping over a learned competence posterior; time-varying threshold; $1/\sqrt{n}$ regret in calibration trajectories | strategies.SelfEscalation + wrap_stream(): Beta competence posterior per streamed chunk, stop when loss(stop) < loss(continue) |
| TACIT-Switch — Aug 2026 | Permanent handoff learned from interval-censored teacher intervention times; mixture-cure threshold model; +7.4–11.1 pp | learning.MixtureCureModel (Weibull mixture-cure, censored MLE) + learning.HandoffPolicy |
| JiSi — Jan 2026 | Query–response mixed routing, support-set aggregator selection, adaptive routing ↔ aggregation switch; 10 open LLMs beat Gemini-3-Pro at 47 % cost | signals.DraftResponseSignal (query–response features); strategies.MixtureOfAgents (routing / aggregation switch) |
| GMTRouter — EMNLP 2026 Findings | Heterogeneous graph (user, LLM, query, response, turn); few-shot personalisation | learning.UserAdaptiveStrategy: user → neighbour users → global shrinkage over hashed profile vectors |
| TRouter — ACL 2026 | Task-taxonomy-guided synthetic data for cold start; latent task-type variables with prior regularisation | osr train --synth |
| SCX Router — Sep 2026 | 0.6 B decoder-KV classifier predicts task type, difficulty, reasoning mode and output length before generation; task ontology (23 families / 115 types / 345 subtypes / 30 domains); separates learned prediction from per-task policies (eligibility, cost, cache, safety, sovereignty) | ontology-based task classifier; policy/prediction separation matches ours |
| Think When Needed — SIGIR 2026 | Pre-generation router head decides Think vs Non-Think; operating point chosen on the validation Pareto frontier; +6.3 % NDCG@10 at −49.5 % tokens | effort as a routable dimension |
| Agent-as-a-Router, SWE-Router, "Routing least learnable where most valuable" — 2026 | Routing gains shrink on the hardest agentic tasks | eval.headroom.learnability_by_difficulty: per-difficulty-bucket headroom and noise floor |
| BudgetMem, Gated-Memory Routing — 2026 | Routing applied to agent memory tiers | strategies.MemoryRouter (tiers with capacity / cost / latency, learned ImportanceGate, budgeted recall) |
2.12 Capability discovery at scale (models, agents, tools, skills)#
| Work | Core idea | What we take |
|---|---|---|
| Enrich-Retrieve-Rank — Aug 2026 | MATS (models · agents · tools · skills) discovery; offline enrichment + retrieve-then-rank; in-context routing Match@1 falls 0.85 → 0.12 at N = 7 278 while retrieve-then-rank holds 0.81 → 0.39; crossover ≈ N = 500; 70× cheaper than full context | two-stage narrowing switched on by catalogue size |
| SCOUT — Aug 2026 (PayPal) | MCP gateway with tool_search / execute_tool meta-tools; BM25 + dense with RRF; 140 k → 1.3 k tool tokens (99 %) across 2 000+ tools / 200+ servers | meta-tools; hybrid retrieval |
| Task2MCP / T2MRec — 2026 | MCP server recommendation as retrieval + rank under engineering constraints | adapters.recommend_servers() over ServerCards with constraint filters |
| SchemaRouter, CacheRouter, Nexus, ToolDNS, Semantic Tool Discovery — 2026 | Field-aware matching, cache-preserving tool subsets, retrieval-decoupled routing, DNS-based discovery | discovery.schema_match() / SchemaAwareStrategy; discovery.CachePreservingSelector; DNS discovery not adopted |
| Best Prefix Selection — Aug 2026 | Skill selection as submodular benefit minus context penalty under a token budget; $(1-1/e,,1)$ bicriteria greedy; 0.73 vs 0.20–0.52 success, −28 % tokens | RoutePlan.skills set selection |
| SkillFeed — Aug 2026 | Profile-conditioned skill retrieval with a counterfactual benchmark; +35 pt when the profile matters | learning.SkillAffinity → select_skill_set(relevance=); profile-swap evaluation |
| Compositional skill routing, CaSKG, systems foundation for agentic skills — 2026 | Skill graphs and composition | discovery.SkillGraph (requires / conflicts / composes + learned co-usage; compose()) |
2.13 Non-stationarity, budgets and latency#
| Work | Core idea | What we take |
|---|---|---|
| Drift-Aware Sparse Routing — Sep 2026 | Non-stationary sparse contextual routing with multiple knapsack constraints; shadow-audit stream for pruned arms; pessimistic reward / optimistic cost; online shadow prices; regret $\tilde O(T\sqrt{s/\rho W} + W V_T + \sqrt T)$ | multi-knapsack CostAwareBandit |
| OrcaRouter — May 2026 | LinUCB over lexical + sentence embeddings; offline full-information reward matrix → ridge per arm → online bandit updates; #2 on RouterArena | offline warm start for LinUCBStrategy |
| Latency-aware routing — Jul 2026 | TTFT estimator that simulates batching; joint latency / accuracy / cost; +40 % utility at equal latency | queue-aware latency in HealthStrategy |
| HW-Router — DAC 2026; energy characterisation — IISWC 2026 | Hardware- and energy-aware routing | math.EnergyModel (per-target Wh regression, gCO2) and hardware_profile() priors; energy as a knapsack resource |
| Federate the Router — Jan 2026 | Federated MLP / K-means routers from decentralised sparse evaluations | mergeable learner state |
| D-UCB / SW-UCB (Garivier & Moulines 2011), delayed-feedback bandits (Joulani et al. 2013), bandits with knapsacks (Badanidiyuru et al. 2013), dueling bandits (Yue et al. 2012), multi-objective bandits (Drugan & Nowé 2013) | Classical machinery for drift, delay, budgets, preferences, Pareto | forgetting, delayed credit, knapsacks, pareto_front |
| Off-policy evaluation: IPS, SNIPS, doubly robust (Dudík et al. 2011) | Score a new policy on logged decisions | eval OPE; propensity logging |
2.14 Economics and mechanism design#
| Work | Core idea | What we take |
|---|---|---|
| EA-RAM — Aug 2026 | Reverse auction: providers bid self-predicted success probability + cost; Bayesian incentive compatible under dual error; explicit welfare-loss bound | strategies.AuctionStrategy: bias-corrected claims, second-price payment |
| Strategy auctions for small agents — ICML 2026; coalition formation and communication pricing in skill-based agentic AI — 2026 | Market mechanisms among agents | research track |
| Dominant-resource fairness (Ghodsi et al. 2011) | Fair multi-resource allocation | tenant fairness under shared budgets |
3. Systems landscape (open source, September 2026)#
| System | Targets | Learning | Explainable trace | Hard policy | Deps |
|---|---|---|---|---|---|
| RouteLLM (lmsys) | 2 LLMs | offline (preference) | no | no | torch |
| vLLM Semantic Router | LLMs (MoM) | classifier (offline) | partial | yes (PII, jailbreak, location) | Go/Rust/Python, models |
| Aurelio semantic-router | routes (intents) | embeddings | no | no | embeddings |
| NVIDIA llm-router | LLMs | classifier | no | no | Triton |
| LiteLLM / Portkey / aisix / nyro | LLMs (gateway) | none (rules, fallback) | no | rate/budget | service |
| NadirClaw / Misceo / ClawRouter / nexus-llm-router | LLMs (cheap↔premium) | heuristics | no | budget | Python/TS |
| LLMRouter (ulab-uiuc) | LLMs | 16+ trainable routers, xRouteBench | no | no | torch |
| SCX Router (checkpoint) | LLMs | 0.6 B zero-shot classifier + task ontology | task labels | per-task policies | torch |
| OrcaRouter | LLMs | LinUCB, offline warm start | no | no | Python, embeddings |
| SCOUT (MCP gateway) | tools | hybrid retrieval | no | gateway ACLs | service |
| OpenSmartRoute | LLM · agent · skill · persona · tool · workflow · human | online (bandit, IRT, BT, LinUCB, MDP) + optional judge | yes, per strategy | yes, before scoring | 0 in core |
Gateways and OpenSmartRoute are complementary: the gateway translates protocols and executes; the router decides.
4. Gaps this project addresses#
- Heterogeneous targets. All surveyed routers choose among LLMs. Real systems choose among a model,
an agent with tools, a deterministic skill, a persona layered on a model, or a human. Treating these
uniformly (one
RouteTargetcontract, one policy layer, one learning loop) is the central design decision. - Composition. MasRouter shows that mode + role + model beats picking a model.
RoutePlandoes this with declarative slots and no training. - Inductive by default. GraphRouter/UniRoute/IRT show it is possible; most shipped routers still retrain on pool change. Every strategy here works from declared capabilities on day one and learns from day two.
- Hard constraints before soft scores. Cost/quality routers routinely trade away privacy or region. Policy runs first and cannot be outweighed.
- Control-plane integrity. Almost no router considers adversarial inputs; Shafran et al. show they must.
- Explainability. Regulated deployments need to say why a request went where it did. Every decision carries per-strategy scores and rationales.
- Honest evaluation. Benchmarks in 2026 show most routers do not beat a static task table and that a third of the "oracle gap" is label noise. The evaluation harness must make these baselines and noise floors unavoidable, or learned strategies will be adopted on illusory gains.
- Task-level supervision. Agentic workloads reward the task, not the call. The data model must
carry
task_idand trajectory position so learners can be credited correctly. - Discovery at scale. Flat similarity over thousands of tools and skills collapses; two-stage retrieve-then-rank and set-valued skill selection under a token budget are required.
- Risk, not just cost. Set-valued decisions with abstention and conformal thresholds give operators a guarantee to sign off on; a tuned scalar threshold does not.
5. Idea → implementation map#
| Idea | Source | Module |
|---|---|---|
| Difficulty-based routing with quality floor | Hybrid LLM | signals.ComplexitySignal, Objective.quality_floor |
| Domain/action preferences, pinned rules | Arch-Router | signals.DomainActionSignal, strategies.RulesStrategy |
| Example-based target representation | UniRoute, GraphRouter | RouteTarget.examples, strategies.SimilarityStrategy |
| Ability × difficulty model | IRT-Router | math.irt, learning.IRTStrategy |
| Per-domain pairwise preference | RouteLLM, P2L | math.preference, learning.PreferenceStrategy |
| Contextual bandit on signal vector | MixLLM, LinUCB | math.bandits.LinUCB, learning.LinUCBStrategy |
| Budget-constrained exploration | C2MAB-V | math.bandits.CostAwareBandit |
| Cheap-first cascade | FrugalGPT | strategies.Cascade |
| LLM-as-router conditioned on descriptors, cost-aware | Router-R1 | strategies.LLMJudgeStrategy + escalate_llm_judge_below |
| Mode/role/model composition | MasRouter | Router._build_plan, RoutePlan |
| Conversation lookahead | dialogue MDPs | math.markov, learning.MarkovStrategy |
| Quality–cost frontier evaluation | RouterBench | eval |
| Confounder-gadget defence | Rerouting LLM Routers | security.InputGuard |
| Capacity sizing for human pools | queueing theory | math.decision.erlang_c, servers_for_sla |
| Drift detection on target quality | Page–Hinkley, ADWIN | math.estimators, learning.AutoLearner |
| Static task-type table as default learner and baseline | "Routing gap is task type" | strategies.TaskTableStrategy (v0.4) |
| Multi-sample oracle, run-to-run flip rate | "How much of the routing gap is real?" | eval --oracle-samples, --repeats (v0.4) |
| Diversity / coreset and paraphrase robustness | "When is routing meaningful?" | eval reports (v0.4) |
| Ontology-based task / difficulty / reasoning-mode / length prediction | SCX Router, TRouter | signals learned classifiers, osr train --synth (v0.4) |
| Calibrated confidence | temperature / isotonic scaling | router confidence (v0.4) |
| Offline reward matrix → LinUCB warm start | OrcaRouter | learning.LinUCBStrategy (v0.5) |
| Effort as a target dimension | Think When Needed | RouteTarget.effort (v0.5) |
| Profile-conditioned routing | GMTRouter, SkillFeed | RouteRequest.profile (v0.5) |
| Task-level admission routing with delayed terminal reward | TRACE-Router | Outcome.task_id, delayed-feedback bandits (v0.6) |
| Cascade as MDP with a stop action | RLCascadeRouter, unified cascade routing | strategies.Cascade on math.markov.RoutingMDP (v0.6) |
| Progress-guided step routing | ProgRouter | v0.6 |
Retrieve-then-rank narrowing, tool_search meta-tool | Enrich-Retrieve-Rank, SCOUT | adapters.mcp, two-stage selection (v0.7) |
| Submodular skill-set selection | Best Prefix Selection | Router._build_plan (v0.7) |
| Conformal thresholds, set-valued routing with abstention | RouteNLP, RACER | RouteDecision.candidate_set (v0.8) |
| Learning to defer | Madras; Mozannar & Sontag | HUMAN deferral objective (v0.8) |
| Multi-knapsack budgets with shadow audit | Drift-Aware Sparse Routing | math.bandits.CostAwareBandit (v0.8) |
| Forgetting in learners | D-UCB / SW-UCB | math.irt, math.preference, math.bandits (v0.8) |
| Queue-aware latency (TTFT) | Latency-aware routing | realtime.HealthStrategy + math.decision (v0.8) |
| Off-policy evaluation, shadow mode | IPS / DR | eval, EnterpriseRouter(shadow=) (v0.8) |
| Origin policy on tool parameters | ROPE | security (v0.9) |
| Contrastive (query, target) training over acceptable sets | RouterDC | learning.ContrastiveRouter, ContrastiveStrategy, acceptable_set |
| Reward-model-distilled soft labels | Zooter | learning.soft_labels, ContrastiveRouter.fit(objective="distilled") |
| A small routing model distilled from the full ensemble, calibrated, learning online | RouteLLM (matrix-factorisation router), Zooter, RouterDC | learning.RouterSLM, distill_router, SLMStrategy |
| Self-improvement from live outcomes and public routing datasets without regressions | RouterBench, RouterEval; champion/challenger evaluation | learning.SelfImprover, eval.DatasetCollector, eval.rows_from_feedback |
| Live cost/quality priors and discovery of new models | LLMRouterBench cost-quality frontier | adapters.ModelCatalogue, adapters.WebKnowledge |
| Decision loss vs prediction loss; end-to-end policy gradient | Router-R1, RLCascadeRouter | learning.PolicyGradientStrategy, decision_reward, decision_regret |
| Oracle gap decomposition, target diversity, minimal catalogue, scaling curve | "How much of the routing gap is real?", "When is routing meaningful?", RouterEval | eval.headroom (routing_headroom, target_diversity, min_catalogue, scaling_curve) |
| Learnability on the hardest tasks | SWE-Router, Agent-as-a-Router | eval.headroom.learnability_by_difficulty |
| Memory-tier routing under a token budget | BudgetMem, Gated-Memory Routing | strategies.MemoryRouter, MemoryTier, ImportanceGate |
| Modality-aware scoring and text-first escalation | LatentRouter, modality escalation, CUA routing | strategies.ModalityStrategy, ModalityEscalation, request_modalities |
| Semantic cache as the cheapest target | GPTCache, vLLM semantic router | strategies.SemanticCache, SemanticCacheStrategy |
| Annotator skill estimation and quorum selection | QUORUM; Dawid & Skene 1979 | strategies.AnnotatorPool, quorum_accuracy, HumanRoutingStrategy |
| Speculative draft/strong cascades with cancellation | speculative cascades; Differential Reasoning Router | strategies.SpeculativeCascade, expected_mode_costs |
6. Open research questions#
Every item of the roadmap research track now has a module (see the research-track table in ROADMAP.md); what remains open is measurement, not implementation:
- How much routing headroom is real on a given catalogue once label noise and task-type effects are
removed —
eval.headroom.routing_headroomreports it; the open question is what the numbers say on your pool, and therefore when a learned router is worth its complexity over a static table. - How few, how different targets must be before routing beats the best single model
(
target_diversity,min_catalogue), and whether accuracy keeps scaling with catalogue size in heterogeneous pools (scaling_curve). - Prediction vs decision:
decision_regretputs end-to-end RL (PolicyGradientStrategy) next to predict-then-optimise ensembles on the same rows; when does decision loss matter in practice? - Where to escalate: before generation (router), during (
SelfEscalation), after (Cascade,MixtureOfAgents) or speculatively in parallel (SpeculativeCascade) — cost/quality frontiers of the four regimes on the same pool. - Contrastive vs pointwise training when several targets are acceptable (
ContrastiveRoutervsTaskTableStrategy/LinUCBStrategy), and how much reward-distilled soft labels help over one-hot. - Routing for the hardest agentic work (web, SWE):
learnability_by_difficultyon real agent traces. - Strategic providers: how
AuctionStrategy's bias correction behaves against adaptive bidders; coalition pricing in multi-vendor catalogues. - Human pools: Dawid-Skene assumes conditionally independent annotators; correlated errors inside
a quorum (
AnnotatorPool.select_quorum) are not yet modelled. - Fairness under shared budgets; energy priors from
hardware_profile()against metered fleets.
7. References#
Entries without author lists are 2026 preprints cited by arXiv identifier; the italic text is a descriptive title and the identifier is authoritative.
Surveys
- Varangot-Reille C., Bouvard C., Gourru A., Ciancone M., Schaeffer M., Jacquenet F. Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in LLM-Based Systems. JAIR 2025. arXiv:2502.00409
Supervised & preference routers
- Šakota M., Peyrard M., West R. Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling (FORC). WSDM 2024. arXiv:2308.06077
- Ding D. et al. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. ICLR 2024. arXiv:2404.14618
- Ong I. et al. RouteLLM: Learning to Route LLMs with Preference Data. 2024. arXiv:2406.18665
- Lu K. et al. Routing to the Expert: Efficient Reward-guided Ensemble of LLMs (Zooter). 2023. arXiv:2311.08692
- Chen S., Jiang W., Lin B., Kwok J.T., Zhang Y. RouterDC: Query-Based Router by Dual Contrastive Learning. NeurIPS 2024. arXiv:2409.19886
- Tran C., Paracha S., Hafeez A., Chen S. Arch-Router: Aligning LLM Routing with Human Preferences. 2025. arXiv:2506.16655
- Frick E. et al. Prompt-to-Leaderboard. 2025. arXiv:2502.14855
- Stepanov I. et al. SCX Router: Streaming Zero-Shot Model Selection with a Decoder-KV Classifier and a Real-World Task Ontology. 2026. arXiv:2609.02292
- TRouter: Task-Taxonomy-Guided Synthetic Data for Cold-Start LLM Routing. ACL 2026. arXiv:2604.09377
- VDAR-Router: Verbalized Difficulty Retrieval for LLM Routing. 2026. arXiv:2607.18098
- Scalable Prompt Routing via Latent Task Discovery. 2026. arXiv:2603.19415
- Guo H. et al. Think When Needed: Model-Aware Reasoning Routing for LLM-based Ranking. SIGIR 2026. arXiv:2601.18146
- When to Think Deeply. 2026. arXiv:2606.06745
Inductive / representation
- Feng T., Shen Y., You J. GraphRouter: A Graph-based Router for LLM Selections. ICLR 2025. arXiv:2410.03834
- Jitkrittum W. et al. Universal Model Routing for Efficient LLM Inference (UniRoute). 2025. arXiv:2502.08773
- Zhuang R. et al. EmbedLLM: Learning Compact Representations of Large Language Models. 2024. arXiv:2410.02223
- Song W. et al. IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory. ACL 2025. arXiv:2506.01048
- Sun Y., Xie E., Feng T., You J. GMTRouter: Personalized LLM Router over Multi-turn User Interactions. EMNLP 2026 Findings. arXiv:2511.08590
- ProbeDirichlet / RouterXBench. 2026. arXiv:2602.11877
Online / bandits
- Wang X. et al. MixLLM: Dynamic Routing in Mixed Large Language Models. NAACL 2025. arXiv:2502.18482
- Dai X., Li J., Liu X., Yu A., Lui J.C.S. Cost-Effective Online Multi-LLM Selection with Versatile Reward Models (C2MAB-V). 2024. arXiv:2405.16587
- Li L., Chu W., Langford J., Schapire R.E. A Contextual-Bandit Approach to Personalized News Article Recommendation. WWW 2010
- Auer P., Cesa-Bianchi N., Fischer P. Finite-time Analysis of the Multiarmed Bandit Problem. Machine Learning 47, 2002
- Thompson W.R. On the Likelihood that One Unknown Probability Exceeds Another. Biometrika 1933
- Bao Z. et al. OrcaRouter: A Production-Oriented LLM Router with Hybrid Offline-Online Learning. 2026. arXiv:2605.30736
- Drift-Aware Sparse Routing: Nonstationary Contextual Routing under Multiple Knapsack Constraints. 2026. arXiv:2609.00662
- Raj R. et al. TRACE-Router: Task-Consistent and Adaptive Online Routing for Agentic AI. 2026. arXiv:2607.22465
- Federate the Router. 2026. arXiv:2601.22318
- Patel S., Parthasarathy A.R., Mallick A., Joshi G. Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads. 2026. arXiv:2607.18253
- HW-Router. DAC 2026. arXiv:2608.14575
- Garivier A., Moulines E. On Upper-Confidence Bound Policies for Switching Bandit Problems. ALT 2011
- Joulani P., György A., Szepesvári C. Online Learning under Delayed Feedback. ICML 2013
- Badanidiyuru A., Kleinberg R., Slivkins A. Bandits with Knapsacks. FOCS 2013
- Yue Y., Broder J., Kleinberg R., Joachims T. The K-armed Dueling Bandits Problem. JCSS 2012
- Drugan M.M., Nowé A. Designing Multi-Objective Multi-Armed Bandits Algorithms. IJCNN 2013
- Dudík M., Langford J., Li L. Doubly Robust Policy Evaluation and Learning. ICML 2011
Cascades & multi-round
- Chen L., Zaharia M., Zou J. FrugalGPT. 2023. arXiv:2305.05176
- Aggarwal P. et al. AutoMix: Automatically Mixing Language Models. NeurIPS 2024. arXiv:2310.12963
- Zhang H., Feng T., You J. Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via RL. NeurIPS 2025. arXiv:2506.09033
- Jiang D., Ren X., Lin B.Y. LLM-Blender. ACL 2023. arXiv:2306.02561
- Dekoninck J., Baader M., Vechev M. A Unified Approach to Routing and Cascading for LLMs. 2024. arXiv:2410.10347
- Huang S., Wang S., Ma H., Xu Z. RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning. 2026. arXiv:2608.15817
- Shaikh N. Knowing When to Ask for Help: Bayesian Self-Escalation in Hierarchical LLM Agents. 2026. arXiv:2608.24087
- Lei J., Huang J. TACIT-Switch: Cost-Aware Model Escalation for LLM Agents from Censored Supervision. 2026. arXiv:2608.27911
- Li S., Abdelmoniem A.M., Wang S. ProgRouter: Online Progress-Guided Orchestration for Multi-Agent LLM Workflows. EMNLP 2026 Findings. arXiv:2608.25992
- MTRouter: Multi-Turn LLM Routing. ACL 2026. arXiv:2604.23530
- Tang S. et al. Beyond Gemini-3-Pro: Revisiting LLM Routing and Aggregation at Scale (JiSi). 2026. arXiv:2601.01330
- Confidence-Guided Stepwise Model Routing. 2025. arXiv:2511.06190
- Differential Reasoning Router. 2026. arXiv:2608.30224
Risk control, calibration, deferral
- RACER: Risk-Controlled LLM Routing with Abstention. 2026. arXiv:2603.06616
- RouteNLP: Conformal Cascade Thresholds and Distillation–Routing Co-optimisation. ACL 2026 Industry. arXiv:2604.23577
- CR²: Cost-Aware Risk-Controlled Routing. 2026. arXiv:2605.12001
- Guo C., Pleiss G., Sun Y., Weinberger K.Q. On Calibration of Modern Neural Networks. ICML 2017
- Kuhn L., Gal Y., Farquhar S. Semantic Uncertainty. ICLR 2023 · Kadavath S. et al. Language Models (Mostly) Know What They Know. 2022
- Madras D., Pitassi T., Zemel R. Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer. NeurIPS 2018
- Mozannar H., Sontag D. Consistent Estimators for Learning to Defer to an Expert. ICML 2020
- Verma R., Barrejón D., Nalisnick E. Learning to Defer to Multiple Experts. AISTATS 2023
Agents, tools, skills
- Yue Y. et al. MasRouter: Learning to Route LLMs for Multi-Agent Systems. 2025. arXiv:2502.11133
- Qin Y. et al. ToolLLM: Facilitating LLMs to Master 16000+ Real-world APIs. 2023. arXiv:2307.16789
- Du Y., Wei F., Zhang H. AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls. 2024. arXiv:2402.04253
- Enrich-Retrieve-Rank: Discovery for Models, Agents, Tools and Skills at Scale. 2026. arXiv:2608.22695
- SCOUT: Scalable Tool Discovery for MCP Gateways. PayPal, 2026. arXiv:2608.23992
- Task2MCP / T2MRec: MCP Server Recommendation. 2026. arXiv:2604.17234
- Semantic Tool Discovery for MCP. 2026. arXiv:2603.20313 · SchemaRouter. arXiv:2608.21375 · CacheRouter. arXiv:2608.22708 · Nexus. arXiv:2608.20397 · ToolDNS. arXiv:2607.18242 · ComplexMCP. arXiv:2605.10787
- Best Prefix Selection: Optimal Skill Selection under Token Budgets. 2026. arXiv:2608.19993
- SkillFeed: Personalized Skill Routing. 2026. arXiv:2608.28241
- Compositional Skill Routing. arXiv:2606.18051 · CaSKG. arXiv:2608.25500 · A Systems Foundation for Agentic Skills. arXiv:2608.29596 · Microskill Architecture. arXiv:2606.05720
- Agent-as-a-Router. arXiv:2606.22902 · SWE-Router. arXiv:2607.00053 · Agentic Routing: a harness-native data flywheel. arXiv:2607.11399 · Routing Is Least Learnable Where It Is Most Valuable. arXiv:2608.06171 · LLMs Predict Failure Risk but Not Which Collaboration Protocol Pays Off. arXiv:2608.14927
- BudgetMem. ICML 2026. arXiv:2602.06025 · Gated-Memory Routing. arXiv:2609.00237
- LatentRouter. arXiv:2605.11301 · VLM Routing for Computer-Use Agents. arXiv:2603.12823 · Modality Escalation. arXiv:2607.05438
Economics & mechanism design
- Chen H. et al. Error-Aware Reverse Auction Mechanism for Large Language Model Routing (EA-RAM). 2026. arXiv:2608.12719
- Strategy Auctions for Small Agents. ICML 2026. arXiv:2602.02751
- Coalition Formation and Communication Pricing in Skill-Based Agentic AI. 2026. arXiv:2608.07532
- Ghodsi A. et al. Dominant Resource Fairness. NSDI 2011
Sequential decisions & queueing
- Levin E., Pieraccini R., Eckert W. A Stochastic Model of Human-Machine Interaction for Learning Dialog Strategies. IEEE TSAP 2000
- Young S., Gašić M., Thomson B., Williams J.D. POMDP-Based Statistical Spoken Dialog Systems: A Review. Proc. IEEE 2013
- Koole G., Mandelbaum A. Queueing Models of Call Centers: An Introduction. Annals of OR 2002
- Kingman J.F.C. The Single Server Queue in Heavy Traffic. 1961 · Little J.D.C. A Proof for the Queuing Formula L = λW. 1961
Security
- Shafran A., Schuster R., Ristenpart T., Shmatikov V. Rerouting LLM Routers. 2025. arXiv:2501.01818
- ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection. 2026. arXiv:2608.27496
- Verifiable Manifest Signing for MCP Pipelines. 2026. arXiv:2601.23132
- Recognition Without Enforcement. 2026. arXiv:2608.28502
- Beyond Max Tokens: Resource Amplification via Tool Chains. 2026. arXiv:2601.10955
- When Safety Routing Breaks. 2026. arXiv:2609.01455
- Malicious Prompt Detection via Bootstrap Aggregation. 2026. arXiv:2602.08062
Benchmarks
- Hu Q.J. et al. RouterBench: A Benchmark for Multi-LLM Routing System. 2024. arXiv:2403.12031
- Huang Z. et al. RouterEval: A Comprehensive Benchmark for Routing LLMs. 2025. arXiv:2503.10657
- LLMRouterBench: A Unified Benchmark for LLM Routing. 2026. arXiv:2601.07206
- Feng T., You J. et al. LLMRouter / xRouteBench: Routing as Sequential Decision Making. 2026. arXiv:2608.06867
- RouterXBench. 2026. arXiv:2602.11877
- How Much of the Routing Gap Is Real? Label Noise and the Oracle. 2026. arXiv:2607.03436
- Most of the LLM Routing Gap Is Task Type. 2026. arXiv:2608.23023
- When is Routing Meaningful? Diversity and Perturbation Robustness. 2026. arXiv:2607.09197
- Characterising Energy per Request and per Token for LLM Serving. IISWC 2026. arXiv:2608.28044
Drift
- Page E.S. Continuous Inspection Schemes. Biometrika 1954 · Hinkley D.V. 1971
- Bifet A., Gavaldà R. Learning from Time-Changing Data with Adaptive Windowing (ADWIN). SDM 2007
Systems
- vLLM Project. vLLM Semantic Router. github.com/vllm-project/semantic-router · technical report arXiv:2603.04444
- LMSYS. RouteLLM. github.com/lm-sys/RouteLLM
- ulab-uiuc. LLMRouter. github.com/ulab-uiuc/LLMRouter · GMTRouter. github.com/ulab-uiuc/GMTRouter
- Generating Workflow DAGs from Natural-Language Routing Rules. 2026. arXiv:2608.30250