Open-source LLM proxy
OpenSmartRoute vs LiteLLM
LiteLLM is an excellent execution layer: one SDK and proxy for 100+ providers with keys, budgets and fallbacks. OpenSmartRoute adds the decision layer on top: which capability, under which policy, explained, and learned from outcomes.
“LiteLLM executes; OpenSmartRoute decides.”
Choose LiteLLM when
- Provider coverage and community matter most
- You already run it and only need model fallbacks
Choose OpenSmartRoute when
- Routing must be explainable in a post-mortem
- Policy must reject a candidate before scoring, not after
- The pool includes agents, tools and humans
- You need learned routing that you own
Capability by capability
Our reading of public documentation; corrections welcome via GitHub issues.
| Capability | OpenSmartRoute | LiteLLM |
|---|---|---|
| Routes to LLMs | Yes | Yes |
| Routes to agents, skills, tools, workflows and humans | Yes | No |
| Hard policy before scoring (region, data boundary, PII, tenant, cost)Budgets and allow-lists per key; no data-boundary or PII gating | Yes | Partial |
| Per-request trace with ranked candidates and rejections | Yes | No |
| Learns from your outcomes (per target, per tenant) | Yes | No |
| Multi-step plans (persona -> skill -> model) | Yes | No |
| Self-hosted, air-gapped, zero runtime dependenciesSelf-hosted, but with a large dependency tree | Yes | Partial |
| Prompt-injection and steering-gadget defences on the router | Yes | No |
| Savings ledger and Routing Audit | Yes | No |
| OpenAI-compatible proxy | Yes | Yes |
| Provider marketplace with credits | No | No |
| Request/response observability UI for every callDecision traces and usage; pair with your APM for payload-level tracing | Partial | Partial |
Run OpenSmartRoute behind LiteLLM
Call /v1/route first, then hand the chosen model to LiteLLM for execution. Nothing in your provider setup changes.
togetherpython
decision = requests.post(f"{OSR}/v1/route", json={"text": prompt, "constraints": {"tenant": tenant}}).json()
resp = litellm.completion(model=decision["target"]["model"], messages=messages)
requests.post(f"{OSR}/v1/feedback", json={"request_id": decision["request_id"], "success": True, "cost_usd": resp._hidden_params["response_cost"]})Prove it on your traffic
Run the Routing Audit on a week of anonymised logs: savings versus your current mix, policy violations your routing allowed, and where agents or humans would have been the better answer.
Other comparisons: OpenRouter, Portkey, Building it yourself.