Imported from yuyuyuyu52/litellm-public-relay (
litellm-rust/crates/ai-gateway/AGENTS.md). Install upstream withnpx skills add yuyuyuyu52/litellm-public-relay --skill ai-gateway. Copyright stays with the author.
ai-gateway — folder architecture
The Axum server that fronts the Rust gateway. It owns transport + config + auth
only; deployment selection lives in core::router, transforms in core/providers.
src/
main.rs # entrypoint: build AppState (router + master key), bind, serve
state.rs # AppState — shared Arc<Router> + master_key
gil.rs # GIL-activity tracker (records Python acquisitions)
auth/ # authentication as an axum extractor — added to handler args
mod.rs # RequireMasterKey: FromRequestParts, single master key (LITELLM_MASTER_KEY)
routes/ # one module per route, all matching the same template
AGENTS.md # ← the route template (read this before adding a route)
mod.rs # app(): merges every module's router()
health.rs # simple route (one file): router() + liveness/readiness
gil.rs # simple route (one file): router() + GET /health/gil
realtime/ # route with logic → axum surface + a no-axum service:
mod.rs # router() + handler + WS<->events adapter (the axum surface)
service.rs # business logic (select deployment, call provider) — no axum, testable
python/ # Python interop (feature: python-config) — load-time only
mod.rs, config.rs, AGENTS.md
Rules
- Routes follow one template. Each route module exposes
pub fn router() -> Router<AppState>;routes/mod.rsonly merges them. Simple routes are one file; non-trivial routes are a folder (handler/service/transport). Seeroutes/AGENTS.md. - Auth is an extractor. Add
crate::auth::RequireMasterKeyto a handler's args; it runs during extraction. Never re-implement the check per route. - Handlers are thin. A handler validates and delegates to its
service. No business logic, no provider calls, no transforms in handlers. - State is shared and cheap to clone. Long-lived handles live behind
Arcinstate.rs; read env/config only inmain.rswhen building state.
Auth (interim)
A single master key (LITELLM_MASTER_KEY), enforced by the
auth::RequireMasterKey extractor: any caller presenting it as
Authorization: Bearer <key> may invoke the gateway. Fails closed (500) when
unset; constant-time compare. The server binds 127.0.0.1 by default (HOST to
override). Full per-key auth + budgets/rate-limits are delegated to the Python
proxy in a later phase. Health routes don't add the extractor (unauthenticated).
Python interop
Anything that calls into Python lives in python/ and is load-time only — see
python/AGENTS.md. The realtime data path never takes the GIL.
