Imported from zwldarren/llm-proxy (
AGENTS.md). Install upstream withnpx skills add zwldarren/llm-proxy. Copyright stays with the author.
LLM Proxy - Agent Guidelines
OVERVIEW
LLM API proxy with multi-provider support (OpenAI, Anthropic, Gemini, Ollama, vLLM, SGLang, OpenRouter), protocol translation, streaming, rate limiting, MCP integration, and Vue 3 admin UI.
COMMANDS
Backend
uv sync # Install dependencies
uv run ruff check --fix # Lint (auto-fix)
uv run ruff format # Format code
uv run ty check # Type check (ty, not mypy/pyright)
uv run pytest # Run tests
uv run llm-proxy # Run production server
uv run llm-proxy-dev # Start backend + frontend dev servers concurrently
Verification order
uv run ruff check --fix && uv run ruff format && uv run ty check && uv run pytest
ARCHITECTURE
Request Flow
- Incoming request → Protocol endpoint (
protocols/) mounts the route - Parse to
InternalRequestvia the protocol serializer (protocols/<name>/serializer.py) - Route via
UnifiedProcessor(core/processing/unified.py) to adapter - Adapter calls provider API via the provider serializer (
serialization/) - Response →
InternalResponse→ Format for client via the protocol serializer
Serialization Pattern
Protocol serializers (client wire format ↔ internal models) and provider serializers (internal models ↔ provider API format) are two separate classes in two separate registries — one serializer never does both sides.
from llm_proxy.protocols.registry import get_protocol_serializer
from llm_proxy.serialization import get_provider_serializer
# Client side: wire format -> InternalRequest
protocol_serializer = get_protocol_serializer("openai")
unified_request = protocol_serializer.parse_request(raw_data)
# Upstream side: InternalRequest -> provider body
provider_serializer = get_provider_serializer("openai")
provider_body = provider_serializer.build_provider_request(unified_request)
MCP Integration
mcp/manager.py: MCP server lifecycle managementmcp/proxy.py: Request proxying to MCP servers- Configure MCP servers in provider config with
mcp_serverskey
DATABASE MIGRATIONS
Migrations are the single source of truth for schema. There is no create_all fallback — if migrations fail, startup fails loudly.
connection.py:init_db()callsalembic upgrade headon startup viaasyncio.to_thread()env.pycreates an async engine withNullPool, runs migrations, and explicitly commitsrender_as_batchis enabled only for SQLite; PostgreSQL uses native DDL- Generated migrations are auto-formatted by the ruff post-write hook in
alembic.ini
Commands
# After modifying models in database/tables.py:
uv run alembic revision --autogenerate -m "description_of_change"
TESTS
tests/conftest.pyprovidesMockResponseandcreate_mock_clientfor mockinghttpx2
DEV SERVER
uv run llm-proxy-dev (llm_proxy.__main__:main_wrapper_dev) starts:
- Backend uvicorn with
--reloadon port9911 - Frontend Vite dev server on
0.0.0.0:5173, proxying API to backend viaVITE_API_BASE_URL
Requires bun in PATH. node_modules is auto-installed on first start if missing.
STYLE & CONVENTIONS
- Python: 3.14+ (
pyproject.toml) - Comments: All code comments must be in English
- Ruff: line-length 100, excludes
**/versions/** - Type checker:
ty(configured inpyproject.tomlwithinclude = ["src"])
Exception Handling
-
Python 3.14 (PEP 758) allows omitting parentheses in multi-exception
exceptclauses when there is noasbinding. Ruff format will therefore collapseexcept (ValueError, ConfigurationError):intoexcept ValueError, ConfigurationError:— this is valid Python 3.14 and catches both exceptions.Correct without
as:except ValueError, ConfigurationError: passCorrect with
as(parentheses are still required):except (ValueError, ConfigurationError) as exc: ... # or, when the exception object is unused: except (ValueError, ConfigurationError) as _: passDo not write:
except ValueError, ConfigurationError as exc: # WRONG: binds only to ValueError ...
DESIGN CONTEXT
Strategic + visual context for the admin UI. Full detail in PRODUCT.md (strategy: register, users, purpose, brand, anti-references, principles, accessibility) and DESIGN.md (visual system: tokens, typography, elevation, components, do's/don'ts). The frontend's own frontend/AGENTS.md carries the tactical DESIGN CONTEXT for the Vue app; PRODUCT.md / DESIGN.md are the root source of truth and win on conflicts.
- Register:
product— design serves the product (admin console / dashboard / tool). - Primary users: operators who self-host the proxy and run the dashboard (configure providers/models/keys, monitor usage, cost, logs). API consumers are a non-design surface.
- Brand personality: Technical Sophistication, Reliable, Modern. Direct, precise, professional; confident, not arrogant.
- Aesthetic: Monochrome Editorial — dark-first (deep cool-neutral, hue 220°), light mode a crisp mirror. Space Grotesk display + Manrope body + IBM Plex Mono for all data. Low-saturation semantic tints (violet/blue/amber/rose + status) for differentiation only, never decoration.
- Design principles: Clarity Over Cleverness · Confidence Through Feedback · Technical Sophistication · Dark-First Design · Respectful of Time.
- Anti-references: NOT playful/consumer; NOT boring/outdated admin panels; NO loud color accents; NOT the generic 2026 AI-tool aesthetic (no cream/sand bodies, gradient-text, identical card grids, hero-metric template).
- Accessibility: WCAG AA both themes;
prefers-reduced-motionrespected; visible focus rings + skip link; color never the sole status signal; i18n (English primary, Chinese secondary). - Frontend stack: Vue 3 + Vite (rolldown) + TypeScript + Tailwind v4 + shadcn-vue (new-york, neutral) + reka-ui + Pinia + vue-router + vue-i18n + vue-sonner. Tokens in
frontend/src/assets/main.css; primitives infrontend/src/components/ui/.
Agent skills
Issue tracker
Issues live as GitHub issues in this repo; use the gh CLI. See docs/agents/issue-tracker.md.
Triage labels
Five canonical labels (needs-triage, needs-info, ready-for-agent, ready-for-human, wontfix). See docs/agents/triage-labels.md.
Domain docs
Single-context: CONTEXT.md + docs/adr/ at the repo root. See docs/agents/domain.md.
