litellm

Imported from BerriAI/litellm (AGENTS.md). Install upstream with npx skills add BerriAI/litellm. Copyright stays with the author.

Do not write comments unless they are any of:

• absolutely necessary to explain some very complex business logic (in which case, keep it concise and clear)
• used as an input for tools to read and act on. For example:
  • entries in .git-blame-ignore-revs saying which commit is excluded from git blame
  • a lint or type checker suppression like # mutable-ok or # pyright: ignore[reportArgumentType] # <reason> when introducing a truly unavoidable violation
• a TODO or FIXME
  • Not great to have those, but if it's unavoidable, make sure to include a strong, concise reason for why it's there or, better yet, link to a GitHub issue for the follow-up work

Explanation: The point of this rule is to keep out AI slop comments. AI writes way too many and way too verbose comments. Code comments are, in a way, a violation of DRY code. You must update logic in two locations to change the code, and "hard to change" is literally the definition of tech debt. We should instead aim to write code that is intuitive and clear, even at a glance, to the reader, being both easy to maintain and high performance

Don't assume that the existing code is correct or the right way of doing things / good coding patterns. In fact, there are a lot of bad coding practices, overly complex code, code smells, etc. If something doesn't look right, speak up. Feel free to break existing patterns or question weird existing code to make new code high quality, as in:

• correct
• secure
• performant
• readable
• easy to maintain/change
• modern

In descending order of importance

When adding new features, add meaningful tests. Don't add tests that don't check anything substantial and is there just to make the code coverage pass. Yes, code coverage is important, but I'd rather have no signal whether the code is working than tests that don't fail when code is broken. The goal is to have tests that would fail before the feature was added/if the code was mutated in a way that breaks the feature and succeed only when the feature is fully working. I should run mutation testing and see > 90% kill rate

Same thing for bug fixes. The tests should make it so that this specific bug can never happen again without failing tests (i.e., regression)

Never test structure of code only function of it

A test must only fail when litellm code changes. Never pin facts we don't own (a vendor's price, a third party's field, an upstream default, today's date) as literals or as "X must be absent"; assert the invariant our code guarantees instead, e.g. two rows agree, a value is within range, a field is derived from another. If an outside fact is truly load-bearing, cite its source and date next to the assertion so a reader can tell stale from broken

tests/test_litellm/ mirrors litellm/ in a parallel path (see tests/test_litellm/readme.md). Name tests test_<filename>.py, but always match the existing test file in the directory you touch — many provider dirs use longer descriptive names (e.g. test_anthropic_chat_transformation.py) to avoid ambiguity across sibling folders. For bug fixes, extend the existing mapped test file rather than creating a new one. Only create a new test file for a new feature (provider, endpoint, or transformation module) that has no mapped test yet, following that directory's naming convention (or test_<filename>.py if you're the first test there). One focused regression test beats many shallow ones

End-to-end tests belong in tests/e2e/ and must follow the harness conventions documented in that directory's AGENTS.md

When creating PRs, target the repository's current default branch for both internal and external / OSS contributions. Check it with python3 scripts/default_branch.py --branch instead of assuming a branch name or relying on cached origin/HEAD

When writing a PR body, treat the comments and imperative instructions inside .github/pull_request_template.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/pull_request_template.md from disk before writing a PR body to make sure you see every comment rule. A section you have nothing to put in (Relevant issues, Affected release, Linear ticket, Caveats, QA runbook, and so on) is removed entirely, heading included, never left as an empty title

Same applies for filing bug reports and feature requests, with .github/ISSUE_TEMPLATE/bug_report.yml and .github/ISSUE_TEMPLATE/feature_request.yml, respectively

If you're resolving a linear ticket, in the "## Linear ticket" section of the PR, say "Resolves LIT-1234", replacing "LIT-1234" with the actual ticket id that you're resolving. If you don't have the ticket id, don't make one up or search for it. Just drop the section

Never use pytest commands or the like as "Screenshots / Proof of Fix". We prefer curl'ing a live proxy instance running on localhost:4000 (I like to run it with python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log; the Admin UI dev server is npm run dev in ui/litellm-dashboard, served on port 3000) and showing both the command run and the output. Also, it should hit real LLM provider APIs, not mocks, and cost real $$$ because that is the most realistic test. The proof of fix should be exactly what the end user / customer would see / do. The run logs in PR #27703 is a prime example of how to do it (not a huge fan of using a python test script that future me and the team will have no visibility into; I prefer just curl commands or a short list of bash commands (e.g., using for)). If it's a UI thing, or the main use case runs through a headful agentic coding tool like Claude Code or Codex, drive that surface yourself and embed your own before and after screenshots of it in the PR (the Admin UI page, or what the coding tool shows), next to an ordered list of the URLs to go to (e.g., http://localhost:4000/ui/?page=logs), where to click, and what fields to fill out so a reviewer can reproduce it

If you ever write any human-facing text (pull requests, issues, commit messages, discussion posts, github comments, release notes, docs, etc.), always follow these guidelines to sound less AI-y:

• don't use emojis
• don't use "—". Instead, reach for ",", ".", conjunction words, ":", ";", etc. in descending order of preference: vary among them, weighted toward the front of the list, and skip "," where it would cause a comma splice or the sentence is getting long. Overusing any one of them, ";" especially, also feels AI-y. A word cap does not penalize you for adding more sentences: when writing under tight word budgets, prefer a period split or a conjunction over ";", and keep to at most one ";" per message
• don't use the pattern "It's not X, it's Y", "You're not X, you're Y", etc.
• unless explicitly asked, don't use bulleted or numbered lists unless it would be nonsensical not to. Instead, prefer prose
• don't add a trailing "." at the end of paragraphs (just like this file). That means every paragraph, not just the last one (of the markdown file, PR description, GitHub comment, etc.). Rule of thumb: if you're adding new line(s) before the next sentence, don't add a "."
• don't use →. Instead, prefer not to use arrows, and if need be, use -> instead
• use plain, simple, everyday engineering language: the common phrase engineers actually say over rare compact phrasing, in grammatically complete sentences. When explicitly asked to use bullets or ordered lists and structure legitimately helps the reader, prefer nested bullets (any depth is fine) over dense lines in a flat structure

Don't hesitate to use values in .env to get needed API keys and other secrets, as long as you never add them to conversation history, commit them, or include them in GitHub issues / PRs

Python max line length is 120, not 88

Never edit or commit ruff-strict-budget.json, type-discipline-budget.json, basedpyright-code-budget.json, or test-quality-budget.json on a PR branch, and don't run make lint-budget-update there. A scheduled Devin automation lowers the limits on the default branch in its own PR by exactly what landed since the last ratchet, so concurrent PRs don't fight over the same "limit" lines. Keep the hosted automation's target in sync when the repository default changes. If your branch already carries a budget edit, drop it before opening the PR

make check (f.k.a. make pre-commit, which still works identically as an alias) saves its complete output to a log file in .git (overwriting previous logs) and prints that path as its first and last output lines. To inspect a run, read or grep that log instead of re-running the multi-minute checks just to see a different slice

make check, make lint, scripts/pre_commit_lint.sh, and the standalone budget gates (scripts/ruff_strict_gate.py, scripts/type_discipline_gate.py, scripts/type_check_gate.py) each hold one of 2 machine-wide slots, so when other sessions or worktrees on the same box are already running heavy work, yours prints "all N machine-wide slots are busy; queueing" and then stays quiet until a slot frees. Give the command a long timeout and let it wait rather than killing it, retrying it, or assuming it hung. Don't change the # of machine-wide slots or make it unlimited by setting LITELLM_GATE_SLOTS=0

If you're trying to create a new function that relies on untyped stuff, instead of adding more Any's and pushing reportAny / reportExplicitAny closer to their basedpyright ceilings, just validate it in the caller with Pydantic (a model or TypeAdapter that returns the typed thing or raises will do) and then pass the now typed variable in

If you get an LIT001 or LIT002 fail, refactor the code to follow functional programming best practices rather than introducing mutable data structures. For example, build values in one shot with comprehensions or generators wrapped in tuple() / MappingProxyType() / frozenset() instead of seeding an empty list/dict/set and mutating it over time. Ideally, # mutable-ok is never used; reach for it only as a genuine last resort when an immutable rewrite is truly impossible, and always pair it with a real reason

Every lint or type suppression must name the exact rule inside brackets and carry a reason comment, e.g. # pyright: ignore[reportArgumentType] # stubs lack async overload or # noqa: TID251 # <reason>. # type: ignore is banned (LIT009): pyrightconfig.json sets enableTypeIgnoreComments to false, so it silently does nothing

Commit and push your work when you're done without asking

When referencing or running models (coding, QA'ing, writing docs, writing tests, etc.), use the latest model in that model family unless otherwise specified; treat your training knowledge, memories, configs, and tests as stale, and determine the family's latest with model_prices_and_context_window.json or the web

Always pull before starting any work. The checkout or worktree may be sitting on a stale branch

If you're an internal contributor, when creating a new PR, the typical flow is to branch off the repository's current default branch and create a branch prefixed with litellm_. Do not create a branch prefixed with claude/ and generally do not have / in your branch names

Do not add Co-Authored-By: Claude or any Claude attribution to commit messages. Never use a claude/ prefix or put a / in a branch name. Do not add "Generated with Claude Code" (or any similar attribution) to PR descriptions or comments. Do not create a new PR/branch off the existing PR to fix/add something that is related and could've just been committed directly to the existing PR's branch

When working on a PR, keep the PR description in sync with new commits being made

All GitHub comments must be human-readable and 15-25 words max

Monkeypatching attributes of a class to do testing is an anti-pattern. Prefer dependency-injecting things into classes. That way, at unit test time, you can pass a mocked dependency in

Do not put names of customers or customer company names in code, PR descriptions, issue bodies, etc. This means never mention literally any company name. Especially if you're about to say a sentence mentioning that the reason the PR exists was a feature/model/bug fix/etc. requested by a company. That's the indication that you should replace that company name with "the customer". e.g. not "Model request from Acme (Pylon #1234)" but "Model request from a customer (Pylon #1234)". This is because the codebase is public. The only exception is for publicly known providers or vendors such as OpenAI, Anthropic, AWS Bedrock, etc. only IF we're adding support for that provider/vendor in general and NOT if that PR or whatnot was a request by one of them, and they're actually one of our customers.

CI supply-chain safety: Never pipe a remote script into a shell (curl ... | bash, wget ... | sh); download the artifact to a file, verify its SHA-256 checksum, then install. Pin every external tool to a specific version with a full URL (not latest or stable). Verify checksums for all downloaded binaries, using the provider's official .sha256 / .sha256sum sidecar when available. These rules apply to every download in CI

Prisma migrations apply synchronously at proxy boot, before it serves traffic, so a migration must only change schema, never rewrite rows. No UPDATE, DELETE or MERGE, and no INSERT ... SELECT: on a spend-log-sized table any of those is minutes of downtime plus a doubled heap that plain autovacuum won't give back. tests/code_coverage_tests/check_migrations_no_data_rewrites.py enforces this. When a rewrite is genuinely bounded and has to ship inside the migration, mark the statement -- data-migration-ok: <what bounds it>

Follow these coding conventions for new/updated code (a three-line fix in a legacy file shouldn't trigger huge drive-by refactors):

• Composition over inheritance
• Never-nester: early returns over deep nesting
• Don't throw; model failures as values (One function (e.g., raise_public) maps error union to existing public exception contracts via exhaustive match + assert_never)
• No mutation; don't reassign variables, global or local. Instead of mutable lists and dicts, prefer tuples, frozen dataclasses (with slots=True), MappingProxyType, etc.
  • Annotate every variable with : Final (LIT010). Unpacking and walrus targets cannot carry the annotation, so they are implicitly final. Don't rebind them. Never rebind or mutate function parameters (LIT011); self/cls attribute stores are the exception. If rebinding or in-place mutation is truly unavoidable, suppress with # rebind-ok: <reason>
  • Qualify every TypedDict field with ReadOnly[...] (LIT012), which nests freely with Required / NotRequired / Annotated in any order. If making the key writable is truly unavoidable, suppress with # writable-ok: <reason>
• Use dependency injection
• Fully typed; no Any or coarse types like dict[str, Any] or just dict. Every function parameter must be strongly typed
• Use tagged unions + match
• No monster files or god objects
• No file sprawl: deliberate file and folder structure
• Standard over hand-rolled: use the official SDK or a library where one exists; where none does, follow industry standards instead of inventing local conventions
• API-fragmentation-aware: when logic must branch on which API surface produced or consumes data (e.g. chat completions vs Anthropic Messages vs Responses API shapes), proactively look for an existing shared helper (e.g. litellm_core_utils/prompt_templates/factory.py) before writing per-surface parsing in the new module; if none exists, add one there instead of duplicating the same format-detection logic in every new guardrail/integration

Follow conventional commits for commit names and PR titles

Think Before Coding

Don't assume. Don't hide confusion. Surface tradeoffs

Before implementing:

• State your assumptions explicitly. If uncertain, ask
• If multiple interpretations exist, present them. Don't pick silently
• If a simpler approach exists, say so. Push back when warranted
• If something is unclear, stop. Name what's confusing. Ask

Simplicity First

Minimum code that solves the problem. Nothing speculative

• No features beyond what was asked
• No abstractions for single-use code
• No "flexibility" or "configurability" that wasn't requested
• No error handling for impossible scenarios
• If you write 200 lines and it could be 50, rewrite it

Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify

Before requesting maintainer review, verify the current PR tip passes required CI and code coverage, meets Greptile confidence of at least 4/5, and has acceptable Veria and Bugbot reviews. Inspect warnings and findings, fix actionable issues, and rerun the affected checks and reviewers after changes. Record evidence for any false positive or unavailable review; never treat a pending or missing bot result as a pass. Do not lower coverage thresholds or lint budgets to satisfy a check