Imported from onlyairnopods/lmms-eval-dev (AGENTS.md). Install upstream with npx skills add onlyairnopods/lmms-eval-dev. Copyright stays with the author.

Agent Guidelines for lmms-eval

This file provides context for AI coding agents (Codex, Devin, SWE-Agent, etc.) working on this codebase.

For detailed development guidelines, see CLAUDE.md.

Quick Reference

Setup: uv sync && pre-commit install Run eval: python -m lmms_eval --model qwen2_5_vl --tasks mme --limit 5 --batch_size 1 Lint: pre-commit run --all-files Test registry: uv run python -m unittest discover -s test/eval -p "test_model_registry_v2.py"

Useful CLI Flags

Environment Variables

export OPENAI_API_KEY="..."      # Required for OpenAI/API-backed models
export HF_TOKEN="..."            # Required for gated HuggingFace datasets
export HF_HOME="/path/to/cache"  # HuggingFace cache directory
export HF_HUB_ENABLE_HF_TRANSFER="1"  # Faster downloads

Project Overview

lmms-eval evaluates Large Multimodal Models (LMMs) across image, video, and audio tasks. It supports 100+ benchmarks and 30+ model backends.

Key Directories

Model Registry V2

Models are registered via two dicts in __init__.py that map model_id -> ClassName:

• AVAILABLE_CHAT_TEMPLATE_MODELS - chat models in models/chat/
• AVAILABLE_SIMPLE_MODELS - simple models in models/simple/

If the same model_id exists in both, the registry creates one ModelManifest with both paths. Resolution prefers chat over simple.

MODEL_ALIASES provides backward-compatible name mappings: {"new_name": ("old_name_1", "old_name_2")}.

Pipeline

Dataset --> doc_to_messages (or doc_to_visual + doc_to_text)
        --> Model.generate_until() or Model.loglikelihood()
        --> process_results()
        --> metric aggregation

Common Tasks

Adding a New Model

1. Create lmms_eval/models/chat/<name>.py
2. Subclass lmms, set is_simple = False, implement generate_until
3. Use @register_model("<name>") decorator
4. Add "<name>": "ClassName" to AVAILABLE_CHAT_TEMPLATE_MODELS in __init__.py
5. Test: python -m lmms_eval --model <name> --model_args pretrained=org/model --tasks mme --limit 5

Adding a New Task

1. Create lmms_eval/tasks/<name>/<name>.yaml + utils.py
2. YAML needs: task, dataset_path, test_split, output_type, doc_to_messages, process_results, metric_list
3. Tasks auto-register from YAML - no manual registration needed
4. Test: python -m lmms_eval --model qwen2_5_vl --tasks <name> --limit 8

Task YAML Advanced Features

lmms_eval_specific_kwargs - Model-specific prompt overrides. Framework selects matching key based on model, falls back to default:

lmms_eval_specific_kwargs:
  default:
    pre_prompt: ""
    post_prompt: "\nAnswer directly."
  qwen3_vl:
    format: "qwen3_vl"
    pre_prompt: "Question: "
    post_prompt: "Answer with the option letter only."

These kwargs are passed to doc_to_messages(doc, lmms_eval_specific_kwargs=...).

include - Inherit shared config from a template file (avoids duplication across variants):

include: _default_template_yaml

group + task list - Define task families:

group: mmmu
task:
- mmmu_val
- mmmu_test

output_type options: generate_until (free-form), loglikelihood (multiple-choice), generate_until_multi_round (multi-turn conversation)

Fixing a Model Bug

1. Find the model file in models/chat/ or models/simple/
2. Check generate_until for generation issues, loglikelihood for multiple-choice
3. Look at req.args unpacking - chat models get 5 elements, simple get 6
4. Run with --limit 5 to verify the fix quickly

Fixing a Task Bug

1. Task YAML is in lmms_eval/tasks/<task_name>/
2. Helper functions are in utils.py next to the YAML
3. process_results handles scoring, doc_to_messages handles input formatting
4. Test with --limit 8 to verify

Debugging

Quick Diagnostics

• Verbose logging: python -m lmms_eval --model ... --verbosity DEBUG - shows detailed traces
• Small test run: --limit 5 evaluates only 5 samples - always use this when testing changes
• Log samples: --log_samples saves per-sample predictions to output directory for inspection

Common Errors and Fixes

Logging

The codebase uses eval_logger from loguru. To add debug logging in your code:

from lmms_eval.utils import eval_logger
eval_logger.debug("Processing batch of {} samples", len(batch))
eval_logger.warning("Missing visual for doc_id={}", doc_id)

Retry Patterns (API Models)

API-backed models (openai, gemini, etc.) support retry configuration:

python -m lmms_eval --model openai --model_args pretrained=gpt-4o,max_retries=5,retry_backoff_s=2.0 --tasks mme

Constraints

• Package manager: uv only, never pip
• Formatting: Black (line-length=240) + isort (profile=black). Run pre-commit run --all-files before committing.
• No type suppression: Never use as any, @ts-ignore, type: ignore to suppress type errors
• Commits: Never mention co-authored-by or AI tools
• Minimal changes: Fix the specific issue, don't refactor unrelated code
• Follow patterns: Match the style of neighboring files exactly