config

Imported from ianshank/Agents (src/eval_harness/config/AGENTS.md). Install upstream with npx skills add ianshank/Agents --skill config. Copyright stays with the author.

AGENTS.md — src/eval_harness/config

Turns a YAML file into a validated EvalConfig. Every behavioural default is declared here.

This package owns the whole load pipeline — migrate, interpolate, override, validate — and the models that carry every threshold and parameter. Nothing in the engine holds a behavioural literal; if a number drives behaviour, its default belongs on a field here.

Map

Diagram

flowchart LR
  accTitle: Config load pipeline
  accDescr: A YAML file is migrated to the current schema, then environment variables are interpolated, then command-line overrides are applied, and only then is it validated into an EvalConfig that the engine reads.

  YAML["config/*.yaml"]
  subgraph CFG["config/ (you are here)"]
    MIG["migrate_to_current<br/>migrations.py"]
    INT["interpolate<br/>env vars"]
    OVR["apply_overrides<br/>run.sample_rate=0.1"]
    VAL["EvalConfig.model_validate<br/>extra forbid"]
  end
  ENG["engine, gating, cli"]
  ERR["ConfigError"]
  YAML --> MIG
  MIG --> INT
  INT --> OVR
  OVR --> VAL
  VAL --> ENG
  VAL -->|unknown key| ERR
  MIG -->|no migration path| ERR
  classDef here fill:#e8f0fe,stroke:#1a73e8,stroke-width:2px
  class CFG here

Rules that bite here

• Loading is strict: model_config = ConfigDict(extra="forbid"). An unknown key raises ConfigError rather than being ignored. Do not add a permissive fallback to make an old config load — add a migration, which is the mechanism that already exists for that.
• SCHEMA_VERSION is single-sourced in ../version.py and is not touched on a feature branch. A bump happens in a dedicated release commit and requires a registered @migration here; migrate_to_current walks the chain and raises on a cycle or a gap.
• A config is untrusted input, not data. Fields that become an import or a filesystem path must route through core/_imports.py or core/_paths.py. Do not add a new config-driven path or import that bypasses both gates.
• No behavioural literal at a call site. A default goes on the field, documented in its description, so an operator can override it without a code change (ADR 0009).

Verify

python -m pytest tests/test_config.py tests/test_backwards_compat_config.py -q

Subagents

See also