Imported from rgranadosd/agent-manager (evaluation-job/AGENTS.md). Install upstream with npx skills add rgranadosd/agent-manager --skill evaluation-job. Copyright stays with the author.

evaluation-job — agent guide

A Dockerized Python job (not a library) that runs Monitor-mode evaluations against agent traces in a time window. Invoked by an Argo ClusterWorkflowTemplate. It wraps amp-evaluation's Monitor runner, fetches traces from the traces API, and publishes scores back to the platform.

What it is (and isn't)

• Two files: main.py (the job) and test_main.py.
• pyproject.toml has no [project] section, no dependencies, no build backend — only ruff/mypy/pytest config. This is intentional: it is never published to PyPI; deps come from the Docker image (which installs amp-evaluation).
• Entry point is python main.py with argparse CLI flags.

main.py structure

• OAuth2TokenManager — client-credentials token caching (refresh when within 30s of expiry).
• handle_sigterm → logs and exits immediately via sys.exit(0) (graceful shutdown for Kubernetes). It does not set a shutdown flag, and nothing polls one.
• Argparse config validation → evaluator registration from a JSON arg → trace fetch (paginated) → Monitor.run() → publish scores.

CLI flags

OAuth2 client credentials for publisher authentication are read from the environment (via OAuth2TokenManager), not passed as CLI flags.

Traces are fetched via TraceFetcher.fetch_traces(...). Page size is configurable via the page_size parameter and defaults to the value set on the TraceFetcher instance; this job constructs the fetcher with page_size=TRACE_FETCH_PAGE_SIZE (a module constant, currently 10, to bound memory). Scores POST to /api/v1/publisher/monitors/{monitor_id}/runs/{run_id}/scores (built from --publisher-endpoint) as an EvaluatorSummary-derived payload (individualScores + aggregatedScores), with exponential backoff (3 retries, 2s initial).

Commands

make lint             # ruff check .
make format-check     # ruff format --check .
make type-check       # mypy main.py --install-types --non-interactive
make test             # pytest test_main.py -v
make docker-build     # image installing amp-evaluation from PyPI (AMP_EVALUATION_VERSION arg)
make docker-build-dev # image installing amp-evaluation from local ../libs (PROJECT_ROOT=..)
make docker-load-k3d  # build-dev + load into k3d

Config: ruff/mypy target py311, line-length 120.

Gotchas

• The EvaluatorSummary JSON published must match the Pydantic schema in amp-evaluation exactly.
• Most runtime config comes from AMP_* env vars consumed by amp-evaluation.Config, not from CLI flags.
• Page size (10) is not a flag — change it in the source if needed.
• For local iteration on the evaluator code, use make docker-build-dev so it picks up your local ../libs/amp-evaluation instead of the published version.

Depends on ../libs/amp-evaluation/AGENTS.md (evaluators + Monitor) and hits ../agent-manager-observer/AGENTS.md for traces.