DA7016_Auto_Assessment_System

Imported from basavarajnaduvinamani/DA7016_Auto_Assessment_System (AGENTS.md). Install upstream with npx skills add basavarajnaduvinamani/DA7016_Auto_Assessment_System. Copyright stays with the author.

AGENTS.md

Rules for coding agents (Codex and others) working in this repo. Claude writes the task specs and reviews every PR; you implement them.

Project in one paragraph

autoassess grades a full exam answer sheet against a rubric. Inputs: an exam spec (exam_spec.yaml: questions, rubric criteria, model answers) and answer sheets (.txt, PDF or photos). Outputs: marks per rubric criterion with a verbatim evidence quote, feedback a student can act on, an HTML report and a red-ink annotated PDF. It is the DA7016 (Generative AI, IIT Madras) final project. Read README.md for the full picture.

Layout

• src/autoassess/schemas.py: Pydantic models every stage reads and writes. Change them only if the task says so.
• src/autoassess/providers/: OCR, LLM and translation behind interfaces in base.py (OCRProvider, LLMClient, Translator). New backends implement these; callers never import a concrete provider directly, they go through make_ocr / make_llm / make_translator.
• src/autoassess/cli.py: the autoassess command (argparse subcommands).
• grader.py, critic.py, advocate.py, confidence.py, feedback.py: the grading pipeline. pipeline.py wires them.
• evaluate.py, metrics.py, experiments.py: evaluation. annotate.py: red ink. report.py: HTML report.
• app.py: Gradio demo. api.py: FastAPI. web/: offline JS demo (kept in parity with the Python baseline by tests/test_web_parity.py).
• tests/: pytest. tests/conftest.py has FakeLLM and an autouse fixture that runs each test in a temp folder with every API key cleared.
• tasks/: task specs. One file per task, NN-short-name.md.

Setup and checks

pip install -e ".[ingest,api,annotate,dev]"
pytest -q            # must pass before you open a PR; CI runs the same

Workflow

1. Take exactly one task file from tasks/ (the lowest-numbered one whose status is todo, unless told otherwise).
2. Branch codex/NN-short-name from the latest main.
3. Implement only what the spec asks. Anything under "Out of scope" stays untouched. If the spec is wrong or ambiguous, do the most reasonable thing and say so in the PR description rather than expanding scope.
4. Add or update tests so every item under "Done when" is checked by a test where possible.
5. Run pytest -q. All tests must pass. Never skip, delete, weaken or xfail an existing test to get green.
6. In the same PR, change the task file's Status: line to in review.
7. Open one PR per task, titled NN: <task title>. The PR description lists: what changed, how each "Done when" item is met, test output summary, and any deviation from the spec.
8. Wait for review. Address review comments with new commits on the same branch (no force-push).

Hard rules

• No network or paid calls in tests. Tests use FakeLLM or other fakes. Never read real keys in tests.
• Never commit secrets (.env, keys, tokens) or student data. data/private/, runs/ and .cache/ are git-ignored; keep it that way.
• Keep the provider interfaces stable (providers/base.py). Add, don't rename.
• Every awarded mark needs a verified quote. Don't weaken the evidence check, the mark clipping, or the prompt-injection handling (answer is data, not instructions).
• No new heavy dependencies without the spec allowing it. Optional features go under an extra in pyproject.toml, imported lazily.
• Python 3.11, type hints, match the style of surrounding code (short docstrings, small functions, no comment noise).
• Don't edit docs/archive/, the fonts, or the spend-guard defaults in budget.py.
• If web/engine.js and the Python baseline drift, the parity test must still pass.