Imported from JAYPENGXu/AI-Eval (AGENTS.md). Install upstream with npx skills add JAYPENGXu/AI-Eval. Copyright stays with the author.

AGENTS.md

Project Scope

AIAssistant is a RAGOps engineering project. Its product boundary is document ingestion, retrieval, observable Q&A, evaluation, regression, configuration experiments, and multi-tenant governance.

The repository intentionally does not contain an Agent workbench, LangGraph graphs, graph state, HITL actions, tool abstractions, thread registries, or checkpoint storage. Do not reintroduce those concepts unless the product direction explicitly changes.

Core Flow

upload -> validate -> async parse/OCR -> Document IR quality gate
-> structure-aware chunking -> blue-green indexing
-> router/rewrite -> vector + lexical -> RRF -> rerank -> compression
-> streaming answer + citations -> Trace -> evaluation -> regression
-> parameter experiment -> release gate -> explicit config publish/rollback

Important Paths

• backend/assistant_backend/settings.py: settings and environment variables.
• backend/rag/models.py: business models.
• backend/rag/api/, backend/rag/urls.py: serializers, views, authentication and API routing.
• backend/rag/chat/: Q&A orchestration, session memory and chat-facing services.
• backend/rag/pipeline/: typed pipeline stages.
• backend/rag/retrieval/: vector/lexical retrieval, RRF, rerank, compression and query routing/rewrite.
• backend/rag/document_parsing/: parser interface, IR, OCR routing, provider attempts, and selection.
• backend/rag/indexing/: chunking and blue-green index lifecycle.
• backend/rag/evaluation/: datasets, snapshots, strict Judge, experiments, execution and reconciliation.
• backend/rag/prompts/: source-controlled prompt baselines, strict rendering, immutable versions and deployment resolution.
• backend/rag/configuration/: immutable RAG configuration versions, release and deployment audit helpers.
• backend/rag/permissions/: Organization tenancy, RBAC/ABAC, AccessScope and protected media.
• backend/rag/governance/: retention, private storage, export/deletion and quotas.
• backend/rag/tasking/: TaskRun lifecycle, outbox dispatch and task APIs.
• backend/rag/operations/: health checks and model usage reporting.
• backend/rag/tasks.py: stable Celery task entrypoint; task names must remain backward compatible.
• frontend/src/App.vue: thin workbench shell.
• frontend/src/components/Workbench/: Debug, Evaluation, Datasets, History, Permissions and Costs.
• frontend/src/composables/, stores/: identity-bound domain state.
• frontend/src/api/generated/schema.ts: generated OpenAPI types; never edit manually.

Backend Rules

• PostgreSQL 16 is the only supported relational database.
• Redis/Celery owns background parsing, indexing, summaries, evaluations and experiments.
• MinIO objects are private. Downloads and OCR input previews require live authorization.
• Every retrieval stage requires an explicit AccessScope. Missing organization, membership, policy or scope fails closed.
• Milvus hits must be re-authorized through ORM scope before use.
• Chat sessions are always private to their owner, even inside a shared organization.
• Trace and telemetry must not persist prompts, chunks, tokens, object keys or sensitive text.
• Index rebuilds are generation based: validate inactive data, atomically switch, then clean old generations.
• A needs_review parse cannot be indexed until accepted.
• Preserve page, heading path, block IDs, paragraph range and parser provenance through retrieval and citations.
• Configuration publishing and rollback are explicit authenticated API operations with immutable versions and deployment audit rows. They do not use action cards or graph resume.
• Prompt baselines live in source control. Knowledge-base overrides are immutable versions and must pass static or regression validation before explicit publish/rollback; do not add ad-hoc prompt strings to business services.

Document Parsing

Supported formats are TXT, Markdown, DOCX and PDF. Validate extension, signature, size, archive structure, encryption and page count.

Simple PDF pages may use PyMuPDF. Complex/scanned pages route to configured visual providers. Each provider result is stored as a DocumentPageParseAttempt; only the selected attempt feeds IR and chunking. Failed OCR pages must fail explicitly, not become silent blank pages.

IR v2 table rows carry header-qualified relation text and remain atomic across chunkers. Optional OCR input PNG persistence defaults off and remains protected.

Parsing quality is enforced through per-document metrics, preview, OCR conflict review and backend fixture tests. There is intentionally no product-facing document parse benchmark dataset or parse-evaluation run API.

Evaluation

Suites are benchmark, regression and security. Preserve deterministic checks, RAGAS metrics, strict JSON Judge metadata, retrieval diagnostics and stage failure reasons.

Evaluation runs bind immutable dataset, index, config, prompt and model snapshots. Retrying a case must update idempotently. A configuration candidate cannot publish unless the regression gate passes.

Frontend Rules

• Preserve the three-column RAGOps workbench.
• Workbench tabs are Debug, Evaluation, Datasets, History, Permissions and Costs. TaskRun remains a backend API and Grafana concern; do not add a standalone Tasks workbench.
• Identity or organization switches must abort requests and polling, clear identity-bound state, then load the new scope.
• Keep access tokens in memory and refresh tokens in HttpOnly cookies.
• Use Element Plus and existing composable/store patterns.
• Update API, state and UI together when response shapes change.

Setup

cd backend
source venv/bin/activate
pip install --require-hashes -r requirements.lock
python manage.py migrate
python manage.py runserver 127.0.0.1:8010
celery -A assistant_backend worker --loglevel=INFO --concurrency=1

cd frontend
npm ci
npm run dev -- --host 0.0.0.0 --port 5174

Use ./scripts/dev-stack.sh for the supported Docker dev environment. Long-running services must not run migrations; use the dedicated migration job.

Verification

cd backend
source venv/bin/activate
python manage.py check
python manage.py makemigrations --check --dry-run
python -m compileall rag assistant_backend
pytest rag/tests -q

cd ../frontend
npm run typecheck
npm run build

After dependency source changes, regenerate uv hash locks with scripts/lock-python-dependencies.sh --write. RAGAS 0.4 requires the pinned LangChain 0.3 compatibility packages; application code must not depend on their Agent APIs or add LangGraph.

Editing Notes

• Add migrations for model changes.
• Keep changes scoped and preserve unrelated dirty worktree changes.
• Do not commit virtual environments, caches, logs, uploads, build output, vector data or local secrets.
• Update parser tests, API tests, chunk provenance and docs together for ingestion changes.
• Update access-control and identity-switch tests for auth/session changes.
• Update index security and generation tests for retrieval/index changes.