llm-evaluation

Imported from YapWH1208/llm-evaluation (AGENTS.md). Install upstream with npx skills add YapWH1208/llm-evaluation. Copyright stays with the author.

Project guidance

LLM/SLM evaluation platform: FastAPI backend (backend/, Python ≥3.12, SQLite-first with an optional MongoDB document store) + React 19 / Vite / TypeScript web app (frontend/). End-to-end flow: docs/evaluation-workflow.md. Ops: docs/deployment.md. Human-facing developer guide: DEVELOPER.md.

Skills — STOP. Load a skill before you act.

Skills (playbooks) are user-level, in ~/.agents/skills/, loaded via the skill tool — never run_skill, and the names have no superpowers- prefix. A repo-local skill (superdesign) may live in .agents/skills/ (gitignored, pinned via skills-lock.json).

The rule: before you do ANYTHING non-trivial — before you explore, run bash, write code, or answer — STOP and check the skills index. If a skill might fit, load it. Loading a skill is cheap. Skipping it is the #1 mistake. Process skill FIRST. Action SECOND.

These skills supplement the native tools — they don't replace them. For dispatching subagents, code review, parallel work, and codebase exploration, use the native tools directly: task (run a subagent), review (code-review a diff), wait (join parallel jobs), explore (investigate the codebase).

If you catch yourself about to explore, fix, or answer without loading a skill — STOP and load it. "Build X" → brainstorming first. "Fix this bug" → systematic-debugging first.

Commands (repo root)

• Install backend deps: python -m pip install -e ".[dev]" (editable install from repo root; pythonpath=["backend"] comes from pyproject.toml). No requirements.txt. MongoDB needs the extra: "…[dev,mongodb]" (pymongo is not in the base dev install).
• Backend tests: python -m pytest -q (config sets pythonpath=["backend"], testpaths=["tests"]). Single file: python -m pytest tests/test_x.py -q.
• Run API: uvicorn app.main:app --app-dir backend --reload (docs at /docs on port 8000).
• DB CLI: python -m app.cli database preview|initialize|validate (migrations also run automatically at startup).
• Docker deploy: docker compose up --build (API with SQLite + web container) needs no environment variables — the container auto-provisions a Fernet key at ./data/.lle-secret-key on first start; see docs/deployment.md. The API has no built-in authentication — keep it on loopback or a trusted network.
• Frontend deps/dev server: npm ci / npm run dev inside frontend/ (vite, port 5173). Root package-lock.json is a stub — never npm install at repo root. Node 22 is what CI uses.
• Frontend tests: npm test -- --run in frontend/ — bare npm test is vitest watch mode (config lives in vite.config.ts: jsdom, src/test-setup.ts).
• Frontend typecheck + build: npm run build (runs tsc -b && vite build). No lint script exists for either end.
• One-command local launch: quick-launch.sh / quick-launch.command / quick-launch.bat (installs deps, starts both services, creates data/.lle-secret-key on first run).

Environment (LLE_*)

• LLE_SECRET_ENCRYPTION_KEY — Fernet key for encrypted endpoint credentials; if unset the launchers persist one at data/.lle-secret-key (gitignored).
• LLE_DATABASE_URL — defaults to sqlite:///./data/llm_evaluation.db; mongodb:// switches to the document store.
• LLE_DATABASE_INIT_MODE — auto_migrate (default) | preview | validate.
• Full list (concurrency ceilings, dataset host allowlists, CORS, etc.): backend/app/core/config.py is the single source of truth.

Architecture

• App factory create_app() in backend/app/main.py; feature routers, services, and ports live in backend/app/modules/*; SQLAlchemy models and forward-only migrations remain under backend/app/db/.
• Application behavior is storage-neutral. SQLite and MongoDB differences live in repository adapters selected once by create_app(); feature APIs and services must not branch on the configured backend.
• Provider protocols have one adapter each under backend/app/infrastructure/providers/adapters/. Built-in benchmark plugins live in backend/app/benchmarks/; their persisted catalog is owned by backend/app/modules/benchmarks/.
• Frontend transport is limited to frontend/src/shared/api/client.ts and errors.ts; feature API operations, DTOs, state, and effects live under frontend/src/features/*.

Testing quirks

• MongoDB tests fake the client — never require a live MongoDB.
• Avoid wall-clock-dependent tests (history: rate-window tests were made deterministic for this reason).
• CI (.github/workflows/ci.yml) runs Ruff lint/format and python -m pytest -q, then in frontend/: npm ci, ESLint, tests, and the production build.

Frontend conventions

• All UI copy goes through the typed i18n catalog frontend/src/i18n/catalog.ts (8 locales, strict key parity — a missing key fails src/i18n/locales.test.ts).
• Tests are colocated as src/*.test.ts(x) (vitest + jsdom + Testing Library).

Repo gotchas

• .code-review-graph/ is a local-only knowledge graph DB (gitignored). A pre-commit hook runs code-review-graph update and detect-changes --brief. If the graph warns it was built on another branch, run code-review-graph build.
• /docs and /data are gitignored (existing docs are force-tracked; new files under docs/ need git add -f). reasonix.toml is gitignored too.
• Default branch is master; feature work is done on agent/... branches merged via PRs. Keep commits conventional (feat:, fix:, docs:, test:, ci:).
• Keep CHANGELOG.md updated under ## Unreleased (or a new version section) when user-facing behavior changes.