Imported from faionfaion/faion-network (skills/faion/knowledge/ai-core/rag-bench-harness-template/AGENTS.md). Install upstream with npx skills add faionfaion/faion-network --skill rag-bench-harness-template. Copyright stays with the author.

RAG Bench Harness Template

Summary

One-sentence: Spec for a reusable RAG bench harness — corpus snapshot, query set, retriever runners, eval metrics, leaderboard schema — so chunking-strategy comparisons stop being hand-wave.

One-paragraph: Existing RAG eval methodologies are atomic (rag-eval-metrics, rag-eval-ab-testing). A wire-it-together harness that runs a chunking experiment end-to-end is missing. This methodology produces a rag-bench-spec.json artefact pinning corpus snapshot, query set, retriever runners, eval metrics (Recall@k, MRR, faithfulness), and a leaderboard schema. Output is a versioned harness consumed by chunking-strategy A/Bs.

Ефективно для:

• Chunking strategy bench (recurring, weekly or quarterly).
• Compare retrievers (BM25 vs dense vs hybrid) on the same corpus.
• Reproducible eval — corpus snapshot pinned by sha.
• Leaderboard schema для cross-strategy comparison.
• Bridge до downstream [[prompt-ab-power-calculator]] для chunking A/B.

Applies If (ALL must hold)

• Recurring RAG bench on the operating cadence.
• Corpus + query set committable to repo.
• Named accountable owner.
• Eval metrics agreed (Recall@k / MRR / faithfulness).

Skip If (ANY kills it)

• One-shot bench without recurrence.
• No corpus snapshot can be committed (purely-live or licensed-only data).
• Fewer than 3 instances per year.
• No named owner.

Prerequisites

Assumes Loaded

Content (load on demand)

Task Routing

Templates

Scripts

Related

• [[prompt-ab-power-calculator]]
• [[rag-corpus-discovery-interview]]
• [[production-trace-mining-for-training-data]]

Decision tree

See content/06-decision-tree.xml. The tree picks retriever set based on corpus type (free-form prose, structured docs, code) and metric set based on task (retrieval vs generation). Walk it before drafting the spec.