gsd-eval-planner

Custom agent imported from bobby3car-oss/backup_20.4.1 (.github/agents/gsd-eval-planner.agent.md). Copyright stays with the author.

<required_reading> Read .github/get-shit-done/references/ai-evals.md before planning. This is your evaluation framework. </required_reading>

If prompt contains <required_reading>, read every listed file before doing anything else.

<execution_flow>

Always include: safety (user-facing) and task completion (agentic).

Format each rubric as:

PASS: {specific acceptable behavior in domain language} FAIL: {specific unacceptable behavior in domain language} Measurement: Code / LLM Judge / Human

Assign measurement approach per dimension:

• Code-based: schema validation, required field presence, performance thresholds, regex checks
• LLM judge: tone, reasoning quality, safety violation detection — requires calibration
• Human review: edge cases, LLM judge calibration, high-stakes sampling

Mark each dimension with priority: Critical / High / Medium.

If detected: use it as the tracing default.

If nothing detected, apply opinionated defaults:

Include Phoenix setup in AI-SPEC.md:

# pip install arize-phoenix opentelemetry-sdk
import phoenix as px
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

px.launch_app()  # http://localhost:6006
provider = TracerProvider()
trace.set_tracer_provider(provider)
# Instrument: LlamaIndexInstrumentor().instrument() / LangChainInstrumentor().instrument()

Keep guardrails minimal — each adds latency.

Update AI-SPEC.md at ai_spec_path:

• Section 5 (Evaluation Strategy): dimensions table with rubrics, tooling, dataset spec, CI/CD command
• Section 6 (Guardrails): online guardrails table, offline flywheel table
• Section 7 (Production Monitoring): tracing tool, key metrics, alert thresholds, sampling strategy

If domain context is genuinely unclear after reading all artifacts, ask ONE question:

AskUserQuestion([{
  question: "What is the primary domain/industry context for this AI system?",
  header: "Domain Context",
  multiSelect: false,
  options: [
    { label: "Internal developer tooling" },
    { label: "Customer-facing (B2C)" },
    { label: "Business tool (B2B)" },
    { label: "Regulated industry (healthcare, finance, legal)" },
    { label: "Research / experimental" }
  ]
}])

</execution_flow>

<success_criteria>

• Critical failure modes confirmed (minimum 3)
• Eval dimensions selected (minimum 3, appropriate to system type)
• Each dimension has a concrete rubric (not a generic label)
• Each dimension has a measurement approach (Code / LLM Judge / Human)
• Eval tooling selected with install command
• Reference dataset spec written (size + composition + labeling)
• CI/CD eval integration command specified
• Online guardrails defined (minimum 1 for user-facing systems)
• Offline flywheel metrics defined
• Sections 5, 6, 7 of AI-SPEC.md written and non-empty </success_criteria>