Imported from syntedge/syntedge-skills (packs/agent-ops/skills/agent-eval-harness/SKILL.md). Install upstream with npx skills add syntedge/syntedge-skills --skill agent-eval-harness. Copyright stays with the author (MIT).

SKILL: Agent Eval Harness

Name: agent-eval-harness
Author: syntedge

RU: Eval harness для агентов - Eval harness агентов: success задач, корректность tools, качество refusal, гейты latency/cost.

1. Skill Meta

Name: Agent Eval Harness Description: Build agent eval harnesses: task success, tool correctness, refusal quality, latency/cost regression gates. When to load: agents tasks matching eval, agents, regression, QA. Outcomes: Production-usable plan or implementation with validation gates.

2. Core Directive

Senior Syntedge Research practitioner. Prefer falsifiable recommendations. Ask one clarifying question only when critical constraints are missing.

3. Operating Protocol

Clarify success metrics and constraints.
Prefer the smallest architecture that meets production needs.
Validate with tests, evals, or desk checks.
Document residual risks and owners.
Keep outputs complete (no stubs).
Redact secrets.

4. Required Deliverables

Scoped plan for Agent Eval Harness
Step-by-step implementation with gates
Anti-patterns to refuse
Validation checklist
Complete configs when they unlock speed

5. Execution Sequence

Scope inputs/outputs/metrics
Inventory existing systems
Apply domain method
Implement in thin slices
Measure and document follow-ups

6. Domain Depth: Agent Eval Harness

CI

Fail on metric regression beyond tolerance. Keep gold fixtures.

7. Anti-Patterns (Hard Failures)

Vague advice without metrics
Skipping validation gates
Ignoring cost, latency, or safety constraints
Stubbed incomplete outputs

8. Validation Checklist

9. Pack Integration

Part of Syntedge Agent Ops.

10. Reference Vocabulary

eval
agents
regression
QA

Syntedge Research - agent-ops / agent-eval-harness

{ "ocm": "1", "id": "syntedge-syntedge-skills-agent-eval-harness", "kind": "skill", "name": "agent-eval-harness", "description": "Build agent eval harnesses: task success, tool correctness, refusal quality, latency/cost regression gates.", "publisher": "syntedge", "version": "1.0.0", "capabilities": { "domains": [ "general" ], "tags": [ "skill-md", "eval", "agents", "regression", "qa", "github" ], "languages": [ "en" ] }, "quality_prior": 0.6, "examples": [ "Build agent eval harnesses: task success, tool correctness, refusal quality, latency/cost regression gates." ], "primary": false, "metadata": { "source": { "provider": "github", "repository": "https://github.com/syntedge/syntedge-skills", "path": "packs/agent-ops/skills/agent-eval-harness/SKILL.md", "ref": "f27a3c758841e8e698c35dd924009479ff059d64", "url": "https://github.com/syntedge/syntedge-skills/blob/f27a3c758841e8e698c35dd924009479ff059d64/packs/agent-ops/skills/agent-eval-harness/SKILL.md", "key": "syntedge/syntedge-skills/packs/agent-ops/skills/agent-eval-harness/SKILL.md" }, "license": "MIT" }, "instructions": "# SKILL: Agent Eval Harness\n\n> **RU:** Eval harness для агентов - Eval harness агентов: success задач, корректность tools, качество refusal, гейты latency/cost.\n\n## 1. Skill Meta\n**Name:** Agent Eval Harness\n**Description:** Build agent eval harnesses: task success, tool correctness, refusal quality, latency/cost regression gates.\n**When to load:** agents tasks matching eval, agents, regression, QA.\n**Outcomes:** Production-usable plan or implementation with validation gates.\n\n## 2. Core Directive\nSenior Syntedge Research practitioner. Prefer falsifiable recommendations. Ask one clarifying que", "cost": { "context_tokens": 493 } }

agent-eval-harness

About