mangomas-eval-harness

Name: mangomas-eval-harness
Author: ianshank

Evaluate AgentRuntime on JSONL (exact-match final_answer, tool ids, truncation, faults). Use when asked to score the local harness. Not evaluate_agent_skill fixture counting.

by ianshank(0) 0 installs

Free

Free account. Installing gives you the manifest plus copy-paste snippets.

See reviews

About

Imported from ianshank/Distilled_Agents (.cursor/skills/mangomas-eval-harness/SKILL.md). Install upstream with npx skills add ianshank/Distilled_Agents --skill mangomas-eval-harness. Copyright stays with the author.

Evaluate the local harness

Runs AgentRuntime (Echo via --scripted in tests). Do not treat evaluate_agent_skill JSON pre-grades as agent quality.

python scripts/harness/eval_harness.py \
  --input "$INPUT_JSONL" \
  --harness-id "${MANGOMAS_HARNESS_ID:-base_react}"

Pass --strict to fail on bad JSON or empty prompts instead of skipping. The --threshold gate uses labeled exact-match plus unlabeled completion (not truncated) over every evaluated row.

Trusted JSONL: injection detection is off. Interactive tasks stay on run_agent.py.

Use it

Copy one of these into your project. Installing also returns the manifest and these snippets.

# after Install: the listing is in your workspace's routing pool - a plan picks it for its slot
curl -s -X POST https://api.opensmartroute.ai/api/v1/route -H 'Authorization: Bearer $OSR_API_KEY' -H 'Content-Type: application/json' -d '{"text": "...", "plan": true}'

Manifest

An Open Capability Manifest: the router reads it to know what this does, what it costs and when to pick it.

ianshank-distilled-agents-mangomas-eval-harness.ocm.jsonjson

{
  "ocm": "1",
  "id": "ianshank-distilled-agents-mangomas-eval-harness",
  "kind": "skill",
  "name": "mangomas-eval-harness",
  "description": "Evaluate AgentRuntime on JSONL (exact-match final_answer, tool ids, truncation, faults). Use when asked to score the local harness. Not evaluate_agent_skill fixture counting.",
  "publisher": "ianshank",
  "version": "1.0.0",
  "capabilities": {
    "domains": [
      "general"
    ],
    "tags": [
      "skill-md",
      "github"
    ],
    "languages": [
      "en"
    ]
  },
  "quality_prior": 0.6,
  "examples": [
    "Evaluate AgentRuntime on JSONL (exact-match final_answer, tool ids, truncation, faults). Use when asked to score the local harness. Not evaluate_agent_skill fixture counting."
  ],
  "primary": false,
  "metadata": {
    "source": {
      "provider": "github",
      "repository": "https://github.com/ianshank/Distilled_Agents",
      "path": ".cursor/skills/mangomas-eval-harness/SKILL.md",
      "ref": "c25c4431a8e1473cfe3502b8aa0c194d33319534",
      "url": "https://github.com/ianshank/Distilled_Agents/blob/c25c4431a8e1473cfe3502b8aa0c194d33319534/.cursor/skills/mangomas-eval-harness/SKILL.md",
      "key": "ianshank/Distilled_Agents/.cursor/skills/mangomas-eval-harness/SKILL.md"
    }
  },
  "instructions": "# Evaluate the local harness\n\nRuns `AgentRuntime` (Echo via `--scripted` in tests). Do not treat `evaluate_agent_skill` JSON pre-grades as agent quality.\n\n```bash\npython scripts/harness/eval_harness.py \\\n  --input \"$INPUT_JSONL\" \\\n  --harness-id \"${MANGOMAS_HARNESS_ID:-base_react}\"\n```\n\nPass `--strict` to fail on bad JSON or empty prompts instead of skipping. The `--threshold` gate uses labeled exact-match plus unlabeled completion (not truncated) over every evaluated row.\n\nTrusted JSONL: injection detection is off. Interactive tasks stay on `run_agent.py`.",
  "cost": {
    "context_tokens": 140
  }
}

Fetch it by URL: GET /api/v1/registry/ianshank-distilled-agents-mangomas-eval-harness/manifest?version=1.0.0

About

Evaluate the local harness

Use it

Manifest

Reviews