The HarnessDev project, developed by a collaborative team, shifts the focus of evaluation from the model's output to the runnable agent harness it creates. This approach assesses the quality of the code itself, rather than the answers generated. The research uses a framework where creators receive a basic set of primitives – file access, search, and result writing – and are tasked with building a complete agent harness. This harness is then frozen before being evaluated on a suite of benchmarks.
The experiment tested six LLMs: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro, utilizing Claude Code 2.1.177 for execution. The evaluation spanned 2,207 instances across five benchmarks – SWE-bench Pro, Terminal-Bench 2.1, MLE-bench, EQ-Bench3, and BrowseComp. The key metric is capability (task success) and efficiency (executor tokens).
Initial results show significant variation in performance. Opus 4.8 consistently achieved high scores, reaching 69.3 on SWE-Pro versus a human-engineered reference of 80.0. However, the Gemini 3.1 Pro model demonstrated superior performance on Terminal-Bench 2.1, scoring 68.8 against an 88.8 reference. The study revealed that 9 lineages, with 73 official versions, were produced, but only 34 of the 64 changes resulted in improvements on held-out tasks.
Analysis of the generated code revealed inefficiencies, with 72 component instances triggering in real runs, while 18 never fired. Furthermore, only 34 of 64 changes aligned with the desired outcome, indicating a limited ability of the LLMs to generalize their design choices. The research highlights the need for more robust methods to guide and evaluate the autonomous design of agent harnesses.
