The research investigated the performance of different agentic coding harnesses, specifically examining the impact of vendor-native pairings. A private suite of 256 repository and contest tasks was used, running under eight different configurations: claude-agent-sdk and deepagents on claude-opus-4-8, and openai-codex SDK and deepagents on gpt-5.5, gemini-3.5-flash and deepseek-v3.2. A total of 800 planned runs were graded by an isolated oracle, with 792 completed. The study found no average advantage for either harness, with a difference of -1.25 pp for Opus 4.8 and +1.25 pp for GPT-5.5, based on task-bootstrap confidence intervals.
The Opus 4.8 harness showed opposite stratification: it trailed by 9.0 pp on repository tasks and led by 23.7 pp on contest tasks, with a label-permutation p-value of 0.003. Furthermore, 22 of 81 runs cancelled due to wall-clock ceiling constraints, but 22 produced passing patches. The cost of the harnesses was observed to be 1.3 to 1.6 times as much per solved task on Opus 4.8 and 1.2 times on GPT-5.5, based on usage estimates. On the Anthropic account, 58 runs generated no usage record, making the cost ratio unresolved.
This research corrected an earlier manuscript that had an error in its cost telemetry. The orchestrator, grading oracle, reanalysis code, and derived aggregates were released. The tasks remain private. The study highlights the complexity of evaluating agentic coding systems and the potential for unresolved billing issues.
Source: https://arxiv.org/abs/2609.11987