A new open-source benchmarking harness measures cost per correct answer, agent trajectory cost, and rubric-graded deliverable quality across OpenAI models on Amazon Bedrock. The harness provides a more comprehensive evaluation than traditional price per token metrics. It allows for a direct comparison of model performance based on the outcomes relevant to production workloads. This approach helps engineers determine the most cost-effective model for their specific needs.
The harness’s metrics include cost per correct answer, which directly relates to the accuracy of the model’s responses. Agent trajectory cost measures the computational resources consumed by the agent during a task. Rubric-graded deliverable quality assesses the overall quality of outputs based on predefined criteria. These metrics provide a holistic view of model performance in a production setting.
