HarvestBench is a new benchmark that assigns a price to avoiding harm to animals in a farm simulation environment. It involves LLM sub-agents controlling tractors in a reinforcement learning gridworld, making decisions without memory.
When animals block a tractor's route, the model chooses whether to drive on, swerve around, or take alternative actions, with a fuel cost attached to harming animals. The environment includes controls like rocks and hay bales, which serve as comparison points for harm and moral decisions.
Across nine models and over 7,200 decisions, kill rates varied from 0.4% to 98.8%. Models showed sensitivity to pricing, with some adjusting their decisions based on the cost of harm. The presence of morality briefings significantly reduced harm, demonstrating the impact of contextual information.
HarvestBench measures what models will pay to avoid harm, using a reproducible scoring method based on game logs. This approach provides insights into model behavior regarding moral decisions without relying on LLM graders.
Source: https://arxiv.org/abs/2609.04444