The research introduces MERIT, a benchmark and harness designed to assess the utility of long-term memory in tool-using LLM agents. It focuses on measuring the marginal cost of memory operations, providing a more realistic evaluation than traditional conversational recall benchmarks. The benchmark incorporates episodic tool-use tasks across three domains, verifying dependencies on earlier-episode facts through automated leak checks, difficulty ladders, and controlled memory corruption. The system utilizes token and dollar metering for every memory operation, tracking costs precisely.
Experiments across 23,440 scored episodes with a total cost of $42.57, involved a two-generation pilot on gpt-4.1-mini and a 3-model x 3-seed grid. Results demonstrated that structured fact stores and LLM summarization consistently achieved success rates of 70-100% when recalling updated facts, while embedding retrieval performed unpredictably, with a maximum seed gap of 0.45.
Specifically, agents acting on correctly retrieved values only succeeded 55% of the time. The implementation of memory significantly impacted task success, with a swap moving performance by up to 60 points. Full replay was never economically viable, with the best condition delivering 2.7-3.9x its marginal utility per dollar.
The benchmark, harness, and all traces are released. Source: https://arxiv.org/abs/2609.05441v1