ZipBench addresses the high cost of evaluating comprehensive benchmarks for large language models (LLMs). The framework focuses on reducing redundancy in benchmark suites, a common issue where many benchmarks share similar data. It achieves this by evaluating a small set of anchor LLMs and then synthesizing pseudo evaluation results to broaden the coverage of the benchmark. This approach learns compact sample representations and selects a small, representative subset for evaluation.
The ZipBench framework creates ZipBench Zoo, a collection of over 100 benchmark proxies. These proxies cover text, multimodal, and agent tasks. The proxies achieve mean absolute errors of 0.002 to 0.02 and average Spearman correlations of approximately 0.98 compared to the full benchmarks. This level of accuracy is achieved with significantly reduced computational cost.
Building on the ZipBench framework, the research team developed a system for creating compact versions of benchmarks. The system synthesizes pseudo evaluation results to broaden coverage, learns compact sample representations, and selects a small yet representative subset. The framework is designed to be easily extensible to newly released benchmarks.
The code for ZipBench and the associated ZipBench Zoo is publicly available. This allows researchers and engineers to implement and utilize the framework for their own LLM evaluation and benchmark compression needs. The framework lowers the barrier to broad LLM research for compute-constrained researchers.
Source: https://arxiv.org/abs/2609.12475
