Establishing Best Practices in Building Rigorous Agentic Benchmarks
Abstract
Benchmarks are essential for quantitatively tracking progress in AI. As AI agents become increasingly capable, researchers and practitioners have introduced agentic benchmarks to evaluate agents on complex, real-world tasks. These benchmarks typically measure agent capabilities by evaluating task outcomes via specific reward designs. However, we show that many agentic benchmarks have issues in task setup or reward design. For example, SWE-bench-Verified uses insufficient test cases, while $\tau$-bench counts empty responses as successes. Such issues can lead to under- or overestimation of agents’ performance by up to 100% in relative terms. To make agentic evaluation rigorous, we introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience, a survey of best practices, and previously reported issues. When applied to CVE-Bench, a benchmark with a particularly complex evaluation design, ABC reduces performance overestimation by 33%.
Cite
Text
Zhu et al. "Establishing Best Practices in Building Rigorous Agentic Benchmarks." Advances in Neural Information Processing Systems, 2025.Markdown
[Zhu et al. "Establishing Best Practices in Building Rigorous Agentic Benchmarks." Advances in Neural Information Processing Systems, 2025.](https://mlanthology.org/neurips/2025/zhu2025neurips-establishing/)BibTeX
@inproceedings{zhu2025neurips-establishing,
title = {{Establishing Best Practices in Building Rigorous Agentic Benchmarks}},
author = {Zhu, Yuxuan and Jin, Tengjun and Pruksachatkun, Yada and Zhang, Andy K and Liu, Shu and Cui, Sasha and Kapoor, Sayash and Longpre, Shayne and Meng, Kevin and Weiss, Rebecca and Barez, Fazl and Gupta, Rahul and Dhamala, Jwala and Merizian, Jacob and Giulianelli, Mario and Coppock, Harry and Ududec, Cozmin and Kellermann, Antony and Sekhon, Jasjeet S and Steinhardt, Jacob and Schwettmann, Sarah and Narayanan, Arvind and Zaharia, Matei and Stoica, Ion and Liang, Percy and Kang, Daniel},
booktitle = {Advances in Neural Information Processing Systems},
year = {2025},
url = {https://mlanthology.org/neurips/2025/zhu2025neurips-establishing/}
}