VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Abstract
Process Reward Models (PRMs) have proven effective at enhancing mathematical reasoning for Large Language Models (LLMs) by leveraging increased inference-time computation. However, they are predominantly trained on mathematical data and their generalizability to non-mathematical domains has not been rigorously studied. In response, this work first shows that current PRMs have poor performance in other domains. To address this limitation, we introduce VersaPRM, a multi-domain PRM trained on synthetic reasoning data generated using our novel data generation and annotation method. VersaPRM achieves consistent performance gains across diverse domains. For instance, in the MMLU-Pro category of Law, VersaPRM via weighted majority voting, achieves a 7.9% performance gain over the majority voting baseline–surpassing Qwen2.5-Math-PRM’s gain of 1.3%. We further contribute to the community by open-sourcing all data, code and models for VersaPRM.
Cite
Text
Zeng et al. "VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data." Proceedings of the 42nd International Conference on Machine Learning, 2025.Markdown
[Zeng et al. "VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data." Proceedings of the 42nd International Conference on Machine Learning, 2025.](https://mlanthology.org/icml/2025/zeng2025icml-versaprm/)BibTeX
@inproceedings{zeng2025icml-versaprm,
title = {{VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data}},
author = {Zeng, Thomas and Zhang, Shuibai and Wu, Shutong and Classen, Christian and Chae, Daewon and Ewer, Ethan and Lee, Minjae and Kim, Heeju and Kang, Wonjun and Kunde, Jackson and Fan, Ying and Kim, Jungtaek and Koo, Hyung Il and Ramchandran, Kannan and Papailiopoulos, Dimitris and Lee, Kangwook},
booktitle = {Proceedings of the 42nd International Conference on Machine Learning},
year = {2025},
pages = {74197-74239},
volume = {267},
url = {https://mlanthology.org/icml/2025/zeng2025icml-versaprm/}
}