Escape Sky-High Cost: Early-Stopping Self-Consistency for Multi-Step Reasoning
Abstract
Self-consistency (SC) has been a widely used decoding strategy for chain-of-thought reasoning. Despite bringing significant performance improvements across a variety of multi-step reasoning tasks, it is a high-cost method that requires multiple sampling with the preset size. In this paper, we propose a simple and scalable sampling process, Early-Stopping Self-Consistency (ESC), to greatly reduce the cost of SC without sacrificing performance. On this basis, one control scheme for ESC is further derivated to dynamically choose the performance-cost balance for different tasks and models. To demonstrate ESC's effectiveness, we conducted extensive experiments on three popular categories of reasoning tasks: arithmetic, commonsense and symbolic reasoning over language models with varying scales. The empirical results show that ESC reduces the average number of sampling of chain-of-thought reasoning by a significant margin on six benchmarks, including MATH (-33.8%), GSM8K (-80.1%), StrategyQA (-76.8%), CommonsenseQA (-78.5%), Coin Flip (-84.2%) and Last Letters (-67.4%), while attaining comparable performances.
Cite
Text
Li et al. "Escape Sky-High Cost: Early-Stopping Self-Consistency for Multi-Step Reasoning." International Conference on Learning Representations, 2024.Markdown
[Li et al. "Escape Sky-High Cost: Early-Stopping Self-Consistency for Multi-Step Reasoning." International Conference on Learning Representations, 2024.](https://mlanthology.org/iclr/2024/li2024iclr-escape/)BibTeX
@inproceedings{li2024iclr-escape,
title = {{Escape Sky-High Cost: Early-Stopping Self-Consistency for Multi-Step Reasoning}},
author = {Li, Yiwei and Yuan, Peiwen and Feng, Shaoxiong and Pan, Boyuan and Wang, Xinglin and Sun, Bin and Wang, Heda and Li, Kan},
booktitle = {International Conference on Learning Representations},
year = {2024},
url = {https://mlanthology.org/iclr/2024/li2024iclr-escape/}
}