BenchOverflow: Measuring Overflow in Large Language Models via Plain-Text Prompts

Abstract

We investigate a failure mode of large language models (LLMs) in which benign, plain-text prompts elicit excessive outputs, a phenomenon we term Overflow. Unlike jailbreaks or prompt injection, Overflow arises under ordinary interaction settings and carries concrete risks for denial-of-wallet, latency, and cross-user performance degradation. We introduce BenchOverflow, a model-agnostic benchmark of nine plain-text prompting strategies that amplify output volume without adversarial suffixes or policy circumvention. Using a standardized protocol with a fixed budget of 5,000 new tokens, we evaluate BenchOverflow on nine open- and closed-source models. Across models, BenchOverflow produces pronounced rightward shifts and heavy tails in length distributions. Cap-saturation rates (CSR@1k/3k/5k) and empirical cumulative distribution functions (ECDFs) quantify tail risk; within-prompt variance and cross-model correlations show that Overflow is broadly reproducible yet heterogeneous across families and attack vectors. A lightweight mitigation—a fixed conciseness reminder—attenuates right tails and lowers CSR for several strategies. Our findings reframe verbosity as a measurable risk to reliability and cost, rather than a mere stylistic quirk. BenchOverflow provides a practical, reproducible protocol for benchmarking length-control robustness in deployed LLMs.