Rainbow Padding: Mitigating Early Termination in Instruction-Tuned Diffusion LLMs

Abstract

Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive models, offering flexible generation orders and strong performance on complex reasoning tasks. However, instruction-tuned dLLMs exhibit a critical vulnerability we term \<eos\> overflow: as allocated sequence length increases, responses paradoxically become shorter, collapsing into early termination or degenerating into streams of \<eos\> tokens. Although noticed in practice, this issue has not been systematically analyzed. We trace its root cause to the dual role of \<eos\> as both termination and padding, which concentrates probability mass on \<eos\> at later positions and propagates backward to trigger early termination. To address this, we introduce Rainbow Padding, a simple remedy that replaces repeated \<eos\> placeholders with a repeating cycle of distinct padding tokens, distributing probability mass and breaking \<eos\> dominance. Experiments show that Rainbow Padding substantially improves length robustness and output quality, with as few as seven padding tokens to prevent early termination. Moreover, the method integrates efficiently into existing instruction-tuned models: LoRA fine-tuning for a single epoch on minimal data yields significant improvements, making this solution highly practical. The project is available at ~\url{https://ai-isl.github.io/rainbow-padding}