Exploring and Improving Drafts in Blockwise Parallel Decoding

Abstract

Blockwise parallel decoding (BPD) was proposed in Stern et al. (2018) as a method to improve the inference speed of language models by simultaneously predicting multiple future tokens, termed block drafts, which are subsequently verified by the autoregressive model. Block drafts are generated by multiple independent prediction heads of blockwise parallel language models. This paper contributes to the understanding and improvement of block drafts in two ways. First, we analyze the token distributions produced by multiple prediction heads. Secondly, we leverage this analysis to develop algorithms to improve BPD inference speed by refining the block drafts using n-gram and neural language models. Experiments demonstrate that refined block drafts yield a +5-21% increase in block efficiency (i.e., the number of accepted tokens from the block draft) across diverse datasets.