ToolDec: Syntax Error-Free and Generalizable Tool Use for LLMs via Finite-State Decoding

Abstract

Large language models (LLMs) have shown promising capabilities in using external tools. However, existing approaches rely on fine-tuning or in-context learning to use tools, which make syntactic mistakes and are difficult to generalize. In this paper, we propose ToolDec, a finite-state machine-guided decoding algorithm for tool-augmented LLMs. ToolDec eliminates tool-related errors by ensuring valid tool names and type-conforming arguments. Furthermore, ToolDec enables LLM to effectively select tools using only the information contained in their names, with no need for tool-specific fine-tuning. Our experiments on multiple word problem datasets show that ToolDec reduces syntactic errors to zero, consequently achieving significantly better performance and as much as a 2x speedup. We also show that ToolDec achieves superior generalization performance on unseen tools, performing up to 8x better than the baseline.