Retro*: Optimizing LLMs for Reasoning-Intensive Document Retrieval

Abstract

With the growing popularity of LLM agents and RAG, it has become increasingly important to retrieve documents that are essential for solving a task, even when their connection to the task is indirect or implicit. Addressing this problem requires fine-grained reasoning to accurately assess the relevance between the task and each candidate document. This capability, however, poses a significant challenge for existing IR techniques. Despite recent progress in reasoning-enhanced IR, existing approaches still face significant challenges in applicability, scalability, and efficiency. In this work, we propose **Retro\***, a novel approach for reasoning-intensive document retrieval. Our method introduces a rubric-based relevance **scoring mechanism**, enabling the model to reason about the relationship between a task and a document based on explicitly defined criteria, whereby producing a fine-grained, interpretable relevance score. Retro\* also supports **test-time scaling** by combining multiple reasoning trajectories via score integration, which produces more reliable relevance estimates. To optimize Retro\*'s reasoning capabilities, we introduce a novel **reinforcement learning** algorithm tailored for its relevance scoring mechanism, which employs two composite rewards to fully exploit the trajectories of each training sample. Our experiments show that Retro\* outperforms existing document retrieval methods with notable advantages, leading to **state-of-the-art** performance on the BRIGHT benchmark.