Huge Frozen Language Models as Readers for Open-Domain Question Answering

Abstract

In the open-book variant of the open domain question answering setting, an answer generator typically attends to 100+ retrieved documents when answering, and is thus often called a "reader". Current readers are fine tuned for this long-context functionality. Because it is prohibitively expensive to fine tune huge models to attend to 100+ retrieved documents, readers tend to be relatively small, typically having fewer than 1B parameters. We introduce huge LMs into this pipeline as frozen readers. To do so, we use a re-ranking stage to condense relevant information from 100+ retrieved documents into the input sequence length of the frozen LM reader. We show that frozen LMs can reach and surpass leading fine tuning approaches on Natural Questions, a prominent open-domain question answering benchmark.