ML Anthology

Releasing the CRaQAn (Coreference Resolution in Question-Answering): An Open-Source Dataset and Dataset Creation Methodology Using Instruction-Following Models

Rob Grzywinski, Joshua D'Arcy, Robert Naidoff, Ashish Shukla, Alex Browne, Ren Gibbons, Brinnae Bent
NeurIPSW 2023
/neuripsw/2023/grzywinski2023neuripsw-releasing/

Abstract

Instruction-following language models demand robust methodologies for information retrieval to augment instructions for question-answering applications. A primary challenge is the resolution of coreferences in the context of chunking strategies for long documents. The critical barrier to experimentation of handling coreferences is a lack of open source datasets, specifically in question-answering tasks that require coreference resolution. In this work we present our Coreference Resolution in Question-Answering (CRaQAn) dataset, an open-source dataset that caters to the nuanced information retrieval requirements of coreference resolution in question-answering tasks by providing over 250 question-answer pairs containing coreferences. To develop this dataset, we developed a novel approach for creating high-quality datasets using an instruction-following model (GPT-4) and a Recursive Criticism and Improvement Loop.

PDF NeurIPSW OpenReview Semantic Scholar

Cite

Text

Grzywinski et al. "Releasing the CRaQAn (Coreference Resolution in Question-Answering): An Open-Source Dataset and Dataset Creation Methodology Using Instruction-Following Models." NeurIPS 2023 Workshops: Instruction, 2023.

Markdown

[Grzywinski et al. "Releasing the CRaQAn (Coreference Resolution in Question-Answering): An Open-Source Dataset and Dataset Creation Methodology Using Instruction-Following Models." NeurIPS 2023 Workshops: Instruction, 2023.](https://mlanthology.org/neuripsw/2023/grzywinski2023neuripsw-releasing/)

BibTeX

@inproceedings{grzywinski2023neuripsw-releasing,
  title     = {{Releasing the CRaQAn (Coreference Resolution in Question-Answering): An Open-Source Dataset and Dataset Creation Methodology Using Instruction-Following Models}},
  author    = {Grzywinski, Rob and D'Arcy, Joshua and Naidoff, Robert and Shukla, Ashish and Browne, Alex and Gibbons, Ren and Bent, Brinnae},
  booktitle = {NeurIPS 2023 Workshops: Instruction},
  year      = {2023},
  url       = {https://mlanthology.org/neuripsw/2023/grzywinski2023neuripsw-releasing/}
}