SCONER: Scoring Negative Candidates\\Before Training Neural Re-Ranker for Question Answering

Abstract

A neural re-ranker aims to re-scores a set of candidates given by a search engine. It is crucial to obtain good performance on many down-stream tasks such as retrieval-based question answering (ReQA). In this work, we introduce a scoring function for negative candidates to train a neural re-ranker and compare models trained by our approach with three baselines on a range of ReQA tasks. We term our approach as SCONER---scoring negative candidates before training neural re-ranker, which includes 1) a scoring function based on the concept of Semantic Textual Similarity (STS) and data augmentation; and 2) a neural re-ranker trained on data using generated negativeness scores as labels. Experimental results show that SCONER outperforms three baselines by up to 13\% absolute improvement on the SearchQA dataset and 5.5\% on average across all datasets in terms of P@1. SCONER demonstrates that using different negativeness scores to train a neural-ranker is better than a single score, and we present a simple yet efficient way to generate the scores.