Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems

Abstract

In task-oriented dialogue (ToD), a user holds a conversation with an artificial agent  with the aim of completing a concrete task. Although this technology represents one of  the central objectives of AI and has been the focus of ever more intense research and  development efforts, it is currently limited to a few narrow domains (e.g., food ordering,  ticket booking) and a handful of languages (e.g., English, Chinese). This work provides an  extensive overview of existing methods and resources in multilingual ToD as an entry point  to this exciting and emerging field. We find that the most critical factor preventing the  creation of truly multilingual ToD systems is the lack of datasets in most languages for  both training and evaluation. In fact, acquiring annotations or human feedback for each  component of modular systems or for data-hungry end-to-end systems is expensive and  tedious. Hence, state-of-the-art approaches to multilingual ToD mostly rely on (zero- or  few-shot) cross-lingual transfer from resource-rich languages (almost exclusively English),  either by means of (i) machine translation or (ii) multilingual representations. These  approaches are currently viable only for typologically similar languages and languages with  parallel / monolingual corpora available. On the other hand, their effectiveness beyond these  boundaries is doubtful or hard to assess due to the lack of linguistically diverse benchmarks  (especially for natural language generation and end-to-end evaluation). To overcome this  limitation, we draw parallels between components of the ToD pipeline and other NLP tasks,  which can inspire solutions for learning in low-resource scenarios. Finally, we list additional  challenges that multilinguality poses for related areas (such as speech, fluency in generated  text, and human-centred evaluation), and indicate future directions that hold promise to  further expand language coverage and dialogue capabilities of current ToD systems.