Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech
Abstract
Learning to understand grounded language, which connects natural language to percepts, is a critical research area. Prior work in grounded language acquisition has focused primarily on textual inputs. In this work, we demonstrate the feasibility of performing grounded language acquisition on paired visual percepts and raw speech inputs. This will allow human-robot interactions in which language about novel tasks and environments is learned from end-users, reducing dependence on textual inputs and potentially mitigating the effects of demographic bias found in widely available speech recognition systems. We leverage recent work in self-supervised speech representation models and show that learned representations of speech can make language grounding systems more inclusive towards specific groups while maintaining or even increasing general performance.
Cite
Text
Kebe et al. "Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech." AAAI Conference on Artificial Intelligence, 2022. doi:10.1609/AAAI.V36I10.21335Markdown
[Kebe et al. "Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech." AAAI Conference on Artificial Intelligence, 2022.](https://mlanthology.org/aaai/2022/kebe2022aaai-bridging/) doi:10.1609/AAAI.V36I10.21335BibTeX
@inproceedings{kebe2022aaai-bridging,
title = {{Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech}},
author = {Kebe, Gaoussou Youssouf and Richards, Luke E. and Raff, Edward and Ferraro, Francis and Matuszek, Cynthia},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2022},
pages = {10884-10893},
doi = {10.1609/AAAI.V36I10.21335},
url = {https://mlanthology.org/aaai/2022/kebe2022aaai-bridging/}
}