Unsupervised Learning of Spoken Language with Visual Context
Abstract
Humans learn to speak before they can read or write, so why can’t computers do the same? In this paper, we present a deep neural network model capable of rudimentary spoken language acquisition using untranscribed audio training data, whose only supervision comes in the form of contextually relevant visual images. We describe the collection of our data comprised of over 120,000 spoken audio captions for the Places image dataset and evaluate our model on an image search and annotation task. We also provide some visualizations which suggest that our model is learning to recognize meaningful words within the caption spectrograms.
Cite
Text
Harwath et al. "Unsupervised Learning of Spoken Language with Visual Context." Neural Information Processing Systems, 2016.Markdown
[Harwath et al. "Unsupervised Learning of Spoken Language with Visual Context." Neural Information Processing Systems, 2016.](https://mlanthology.org/neurips/2016/harwath2016neurips-unsupervised/)BibTeX
@inproceedings{harwath2016neurips-unsupervised,
title = {{Unsupervised Learning of Spoken Language with Visual Context}},
author = {Harwath, David and Torralba, Antonio and Glass, James},
booktitle = {Neural Information Processing Systems},
year = {2016},
pages = {1858-1866},
url = {https://mlanthology.org/neurips/2016/harwath2016neurips-unsupervised/}
}