Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments
Abstract
We develop a language-guided navigation task set in a continuous 3D environment where agents must execute low-level actions to follow natural language navigation directions. By being situated in continuous environments, this setting lifts a number of assumptions implicit in prior work that represents environments as a sparse graph of panoramas with edges corresponding to navigability. Specifically, our setting drops the presumptions of known environment topologies, short-range oracle navigation, and perfect agent localization. To contextualize this new task, we develop models that mirror many of the advances made in prior settings as well as single-modality baselines. While some transfer, we find significantly lower absolute performance in the continuous setting – suggesting that performance in prior ‘navigation-graph’ settings may be inflated by the strong implicit assumptions.
Cite
Text
Krantz et al. "Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments." Proceedings of the European Conference on Computer Vision (ECCV), 2020. doi:10.1007/978-3-030-58604-1_7Markdown
[Krantz et al. "Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments." Proceedings of the European Conference on Computer Vision (ECCV), 2020.](https://mlanthology.org/eccv/2020/krantz2020eccv-beyond/) doi:10.1007/978-3-030-58604-1_7BibTeX
@inproceedings{krantz2020eccv-beyond,
title = {{Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments}},
author = {Krantz, Jacob and Wijmans, Erik and Majumdar, Arjun and Batra, Dhruv and Lee, Stefan},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2020},
doi = {10.1007/978-3-030-58604-1_7},
url = {https://mlanthology.org/eccv/2020/krantz2020eccv-beyond/}
}