Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environments
Abstract
Recent work in Vision-and-Language Navigation (VLN) has presented two environmental paradigms with differing realism -- the standard VLN setting built on topological environments where navigation is abstracted away, and the VLN-CE setting where agents must navigate continuous 3D environments using low-level actions. Despite sharing the high-level task and even the underlying instruction-path data, performance on VLN-CE lags behind VLN significantly. In this work, we explore this gap by transferring an agent from the abstract environment of VLN to the continuous environment of VLN-CE. We find that this sim-2-sim transfer is highly effective, improving over the prior state of the art in VLN-CE by +12% success rate. While this demonstrates the potential for this direction, the transfer does not fully retain the original performance of the agent in the abstract setting. We present a sequence of experiments to identify what differences result in performance degradation, providing clear directions for further improvement.
Cite
Text
Krantz and Lee. "Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environments." Proceedings of the European Conference on Computer Vision (ECCV), 2022. doi:10.1007/978-3-031-19842-7_34Markdown
[Krantz and Lee. "Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environments." Proceedings of the European Conference on Computer Vision (ECCV), 2022.](https://mlanthology.org/eccv/2022/krantz2022eccv-sim2sim/) doi:10.1007/978-3-031-19842-7_34BibTeX
@inproceedings{krantz2022eccv-sim2sim,
title = {{Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environments}},
author = {Krantz, Jacob and Lee, Stefan},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2022},
doi = {10.1007/978-3-031-19842-7_34},
url = {https://mlanthology.org/eccv/2022/krantz2022eccv-sim2sim/}
}