The Curious Case of Representational Alignment: Unravelling Visio-Linguistic Tasks in Emergent Communication

Abstract

Natural language has the universal properties of being compositional and grounded in the real world. A popular method to investigate the emergence of linguistic properties is by simulating emergent communication setups with deep neural agents in referential games. Despite growing interest, experiments have yielded mixed results compared to similar experiments addressing linguistic properties of human language. Here we address representational alignment as a potential contributing factor to these results. Specifically, we investigate the alignment between agent image representations and between agent representations and the input images. We first revisit and confirm that the emergent language in the common referential game does not appear to encode conceptual visual features, since agent image representations drift away from the input whilst inter-agent alignment increases. We further find a strong relationship between inter-agent alignment and topographic similarity, a common metric for compositionality, and address its consequences. We then introduce an alignment penalty that results in equivalent communicative success but prevents representational drift. Overall, we show critical differences between emergent solutions from humans and neural agents and highlight the importance of representational alignment in simulations of language emergence.