Positional Information Can Emerge Through Causal Attention Making Nearby Token Embeddings Similar Even Without Positional Encodings
Abstract
Transformers with causal attention can solve tasks that require positional information without using positional encodings. In this work, we propose and investigate a new hypothesis about how positional information can be stored without using explicit positional encoding. We observe that nearby embeddings are more similar to each other than faraway embeddings, allowing the transformer to potentially reconstruct the positions of tokens. We show that this pattern can occur in both the trained and the randomly initialized Transformer models with causal attention and no positional encodings over a common range of hyperparameters.
Cite
Text
Zuo et al. "Positional Information Can Emerge Through Causal Attention Making Nearby Token Embeddings Similar Even Without Positional Encodings." NeurIPS 2024 Workshops: InterpretableAI, 2024.Markdown
[Zuo et al. "Positional Information Can Emerge Through Causal Attention Making Nearby Token Embeddings Similar Even Without Positional Encodings." NeurIPS 2024 Workshops: InterpretableAI, 2024.](https://mlanthology.org/neuripsw/2024/zuo2024neuripsw-positional/)BibTeX
@inproceedings{zuo2024neuripsw-positional,
title = {{Positional Information Can Emerge Through Causal Attention Making Nearby Token Embeddings Similar Even Without Positional Encodings}},
author = {Zuo, Chunsheng and Guerzhoy, Pavel and Guerzhoy, Michael},
booktitle = {NeurIPS 2024 Workshops: InterpretableAI},
year = {2024},
url = {https://mlanthology.org/neuripsw/2024/zuo2024neuripsw-positional/}
}