ML Anthology

Cached Transformers: Improving Transformers with Differentiable Memory Cachde

Zhaoyang Zhang, Wenqi Shao, Yixiao Ge, Xiaogang Wang, Jinwei Gu, Ping Luo
AAAI 2024 pp. 16935-16943
doi:10.1609/AAAI.V38I15.29636 /aaai/2024/zhang2024aaai-cached/

Abstract

This work introduces a new Transformer model called Cached Transformer, which uses Gated Recurrent Cached (GRC) attention to extend the self-attention mechanism with a differentiable memory cache of tokens. GRC attention enables attending to both past and current tokens, increasing the receptive field of attention and allowing for exploring long-range dependencies. By utilizing a recurrent gating unit to continuously update the cache, our model achieves significant advancements in \textbf{six} language and vision tasks, including language modeling, machine translation, ListOPs, image classification, object detection, and instance segmentation. Furthermore, our approach surpasses previous memory-based techniques in tasks such as language modeling and displays the ability to be applied to a broader range of situations.

PDF AAAI Semantic Scholar

Cite

Text

Zhang et al. "Cached Transformers: Improving Transformers with Differentiable Memory Cachde." AAAI Conference on Artificial Intelligence, 2024. doi:10.1609/AAAI.V38I15.29636

Markdown

[Zhang et al. "Cached Transformers: Improving Transformers with Differentiable Memory Cachde." AAAI Conference on Artificial Intelligence, 2024.](https://mlanthology.org/aaai/2024/zhang2024aaai-cached/) doi:10.1609/AAAI.V38I15.29636

BibTeX

@inproceedings{zhang2024aaai-cached,
  title     = {{Cached Transformers: Improving Transformers with Differentiable Memory Cachde}},
  author    = {Zhang, Zhaoyang and Shao, Wenqi and Ge, Yixiao and Wang, Xiaogang and Gu, Jinwei and Luo, Ping},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year      = {2024},
  pages     = {16935-16943},
  doi       = {10.1609/AAAI.V38I15.29636},
  url       = {https://mlanthology.org/aaai/2024/zhang2024aaai-cached/}
}