Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon
Abstract
Memorization in language models is typically treated as a homogenous phenomenon, neglecting the specifics of the memorized data. We instead model memorization as the effect of a set of complex factors that describe each sample and relate it to the model and corpus. To build intuition around these factors, we break memorization down into a taxonomy: recitation of highly duplicated sequences, reconstruction of inherently predictable sequences, and recollection of sequences that are neither. We demonstrate the usefulness of our taxonomy by using it to construct a predictive model for memorization. By analyzing dependencies and inspecting the weights of the predictive model, we find that different factors have different influences on the likelihood of memorization depending on the taxonomic category.
Cite
Text
Prashanth et al. "Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon." International Conference on Learning Representations, 2025.Markdown
[Prashanth et al. "Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon." International Conference on Learning Representations, 2025.](https://mlanthology.org/iclr/2025/prashanth2025iclr-recite/)BibTeX
@inproceedings{prashanth2025iclr-recite,
title = {{Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon}},
author = {Prashanth, USVSN Sai and Deng, Alvin and O'Brien, Kyle and Jyothir, S V and Khan, Mohammad Aflah and Borkar, Jaydeep and Choquette-Choo, Christopher A. and Fuehne, Jacob Ray and Biderman, Stella and Ke, Tracy and Lee, Katherine and Saphra, Naomi},
booktitle = {International Conference on Learning Representations},
year = {2025},
url = {https://mlanthology.org/iclr/2025/prashanth2025iclr-recite/}
}