Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models
Abstract
What latent features are encoded in language model (LM) representations? Recent work on training sparse autoencoders (SAEs) to disentangle interpretable features in LM representations has shown significant promise. However, evaluating the quality of these SAEs is difficult because we lack a ground-truth collection of interpretable features which we expect good SAEs to identify. We thus propose to measure progress in interpretable dictionary learning by working in the setting of LMs trained on Chess and Othello transcripts. These settings carry natural collections of interpretable features—for example, “there is a knight on F3”—which we leverage into metrics for SAE quality. To guide progress in interpretable dictionary learning, we introduce a new SAE training technique, $p$-annealing, which demonstrates improved performance on our metric.
Cite
Text
Karvonen et al. "Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models." Neural Information Processing Systems, 2024. doi:10.52202/079017-2644Markdown
[Karvonen et al. "Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models." Neural Information Processing Systems, 2024.](https://mlanthology.org/neurips/2024/karvonen2024neurips-measuring/) doi:10.52202/079017-2644BibTeX
@inproceedings{karvonen2024neurips-measuring,
title = {{Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models}},
author = {Karvonen, Adam and Wright, Benjamin and Rager, Can and Angell, Rico and Brinkmann, Jannik and Smith, Logan and Verdun, Claudio Mayrink and Bau, David and Marks, Samuel},
booktitle = {Neural Information Processing Systems},
year = {2024},
doi = {10.52202/079017-2644},
url = {https://mlanthology.org/neurips/2024/karvonen2024neurips-measuring/}
}