GPT-2 Small Fine-Tuned on Logical Reasoning Summarizes Information on Punctuation Tokens
Abstract
How is information stored and aggregated within a language model performing inference? Preliminary evidence suggests that representations of punctuation tokens might serve as ``summary points'' for information about preceding text. We add to this body of evidence by demonstrating that GPT-2 small fine-tuned on the RuleTaker logical inference dataset aggregates crucial information about rules and sentences above period tokens.
Cite
Text
Chauhan and Geiger. "GPT-2 Small Fine-Tuned on Logical Reasoning Summarizes Information on Punctuation Tokens." NeurIPS 2024 Workshops: MINT, 2024.Markdown
[Chauhan and Geiger. "GPT-2 Small Fine-Tuned on Logical Reasoning Summarizes Information on Punctuation Tokens." NeurIPS 2024 Workshops: MINT, 2024.](https://mlanthology.org/neuripsw/2024/chauhan2024neuripsw-gpt2/)BibTeX
@inproceedings{chauhan2024neuripsw-gpt2,
title = {{GPT-2 Small Fine-Tuned on Logical Reasoning Summarizes Information on Punctuation Tokens}},
author = {Chauhan, Sonakshi and Geiger, Atticus},
booktitle = {NeurIPS 2024 Workshops: MINT},
year = {2024},
url = {https://mlanthology.org/neuripsw/2024/chauhan2024neuripsw-gpt2/}
}