Linear Decoding of Morphology Relations in Language Models (Student Abstract)

AAAI 2025 pp. 29532-29534

doi:10.1609/AAAI.V39I28.35315 /aaai/2025/xia2025aaai-linear/

Abstract

The recent success of transformer language models owes much to their conversational fluency, which includes linguistic and morphological proficiency. An affine Taylor approximation has been found to be a good approximation for transformer computations over certain factual and encyclopedic relations. We show that the truly linear approximation W s, where s is a early layer representation of the base form and W is a local model derivative, is necessary and sufficient to approximate morphological derivation, achieving above 80% top-1 accuracy across most morphological tasks in the Bigger Analogy Test Set. We argue that many morphological forms in transformer models are likely linearly encoded.

PDF AAAI Semantic Scholar

Cite

Text

Xia and Kalita. "Linear Decoding of Morphology Relations in Language Models (Student Abstract)." AAAI Conference on Artificial Intelligence, 2025. doi:10.1609/AAAI.V39I28.35315

Markdown

[Xia and Kalita. "Linear Decoding of Morphology Relations in Language Models (Student Abstract)." AAAI Conference on Artificial Intelligence, 2025.](https://mlanthology.org/aaai/2025/xia2025aaai-linear/) doi:10.1609/AAAI.V39I28.35315

BibTeX

@inproceedings{xia2025aaai-linear,
  title     = {{Linear Decoding of Morphology Relations in Language Models (Student Abstract)}},
  author    = {Xia, Eric and Kalita, Jugal},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year      = {2025},
  pages     = {29532-29534},
  doi       = {10.1609/AAAI.V39I28.35315},
  url       = {https://mlanthology.org/aaai/2025/xia2025aaai-linear/}
}