Learning the Greatest Divisor - Explainable Predictions in Transformers
Abstract
We train small transformers to calculate the greatest common divisor (GCD) of two positive integers, and show that their predictions are fully explainable. During training, models learn a list $\mathcal D$ of divisors, and predict the largest element of $\mathcal D$ that divides both inputs. We also show that training distributions have a large impact on performance. Models trained from uniform operands only learn a handful of GCD (up to $38$ out of $100$). Training from log-uniform operands boosts performance to $73$ correct GCD, and training from a log-uniform distribution of GCD to $91$.
Cite
Text
Charton. "Learning the Greatest Divisor - Explainable Predictions in Transformers." NeurIPS 2023 Workshops: MATH-AI, 2023.Markdown
[Charton. "Learning the Greatest Divisor - Explainable Predictions in Transformers." NeurIPS 2023 Workshops: MATH-AI, 2023.](https://mlanthology.org/neuripsw/2023/charton2023neuripsw-learning/)BibTeX
@inproceedings{charton2023neuripsw-learning,
title = {{Learning the Greatest Divisor - Explainable Predictions in Transformers}},
author = {Charton, Francois},
booktitle = {NeurIPS 2023 Workshops: MATH-AI},
year = {2023},
url = {https://mlanthology.org/neuripsw/2023/charton2023neuripsw-learning/}
}