The Concept Percolation Hypothesis: Analyzing the Emergence of Capabilities in Neural Networks Trained on Formal Grammars
Abstract
We analyze emergence of capabilities as a function of learning time, i.e., learning curve analysis. Training models on a well-defined, synthetic context-sensitive formal language, we find the existence of precise phases that separate the learning dynamics. Motivated by our results, we propose a qualitative theory grounded in the process of graph percolation that describes a mechanistic basis for how capabilities may be emerging in neural networks as they are trained on increasingly larger datasets.
Cite
Text
Lubana et al. "The Concept Percolation Hypothesis: Analyzing the Emergence of Capabilities in Neural Networks Trained on Formal Grammars." ICML 2024 Workshops: MI, 2024.Markdown
[Lubana et al. "The Concept Percolation Hypothesis: Analyzing the Emergence of Capabilities in Neural Networks Trained on Formal Grammars." ICML 2024 Workshops: MI, 2024.](https://mlanthology.org/icmlw/2024/lubana2024icmlw-concept/)BibTeX
@inproceedings{lubana2024icmlw-concept,
title = {{The Concept Percolation Hypothesis: Analyzing the Emergence of Capabilities in Neural Networks Trained on Formal Grammars}},
author = {Lubana, Ekdeep Singh and Kawaguchi, Kyogo and Dick, Robert P. and Tanaka, Hidenori},
booktitle = {ICML 2024 Workshops: MI},
year = {2024},
url = {https://mlanthology.org/icmlw/2024/lubana2024icmlw-concept/}
}