Banyan: Improved Representation Learning with Explicit Structure
Abstract
We present Banyan, a model that efficiently learns semantic representations by leveraging explicit hierarchical structure. While transformers excel at scale, they struggle in low-resource settings. Conversely recent structured models have shown promise as efficient learners, but lack performance. Banyan bridges this gap with two key innovations: an entangled hierarchical tree structure and diagonalized message passing, enabling it to outperform larger transformer models with just 14 non-embedding parameters. It excels in low-resource settings, offering a viable alternative for under-represented languages and highlighting its potential for efficient, interpretable NLP in resource-constrained environments.
Cite
Text
Opper and N. "Banyan: Improved Representation Learning with Explicit Structure." Proceedings of the 42nd International Conference on Machine Learning, 2025.Markdown
[Opper and N. "Banyan: Improved Representation Learning with Explicit Structure." Proceedings of the 42nd International Conference on Machine Learning, 2025.](https://mlanthology.org/icml/2025/opper2025icml-banyan/)BibTeX
@inproceedings{opper2025icml-banyan,
title = {{Banyan: Improved Representation Learning with Explicit Structure}},
author = {Opper, Mattia and N, Siddharth},
booktitle = {Proceedings of the 42nd International Conference on Machine Learning},
year = {2025},
pages = {47287-47301},
volume = {267},
url = {https://mlanthology.org/icml/2025/opper2025icml-banyan/}
}