The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben Allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, Yacine Jernite

NeurIPS 2022

/neurips/2022/laurencon2022neurips-bigscience/

Abstract

As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

PDF NeurIPS OpenReview Semantic Scholar

Cite

Text

Laurençon et al. "The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset." Neural Information Processing Systems, 2022.

Markdown

[Laurençon et al. "The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset." Neural Information Processing Systems, 2022.](https://mlanthology.org/neurips/2022/laurencon2022neurips-bigscience/)

BibTeX

@inproceedings{laurencon2022neurips-bigscience,
  title     = {{The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset}},
  author    = {Laurençon, Hugo and Saulnier, Lucile and Wang, Thomas and Akiki, Christopher and del Moral, Albert Villanova and Le Scao, Teven and Von Werra, Leandro and Mou, Chenghao and Ponferrada, Eduardo González and Nguyen, Huu and Frohberg, Jörg and Šaško, Mario and Lhoest, Quentin and McMillan-Major, Angelina and Dupont, Gerard and Biderman, Stella and Rogers, Anna and Allal, Loubna Ben and De Toni, Francesco and Pistilli, Giada and Nguyen, Olivier and Nikpoor, Somaieh and Masoud, Maraim and Colombo, Pierre and de la Rosa, Javier and Villegas, Paulo and Thrush, Tristan and Longpre, Shayne and Nagel, Sebastian and Weber, Leon and Muñoz, Manuel and Zhu, Jian and Van Strien, Daniel and Alyafeai, Zaid and Almubarak, Khalid and Vu, Minh Chien and Gonzalez-Dios, Itziar and Soroa, Aitor and Lo, Kyle and Dey, Manan and Suarez, Pedro Ortiz and Gokaslan, Aaron and Bose, Shamik and Adelani, David and Phan, Long and Tran, Hieu and Yu, Ian and Pai, Suhas and Chim, Jenny and Lepercq, Violette and Ilic, Suzana and Mitchell, Margaret and Luccioni, Sasha Alexandra and Jernite, Yacine},
  booktitle = {Neural Information Processing Systems},
  year      = {2022},
  url       = {https://mlanthology.org/neurips/2022/laurencon2022neurips-bigscience/}
}