ML Anthology

DataPerf: Benchmarks for Data-Centric AI Development

Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Alicia Parrish, Hannah Rose Kirk, Jessica Quaye, Charvi Rastogi, Douwe Kiela, David Jurado, David Kanter, Rafael Mosquera, Will Cukierski, Juan Ciro, Lora Aroyo, Bilge Acun, Lingjiao Chen, Mehul Raje, Max Bartolo, Evan Sabri Eyuboglu, Amirata Ghorbani, Emmett Goodman, Addison Howard, Oana Inel, Tariq Kane, Christine R. Kirkpatrick, D. Sculley, Tzu-Sheng Kuo, Jonas W Mueller, Tristan Thrush, Joaquin Vanschoren, Margaret Warren, Adina Williams, Serena Yeung, Newsha Ardalani, Praveen Paritosh, Ce Zhang, James Y Zou, Carole-Jean Wu, Cody Coleman, Andrew Y. Ng, Peter Mattson, Vijay Janapa Reddi
NeurIPS 2023
/neurips/2023/mazumder2023neurips-dataperf/

Abstract

Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithfulness of the underlying problems. Neglecting the fundamental importance of data has given rise to inaccuracy, bias, and fragility in real-world applications, and research is hindered by saturation across existing dataset benchmarks. In response, we present DataPerf, a community-led benchmark suite for evaluating ML datasets and data-centric algorithms. We aim to foster innovation in data-centric AI through competition, comparability, and reproducibility. We enable the ML community to iterate on datasets, instead of just architectures, and we provide an open, online platform with multiple rounds of challenges to support this iterative development. The first iteration of DataPerf contains five benchmarks covering a wide spectrum of data-centric techniques, tasks, and modalities in vision, speech, acquisition, debugging, and diffusion prompting, and we support hosting new contributed benchmarks from the community. The benchmarks, online evaluation platform, and baseline implementations are open source, and the MLCommons Association will maintain DataPerf to ensure long-term benefits to academia and industry.

PDF NeurIPS OpenReview Semantic Scholar

Cite

Text

Mazumder et al. "DataPerf: Benchmarks for Data-Centric AI Development." Neural Information Processing Systems, 2023.

Markdown

[Mazumder et al. "DataPerf: Benchmarks for Data-Centric AI Development." Neural Information Processing Systems, 2023.](https://mlanthology.org/neurips/2023/mazumder2023neurips-dataperf/)

BibTeX

@inproceedings{mazumder2023neurips-dataperf,
  title     = {{DataPerf: Benchmarks for Data-Centric AI Development}},
  author    = {Mazumder, Mark and Banbury, Colby and Yao, Xiaozhe and Karlaš, Bojan and Rojas, William Gaviria and Diamos, Sudnya and Diamos, Greg and He, Lynn and Parrish, Alicia and Kirk, Hannah Rose and Quaye, Jessica and Rastogi, Charvi and Kiela, Douwe and Jurado, David and Kanter, David and Mosquera, Rafael and Cukierski, Will and Ciro, Juan and Aroyo, Lora and Acun, Bilge and Chen, Lingjiao and Raje, Mehul and Bartolo, Max and Eyuboglu, Evan Sabri and Ghorbani, Amirata and Goodman, Emmett and Howard, Addison and Inel, Oana and Kane, Tariq and Kirkpatrick, Christine R. and Sculley, D. and Kuo, Tzu-Sheng and Mueller, Jonas W and Thrush, Tristan and Vanschoren, Joaquin and Warren, Margaret and Williams, Adina and Yeung, Serena and Ardalani, Newsha and Paritosh, Praveen and Zhang, Ce and Zou, James Y and Wu, Carole-Jean and Coleman, Cody and Ng, Andrew Y. and Mattson, Peter and Reddi, Vijay Janapa},
  booktitle = {Neural Information Processing Systems},
  year      = {2023},
  url       = {https://mlanthology.org/neurips/2023/mazumder2023neurips-dataperf/}
}