ML Anthology

Towards Memory-Efficient Training for Extremely Large Output Spaces - Learning with 670k Labels on a Single Commodity GPU

Erik Schultheis, Rohit Babbar
ECML-PKDD 2023 pp. 689-704
doi:10.1007/978-3-031-43418-1_41 /ecmlpkdd/2023/schultheis2023ecmlpkdd-memoryefficient/

Abstract

In classification problems with large output spaces (up to millions of labels), the last layer can require an enormous amount of memory. Using sparse connectivity would drastically reduce the memory requirements, but as we show below, applied naïvely it can result in much diminished predictive performance. Fortunately, we found that this can be mitigated by introducing an intermediate layer of intermediate size. We further demonstrate that one can constrain the connectivity of the sparse layer to be of constant fan-in, in the sense that each output neuron will have the exact same number of incoming connections, which allows for more efficient implementations, especially on GPU hardware. The CUDA implementation of our approach is provided at https://github.com/xmc-aalto/ecml23-sparse .

PDF ECML-PKDD Semantic Scholar

Cite

Text

Schultheis and Babbar. "Towards Memory-Efficient Training for Extremely Large Output Spaces - Learning with 670k Labels on a Single Commodity GPU." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2023. doi:10.1007/978-3-031-43418-1_41

Markdown

[Schultheis and Babbar. "Towards Memory-Efficient Training for Extremely Large Output Spaces - Learning with 670k Labels on a Single Commodity GPU." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2023.](https://mlanthology.org/ecmlpkdd/2023/schultheis2023ecmlpkdd-memoryefficient/) doi:10.1007/978-3-031-43418-1_41

BibTeX

@inproceedings{schultheis2023ecmlpkdd-memoryefficient,
  title     = {{Towards Memory-Efficient Training for Extremely Large Output Spaces - Learning with 670k Labels on a Single Commodity GPU}},
  author    = {Schultheis, Erik and Babbar, Rohit},
  booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
  year      = {2023},
  pages     = {689-704},
  doi       = {10.1007/978-3-031-43418-1_41},
  url       = {https://mlanthology.org/ecmlpkdd/2023/schultheis2023ecmlpkdd-memoryefficient/}
}