Layer-Wise Quantization for Distributed Variational Inequalities

Abstract

We develop a general layer-wise and adaptive quantization framework with with error and code-length guarantees and applications to solving large-scale distributed variational inequality problems. We also propose Quantized and Generalized Optimistic Dual Averaging (QODA) which achieves the optimal rate of convergence for distributed monotone VIs under absolute noise. We empirically show that the adaptive layer-wise quantization achieves up to a $47$% speedup in end-to-end training time for training Wasserstein GAN on $4$ GPUs.

PDF NeurIPSW OpenReview Semantic Scholar

Cite

Text

Nguyen et al. "Layer-Wise Quantization for Distributed Variational Inequalities." NeurIPS 2024 Workshops: Compression, 2024.

Markdown

[Nguyen et al. "Layer-Wise Quantization for Distributed Variational Inequalities." NeurIPS 2024 Workshops: Compression, 2024.](https://mlanthology.org/neuripsw/2024/nguyen2024neuripsw-layerwise/)

BibTeX

@inproceedings{nguyen2024neuripsw-layerwise,
  title     = {{Layer-Wise Quantization for Distributed Variational Inequalities}},
  author    = {Nguyen, Anh Duc and Markov, Ilia and Ramezani-Kebrya, Ali and Antonakopoulos, Kimon and Alistarh, Dan and Cevher, Volkan},
  booktitle = {NeurIPS 2024 Workshops: Compression},
  year      = {2024},
  url       = {https://mlanthology.org/neuripsw/2024/nguyen2024neuripsw-layerwise/}
}