WoodFisher: Efficient Second-Order Approximation for Neural Network Compression

Abstract

Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been significant interest in utilizing this information in the context of deep neural networks; however, relatively little is known about the quality of existing approximations in this context. Our work considers this question, examines the accuracy of existing approaches, and proposes a method called WoodFisher to compute a faithful and efficient estimate of the inverse Hessian.

PDF NeurIPS Semantic Scholar

Cite

Text

Singh and Alistarh. "WoodFisher: Efficient Second-Order Approximation for Neural Network Compression." Neural Information Processing Systems, 2020.

Markdown

[Singh and Alistarh. "WoodFisher: Efficient Second-Order Approximation for Neural Network Compression." Neural Information Processing Systems, 2020.](https://mlanthology.org/neurips/2020/singh2020neurips-woodfisher/)

BibTeX

@inproceedings{singh2020neurips-woodfisher,
  title     = {{WoodFisher: Efficient Second-Order Approximation for Neural Network Compression}},
  author    = {Singh, Sidak Pal and Alistarh, Dan},
  booktitle = {Neural Information Processing Systems},
  year      = {2020},
  url       = {https://mlanthology.org/neurips/2020/singh2020neurips-woodfisher/}
}