ML Anthology

What's Hidden in a Randomly Weighted Neural Network?

Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, Mohammad Rastegari
CVPR 2020
/cvpr/2020/ramanujan2020cvpr-hidden/

Abstract

Training a neural network is synonymous with learning the values of the weights. By contrast, we demonstrate that randomly weighted neural networks contain subnetworks which achieve impressive performance without ever training the weight values. Hidden in a randomly weighted Wide ResNet-50 is a subnetwork (with random weights) that is smaller than, but matches the performance of a ResNet-34 trained on ImageNet. Not only do these "untrained subnetworks" exist, but we provide an algorithm to effectively find them. We empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an "untrained subnetwork" approaches a network with learned weights in accuracy.

PDF CVPR Semantic Scholar

Cite

Text

Ramanujan et al. "What's Hidden in a Randomly Weighted Neural Network?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.

Markdown

[Ramanujan et al. "What's Hidden in a Randomly Weighted Neural Network?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.](https://mlanthology.org/cvpr/2020/ramanujan2020cvpr-hidden/)

BibTeX

@inproceedings{ramanujan2020cvpr-hidden,
  title     = {{What's Hidden in a Randomly Weighted Neural Network?}},
  author    = {Ramanujan, Vivek and Wortsman, Mitchell and Kembhavi, Aniruddha and Farhadi, Ali and Rastegari, Mohammad},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year      = {2020},
  url       = {https://mlanthology.org/cvpr/2020/ramanujan2020cvpr-hidden/}
}