Neural Networks Classify Through the Class-Wise Means of Their Representations

Mohamed El Amine Seddik, Mohamed Tamaazousti

AAAI 2022 pp. 8204-8211

doi:10.1609/AAAI.V36I8.20794 /aaai/2022/seddik2022aaai-neural/

Abstract

In this paper, based on an asymptotic analysis of the Softmax layer, we show that when training neural networks for classification tasks, the weight vectors corre sponding to each class of the Softmax layer tend to converge to the class-wise means computed at the representation layer (for specific choices of the representation activation). We further show some consequences of our findings to the context of transfer learning, essentially by proposing a simple yet effective initialization procedure that significantly accelerates the learning of the Softmax layer weights as the target domain gets closer to the source one. Experiments are notably performed on the datasets: MNIST, Fashion MNIST, Cifar10, and Cifar100 and using a standard CNN architecture.

PDF AAAI Semantic Scholar

Cite

Text

Seddik and Tamaazousti. "Neural Networks Classify Through the Class-Wise Means of Their Representations." AAAI Conference on Artificial Intelligence, 2022. doi:10.1609/AAAI.V36I8.20794

Markdown

[Seddik and Tamaazousti. "Neural Networks Classify Through the Class-Wise Means of Their Representations." AAAI Conference on Artificial Intelligence, 2022.](https://mlanthology.org/aaai/2022/seddik2022aaai-neural/) doi:10.1609/AAAI.V36I8.20794

BibTeX

@inproceedings{seddik2022aaai-neural,
  title     = {{Neural Networks Classify Through the Class-Wise Means of Their Representations}},
  author    = {Seddik, Mohamed El Amine and Tamaazousti, Mohamed},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year      = {2022},
  pages     = {8204-8211},
  doi       = {10.1609/AAAI.V36I8.20794},
  url       = {https://mlanthology.org/aaai/2022/seddik2022aaai-neural/}
}