Geometric Interpretation of a CNN's Last Layer

Abstract

Training Convolutional Neural Networks (CNNs) remains a non-trivial task that in many cases relies on the skills and experience of the person conducting the training. Choosing hyperparameters, knowing when the training should be interrupted, or even when to stop trying training strategies are some difficult decisions that have to be made. These decisions are difficult partly because we still know little about the internal behaviour of CNNs, especially during training. In this work we conduct a methodical experimentation on MNIST public database of handwritten digits to better understand the evolution of the last layer from a geometric perspective: namely the classification vectors and the image embedding vectors. Within this context we present the problem of the variability across equal set-up trainings due to the random component of the initialisation method. We propose a novel approach that guides the initialisation of the parameters in the classification layer. This method reduces 12% the variability across repetitions and leads to accuracies 18% higher on average.