A Reality Check on Pre-Training for Exemplar-Free Class-Incremental Learning

Abstract

Exemplar-free class-incremental learning (EFCIL) aims to classify streaming data without storing examples from the past. Recent EFCIL works suggest that (i) models pre-trained with large amounts of data should be used to initialize learning (ii) self-supervised learned transformers generalize better than supervised convolutional models (iii) adding generated data to the pre-training dataset can improve incremental accuracy. In this article we question the above assertions by comprehensively evaluating various initial training strategies combined with four EFCIL algorithms using four large-scale datasets. Our results indicate that: (i) pre-trained models are preferable when the domain of the incremental classification task is well represented in the pre-training datasets but training with initial data remains useful when the domain shift is significant (ii) supervised convolutional networks remain competitive particularly when improving representation transferability using data augmentation or a projector (iii) adding classes from an external dataset to train the initial model boosts performance when the initial set of classes is small but has a limited effect otherwise (iv) additional classes generated with a diffusion model are not necessarily more useful than a well-chosen set of ImageNet classes to improve model transferability. We provide a nuanced analysis of these results and formulate recommendations to facilitate the practical adoption of EFCIL algorithms.