PivotAlign: Improve Semi-Supervised Learning by Learning Intra-Class Heterogeneity and Aligning with Pivots

Abstract

Self-supervised learning plays an important role in current state-of-the-art semi-supervised learning (SSL) methods. These methods learn inter-class heterogeneity among data and generate pseudo-labels based on class level representations. However they often neglect intra-class heterogeneity resulting in the under-exploitation of finer-grained semantic relationships within classes. To address this limitation we introduce PivotAlign a novel SSL approach that aims to 1) learn hierarchical representations to detect both inter-class and intra-class semantic relationships and 2) refine pseudo-labels based on learned representations with a class-debiasing strategy. Specifically we first learn a set of pivots as sub-prototypes of classes. We then train representations so that features align with the assigned pivot and are hierarchically grouped based on both inter-class and intra-class heterogeneity. This allows us to capture both inter-class and intra-class semantic relationships among data and leverage them to better assign and refine pseudo-labels. Additionally since SSL methods are prone to bias toward classes that are easier to learn we further re-balance class predictions to alleviate this class bias. We demonstrate the effectiveness of PivotAlign on various SSL benchmarks where PivotAlign achieves state-of-the-art performances. The source code will be released upon publication of the work.