Adaptively Denoising Graph Neural Networks for Knowledge Distillation

Abstract

Graph Neural Networks (GNNs) have excelled in various graph-based applications. Recently, knowledge distillation (KD) has provided a new approach to further boost GNNs performance. However, in the KD process, the GNN student may encounter noise issues while learning from GNN teacher and input graph. GNN teachers may carry noise as deep models inevitably introduce noise during training, leading to error propagation in GNN students. Besides, noisy structures in input graph may also disrupt information during message-passing in GNNs. Hence, we propose DKDG to adaptively remove noise in GNN teacher and graph structure for better distillation. DKDG comprises two modules: (1) teacher knowledge denoising module, which separates GNN teacher knowledge into noise and label knowledge, and removes parameters fitting noise knowledge in the GNN student. (2) graph structure denoising module is designed to enhance node representations discrimination. Detailly, we propose a discrimination-preserving objective based on total variation loss and update edge weights between adjacent nodes to minimize this objective. These two modules are integrated through GNN’s forward propagation and trained iteratively. Experiments on five benchmark datasets and three GNNs demonstrate the GNN student distilled by DKDG gains $1.86\%$ 1.86 % relative improvement compared to the best baseline of recent state-of-the-art GNN-based KD methods.