On the Prediction Instability of Graph Neural Networks
Abstract
Instability of trained models, i.e., the dependence of individual node predictions on random factors, can affect reproducibility, reliability, and trust in machine learning systems. In this paper, we systematically assess the prediction instability of node classification with state-of-the-art Graph Neural Networks (GNNs). With our experiments, we establish that multiple instantiations of popular GNN models trained on the same data with the same model hyperparameters result in almost identical aggregated performance, but display substantial disagreement in the predictions for individual nodes. We find that up to 30% of the incorrectly classified nodes differ across algorithm runs. We identify correlations between hyperparameters, node properties, and the size of the training set with the stability of predictions. In general, maximizing model performance implicitly also reduces model instability.
Cite
Text
Klabunde and Lemmerich. "On the Prediction Instability of Graph Neural Networks." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2022. doi:10.1007/978-3-031-26409-2_12Markdown
[Klabunde and Lemmerich. "On the Prediction Instability of Graph Neural Networks." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2022.](https://mlanthology.org/ecmlpkdd/2022/klabunde2022ecmlpkdd-prediction/) doi:10.1007/978-3-031-26409-2_12BibTeX
@inproceedings{klabunde2022ecmlpkdd-prediction,
title = {{On the Prediction Instability of Graph Neural Networks}},
author = {Klabunde, Max and Lemmerich, Florian},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2022},
pages = {187-202},
doi = {10.1007/978-3-031-26409-2_12},
url = {https://mlanthology.org/ecmlpkdd/2022/klabunde2022ecmlpkdd-prediction/}
}