Influence Estimation in Statistical Models Using the Fisher Information Matrix

Abstract

Quantifying how infinitesimal perturbations of training data affect a model is key to diagnosing and improving learning systems. This task was addressed via the notion of influence functions \citep{Hampel_Influence, KohLiang_Influence_DL, koh2019accuracy, Bae_PBRF_Influence}. Following classical works, whenever the underlying problem can be cast as a weighted empirical risk minimization problem, many such influence estimators rely on the Fisher Information Matrix (FIM). Following these lines, we provide a new accuracy analysis that characterizes the asymptotic behavior of FIM-based influence estimators and compare these to Hessian-based influence estimators, while we further extend the theory to objectives with non-differentiable regularizers. The results obtained are broadly applicable and admit an efficient algorithm with favorable computational complexity. Simulations on realistic setups demonstrate its usefulness in terms of accuracy and computational efficiency in many learning settings.