Quantifying Model Predictive Uncertainty with Perturbation Theory Rishabh Singh Jose C. Principe Computational NeuroEngineering Lab (CNEL) University of Florida Gainesville, Florida, USA rish283@ufl.edu, principe@cnel.ufl.edu Abstract We propose a framework for predictive uncer- tainty quantification of a neural network that replaces the conventional Bayesian notion of weight probability density function (PDF) with a physics based potential field representation of the model weights in a Gaussian reproducing kernel Hilbert space (RKHS) embedding. This al- lows us to use perturbation theory from quan- tum physics to formulate a moment decomposi- tion problem over the model weight-output re- lationship. The extracted moments reveal suc- cessive degrees of regularization of the weight potential field around the local neighborhood of the model output. Such localized moments represent well the PDF tails and provide signifi- cantly greater accuracy of the model’s predictive uncertainty than the central moments character- ized by Bayesian and ensemble methods or their variants. We show that this consequently leads to a better ability to detect false model predic- tions of test data that has undergone a covariate shift away from the training PDF learned by the model. We evaluate our approach against base- line uncertainty quantification methods on sev- eral benchmark datasets that are corrupted using common distortion techniques. Our approach provides fast model predictive uncertainty esti- mates with much greater precision and calibra- tion. Deep neural network (DNN) models have become the predominant choice for pattern representation in a wide variety of machine learning applications due to their re- markable performance advantages in the presence of large amount data (LeCun et al., 2015). The increased adoption of DNNs in safety critical and high stake problems such as medical diagnosis, chemical plant control, defense sys- Figure 1: Proposed approach: Moments extracted from the local interaction of the model output with the RKHS potential field of the weights quantify the output uncer- tainty. tems and autonomous driving has led to growing concerns within the research community on the performance trust- worthiness of such models (Kendall & Gal, 2017; Lundberg & Lee, 2017). This becomes particularly imperative in situ- ations involving data distributional shifts or the presence of out of distribution data (OOD) during testing towards which the model may lack robustness due to poor choice of training parameters or lack of sufficiently labeled train- ing data, especially since machine learning algorithms do not have extensive prior information like humans to deal with such situations (Amodei et al., 2016). An im- portant way through which trust in the performance of machine learning algorithms (particularly DNNs) can be established is through accurate techniques of predictive uncertainty quantification of models that allow practition- ers to determine how much they should rely on model predictions. Although there have been several categories of methods developed in the recent years, the Bayesian approach arXiv:2109.10888v1 [cs.LG] 22 Sep 2021