A Hybrid Post Hoc Interpretability Approach for Deep Neural Networks Fl´ avio Arthur Oliveira Santos 1,3(B ) , Cleber Zanchettin 1 , Jos´ e Vitor Santos Silva 2 , Leonardo Nogueira Matos 2 , and Paulo Novais 3 1 Universidade Federal de Pernambuco, Recife, Brazil {faos,cz}@cin.ufpe.br 2 Universidade Federal de Sergipe, S˜ao Crist´ov˜ao, Brazil {jose.silva,leonardo}@dcomp.ufs.br 3 University of Minho, Braga, Portugal pjon@di.uminho.pt, flavio.santos@algoritmi.uminho.pt Abstract. Every day researchers publish works with state-of-the-art results using deep learning models, however as these models become common even in production, ensuring fairness is a main concern of the deep learning models. One way to analyze the model fairness is based on the model interpretability, obtaining the essential features to the model decision. There are many interpretability methods to produce the deep learning model interpretation, such as Saliency, GradCam, Integrated Gradients, Layer-wise relevance propagation, and others. Although those methods make the feature importance map, different methods have dif- ferent interpretations, and their evaluation relies on qualitative analysis. In this work, we propose the Iterative post hoc attribution approach, which consists of seeing the interpretability problem as an optimization view guided by two objective definitions of what our solution consid- ers important. We solve the optimization problem with a hybrid app- roach considering the optimization algorithm and the deep neural net- work model. The obtained results show that our approach can select the features essential to the model prediction more accurately than the traditional interpretability methods. Keywords: Deep learning · Optimization · Interpretability · Fairness 1 Introduction Deep learning [6] models shows state-of-the-art results in a wide range of domains, for example natural language processing [1], computer vision [3], and recommendation systems [23]. Its success is not limited to research projects. Nowadays, deep learning models have become part of people’s daily lives through the countless products they use on the web and that are present on their smart- phones 1 . Despite this success, these models are hard to be adopted in some 1 https://www.technologyreview.com/2015/02/09/169415/deep-learning-squeezed- onto-a-phone/. c  Springer Nature Switzerland AG 2021 H. Sanjurjo Gonz´alez et al. (Eds.): HAIS 2021, LNAI 12886, pp. 600–610, 2021. https://doi.org/10.1007/978-3-030-86271-8_50