1 Abstract—The comparison between improved FMECA methods is commonly conducted by comparing qualitatively the resulting rankings and the potential balance between the three risk factors. This paper introduces the application of Cohen’s kappa concordance coefficient as part of a comparative approach between different methods used to improve the FMECA analysis. The use of ranking agreement metrics allows comparing the rankings generated by independent raters; in this context, the application of Cohen’s kappa in medical and social sciences is broad, but despite its relevance, its application in the FMECA context is limited. The proposed approach considers the concordance assessment between different methodologies used in FMECA (Risk Isosurface function, VIKOR, ITWH, FWGM, Type-I and Type-II Fuzzy Inference System) when applied to the same problem and regarding an FMECA ranking selected as the reference one. The analyzed problem is a blood transfusion case study consisting of eleven failure modes widely used for benchmarking. Results show that Type-II fuzzy inference systems achieve the highest agreement regarding the reference FMECA ranking; one possible explanation for this result is that Type-II FIS considers uncertainty as an additional parameter. This approach proves effective to compare statistically different FMECA methods instead of the classical qualitative comparison between rankings. Index Terms— FMECA; Risk assessment; Type-II fuzzy inference systems; Fuzzy weighted geometric mean; Concordance measurement; Cohen’s kappa. I. INTRODUCTION AILURE Modes, Effects and Criticality Analysis is a qualitative risk assessment method designed to identify potential failure modes, their causes, and systems performance effects [1]. The objective of FMECA is to identify the possible ways a failure can occur, how often it occurs, how severe the failure affects the system performance, and what should be the preventive measures to avoid the failure. The classical FMECA analysis is based on three factors, called risk factors, to characterize each failure mode [1]: the Manuscript received August 15, 2021. This work was supported by national funds through the Fundação para a Ciência e a Tecnologia, I.P., through IDMEC, under LAETA, Pproject UID/EMS/50022/2020, and by Secretaría Nacional de Educación Superior, Ciencia, Tecnología e Innovación (SENESCYT) of the Ecuadorian Government through fellowship CZ05- 000291-2017. (Corresponding author: Andrés A. Zúñiga.) Severity (SEV) that characterize qualitatively the effect of the failure mode, the Frequency of Occurrence (OCC) that characterize how likely is it the failure mode to occur, and the Detectability (DET) that characterize how detectable is the failure mode before to occur. Each risk factor is classified in specific risk categories represented by a numerical scale, it can be a 1 to 10 scale as used in [1], or a 1 to 5 scale as in [2]. Each failure mode is assessed through a risk priority number (RPN); in general terms, the RPN results from the composition between SEV, OCC and DET as in (1), being the product the generally adopted approach.       RPN SEV OCC DET    (1) Because the RPN calculation in the classical FMECA approach results from the unique product between three integers, there is no associated computational complexity. Although FMECA is a very popular qualitative method for failure analysis, computation of the RPN has some disadvantages [3]–[5]. They are: 1) The RPN computation does not consider any difference degree between the three risk factors OCC, SEV, and DET (i.e., no weight averaging each risk factor). 2) Although a higher RPN is usually associated with a more critical failure modes, this is not always true [6], [7], and. 3) The scales for the three risk factors are generally considered arbitrarily and may not accurately represent the risk characteristics in specific problems. To deal with these FMECA shortcomings, some approaches based on computational intelligence and decision-making methods were proposed in the past years. Bowles and Peláez [3] present one of the firsts applications of fuzzy inference systems FIS to improve the FMECA analysis; results shown that proposed FIS approach allows to overcome some FMECA issues like imprecise information related to the risk factors. In [6] authors conducted a literature review about FMECA methods published between 1998 and 2018; the review shows that publications about FMECA improvements increased in last Andrés A. Zúñiga, João F. P. Fernandes and Paulo J. Costa Branco are with the IDMEC, Instituto Superior Técnico, University of Lisbon, 1049-001 Lisbon, Portugal (e-mail: andres.zuniga@tecnico.ulisboa.pt; joao.f.p.fernandes@tecnico.ulisboa.pt; pbranco@tecnico.ulisboa.pt). A New Concordance Coefficients-Based Approach to Compare Improved FMECA Methods Andrés A. Zúñiga, Student Member, IEEE, João F. P. Fernandes, Member, IEEE and Paulo J. Costa Branco, Member, IEEE F