Performance Metrics in the Perspective of Prognosis Uncertainty Bruno P. Le˜ ao 1 , Takashi Yoneyama 2 1 GE Global Research - Brazil Technology Center, Rio de Janeiro, RJ, 21941-615, Brazil leao@ge.com 2 Instituto Tecnol´ ogico de Aeron´ autica, S˜ ao Jos´ e dos Campos, SP, 12228-900, Brazil takashi@ita.br ABSTRACT The subject of uncertainty in failure prognosis, including the importance of estimating and managing it, is a recurring topic in PHM literature. Considering that the prognosis task com- prises forecasting, this could not be any different. However, prognosis performance metrics proposed in literature are usu- ally concerned with measuring adherence to requirements, but not the adequate representation of the true uncertainty that arises from various sources in a prognosis problem. This pa- per presents statistically sound means for evaluating the per- formance of prognosis methods in the perspective of compar- ing the true uncertainty to its estimates. This provides a use- ful yet simple framework for failure prognosis performance evaluation. 1. I NTRODUCTION Failure prognosis is a subject which draws increasing atten- tion from academia and industry over the years. Improve- ments in computational and sensing capabilities has led to un- precedented possibilities for employing data analytics tools in creating value for asset users and maintainers. Reliable esti- mates of remaining useful life (RUL) of equipment can yield benefits not only for maintenance but also for logistics, spare parts management and equipment operation. However, var- ious challenges arise during the development and validation of prognosis solutions. Many of these challenges are associ- ated with the intrinsic uncertainty associated with the progno- sis task. Prognosis comprises forecasting, and future opera- tional and ambient conditions are usually difficult to estimate in advance. Besides that, the estimation of the degradation state and its trend cannot usually be performed in a reliable way when uncertainty is not properly taken into considera- tion. Because of these factors, uncertainty is an important topic in failure prognosis literature. Bruno Le˜ ao et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. In one of the seminal works that discussed about uncertainty in failure prognosis (Engel, Gilmartin, Bongort, & Hess, 2000) the authors describe the role of uncertainty and the importance of its estimation as part of the prognosis task. The important trade-off between the precision of prognosis estimates and the probability that they will capture the ac- tual failure (sometimes referred to as Engel’s paradox) and discussions on the true versus the estimated uncertainty in RUL are also present in the referred paper. Other authors, such as Orchard, Kacprzynski, Goebel, Saha, and Vachtse- vanos (2008), discuss about uncertainty in the context of specific prognosis methods. More recently, Celaya, Saxena, and Goebel (2012) recalled the topic of estimated versus true prognosis uncertainty considering also a sample application. In the referred paper, the authors highlight that there is part of prognosis uncertainty that is intrinsic to the problem un- der consideration and should be properly accounted for and represented. This reinforces the claim that the prognosis task should not be aimed at estimating RUL with minimal uncer- tainty but rather at making the estimated uncertainty as close as possible to the true one. Another topic of active research in the failure prognosis field is the definition of proper metrics for performance evalua- tion. Vachtsevanos, Lewis, Roemer, Hess, and Wu (2006) presented one of the first compiled lists of prognosis perfor- mance metrics. This and other seminal works present the metrics with focus on accuracy and precision. More recently, Le˜ ao, Yoneyama, Rocha, and Fitzgibon (2008) and Saxena, Celaya, Saha, Saha, and Goebel (2010) proposed more elab- orate metrics that consider also other aspects such as design tradeoffs and convergence of estimates. The work presented herein is based on the method described by Le˜ ao, Gomes, Galv˜ ao, and Yoneyama (2010); Le˜ ao and Yoneyama (2011) for prognosis performance evaluation using the Probability Integral Transform (PIT). This approach is presented here as an adequate and statistically sound method for evaluating the quality of prognosis results in terms of fitting the true uncer- tainty as described in the aforementioned references. 1