On Applying the Prognostic Performance Metrics



Published Mar 26, 2021
Abhinav Saxena Jose Celaya Bhaskar Saha Sankalita Saha Kai Goebel


Prognostics performance evaluation has gained significant attention in the past few years.As prognostics technology matures and more sophisticated methods for prognostic uncertainty management are developed, a standardized methodology for performance evaluation becomes extremely important to guide improvement efforts in a constructive manner. This paper is in continuation of previous efforts where several new evaluation metrics tailored for prognostics were introduced and were shown to effectively evaluate various algorithms as compared to other conventional metrics. Specifically, this paper presents a detailed discussion on how these metrics should be interpreted and used. Several shortcomings identified, while applying these metrics to a variety of real applications, are also summarized along with discussions that attempt to alleviate these problems. Further, these metrics have been enhanced to include the capability of incorporating probability distribution information from prognostic algorithms as opposed to evaluation based on point estimates only. Several methods have been suggested and guidelines have been provided to help choose one method over another based on probability distribution characteristics.These approaches also offer a convenient and intuitive visualization of algorithm performance with respect to some of these new metrics like prognostic horizon and α-λ performance, and also quantify the corresponding performance while incorporating the uncertainty information.

How to Cite

Saxena, A. ., Celaya, J. ., Saha, B. ., Saha, S. ., & Goebel, K. . (2021). On Applying the Prognostic Performance Metrics. Annual Conference of the PHM Society, 1(1). Retrieved from https://papers.phmsociety.org/index.php/phmconf/article/view/1621
Abstract 1034 | PDF Downloads 353



data driven prognostics, diagnostic performance, model based prognostics, performance metrics, PHM system design and engineering,, prognostic performance, remaining useful life (RUL), return on investment (ROI)

(Banks and Merenich, 2007) J. Banks and J. Merenich, “Cost Benefit Analysis for Asset Health Management Technology”, Annual Reliability and Maintainability Symposium, RAMS '07, pp. 95-100, 2007
(Devore, 2004) J. L. Devore, Probability and Statistics for Engineering and the Sciences, 6th ed.: Thomson, 2004.
(Feldman et al., 2008) K. Feldman, P. Sandborn, and T. Jazouli, “The analysis of Return on Investment for PHM Applied to Electronic Systems”, in International Conference on Prognostics and Health Management (PHM08), Denver CO, pp. 1-9, 2008.
(Goebel et al., 2008) K. Goebel, B. Saha, and A. Saxena, “A Comparison of Three Data-Driven Techniques for Prognostics,” in 62nd Meeting of the Society For Machinery Failure Prevention Technology (MFPT), Virginia Beach, VA, pp. 191- 131, 2008.
(Guan et al., 2009) X. Guan, Y. Liu, A. Saxena, J. Celaya, and K. Goebel, “Entropy-BasedProbabilistic Fatigue Damage Prognosis and Algorithmic Performance Comparison” in Annual Conference of the Prognostics and Health Management Society 09, San Diego, CA, pp. 1-11, 2009.
(Hoaglin et al., 1983) D. C. Hoaglin, F. Mosteller, and J. W. Tukey, Understanding Robust and Exploratory Data Analysis: Wiley, 1983.
(Leao et al, 2008) B. P. Leao, T. Yoneyama, G. C. Rocha, K. T. Fitzgibbon, “Prognostics performance metrics and their relation to requirements, design, verification and cost-benefit,” in International Conference on Prognostics and Health Management (PHM08), Denver, CO, pp. 1-8, 2008.
(Martinez, 2004) A. R. Martinez, "Exploratory data analysis with MATLAB," A. R. Martinez, ed., CRC Press, 2004.
(MIL-STD-1629A, 1980) Military Standard: Procedures for Performing A failure Mode, Effects and Criticality Analysis, MIL-STD-1629A, Department of Defense Washington DC, Nov 1980.
(Orchard and Vachtsevanos, 2009) M. E. Orchard, and G. J. Vachtsevanos, “A particle-filtering approach for on-line fault diagnosis and failure prognosis,” Transactions of the Institute of Measurement and Control, vol. 31, no. 3-4, pp. 221-246, 2009.
(Saha et al., 2009) B. Saha, K. Goebel, S. Poll, and J. Christophersen “Prognostics Methods for Battery Health Monitoring Using a Bayesian Framework,” IEEE Transactions on Instrumentation and Measurement, vol. 58, no. 2, 2009.
(Saxena et al., 2008) A. Saxena, J. Celaya, E. Balaban, B. Saha, S. Saha, and K. Goebel, “Metrics for evaluating performance of prognostic techniques,” in International Conference on Prognostics and Health Management (PHM08), Denver CO, pp. 1- 17, 2008.
(Saxena et al., 2009) A. Saxena, J. Celaya, B. Saha, S. Saha, and K. Goebel, "Evaluating Algorithmic Performance Metrics Tailored for Prognostics", in Proceedings of IEEE Aerospace Conference, Big Sky, MO, 2009.
(Wang and Lee, 2009) T. Wang, and J. Lee, “On Performance Evaluation of Prognostics Algorithms,” in Machinery Failure Prevention Technology, Dayton OH, 2009.
(Yang and Letourneau, 2007) C. Yang, and S. Letourneau, “Model evaluation for prognostics: estimating cost saving for the end users,” in Sixth International Conference on Machine Learning and Applications, pp. 304-309, 2007.
Technical Research Papers

Most read articles by the same author(s)

<< < 1 2 3 4 5 6 > >>