Prognostics performance evaluation has gained significant attention in the past few years.As prognostics technology matures and more sophisticated methods for prognostic uncertainty management are developed, a standardized methodology for performance evaluation becomes extremely important to guide improvement efforts in a constructive manner. This paper is in continuation of previous efforts where several new evaluation metrics tailored for prognostics were introduced and were shown to effectively evaluate various algorithms as compared to other conventional metrics. Specifically, this paper presents a detailed discussion on how these metrics should be interpreted and used. Several shortcomings identified, while applying these metrics to a variety of real applications, are also summarized along with discussions that attempt to alleviate these problems. Further, these metrics have been enhanced to include the capability of incorporating probability distribution information from prognostic algorithms as opposed to evaluation based on point estimates only. Several methods have been suggested and guidelines have been provided to help choose one method over another based on probability distribution characteristics.These approaches also offer a convenient and intuitive visualization of algorithm performance with respect to some of these new metrics like prognostic horizon and α-λ performance, and also quantify the corresponding performance while incorporating the uncertainty information.
How to Cite
data driven prognostics, diagnostic performance, model based prognostics, performance metrics, PHM system design and engineering,, prognostic performance, remaining useful life (RUL), return on investment (ROI)
(Devore, 2004) J. L. Devore, Probability and Statistics for Engineering and the Sciences, 6th ed.: Thomson, 2004.
(Feldman et al., 2008) K. Feldman, P. Sandborn, and T. Jazouli, “The analysis of Return on Investment for PHM Applied to Electronic Systems”, in International Conference on Prognostics and Health Management (PHM08), Denver CO, pp. 1-9, 2008.
(Goebel et al., 2008) K. Goebel, B. Saha, and A. Saxena, “A Comparison of Three Data-Driven Techniques for Prognostics,” in 62nd Meeting of the Society For Machinery Failure Prevention Technology (MFPT), Virginia Beach, VA, pp. 191- 131, 2008.
(Guan et al., 2009) X. Guan, Y. Liu, A. Saxena, J. Celaya, and K. Goebel, “Entropy-BasedProbabilistic Fatigue Damage Prognosis and Algorithmic Performance Comparison” in Annual Conference of the Prognostics and Health Management Society 09, San Diego, CA, pp. 1-11, 2009.
(Hoaglin et al., 1983) D. C. Hoaglin, F. Mosteller, and J. W. Tukey, Understanding Robust and Exploratory Data Analysis: Wiley, 1983.
(Leao et al, 2008) B. P. Leao, T. Yoneyama, G. C. Rocha, K. T. Fitzgibbon, “Prognostics performance metrics and their relation to requirements, design, verification and cost-benefit,” in International Conference on Prognostics and Health Management (PHM08), Denver, CO, pp. 1-8, 2008.
(Martinez, 2004) A. R. Martinez, "Exploratory data analysis with MATLAB," A. R. Martinez, ed., CRC Press, 2004.
(MIL-STD-1629A, 1980) Military Standard: Procedures for Performing A failure Mode, Effects and Criticality Analysis, MIL-STD-1629A, Department of Defense Washington DC, Nov 1980.
(Orchard and Vachtsevanos, 2009) M. E. Orchard, and G. J. Vachtsevanos, “A particle-filtering approach for on-line fault diagnosis and failure prognosis,” Transactions of the Institute of Measurement and Control, vol. 31, no. 3-4, pp. 221-246, 2009.
(Saha et al., 2009) B. Saha, K. Goebel, S. Poll, and J. Christophersen “Prognostics Methods for Battery Health Monitoring Using a Bayesian Framework,” IEEE Transactions on Instrumentation and Measurement, vol. 58, no. 2, 2009.
(Saxena et al., 2008) A. Saxena, J. Celaya, E. Balaban, B. Saha, S. Saha, and K. Goebel, “Metrics for evaluating performance of prognostic techniques,” in International Conference on Prognostics and Health Management (PHM08), Denver CO, pp. 1- 17, 2008.
(Saxena et al., 2009) A. Saxena, J. Celaya, B. Saha, S. Saha, and K. Goebel, "Evaluating Algorithmic Performance Metrics Tailored for Prognostics", in Proceedings of IEEE Aerospace Conference, Big Sky, MO, 2009.
(Wang and Lee, 2009) T. Wang, and J. Lee, “On Performance Evaluation of Prognostics Algorithms,” in Machinery Failure Prevention Technology, Dayton OH, 2009.
(Yang and Letourneau, 2007) C. Yang, and S. Letourneau, “Model evaluation for prognostics: estimating cost saving for the end users,” in Sixth International Conference on Machine Learning and Applications, pp. 304-309, 2007.
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
The Prognostic and Health Management Society advocates open-access to scientific data and uses a Creative Commons license for publishing and distributing any papers. A Creative Commons license does not relinquish the author’s copyright; rather it allows them to share some of their rights with any member of the public under certain conditions whilst enjoying full legal protection. By submitting an article to the International Conference of the Prognostics and Health Management Society, the authors agree to be bound by the associated terms and conditions including the following:
As the author, you retain the copyright to your Work. By submitting your Work, you are granting anybody the right to copy, distribute and transmit your Work and to adapt your Work with proper attribution under the terms of the Creative Commons Attribution 3.0 United States license. You assign rights to the Prognostics and Health Management Society to publish and disseminate your Work through electronic and print media if it is accepted for publication. A license note citing the Creative Commons Attribution 3.0 United States License as shown below needs to be placed in the footnote on the first page of the article.
First Author et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.