Metrics for Offline Evaluation of Prognostic Performance
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Mar 22, 2021
Abhinav Saxena
Jose Celaya
Bhaskar Saha
Sankalita Saha
Kai Goebel
Abstract
Prognostic performance evaluation has gained significant attention in the past few years.*Currently, prognostics concepts lack standard definitions and suffer from ambiguous and inconsistent interpretations. This lack of standards is in part due to the varied end-user requirements for different applications, time scales, available information, domain dynamics, etc. to name a few. The research community has used a variety of metrics largely based on convenience and their respective requirements. Very little attention has been focused on establishing a standardized approach to compare different efforts. This paper presents several new evaluation metrics tailored for prognostics that were recently introduced and were shown to effectively evaluate various algorithms as compared to other conventional metrics. Specifically, this paper presents a detailed discussion on how these metrics should be interpreted and used. These metrics have the capability of incorporating probabilistic uncertainty estimates from prognostic algorithms. In addition to quantitative assessment they also offer a comprehensive visual perspective that can be used in designing the prognostic system. Several methods are suggested to customize these metrics for different applications. Guidelines are provided to help choose one method over another based on distribution characteristics. Various issues faced by prognostics and its performance evaluation are discussed followed by a formal notational framework to help standardize subsequent developments.
##plugins.themes.bootstrap3.article.details##
Keywords
preventive maintenance, prognostic performance, prognostics, remaining useful life (RUL)
References
Banks, J., & Merenich, J. (2007). Cost benefit analysis for asset health management technology. Reliability and Maintainability Symposium (RAMS), Orlando, FL.
Carrasco, M., & Cassady, C. R. (2006). A study of the impact of prognostic errors on system performance. Annual Reliability and Maintainability Symposium, RAMS06.
Coble, J. B., & Hines, J. W. (2008). Prognostic Algorithm Categorization with PHM Challenge Application. 1st International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Coppe, A., Haftka, R. T., Kim, N., & Yuan, F. (2009). Reducing Uncertainty in Damage Growth Properties by Structural Health Monitoring. Annual Conference of the Prognostics and Health Management Society (PHM09) San Diego, CA.
DeNeufville, R. (2004). Uncertainty Management for Engineering Systems Planning and Design. Engineering Systems Symposium MIT, Cambridge, MA.
Devore, J. L. (2004). Probability and Statistics for Engineering and the Sciences (6th ed.): Thomson.
Drummond, C., & Yang, C. (2008). Reverse Engineering Costs: How Much will a Prognostic Algorithm Save? International Conference on Prognostics and Health Management, Denver, CO.
Engel, S. J. (2008). Prognosis Requirements and V&V: Panel Discussion on PHM Capabilities: Verification, Validation, and Certification Issues. International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Engel, S. J., Gilmartin, B. J., Bongort, K., & Hess, A. (2000). Prognostics, the Real Issues Involved with Predicting Life Remaining. IEEE Aerospace Conference, Big Sky, MT.
Feldman, K., Sandborn, P., & Jazouli, T. (2008). The Analysis of Return on Investment for PHM Applied to Electronic Systems. International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Goebel, K., & Bonissone, P. (2005). Prognostic Information Fusion for Constant Load Systems. 7th Annual Conference on Information Fusion.
Goebel, K., Saha, B., & Saxena, A. (2008). A Comparison of Three Data-Driven Techniques for Prognostics. 62nd Meeting of the Society For Machinery Failure Prevention Technology (MFPT), Virginia Beach, VA.
Guan, X., Liu, Y., Saxena, A., Celaya, J., & Goebel, K. (2009). Entropy-Based Probabilistic Fatigue Damage Prognosis and Algorithmic Performance Comparison. Annual Conference of the Prognostics and Health Management Society (PHM09), San Diego, CA.
Hastings, D., & McManus, H. (2004). A Framework for Understanding Uncertainty and its Mitigation and Exploitation in Complex Systems. Engineering Systems Symposium MIT, Cambridge MA.
Hoaglin, D. C., Mosteller, F., & Tukey, J. W. (Eds.). (1983). Understanding Robust and Exploratory Data Analysis: John Wiley & Sons.
ISO (2004). Condition Monitoring and Diagnostics of Machines - Prognostics part 1: General Guidelines, ISO/IEC Directives Part 2 C.F.R..
Leao, B. P., Yoneyama, T., Rocha, G. C., & Fitzgibbon, K. T. (2008). Prognostics Performance Metrics and Their Relation to Requirements, Design, Verification and Cost-Benefit. International Conference on Prognostics and Health Management (PHM08), Denver CO.
Martinez, A. R. (2004). Exploratory Data Analysis with MATLAB. In A. R. Martinez (Ed.): CRC Press.
MIL-STD-1629A. (1980). Military Standard: Procedures for Performing A Failure Mode, Effects and Criticality Analysis. Washington DC: Department of Defense.
NASA. (2009). NASA Aviation Safety Program Retrieved December 2009, from http://www.aeronautics.nasa.gov/programs_avsafe.htm
Ng, K.-C., & Abramson, B. (1990). Uncertainty Management in Expert Systems. IEEE Expert Systems, 5, 20.
Orchard, M., Kacprzynski, G., Goebel, K., Saha, B., & Vachtsevanos, G. (2008). Advances in Uncertainty Representation and Management for Particle Filtering Applied to Prognostics. International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Orchard, M. E., Tang, L., Goebel, K., & Vachtsevanos, G. (2009). A Novel RSPF Approach to Prediction of High-Risk, Low-Probability Failure Events. Annual Conference of the Prognostics and Health Management Society (PHM09), San Diego, CA.
Orchard, M. E., & Vachtsevanos, G. J. (2009). A Particle-Filtering Approach for On-line Fault Diagnosis and Failure Prognosis. Transactions of the Institute of Measurement and Control, 31(3-4), 221-246.
Orsagh, R. F., Roemer, M. J., Savage, C. J., & McClintic, K. (2001). Development of Effectiveness and Performance Metrics for Mechanical Diagnostic Techniques. 55th Meeting of the Society for Machinery Failure Prevention Technology, Virginia Beach, VA.
Pipe, K. (2008). Practical Prognostics for Condition Based Maintenance. International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Saha, B., & Goebel, K. (2009). Modeling Li-ion Battery Capacity Depletion in a Particle Filtering Framework. Annual Conference of the Prognostics and Health Management Society (PHM09), San Diego, CA.
Sankararaman, S., Ling, Y., Shantz, C., & Mahadevan, S. (2009). Uncertainty Quantification in Fatigue Damage Prognosis. Annual Conference of the Prognostics and Health Management Society (PHM09), San Diego, CA.
Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., et al. (2008). Metrics for Evaluating Performance of Prognostics Techniques. 1st International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2009a). Evaluating Algorithmic Performance Metrics Tailored for Prognostics. IEEE Aerospace Conference, Big Sky, MT.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2009b). On Applying the Prognostics Performance Metrics. Annual Conference of the Prognostics and Health Management Society (PHM09) San Diego, CA.
Schwabacher, M. (2005). A Survey of Data Driven Prognostics. AIAA Infotech@Aerospace Conference, Arlington, VA.
Schwabacher, M., & Goebel, K. (2007). A Survey of Artificial Intelligence for Prognostics. AAAI Fall Symposium, Arlington, VA.
Tang, L., Kacprzynski, G. J., Goebel, K., & Vachtsevanos, G. (2009). Methodologies for Uncertainty Management in Prognostics. IEEE Aerospace Conference, Big Sky, MT.
Uckun, S., Goebel, K., & Lucas, P. J. F. (2008). Standardizing Research Methods for Prognostics. International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Wang, T., & Lee, J. (2009). On Performance Evaluation of Prognostics Algorithms. Machinery Failure Prevention Technology, Dayton, OH.
Wheeler, K. R., Kurtoglu, T., & Poll, S. (2009). A Survey of Health Management User Objectives Related to Diagnostic and Prognostic Metrics. ASME 2009 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (IDETC/CIE), San Diego, CA.
Yang, C., & Letourneau, S. (2007). Model Evaluation for Prognostics: Estimating Cost Saving for the End Users. Sixth International Conference on Machine Learning and Applications.
Carrasco, M., & Cassady, C. R. (2006). A study of the impact of prognostic errors on system performance. Annual Reliability and Maintainability Symposium, RAMS06.
Coble, J. B., & Hines, J. W. (2008). Prognostic Algorithm Categorization with PHM Challenge Application. 1st International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Coppe, A., Haftka, R. T., Kim, N., & Yuan, F. (2009). Reducing Uncertainty in Damage Growth Properties by Structural Health Monitoring. Annual Conference of the Prognostics and Health Management Society (PHM09) San Diego, CA.
DeNeufville, R. (2004). Uncertainty Management for Engineering Systems Planning and Design. Engineering Systems Symposium MIT, Cambridge, MA.
Devore, J. L. (2004). Probability and Statistics for Engineering and the Sciences (6th ed.): Thomson.
Drummond, C., & Yang, C. (2008). Reverse Engineering Costs: How Much will a Prognostic Algorithm Save? International Conference on Prognostics and Health Management, Denver, CO.
Engel, S. J. (2008). Prognosis Requirements and V&V: Panel Discussion on PHM Capabilities: Verification, Validation, and Certification Issues. International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Engel, S. J., Gilmartin, B. J., Bongort, K., & Hess, A. (2000). Prognostics, the Real Issues Involved with Predicting Life Remaining. IEEE Aerospace Conference, Big Sky, MT.
Feldman, K., Sandborn, P., & Jazouli, T. (2008). The Analysis of Return on Investment for PHM Applied to Electronic Systems. International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Goebel, K., & Bonissone, P. (2005). Prognostic Information Fusion for Constant Load Systems. 7th Annual Conference on Information Fusion.
Goebel, K., Saha, B., & Saxena, A. (2008). A Comparison of Three Data-Driven Techniques for Prognostics. 62nd Meeting of the Society For Machinery Failure Prevention Technology (MFPT), Virginia Beach, VA.
Guan, X., Liu, Y., Saxena, A., Celaya, J., & Goebel, K. (2009). Entropy-Based Probabilistic Fatigue Damage Prognosis and Algorithmic Performance Comparison. Annual Conference of the Prognostics and Health Management Society (PHM09), San Diego, CA.
Hastings, D., & McManus, H. (2004). A Framework for Understanding Uncertainty and its Mitigation and Exploitation in Complex Systems. Engineering Systems Symposium MIT, Cambridge MA.
Hoaglin, D. C., Mosteller, F., & Tukey, J. W. (Eds.). (1983). Understanding Robust and Exploratory Data Analysis: John Wiley & Sons.
ISO (2004). Condition Monitoring and Diagnostics of Machines - Prognostics part 1: General Guidelines, ISO/IEC Directives Part 2 C.F.R..
Leao, B. P., Yoneyama, T., Rocha, G. C., & Fitzgibbon, K. T. (2008). Prognostics Performance Metrics and Their Relation to Requirements, Design, Verification and Cost-Benefit. International Conference on Prognostics and Health Management (PHM08), Denver CO.
Martinez, A. R. (2004). Exploratory Data Analysis with MATLAB. In A. R. Martinez (Ed.): CRC Press.
MIL-STD-1629A. (1980). Military Standard: Procedures for Performing A Failure Mode, Effects and Criticality Analysis. Washington DC: Department of Defense.
NASA. (2009). NASA Aviation Safety Program Retrieved December 2009, from http://www.aeronautics.nasa.gov/programs_avsafe.htm
Ng, K.-C., & Abramson, B. (1990). Uncertainty Management in Expert Systems. IEEE Expert Systems, 5, 20.
Orchard, M., Kacprzynski, G., Goebel, K., Saha, B., & Vachtsevanos, G. (2008). Advances in Uncertainty Representation and Management for Particle Filtering Applied to Prognostics. International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Orchard, M. E., Tang, L., Goebel, K., & Vachtsevanos, G. (2009). A Novel RSPF Approach to Prediction of High-Risk, Low-Probability Failure Events. Annual Conference of the Prognostics and Health Management Society (PHM09), San Diego, CA.
Orchard, M. E., & Vachtsevanos, G. J. (2009). A Particle-Filtering Approach for On-line Fault Diagnosis and Failure Prognosis. Transactions of the Institute of Measurement and Control, 31(3-4), 221-246.
Orsagh, R. F., Roemer, M. J., Savage, C. J., & McClintic, K. (2001). Development of Effectiveness and Performance Metrics for Mechanical Diagnostic Techniques. 55th Meeting of the Society for Machinery Failure Prevention Technology, Virginia Beach, VA.
Pipe, K. (2008). Practical Prognostics for Condition Based Maintenance. International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Saha, B., & Goebel, K. (2009). Modeling Li-ion Battery Capacity Depletion in a Particle Filtering Framework. Annual Conference of the Prognostics and Health Management Society (PHM09), San Diego, CA.
Sankararaman, S., Ling, Y., Shantz, C., & Mahadevan, S. (2009). Uncertainty Quantification in Fatigue Damage Prognosis. Annual Conference of the Prognostics and Health Management Society (PHM09), San Diego, CA.
Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., et al. (2008). Metrics for Evaluating Performance of Prognostics Techniques. 1st International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2009a). Evaluating Algorithmic Performance Metrics Tailored for Prognostics. IEEE Aerospace Conference, Big Sky, MT.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2009b). On Applying the Prognostics Performance Metrics. Annual Conference of the Prognostics and Health Management Society (PHM09) San Diego, CA.
Schwabacher, M. (2005). A Survey of Data Driven Prognostics. AIAA Infotech@Aerospace Conference, Arlington, VA.
Schwabacher, M., & Goebel, K. (2007). A Survey of Artificial Intelligence for Prognostics. AAAI Fall Symposium, Arlington, VA.
Tang, L., Kacprzynski, G. J., Goebel, K., & Vachtsevanos, G. (2009). Methodologies for Uncertainty Management in Prognostics. IEEE Aerospace Conference, Big Sky, MT.
Uckun, S., Goebel, K., & Lucas, P. J. F. (2008). Standardizing Research Methods for Prognostics. International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Wang, T., & Lee, J. (2009). On Performance Evaluation of Prognostics Algorithms. Machinery Failure Prevention Technology, Dayton, OH.
Wheeler, K. R., Kurtoglu, T., & Poll, S. (2009). A Survey of Health Management User Objectives Related to Diagnostic and Prognostic Metrics. ASME 2009 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (IDETC/CIE), San Diego, CA.
Yang, C., & Letourneau, S. (2007). Model Evaluation for Prognostics: Estimating Cost Saving for the End Users. Sixth International Conference on Machine Learning and Applications.
Section
Technical Papers