Performance Evaluation for Fleet-based and Unit-based Prognostic Methods

##plugins.themes.bootstrap3.article.main##

##plugins.themes.bootstrap3.article.sidebar##

Abhinav Saxena Shankar Sankararaman Kai Goebel

Abstract

Within the last decade several new methods for prognostics have been developed and an overall understanding of the various issues involved in predictions for health management has significantly improved. However, it appears that there is still a lack of consensus on how prognostics is defined and what constitutes good performance for prognostics. This paper first differentiates prognostics from other prediction approaches before highlighting key attributes of performance for prediction methods. Then it argues that it is important to understand what factors affect the performance of a prognostic approach. Factors such as the application and end use of a prognostic output, the various methods to make predictions, purpose of performance evaluation, etc. are discussed. This paper presents a comprehensive view of various such aspects that dictate or should dictate what performance evaluation must be as far as prognostics is concerned. It is also discussed what should be used as baseline to assess performance and how to interpret commonly used comparisons of algorithm predictions to observed failure times. The primary goal of this paper is to present some arguments of how these issues can be addressed and to stimulate a discussion about meaningful evaluation of prognostic performance. These discussions are followed by a brief description of prognostics metrics proposed recently, their applicability, and limitations. This paper does not intend to suggest any metrics in particular rather highlights important aspects that must be covered by any performance evaluation method for prognostics.

How to Cite

Saxena, A., Sankararaman, S., & Goebel, K. (2014). Performance Evaluation for Fleet-based and Unit-based Prognostic Methods. PHM Society European Conference, 2(1). https://doi.org/10.36001/phme.2014.v2i1.1511
Abstract 64 | PDF Downloads 52

##plugins.themes.bootstrap3.article.details##

Keywords

PHM

References
Celaya, J., Saxena, A., & Goebel, K. (2012). Uncertainty Representation and Interpretation in Model-based Prognostics Algorithms Based on Kalman Filter Estimation. Paper presented at the Annual Conference of the Prognostics and Health Management Society (PHM12), Minneapolis, MN.
Coble, J. B. (2010). Merging Data Sources to Predict Remaining Useful Life – An Automated Method to Identify Prognostic Parameters. PhD Dissertation, The University of Tennessee, Knoxville.
Coble, J. B., & Hines, J. W. (2008). Prognostic Algorithm Categorization with PHM Challenge Application. Paper presented at the 1st International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Engel, S. J., Gilmartin, B. J., Bongort, K., & Hess, A. (2000). Prognostics, the Real Issues Involved with Predicting Life Remaining. Paper presented at the IEEE Aerospace Conference, Big Sky, MT.
Goebel, K., Saha, B., & Saxena, A. (2008). A Comparison of Three Data-Driven Techniques for Prognostics. Paper presented at the 62nd Meeting of the Society For Machinery Failure Prevention Technology (MFPT), Virginia Beach, VA.
Goebel, K., Saxena, A., Saha, S., Saha, B., & Celaya, J. (2011). Prognostic Performance Metrics. Machine Learning and Knowledge Discovery for Engineering Systems Health Management, 147.
Guan, X., Jha, R., Liu, Y., Saxena, A., Celaya, J., & Goebel, K. (2010). Comparison of Two Probabilistic Fatigue Damage Assessment Approaches Using Prognostic Performance Metrics. International Journal of Prognostics and Health Management, 2(1)(5), 11.
Johnson, S. B., Gormley, T., Kessler, S., Mott, C., Patterson-Hine, A., Reichard, K., & Scandura Jr, P. (2011). System health management: with aerospace applications: John Wiley & Sons.
Leao, B. P., Gomes, J. P., & Yoneyama, T. (2011). Improvements on the offline performance evaluation of fault prognostics methods. Paper presented at the Aerospace Conference, 2011 IEEE.
Leao, B. P., & Yoneyama, T. (2013). Performance Metrics in the Perspective of Prognosis Uncertainty. Paper presented at the Annual Conference of the Prognostics and Health Management Society (PHM13), New Orleans, LA.
Liu, S., & Sun, B. (2012). A Novel method for online prognostics performance evaluation. Paper presented at the Prognostics and System Health Management (PHM), 2012 IEEE Conference on.
Olivares, B. E., Muñoz, M. A. C., & Orchard, M. E. (2013). Particle-Filtering-Based Prognosis Framework for Energy Storage Devices With a Statistical Characterization of State-of-Health Regeneration Phenomena. IEEE Transactions on Instrumentation and Measurement, 62(2), 13.
Orchard, M. E., Tang, L., Goebel, K., & Vachtsevanos, G. (2009). A Novel RSPF Approach to Prediction of High-Risk, Low-Probability Failure Events. Paper presented at the Annual Conference of the Prognostics and Health Management Society (PHM09), San Diego, CA.
Roychoudhury, I., Saxena, A., Celaya, J. R., & Goebel, K. (2013). Distilling the Verification Process for Prognostics Algorithms. Paper presented at the Annual Conference of the Prognostics and Health Management Society (PHM13), New Orleans, LA.
Sankararaman, S., & Goebel, K. (2013, October 2013). Why is the Remaining Useful Life Prediction Uncertain? Paper presented at the Annual Conference of the Prognostics and Health Management Society, New Orleans, LA.
Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, M. (2008). Metrics for Evaluating Performance of Prognostics Techniques. Paper presented at the 1st International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2009a). Evaluating Algorithmic Performance Metrics Tailored for Prognostics. Paper presented at the IEEE Aerospace Conference, Big Sky, MT.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2009b). On Applying the Prognostics Performance Metrics. Paper presented at the Annual Conference of the Prognostics and Health Management Society (PHM09) San Diego, CA.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2010). Metrics for Offline Evaluation of Prognostic Performance. International Journal of Prognostics and Health Management, 1(1), 21.
Saxena, A., & Roemer, M. (2013). IVHM Assessment Metrics: SAE International.
Saxena, A., Roychoudhury, I., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2012). Requirement Flowdown for Prognostics Health Management. Paper presented at the AIAA Infotech@Aerospace, Garden Grove, CA.
Sharp, M. E. (2013). Simple Metrics for Evaluating and Conveying Prognostic Model Performance To Users With Varied Backgrounds. Paper presented at the Annual Conference of the Prognostics and Health Management Society (PHM13), New Orleans, LA.
Vachtsevanos, G., Lewis, F. L., Roemer, M., Hess, A., & Wu, B. (2006). Intelligent Fault Diagnosis and Prognosis for Engineering Systems (1st ed.). Hoboken, New Jersey: John Wiley & Sons, Inc.
Wang, T., & Lee, J. (2009). On Performance Evaluation of Prognostics Algorithms. Paper presented at the Machinery Failure Prevention Technology, Dayton, OH.
Section
Technical Papers