Performance Evaluation for Fleet-based and Unit-based Prognostic Methods
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Abstract
Within the last decade several new methods for prognostics have been developed and an overall understanding of the various issues involved in predictions for health management has significantly improved. However, it appears that there is still a lack of consensus on how prognostics is defined and what constitutes good performance for prognostics. This paper first differentiates prognostics from other prediction approaches before highlighting key attributes of performance for prediction methods. Then it argues that it is important to understand what factors affect the performance of a prognostic approach. Factors such as the application and end use of a prognostic output, the various methods to make predictions, purpose of performance evaluation, etc. are discussed. This paper presents a comprehensive view of various such aspects that dictate or should dictate what performance evaluation must be as far as prognostics is concerned. It is also discussed what should be used as baseline to assess performance and how to interpret commonly used comparisons of algorithm predictions to observed failure times. The primary goal of this paper is to present some arguments of how these issues can be addressed and to stimulate a discussion about meaningful evaluation of prognostic performance. These discussions are followed by a brief description of prognostics metrics proposed recently, their applicability, and limitations. This paper does not intend to suggest any metrics in particular rather highlights important aspects that must be covered by any performance evaluation method for prognostics.
How to Cite
##plugins.themes.bootstrap3.article.details##
PHM
Coble, J. B. (2010). Merging Data Sources to Predict Remaining Useful Life – An Automated Method to Identify Prognostic Parameters. PhD Dissertation, The University of Tennessee, Knoxville.
Coble, J. B., & Hines, J. W. (2008). Prognostic Algorithm Categorization with PHM Challenge Application. Paper presented at the 1st International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Engel, S. J., Gilmartin, B. J., Bongort, K., & Hess, A. (2000). Prognostics, the Real Issues Involved with Predicting Life Remaining. Paper presented at the IEEE Aerospace Conference, Big Sky, MT.
Goebel, K., Saha, B., & Saxena, A. (2008). A Comparison of Three Data-Driven Techniques for Prognostics. Paper presented at the 62nd Meeting of the Society For Machinery Failure Prevention Technology (MFPT), Virginia Beach, VA.
Goebel, K., Saxena, A., Saha, S., Saha, B., & Celaya, J. (2011). Prognostic Performance Metrics. Machine Learning and Knowledge Discovery for Engineering Systems Health Management, 147.
Guan, X., Jha, R., Liu, Y., Saxena, A., Celaya, J., & Goebel, K. (2010). Comparison of Two Probabilistic Fatigue Damage Assessment Approaches Using Prognostic Performance Metrics. International Journal of Prognostics and Health Management, 2(1)(5), 11.
Johnson, S. B., Gormley, T., Kessler, S., Mott, C., Patterson-Hine, A., Reichard, K., & Scandura Jr, P. (2011). System health management: with aerospace applications: John Wiley & Sons.
Leao, B. P., Gomes, J. P., & Yoneyama, T. (2011). Improvements on the offline performance evaluation of fault prognostics methods. Paper presented at the Aerospace Conference, 2011 IEEE.
Leao, B. P., & Yoneyama, T. (2013). Performance Metrics in the Perspective of Prognosis Uncertainty. Paper presented at the Annual Conference of the Prognostics and Health Management Society (PHM13), New Orleans, LA.
Liu, S., & Sun, B. (2012). A Novel method for online prognostics performance evaluation. Paper presented at the Prognostics and System Health Management (PHM), 2012 IEEE Conference on.
Olivares, B. E., Muñoz, M. A. C., & Orchard, M. E. (2013). Particle-Filtering-Based Prognosis Framework for Energy Storage Devices With a Statistical Characterization of State-of-Health Regeneration Phenomena. IEEE Transactions on Instrumentation and Measurement, 62(2), 13.
Orchard, M. E., Tang, L., Goebel, K., & Vachtsevanos, G. (2009). A Novel RSPF Approach to Prediction of High-Risk, Low-Probability Failure Events. Paper presented at the Annual Conference of the Prognostics and Health Management Society (PHM09), San Diego, CA.
Roychoudhury, I., Saxena, A., Celaya, J. R., & Goebel, K. (2013). Distilling the Verification Process for Prognostics Algorithms. Paper presented at the Annual Conference of the Prognostics and Health Management Society (PHM13), New Orleans, LA.
Sankararaman, S., & Goebel, K. (2013, October 2013). Why is the Remaining Useful Life Prediction Uncertain? Paper presented at the Annual Conference of the Prognostics and Health Management Society, New Orleans, LA.
Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, M. (2008). Metrics for Evaluating Performance of Prognostics Techniques. Paper presented at the 1st International Conference on Prognostics and Health Management (PHM08), Denver, CO.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2009a). Evaluating Algorithmic Performance Metrics Tailored for Prognostics. Paper presented at the IEEE Aerospace Conference, Big Sky, MT.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2009b). On Applying the Prognostics Performance Metrics. Paper presented at the Annual Conference of the Prognostics and Health Management Society (PHM09) San Diego, CA.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2010). Metrics for Offline Evaluation of Prognostic Performance. International Journal of Prognostics and Health Management, 1(1), 21.
Saxena, A., & Roemer, M. (2013). IVHM Assessment Metrics: SAE International.
Saxena, A., Roychoudhury, I., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2012). Requirement Flowdown for Prognostics Health Management. Paper presented at the AIAA Infotech@Aerospace, Garden Grove, CA.
Sharp, M. E. (2013). Simple Metrics for Evaluating and Conveying Prognostic Model Performance To Users With Varied Backgrounds. Paper presented at the Annual Conference of the Prognostics and Health Management Society (PHM13), New Orleans, LA.
Vachtsevanos, G., Lewis, F. L., Roemer, M., Hess, A., & Wu, B. (2006). Intelligent Fault Diagnosis and Prognosis for Engineering Systems (1st ed.). Hoboken, New Jersey: John Wiley & Sons, Inc.
Wang, T., & Lee, J. (2009). On Performance Evaluation of Prognostics Algorithms. Paper presented at the Machinery Failure Prevention Technology, Dayton, OH.
The Prognostic and Health Management Society advocates open-access to scientific data and uses a Creative Commons license for publishing and distributing any papers. A Creative Commons license does not relinquish the author’s copyright; rather it allows them to share some of their rights with any member of the public under certain conditions whilst enjoying full legal protection. By submitting an article to the International Conference of the Prognostics and Health Management Society, the authors agree to be bound by the associated terms and conditions including the following:
As the author, you retain the copyright to your Work. By submitting your Work, you are granting anybody the right to copy, distribute and transmit your Work and to adapt your Work with proper attribution under the terms of the Creative Commons Attribution 3.0 United States license. You assign rights to the Prognostics and Health Management Society to publish and disseminate your Work through electronic and print media if it is accepted for publication. A license note citing the Creative Commons Attribution 3.0 United States License as shown below needs to be placed in the footnote on the first page of the article.
First Author et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.