Predictive Model Evaluation for PHM
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Nov 1, 2020
Chunsheng Yang
Yanni Zou
Jie Liu
Kyle R Mulligan
Abstract
In the past decades, machine learning techniques or algorithms, particularly, classifiers have been widely applied to various real-world applications such as PHM. In developing high-performance classifiers, or machine learning-based models, i.e. predictive model for PHM, the predictive model evaluation remains a challenge. Generic methods such as accuracy may not fully meet the needs of models evaluation for prognostic applications. This paper addresses this issue from the point of view of PHM systems. Generic methods are first reviewed while outlining their limitations or deficiencies with respect to PHM. Then, two approaches developed for evaluating predictive models are presented with emphasis on specificities and requirements of PHM. A case of real prognostic application is studies to demonstrate the usefulness of two proposed methods for predictive model evaluation. We argue that predictive models for PHM must be evaluated not only using generic methods, but also domain-oriented approaches in order to deploy the models in real-world applications.
##plugins.themes.bootstrap3.article.details##
Keywords
prognostics, prognostics and health management (PHM), binary classifier, Machine Learning Algorithms, Generic Methods, Time to Failure, Predictive Models Evaluation
References
Banker R, Chanes A, Cooper W, et al. (1984). Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis, Management Science, Vol. 30 No. 9, 1078-1092
Bradley A (1997). The Use of the Area under the ROC Curve in the Evaluation of Machine Learning Algorithms, Pattern Recognition, Vol. 30, 1145-1159
Caruana R and Niculescu-Mizil A (2006). An Empirical Comparison of Supervised Learning Algorithms. Proceedings of the 23rd International Conference on Machine Learning (ICML2006)
Caruana R and Niculescu-Mizil A (2004). Data Mining in Metric Space: An empirical Analysis of Supervised Learning Performance Criteria, Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD2004), Seattle, Washington, USA, 69-78
Drummond C and Yang C (2008). Reverse-Engineering Costs: How much will a Prognostic Algorithm save?, Proceedings of the 1st International Conference on Prognostics and Health Management. Denver, USA
Drummond C and Holte R (2004). What ROC Curve Can’t Do (and Cost Curve Can), ECAI Workshop on ROC Analysis in Artificial Intelligence
Drummond C and Holte R (2000). Explicitly Representing Expected Cost: An Alternative to ROC Representation, Proceedings of the 6th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD2000), New York, USA, 155-164
Egan J (1975). Signal Detection Theory and ROC Analysis, New York Academic Press
Fawcett T (2003). ROC Graphs: Notes and Practical Considerations for Data Mining Researchers, Technical report, Intelligent Enterprise Technologies Laboratory, HP
Flach P, (2004). The Many Faces of ROC Analysis in Machine Learning, ICML’04 tutorial, http://www.cs.bris.ac.uk/~flach/ICML04tutorial/
Flach P, (2003). The Geometry of ROC Space: Understanding Machine Learning Metrics through ROC Isometrics. Proceedings of the 20th International Conference on Machine Learning (ICML03), Washington, DC, USA
Flexer A (1996). Statistical Evaluation of Neural Network Experiments: Minimum Requirement and Current Practice. Proceedings of the 13rd European Meeting on Cybernetics and Systems Research. Austrian Society for Cybernetic Studies, 1005-1008
Giudici P (2003). Applied Data Mining, John Wiley and Sons, New York
Hand D and Till R (2001). A Simple Generalization of the Area Under the ROC Curve for Multiple Class Classification Problems, Machine Learning, Vol.45, 171-186
Huang J and Ling C (2005). Using AUC and Accuracy in Evaluating Learning Algorithms, IEEE Transactions on Knowledge and Data Engineering, Vol. 17, No. 3, 299-310
Japkowicz N. and Shah Mohak (2011). Evaluation of Learning Algorithms, A Classification Perspective, Cambridge University Press
Jardine A., Lin D., and Banjevic D. (2006). A review on machinery diagnostics and prognostics implementing condition-based maintenance. Mechanical Systems and Signal Processing Vol.20, 1483-1510.
Létourneau S, Famili F, et al. (1999). Data Mining for Prediction of Aircraft Component Replacement, IEEE Intelligent Systems Journal, Special Issue on Data Mining. 59-6
Prechelt L (1996). A Quantative Study of Experimental Evaluations of Neural Network Algorithms: Current Research Practice, Neural Networks, Vol. 9
Provost F, Fawcett T and Kohavi R (1998). The case Against Accuracy Estimation for Comparing Induction Algorithms, Proceedings of the 15th International Conference on Machine Learning, 445 – 453
Provost F and Fawcett T (2001). Robust Classification for Imprecise Environment, Machine Learning, Vol.42, No.3, 203-231
Provost F and Fawcett T (1997). Analysis and Visualization of Classifier Performance: Combination under Imprecise Class and Cost Distributions, Proceedings of International Conference on Knowledge Discovery and Data Mining (KDD1997)
Provost F, Fawcett T, and Kohavi R (1998). The case Against Accuracy Estimation for Comparing Induction Algorithms, Proceedings of the 15th International Conference on Machine Learning, 445 – 453
Salzberg S (1997). On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach. Data Mining & Knowledge Discovery, Vol. 1, 317-328
Saxena A, Sankararaman S, and Goebel K,, (2014). Performance Evaluation for Fleet-based and Unit-based Prognostic Methods, European Conference of the Prognostics and Health Management Society
Saxena A, Celaya J., Saha B, Saha S, and Goebel K. (2009). On Applying the Prognostic Performance Metrics, Proceedings of International Conference on Prognostics and Health Management
Schwabacher M and Goebel K (2007). A Survey of Artificial Intelligence for Prognostics, The 2007 AAAI Fall Symposium, Arlington, Virginal, USA
Srinivasan A (1999). Note on the location of optimal classifiers in n-dimensional ROC space. Technical Report PRG-TR-2-99, Oxford University Computing Laboratory
Swets J, Dwaes R, et al. (2000). Psychological Science Can Improve Diagnostic decisions, Psychological Science in the Public Interest, Vol. 1, 1-26
Yang C and Létourneau S (2007). Model Evaluation for Prognostics: Estimating Cost Saving for the End Users, The Proceedings of the 6th International Conference on Machine Learning and Applications (ICMLA 2007), Cincinnati, OH, USA
Yang C and Létourneau S (2005). Learning to Predict Train Wheel Failures, Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD2005), 516-525
Zheng Z, Padmanabhan B, et al. (2004). A DEA Approach for Model Combination, Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD2004), Seattle, Washington, USA, 755-758
Bradley A (1997). The Use of the Area under the ROC Curve in the Evaluation of Machine Learning Algorithms, Pattern Recognition, Vol. 30, 1145-1159
Caruana R and Niculescu-Mizil A (2006). An Empirical Comparison of Supervised Learning Algorithms. Proceedings of the 23rd International Conference on Machine Learning (ICML2006)
Caruana R and Niculescu-Mizil A (2004). Data Mining in Metric Space: An empirical Analysis of Supervised Learning Performance Criteria, Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD2004), Seattle, Washington, USA, 69-78
Drummond C and Yang C (2008). Reverse-Engineering Costs: How much will a Prognostic Algorithm save?, Proceedings of the 1st International Conference on Prognostics and Health Management. Denver, USA
Drummond C and Holte R (2004). What ROC Curve Can’t Do (and Cost Curve Can), ECAI Workshop on ROC Analysis in Artificial Intelligence
Drummond C and Holte R (2000). Explicitly Representing Expected Cost: An Alternative to ROC Representation, Proceedings of the 6th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD2000), New York, USA, 155-164
Egan J (1975). Signal Detection Theory and ROC Analysis, New York Academic Press
Fawcett T (2003). ROC Graphs: Notes and Practical Considerations for Data Mining Researchers, Technical report, Intelligent Enterprise Technologies Laboratory, HP
Flach P, (2004). The Many Faces of ROC Analysis in Machine Learning, ICML’04 tutorial, http://www.cs.bris.ac.uk/~flach/ICML04tutorial/
Flach P, (2003). The Geometry of ROC Space: Understanding Machine Learning Metrics through ROC Isometrics. Proceedings of the 20th International Conference on Machine Learning (ICML03), Washington, DC, USA
Flexer A (1996). Statistical Evaluation of Neural Network Experiments: Minimum Requirement and Current Practice. Proceedings of the 13rd European Meeting on Cybernetics and Systems Research. Austrian Society for Cybernetic Studies, 1005-1008
Giudici P (2003). Applied Data Mining, John Wiley and Sons, New York
Hand D and Till R (2001). A Simple Generalization of the Area Under the ROC Curve for Multiple Class Classification Problems, Machine Learning, Vol.45, 171-186
Huang J and Ling C (2005). Using AUC and Accuracy in Evaluating Learning Algorithms, IEEE Transactions on Knowledge and Data Engineering, Vol. 17, No. 3, 299-310
Japkowicz N. and Shah Mohak (2011). Evaluation of Learning Algorithms, A Classification Perspective, Cambridge University Press
Jardine A., Lin D., and Banjevic D. (2006). A review on machinery diagnostics and prognostics implementing condition-based maintenance. Mechanical Systems and Signal Processing Vol.20, 1483-1510.
Létourneau S, Famili F, et al. (1999). Data Mining for Prediction of Aircraft Component Replacement, IEEE Intelligent Systems Journal, Special Issue on Data Mining. 59-6
Prechelt L (1996). A Quantative Study of Experimental Evaluations of Neural Network Algorithms: Current Research Practice, Neural Networks, Vol. 9
Provost F, Fawcett T and Kohavi R (1998). The case Against Accuracy Estimation for Comparing Induction Algorithms, Proceedings of the 15th International Conference on Machine Learning, 445 – 453
Provost F and Fawcett T (2001). Robust Classification for Imprecise Environment, Machine Learning, Vol.42, No.3, 203-231
Provost F and Fawcett T (1997). Analysis and Visualization of Classifier Performance: Combination under Imprecise Class and Cost Distributions, Proceedings of International Conference on Knowledge Discovery and Data Mining (KDD1997)
Provost F, Fawcett T, and Kohavi R (1998). The case Against Accuracy Estimation for Comparing Induction Algorithms, Proceedings of the 15th International Conference on Machine Learning, 445 – 453
Salzberg S (1997). On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach. Data Mining & Knowledge Discovery, Vol. 1, 317-328
Saxena A, Sankararaman S, and Goebel K,, (2014). Performance Evaluation for Fleet-based and Unit-based Prognostic Methods, European Conference of the Prognostics and Health Management Society
Saxena A, Celaya J., Saha B, Saha S, and Goebel K. (2009). On Applying the Prognostic Performance Metrics, Proceedings of International Conference on Prognostics and Health Management
Schwabacher M and Goebel K (2007). A Survey of Artificial Intelligence for Prognostics, The 2007 AAAI Fall Symposium, Arlington, Virginal, USA
Srinivasan A (1999). Note on the location of optimal classifiers in n-dimensional ROC space. Technical Report PRG-TR-2-99, Oxford University Computing Laboratory
Swets J, Dwaes R, et al. (2000). Psychological Science Can Improve Diagnostic decisions, Psychological Science in the Public Interest, Vol. 1, 1-26
Yang C and Létourneau S (2007). Model Evaluation for Prognostics: Estimating Cost Saving for the End Users, The Proceedings of the 6th International Conference on Machine Learning and Applications (ICMLA 2007), Cincinnati, OH, USA
Yang C and Létourneau S (2005). Learning to Predict Train Wheel Failures, Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD2005), 516-525
Zheng Z, Padmanabhan B, et al. (2004). A DEA Approach for Model Combination, Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD2004), Seattle, Washington, USA, 755-758
Section
Technical Papers