Operational Prognostic Model Evaluation
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Sep 6, 2023
Shashvat Prakash
Katarina Vuckovic
Sanket Amin
Abstract
Prognostic analytic models have become a viable way to reduce operational interruptions when sufficient timely data is available. This work describes a set of evaluation metrics which can characterize model performance as a degradation estimate and as a decision enabler. The model accuracy over time is assessed against a correlation with the remaining useful life. This yields both a prediction accuracy and confidence interval. The decision can be based on the level of confidence around the prediction, which is based on both how far into the future the event is predicted and how well the current health and its deterioration is estimated. With an effective means of evaluating prognostic models, better benchmarks can be established to communicate model effectiveness and appropriately schedule routine service.
##plugins.themes.bootstrap3.article.details##
Keywords
Model Evaluation, Remaining Useful Life, Binary Classifier, Model Precision, Model Accuracy
References
Biggio, L., Wieland, A., Chao, M. A., Kastanis, I., and Fink, O., (2021) Uncertainty-Aware Prognosis via Deep Gaussian Process. IEEE Access, vol. 9, pp. 123517- 123527, 2021, doi: 10.1109/ACCESS.2021.3110049.
Deng, K., Zhang, X., Cheng, Y., Zheng, Z., Jiang, F., Liu, W. and Peng, J., (2019). A remaining useful life prediction method with automatic feature extraction for aircraft engines. Proceedings of 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/13th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE). IEEE, 2019, pp. 686–692.
Ding, Y., and Jia, M., (2021). A convolutional transformer architecture for remaining useful life estimation. Global Reliability and Prognostics and Health Management (PHM-Nanjing), pp. 1–7
Dua, D., and Graff, C., (2017). UCI machine learning repository. 2017. [Online]. Available: http://archive.ics.uci.edu/ml
Fawcett T. (2006). An Introduction to ROC Analysis, Pattern Recognition Letters. 27 (8): 861–874.
Fornlöf, V., Galar, D., Syberfeldt, A. and Almgren, T. (2016). Maintenance, prognostics and diagnostic approaches for aircraft engines, 2016 IEEE Metrology for Aerospace (MetroAeroSpace), pp. 403-407, doi: 10.1109/MetroAeroSpace.2016.7573249
International Maintenance Review Board Policy Board, Aircraft Health Monitoring (AHM) Integration in MSG 3, Issue Paper 180. (2018) MSG-3: Operator/ Manufacturer scheduled Maintenance Development, Vol. 1- Fixed Wing Aircraft Rev. 2018.1, Air Transport Association of America, https://www.easa.europa.eu/document-library/imrbpbissue-papers.
Le, D.D., Ghoshal, A., and Cuevas, E. (2011). Conditionbased maintenance plus and maintenance credit validation. Army Research Laboratory, Dept. of Defense, Aberdeen Proving Ground, MD 21005. https://apps.dtic.mil/ sti/pdfs/ADA589574.pdf
Li, G., Yang, L., Lee, C.-G., Wang, X., and Rong, M., (2021). A Bayesian deep learning RUL framework integrating epistemic and aleatoric uncertainties. IEEE Transactions on Industrial Electronics, vol. 68, no. 9, pp. 8829–8841.
Li, X., Ding, Q., and Sun, J.-Q., (2018) “Remaining useful life estimation in prognostics using deep convolution neural networks,” Reliability Engineering System Safety, vol. 172, pp. 1–11.
Liu, J., & Chen, Z. (2019). Remaining useful life prediction of lithium-ion batteries based on health indicator and gaussian process regression model. IEEE Access, 7, 39474-39484. doi: 10.1109/ACCESS.2019.2905740
Luo, J. H., Namburu, M., Pattipati, K., Liu, Q., Kawamoto, M., and Chigusa, S., (2003). Model-based prognostic techniques [maintenance applications], Proc. AUTOTESTCON 2003. IEEE Systems Readiness Technology Conference., pp. 330-340, (22-25 Sept.) doi: 10.1109/AUTEST.2003.1243596.
Mazaev, G., Crevecoeur, G., and Hoecke, S. V., (2021). Bayesian convolutional neural networks for remaining useful life prognostics of solenoid valves with uncertainty estimations. IEEE Transactions on Industrial Informatics, vol. 17, no. 12, pp. 8418–8428.
Pipe K. (2008). Practical prognostics for condition based maintenance, International Conference on Prognostics and Health Management.
Prakash, S., & Brzoska, A. (2021). Evaluating and Optimizing Analytic Signals. Annual Conference of the PHM Society, Virtual, 13(1). November 11-13, doi: 10.36001/phmconf.2021.v13i1.2968
Prakash, S., Brzoska A., and Ensberg, J. (2022). A Framework for Evaluating PHM Models. Proc. IEEE Aerospace Sciences Conference
Ren, L., Sun, Y., Wang, H., and Zhang, L., (2018) Prediction of bearing remaining useful life with deep convolution neural network. IEEE Access, vol. 6, pp. 13 041–13 049.
Song, Y., Li, L., Peng, Y., and Liu, D. (2018) Lithium-ion battery remaining useful life prediction based on GRURNN. Proceedings of 12th International Conference on Reliability, Maintainability, and Safety (ICRMS), pp. 317–322.
Sprong, J. P., Jiang, X., & Polinder, H. (2019). A Deployment of Prognostics to Optimize Aircraft Maintenance: A Literature Review. Annual Conference of the PHM Society, 11(1). doi: 10.36001/phmconf.2019.v11i1.776.
Vuckovic, K., and Prakash, S. (2022) Remaining useful life prediction using gaussian process regression model. Annual Conference of the PHM Society, vol. 14, no. 1.
Zhang, C., Lim, P., Qin, A. K., and Tan, K. C., (2017). Multiobjective deep belief networks ensemble for remaining useful life estimation in prognostics. IEEE Transactions on Neural Networks and Learning Systems, vol. 28, no. 10, pp. 2306–2318, 2017.
Zhang, X., Xu, R.., Kwan, C., Liang, S. Y., Xie, Q., and Haynes, L. (2005). An integrated approach to bearing fault diagnostics and prognostics. Proceedings of the 2005 American Control Conference. IEEE, pp. 2750– 2755.
Zhang, Y., Xiong, R., He, H., and Pecht, M. G., (2018). Long short-term memory recurrent neural network for remaining useful life prediction of lithium-ion batteries. IEEE Transactions on Vehicular Technology, vol. 67, no. 7, pp. 5695–5705.
Zhu, J., Chen, N., and Peng, W. (2018). Estimation of bearing remaining useful life based on multiscale convolutional neural network. IEEE Transactions on Industrial Electronics, vol. 66, no. 4, pp. 3208–3216.
Deng, K., Zhang, X., Cheng, Y., Zheng, Z., Jiang, F., Liu, W. and Peng, J., (2019). A remaining useful life prediction method with automatic feature extraction for aircraft engines. Proceedings of 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/13th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE). IEEE, 2019, pp. 686–692.
Ding, Y., and Jia, M., (2021). A convolutional transformer architecture for remaining useful life estimation. Global Reliability and Prognostics and Health Management (PHM-Nanjing), pp. 1–7
Dua, D., and Graff, C., (2017). UCI machine learning repository. 2017. [Online]. Available: http://archive.ics.uci.edu/ml
Fawcett T. (2006). An Introduction to ROC Analysis, Pattern Recognition Letters. 27 (8): 861–874.
Fornlöf, V., Galar, D., Syberfeldt, A. and Almgren, T. (2016). Maintenance, prognostics and diagnostic approaches for aircraft engines, 2016 IEEE Metrology for Aerospace (MetroAeroSpace), pp. 403-407, doi: 10.1109/MetroAeroSpace.2016.7573249
International Maintenance Review Board Policy Board, Aircraft Health Monitoring (AHM) Integration in MSG 3, Issue Paper 180. (2018) MSG-3: Operator/ Manufacturer scheduled Maintenance Development, Vol. 1- Fixed Wing Aircraft Rev. 2018.1, Air Transport Association of America, https://www.easa.europa.eu/document-library/imrbpbissue-papers.
Le, D.D., Ghoshal, A., and Cuevas, E. (2011). Conditionbased maintenance plus and maintenance credit validation. Army Research Laboratory, Dept. of Defense, Aberdeen Proving Ground, MD 21005. https://apps.dtic.mil/ sti/pdfs/ADA589574.pdf
Li, G., Yang, L., Lee, C.-G., Wang, X., and Rong, M., (2021). A Bayesian deep learning RUL framework integrating epistemic and aleatoric uncertainties. IEEE Transactions on Industrial Electronics, vol. 68, no. 9, pp. 8829–8841.
Li, X., Ding, Q., and Sun, J.-Q., (2018) “Remaining useful life estimation in prognostics using deep convolution neural networks,” Reliability Engineering System Safety, vol. 172, pp. 1–11.
Liu, J., & Chen, Z. (2019). Remaining useful life prediction of lithium-ion batteries based on health indicator and gaussian process regression model. IEEE Access, 7, 39474-39484. doi: 10.1109/ACCESS.2019.2905740
Luo, J. H., Namburu, M., Pattipati, K., Liu, Q., Kawamoto, M., and Chigusa, S., (2003). Model-based prognostic techniques [maintenance applications], Proc. AUTOTESTCON 2003. IEEE Systems Readiness Technology Conference., pp. 330-340, (22-25 Sept.) doi: 10.1109/AUTEST.2003.1243596.
Mazaev, G., Crevecoeur, G., and Hoecke, S. V., (2021). Bayesian convolutional neural networks for remaining useful life prognostics of solenoid valves with uncertainty estimations. IEEE Transactions on Industrial Informatics, vol. 17, no. 12, pp. 8418–8428.
Pipe K. (2008). Practical prognostics for condition based maintenance, International Conference on Prognostics and Health Management.
Prakash, S., & Brzoska, A. (2021). Evaluating and Optimizing Analytic Signals. Annual Conference of the PHM Society, Virtual, 13(1). November 11-13, doi: 10.36001/phmconf.2021.v13i1.2968
Prakash, S., Brzoska A., and Ensberg, J. (2022). A Framework for Evaluating PHM Models. Proc. IEEE Aerospace Sciences Conference
Ren, L., Sun, Y., Wang, H., and Zhang, L., (2018) Prediction of bearing remaining useful life with deep convolution neural network. IEEE Access, vol. 6, pp. 13 041–13 049.
Song, Y., Li, L., Peng, Y., and Liu, D. (2018) Lithium-ion battery remaining useful life prediction based on GRURNN. Proceedings of 12th International Conference on Reliability, Maintainability, and Safety (ICRMS), pp. 317–322.
Sprong, J. P., Jiang, X., & Polinder, H. (2019). A Deployment of Prognostics to Optimize Aircraft Maintenance: A Literature Review. Annual Conference of the PHM Society, 11(1). doi: 10.36001/phmconf.2019.v11i1.776.
Vuckovic, K., and Prakash, S. (2022) Remaining useful life prediction using gaussian process regression model. Annual Conference of the PHM Society, vol. 14, no. 1.
Zhang, C., Lim, P., Qin, A. K., and Tan, K. C., (2017). Multiobjective deep belief networks ensemble for remaining useful life estimation in prognostics. IEEE Transactions on Neural Networks and Learning Systems, vol. 28, no. 10, pp. 2306–2318, 2017.
Zhang, X., Xu, R.., Kwan, C., Liang, S. Y., Xie, Q., and Haynes, L. (2005). An integrated approach to bearing fault diagnostics and prognostics. Proceedings of the 2005 American Control Conference. IEEE, pp. 2750– 2755.
Zhang, Y., Xiong, R., He, H., and Pecht, M. G., (2018). Long short-term memory recurrent neural network for remaining useful life prediction of lithium-ion batteries. IEEE Transactions on Vehicular Technology, vol. 67, no. 7, pp. 5695–5705.
Zhu, J., Chen, N., and Peng, W. (2018). Estimation of bearing remaining useful life based on multiscale convolutional neural network. IEEE Transactions on Industrial Electronics, vol. 66, no. 4, pp. 3208–3216.
Section
Special Session Papers
This work is licensed under a Creative Commons Attribution 3.0 Unported License.