Evaluation of Remaining Useful Life Prediction Algorithms in the Absence of Run-to-Failure Ground Truth Data
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Abstract
Accurate evaluation of Remaining Useful Life (RUL) prediction algorithms is fundamental to the deployment of Prognostics and Health Management solutions. However, for critical industrial assets with extended operational lifespans, run-to-failure ground truth data is typically not available. Preventive maintenance intentionally precludes failure events, creating a fundamental challenge of assessing prognostic accuracy without observing actual end-of-life. This paper presents an algorithm-agnostic framework for the continuous online evaluation of RUL predictions in the absence of run-to-failure data. The innovation is a retrospective methodology that treats the asset’s current sensor state as a pseudo ground truth, enabling the evaluation of whether past predictions correctly anticipated the trajectory leading to the present condition. The framework includes two evaluation modes: (1) Measurement-based evaluation that assesses past sensor forecast accuracy against current observations, and (2) RUL-based evaluation that treats the current sensor value as a virtual degradation threshold and evaluates whether past RUL estimates correctly predicted the time to reach the present condition. The RUL-based evaluation adapts the well-established α–λ accuracy framework (Saxena, Celaya, et al., 2008) by replacing the unknown end-of-life with the current time as a pseudo ground truth reference, enabling continuous online assessment without failure observations. Individual prediction verdicts are aggregated using configurable weighting schemes into a single Service-Level Indicator suitable for performance monitoring. Experimental results across several industrial systems demonstrate the framework’s generalizability across diverse degradation mechanisms, sensor modalities, and prediction algorithms. The framework requires only historical sensor measurements and RUL predictions at different times.
How to Cite
##plugins.themes.bootstrap3.article.details##
Prognostics evaluation, Online performance monitoring
Chao, M. A., Kulkarni, C. S., Goebel, K. F., & Fink, O. (2020). Fusing physics-based and deep learning models for prognostics. Reliability Engineering & System Safety, 217, 107961.
Chen, Z., Wu, M., Zhao, R., Guretno, F., Yan, R., & Li, X. (2021). Machine remaining useful life prediction via an attention-based deep learning approach. IEEE Transactions on Industrial Electronics, 68(3), 2521–2531.
Cheng, H., Kong, X., Wang, Q., Ma, H., Yang, S., & Chen, G. (2023). Deep transfer learning based on dynamic domain adaptation for remaining useful life prediction under different working conditions. Journal of Intelligent Manufacturing, 34(2), 587–613.
da Costa, P. R. d. O., Akçay, A., Zhang, Y., & Kaymak, U. (2020). Remaining useful lifetime prediction via deep domain adaptation. Reliability Engineering & System Safety, 195, 106682.
Ding, Y., Ding, P., Zhao, X., Cao, Y., & Jia, M. (2022). Transfer learning for remaining useful life prediction across operating conditions based on multisource domain adaptation. IEEE/ASME Transactions on Mechatronics, 27(5), 4143–4152.
Forgione, M., Muni, A., Piga, D., & Gallieri, M. (2023). On the adaptation of recurrent neural networks for system identification. Automatica, 155, 111092.
Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2), 243–268.
Goebel, K., Daigle, M. J., Saxena, A., Roychoudhury, I., Sankararaman, S., & Celaya, J. R. (2017). Prognostics: The science of making predictions.
Goebel, K., Saxena, A., Saha, S., Saha, B., & Celaya, J. (2012). Prognostic performance metrics. In M. Pecht & M. Kang (Eds.). CRC Press.
Huang, Z., Xu, Z., Wang, W., & Sun, Y. (2015). Remaining useful life prediction for a nonlinear heterogeneous Wiener process model with an adaptive drift. IEEE Transactions on Reliability, 64(2), 687–700.
Jardine, A. K., Lin, D., & Banjevic, D. (2006). A review on machinery diagnostics and prognostics implementing condition-based maintenance. Mechanical Systems and Signal Processing, 20(7), 1483–1510.
Lee, J., Wu, F., Zhao, W., Ghaffari, M., Liao, L., & Siegel, D. (2014). Prognostics and health management design for rotary machinery systems: Reviews, methodology and applications. Mechanical Systems and Signal Processing, 42(1–2), 314–334.
Lei, Y., Li, N., Gontarz, S., Lin, J., Radkowski, S., & Dybala, J. (2016). A model-based method for remaining useful life prediction of machinery. IEEE Transactions on Reliability, 65(3), 1314–1326.
Lei, Y., Li, N., Guo, L., Li, N., Yan, T., & Lin, J. (2018). Machinery health prognostics: A systematic review from data acquisition to RUL prediction. Mechanical Systems and Signal Processing, 104, 799–834.
Ma, M., & Mao, Z. (2021). Deep-convolution-based LSTM network for remaining useful life prediction. IEEE Transactions on Industrial Informatics, 17(3), 1658–1667.
Ramasso, E., & Saxena, A. (2014). Performance benchmarking and analysis of prognostic methods for CMAPSS datasets. International Journal of Prognostics and Health Management, 5, 1–15.
Roychoudhury, I., Hafiychuk, V., & Goebel, K. (2013). Model-based diagnosis and prognosis of a water recycling system. In 2013 IEEE Aerospace Conference (pp. 1–9).
Sankararaman, S., & Goebel, K. (2015). Significance, interpretation, and quantification of uncertainty in prognostics and remaining useful life prediction. Mechanical Systems and Signal Processing, 52–53, 228–247.
Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, M. (2008). Metrics for evaluating performance of prognostic techniques. In 2008 International Conference on Prognostics and Health Management (pp. 1–17). IEEE.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2010). Metrics for offline evaluation of prognostic performance (Vol. 1, pp. 1–20).
Saxena, A., & Goebel, K. (2008). Turbofan engine degradation simulation data set [Data set]. NASA Ames Prognostics Data Repository. NASA Ames Research Center, Moffett Field, CA.
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008). Damage propagation modeling for aircraft engine run-to-failure simulation. In 2008 International Conference on Prognostics and Health Management (pp. 1–9). IEEE.
Sheth, P., & Roychoudhury, I. (2024). Robust remaining useful life prediction using Jacobian feature regression-based model adaptation. PHM Society European Conference, 8, 11.
Si, X.-S., Hu, C.-H., Chen, M.-Y., & Wang, W. (2011). An adaptive and nonlinear drift-based Wiener process for remaining useful life estimation. In 2011 Prognostics and System Health Management Conference (pp. 1–5).
Sikorska, J. Z., Hodkiewicz, M., & Ma, L. (2011). Prognostic modelling options for remaining useful life estimation by industry. Mechanical Systems and Signal Processing, 25(5), 1803–1836.
Taylor, S. J., & Letham, B. (2018). Forecasting at scale. The American Statistician, 72(1), 37–45.
Vachtsevanos, G., Lewis, F. L., Roemer, M., Hess, A., & Wu, B. (2006). Intelligent fault diagnosis and prognosis for engineering systems. John Wiley & Sons.
Wang, Y., Zhao, Y., & Addepalli, S. (2020). Remaining useful life prediction using deep learning approaches: A review. Procedia Manufacturing, 49, 81–88.
Zhang, Y., Xiong, R., He, H., & Liu, Z. (2017). A LSTM-RNN method for the lithium-ion battery remaining useful life prediction. In 2017 Prognostics and System Health Management Conference (PHM-Harbin) (pp. 1–4).
Zio, E. (2022). Prognostics and health management (PHM): Where are we and where do we need to go in theory and practice? Reliability Engineering & System Safety, 218, 108119.
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
The Prognostic and Health Management Society advocates open-access to scientific data and uses a Creative Commons license for publishing and distributing any papers. A Creative Commons license does not relinquish the author’s copyright; rather it allows them to share some of their rights with any member of the public under certain conditions whilst enjoying full legal protection. By submitting an article to the International Conference of the Prognostics and Health Management Society, the authors agree to be bound by the associated terms and conditions including the following:
As the author, you retain the copyright to your Work. By submitting your Work, you are granting anybody the right to copy, distribute and transmit your Work and to adapt your Work with proper attribution under the terms of the Creative Commons Attribution 3.0 United States license. You assign rights to the Prognostics and Health Management Society to publish and disseminate your Work through electronic and print media if it is accepted for publication. A license note citing the Creative Commons Attribution 3.0 United States License as shown below needs to be placed in the footnote on the first page of the article.
First Author et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.