Remaining Useful Life Estimation of Bearings Meta-analysis of Experimental Procedure



Published Mar 24, 2021
Hugo M. Ferreira Alexandre C. de Sousa


In the domain of predictive maintenance, when trying to repli- cate and compare research in remaining useful life estimation (RUL), several inconsistencies and errors were identified in the experimental methodology used by various researchers. This makes the replication and the comparison of results diffi- cult, thus severely hindering both progress in this research do- main and its practical application to industry. We survey the literature to evaluate the experimental procedures that were used, and identify the most common errors and omission in both experimental procedures and reporting.

A total of 70 papers on RUL were audited. From this meta- analysis we estimate that approximately 11% of the papers present work that will allow for replication and comparison. Surprisingly, only about 24.3% (17 of the 70 articles) com- pared their results with previous work. Of the remaining work, 41.4% generated and compared several models of their own and, somewhat unsettling, 31.4% of the researchers made no comparison whatsoever. The remaining 2.9% did not use the same data set for comparisons. The results of this study were also aggregated into 3 categories: problem class selec- tion, model fitting best practices and evaluation best practices. We conclude that model evaluation is the most problematic one.

The main contribution of the article is a proposal of an ex- perimental protocol and several recommendations that specif- ically target model evaluation. Adherence to this protocol should substantially facilitate the research and application of RUL prediction models. The goals are to promote the collab- oration between scholars and practitioners alike and advance the research in this domain.

Abstract 352 | PDF Downloads 339



data mining and machine learning, methodology, meta-analysis, prognostics, bearings, remaining useful life

Ben Ali, J., Saidi, L., Harrath, S., Bechhoefer, E., & Ben- bouzid, M. (2018). Online automatic diagnosis of wind turbine bearings progressive degradations under real experimental conditions based on unsupervised machine learning. Applied Acoustics, 132(November 2017), 167–181. doi: 10.1016/j.apacoust.2017.11.021
Benkedjouh, T., Medjaher, K., Zerhouni, N., & Rechak, S. (2013, aug). Remaining useful life estimation based on nonlinear feature reduction and support vector regression. Engineering Applications of Artificial Intelligence, 26(7), 1751–1760. Retrieved from{\%}3Dihub doi: 10.1016/j.engappai.2013.02.006
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer-Verlag New York.
Bozchalooi, I. S., & Liang, M. (2008). A joint resonance frequency estimation and in-band noise reduction method for enhancing the detectability of bearing fault signals. Mechanical Systems and Signal Processing, 22(4), 915–933. doi: 10.1016/j.ymssp.2007.10.006
Cheng, Z. (2017, aug). Residual useful life prediction for rolling element bearings based on multi-feature fusion regression. In Proceedings - 2017 international conference on sensing, diagnostics, prognostics, and control, sdpc 2017 (Vol. 2017-Decem, pp. 246–250). IEEE. Retrieved from document/8186510/ doi: 10.1109/SDPC.2017.54
Colquhoun, D. (2018). Correction to ‘the reproducibility of research and the misinterpretation of p-values’. Royal Society Open Science, 5.
Cortez, P., & Embrechts, M. J. (2013). Using sensitivity analysis and visualization techniques to open black box data mining models. Information Sciences, 225, 1 - 17. Retrieved from http:// article/pii/S0020025512007098 doi:
Dong, G., & Chen, J. (2012). Noise resistant time frequency analysis and application in fault diagnosis of rolling element bearings. Mechanical Systems and Signal Processing, 33, 212–236. Retrieved from doi: 10.1016/j.ymssp.2012.06.008
Duong, B. P., Khan, S. A., Shon, D., Im, K., Park, J., Lim, D. S., . . . Kim, J. M. (2018, nov). A reliable health indicator for fault prognosis of bearings. Sensors (Switzerland), 18(11), 3740. Retrieved from http:// doi: 10.3390/s18113740
England, J. R., & Cheng, P. M. (2019, mar). Artificial intelligence for medical image analysis: A guide for authors and reviewers (Vol. 212) (No. 3). American Roentgen Ray Society. Retrieved from doi: 10.2214/AJR.18.20490
Gama, J. (2010). Knowledge discovery from data streams (1st ed.). Chapman & Hall/CRC. doi: 10.1201/EBK1439826119
Giles, C. L., & Lawrence, S. (1997). Presenting and analyzing the results of AI experiments: data averaging and data snooping. Proceedings of the National Conference on Artificial Intelligence, 362–367.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Retrieved from
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer-Verlag New York. doi: 10.1007/978-0-387-84858-7
Heng, A., Zhang, S., Tan, A. C., & Mathew, J. (2009). Rotating machinery prognostics: State of the art, challenges and opportunities. Mechanical Systems and Signal Processing, 23(3), 724–739. doi: 10.1016/j.ymssp.2008.06.009
Ioannidis, J. P. A. (2005, 08). Why most published research findings are false. PLoS Med, 2(8), e124. Retrieved from 2Fjournal.pmed.0020124 doi: 10.1371/journal.pmed.0020124
Jensen, D. (2000, jan). Data snooping, dredging and fishing. ACM SIGKDD Explorations Newsletter, 1(2), 52. Retrieved from citation.cfm?doid=846183.846195 doi: 10.1145/846183.846195
Jiang, J.-R., Lee, J.-E., & Zeng, Y.-M. (2019, dec). Time Series Multiple Channel Convolutional Neural Network with Attention-Based Long Short-Term Memory for Predicting Bearing Remaining Useful Life. Sensors, 20(1), 166. Retrieved from doi: 10.3390/s20010166
Kan, M. S., Tan, A. C., & Mathew, J. (2015). A review on prognostic techniques for non-stationary and non-linear rotating systems. Mechanical Systems and Signal Processing, 62, 1–20. retrieved from doi: 10.1016/j.ymssp.2015.02.016
Kaufman, S., Rosset, S., & Perlich, C. (2011). Leakage in data mining: Formulation, detection, and avoidance. In Proceedings of the 17th acm sigkdd international conference on knowledge discovery and data mining (p. 556–563). New York, NY, USA: Association for Computing Machinery. Retrieved from https:// doi: 10.1145/2020408.2020496
Khelif, R., Chebel-Morello, B., Malinowski, S., Laajili, E., Fnaiech, F., & Zerhouni, N. (2017). Direct Remaining Useful Life Estimation Based on Support Vector regression. IEEE Transactions on Industrial Electronics. doi: 10.1109/TIE.2016.2623260
Lei, Y., Li, N., Guo, L., Li, N., Yan, T., & Lin, J. (2018). Machinery health prognostics: A systematic review from data acquisition to RUL prediction. Mechanical Systems and Signal Processing, 104, 799–834. Retrieved from doi: 10.1016/j.ymssp.2017.11.016
Li, X., Zhang, W., & Ding, Q. (2019). Deep learning-based remaining useful life estimation of bearings using multi-scale feature extraction. Reliability Engineering and System Safety, 182, 208–218. doi: 10.1016/j.ress.2018.11.011
Li, X., Zhang, W., Ma, H., Luo, Z., & Li, X. (2020, jun). Data alignments in machinery remaining useful life prediction using deep adversarial neural networks. Knowledge-Based Systems, 197, 105843. retrieved from{\%}3Dihub doi: 10.1016/J.KNOSYS.2020.105843
Liu, C., Zhang, L., & Wu, C. (2019, dec). Direct remaining Useful Life Prediction for Rolling Bearing Using Temporal Convolutional Networks. In 2019 ieee symposium series on computational intelligence, ssci 2019 (pp. 2965–2971). Institute of Electrical and Electronics Engineers Inc. doi: 10.1109/SSCI44817.2019.9003163
Liu, Z., Zuo, M. J., & Qin, Y. (2015, jun). Remaining useful life prediction of rolling element bearings based on health state assessment. Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science, 230(2), 314–330. Retrieved from 10.1177/0954406215590167 doi: 10.1177/0954406215590167
Machacek, V., & Srholec, M. (2019). Predatory publications in scopus: Evidence on cross-country differences (Working Papers IES No. 2019/20). Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies. Retrieved from fau:wpaper:wp2019 20
Mao, W., He, J., Tang, J., & Li, Y. (2018, dec). Predicting remaining useful life of rolling bearings based on deep feature representation and long short-term memory neural network. Advances in Mechanical Engineering, 10(12), 168781401881718. retrieved from doi/10.1177/1687814018817184 doi: 10.1177/1687814018817184
Measurement Computing Corporation. (2004). Data Acquisition Handbook. A Reference for DAQ And Analog & Digital Signal conditioning (3rd ed.). MA: Author. retrieved from anpdf/Data-Acquisition-Handbook.pdf
Moreno-Torres, J. G., Saez, J. A., & Herrera, F. (2012). Study on the impact of partition-induced dataset shift on k-fold cross-validation. IEEE Transactions on Neural Networks and Learning Systems, 23(8), 1304-1312.
Munafo`, M., Nosek, B., Bishop, D., Button, K., Chambers, C., Percie Du Sert, N., . . . Ioannidis, J. (2017, 1 10). A manifesto for reproducible science. Nature Human Behaviour, 1(1). doi: 10.1038/s41562-016-0021
Murphy, K. (2017, 12). Harking: How badly can cherry-picking and question trolling produce bias in published results? Journal of Business and Psychology. doi: 10.1007/s10869-017-9524-7
Nectoux, P., Gouriveau, R., Medjaher, K., Ramasso, E., Chebel-morello, B., Zerhouni, N., . . . Varnier, C. (2012). PRONOSTIA : An experimental platform for bearings accelerated degradation tests. IEEE International Conference on Prognostics and Health Management, 1–8.
P. Murphy, K. (2012). Machine Learning: A Probabilistic Perspective. Cambridge, MA: The MIT Press.
Peng, Y., Cheng, J., Liu, Y., Li, X., & Peng, Z. (2018, jun). An adaptive data-driven method for accurate prediction of remaining useful life of rolling bearings. Frontiers of Mechanical Engineering, 13(2), 301–310. doi: 10.1007/s11465-017-0449-7
Qiu, H., Lee, J., Lin, J., & Yu, G. (2006). Wavelet filter-based weak signature detection method and its application on rolling element bearing prognostics. Journal of Sound and Vibration, 289(4), 1066 - 1090. Retrieved from http:// article/pii/S0022460X0500221X doi:
Russell, S., & Norvig, P. (2010). Artificial Intelligence A Modern Approach (3rd ed.). Pearson. Retrieved from
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2009). On applying the prognostic performance metrics. Annual Conference of the Prognostics and Health Management Society, PHM 2009, 1–16.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2010). Metrics for offline evaluation of prognostic performance. International Journal of Prognostics and Health Management, 1(1), 1–20.
Shan, P., Hou, P., Ge, H., Yu, L., Li, Y., & Gu, L. (2019, oct). Image Feature-Based for Bearing Health Monitoring with Deep-Learning Method. In 2019 prognostics and system health management conference (phm-qingdao) (pp. 1–6). IEEE. retrieved from document/8942972/ doi: 10.1109/PHM-Qingdao46334.2019.8942972
Shepperd, M. J., Guo, Y., Li, N., Arzoky, M., Capiluppi, A., Counsell, S., . . . Yousefi, L. (2019). The prevalence of errors in machine learning experiments. In H. Yin, D. Camacho, P. Tin˜o, A. J. Tallo´n-Ballesteros, R. Menezes, & R. Allmendinger (Eds.), Intelligent data engineering and automated learning - IDEAL 2019 - 20th international conference, manchester, uk, november 14-16, 2019, proceedings, part I (Vol. 11871, pp. 102–109). Springer. retrieved from\ 12 doi: 10.1007/978-3-030-33607-3 12
Soualhi, A., Medjaher, K., & Zerhouni, N. (2015, jan). Bearing Health Monitoring Based on Hilbert–Huang Transform, Support Vector Machine, and regression. IEEE Transactions on Instrumentation and Measurement, 64(1), 52–62. Retrieved from 6847199/ doi: 10.1109/TIM.2014.2330494
Sutrisno, E., Oh, H., Vasan, A. S. S., & Pecht, M. (2012). Estimation of remaining useful life of ball bearings using data driven methodologies. In 2012 ieee conference on prognostics and health management (pp. 1–7). doi: 10.1109/ICPHM.2012.6299548
Taylor, H. R. (1997). Data Acquisition for Sensor Systems (1st ed.). Springer US. doi: 10.1007/978-1-4757-4905-2
Verstraete, D., Droguett, E., & Modarres, M. (2019, dec). A Deep Adversarial Approach Based on Multi-Sensor Fusion for Semi-Supervised Remaining Useful Life Prognostics. Sensors, 20(1), 176. Retrieved from doi: 10.3390/s20010176
Vlcek, B. L., Hendricks, R. C., & Zaretsky, E. V. (2003, jan). Determination of Rolling-Element Fatigue Life From Computer Generated Bearing Tests. Tribology Transactions, 46(4), 479–493. retrieved from doi/abs/10.1080/10402000308982654 doi: 10.1080/10402000308982654
Wang, B., Lei, Y., Li, N., & Li, N. (2020). A Hybrid Prognostics Approach for Estimating Remaining Useful Life of Rolling Element Bearings. IEEE Transactions on Reliability, 69(1), 401–412. doi: 10.1109/TR.2018.2882682
Wang, B., Lei, Y., Yan, T., Li, N., & Guo, L. (2020, feb). Recurrent convolutional neural network: A new framework for remaining useful life prediction of machinery. Neurocomputing, 379, 117–129. doi: 10.1016/j.neucom.2019.10.064
Wang, T. (2012, jun). Bearing life prediction based on vibration signals: A case study and lessons learned. In Phm 2012 - 2012 ieee int. conf.on prognostics and health management: Enhancing safety, efficiency, availability, and effectiveness of systems through phm technology and application, conference program (pp. 1– 7). IEEE. Retrieved from doi: 10.1109/ICPHM.2012.6299547
Wang, Y., Peng, Y., Zi, Y., Jin, X., & Tsui, K.-L. (2015, jul). An integrated Bayesian approach to prognositics of the remaining useful life and its application on bearing degradation problem. In 2015 ieee 13th international conference on industrial informatics (indin) (pp. 1090– 1095). IEEE. Retrieved from doi: 10.1109/INDIN.2015.7281887
Williams, T., Ribadeneira, X., Billington, S., & Kurfess, T. (2001). Rolling element bearing diagnostics in run-to-failure lifetime testing. Mechanical Systems and Signal Processing, 15(5), 979 - 993. Retrieved from http:// article/pii/S0888327001914189 doi:
Xia, M., Li, T., Shu, T., Wan, J., De Silva, C. W., & Wang, Z. (2019). A Two-Stage Approach for the Remaining Useful Life Prediction of Bearings Using Deep Neural Networks. IEEE Transactions on Industrial Informatics, 15(6), 3703–3711. doi: 10.1109/TII.2018.2868687
Zhang, Y., Hutchinson, P., Lieven, N. A., & Nunez-Yanez, J. (2020). Remaining useful life estimation using long short-term memory neural networks and deep fusion. IEEE Access, 8, 19033–19045. doi: 10.1109/ACCESS.2020.2966827
Zheng, Y. (2019, dec). Predicting Remaining Useful Life Using Continuous Wavelet Transform Integrated Discrete Teager Energy Operator with Degradation Model. In 2019 ieee 5th international conference on computer and communications, iccc 2019 (pp. 240–244). IEEE. Retrieved from doi: 10.1109/ICCC47050.2019.9064232
Technical Papers