Test-Training Leakage in Evaluation of Machine Learning Algorithms for Condition-Based Maintenance
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Abstract
Many articles have been published utilizing machine learning algorithms for condition-based maintenance through the analysis of vibration signals. One extensively researched topic is the classification of fault types in rolling bearings. There is a fairly widespread problem in the evaluation of these learning algorithms, where the separation of examples between the test and training sets is incorrect, leading to an optimistic conclusion about the algorithm's performance even when it is not the case. In this article, we will review this issue and explain how the data should be properly divided between the test and training sets to avoid this occurrence.
How to Cite
##plugins.themes.bootstrap3.article.details##
Test-Training Leakage, Machine learning, Condition-based maintenance, Bearing diagnosis
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press. https://www.deeplearningbook.org/
Hendriks, J., Dumond, P., & Knox, D. A. (2022). Towards better benchmarking using the CWRU bearing fault dataset. Mechanical Systems and Signal Processing, 169, 108732. https://doi.org/10.1016/J.YMSSP.2021.108732
Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. https://doi.org/10.1016/J.PATTER.2023.100804
Lei, Y. (2017). Intelligent fault diagnosis and remaining useful life prediction of rotating machinery. In Intelligent Fault Diagnosis and Remaining Useful Life Prediction of Rotating Machinery (1st ed.). Butterworth-Heinemann. https://doi.org/10.1016/C2016-0-00367-4
Lei, Y., Yang, B., Jiang, X., Jia, F., Li, N., & Nandi, A. K. (2020). Applications of machine learning to machine fault diagnosis: A review and roadmap. Mechanical Systems and Signal Processing, 138. https://doi.org/10.1016/j.ymssp.2019.106587
Lessmeier, C., Kimotho, J. K., Zimmer, D., & Sextro, W. (2016). Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. PHM Society European Conference, 3(1). https://doi.org/10.36001/PHME.2016.V3I1.1577
Liefstingh, M., Taal, C., Restrepo, S. E., & Azarfar, A. (2021). Interpretation of Deep Learning Models in Bearing Fault Diagnosis. Annual Conference of the PHM Society, 13(1). https://doi.org/10.36001/PHMCONF.2021.V13I1.3047
Matania, O., Bachar, L., Bechhoefer, E., & Bortman, J. (2024). Signal Processing for the Condition-Based Maintenance of Rotating Machines via Vibration Analysis: A Tutorial. Sensors 2024, Vol. 24, Page 454, 24(2), 454. https://doi.org/10.3390/S24020454
Matania, O., Bachar, L., Khemani, V., Das, D., Azarian, M. H., & Bortman, J. (2023). One-fault-shot learning for fault severity estimation of gears that addresses differences between simulation and experimental signals and transfer function effects. Advanced Engineering Informatics, 56, 101945. https://doi.org/10.1016/J.AEI.2023.101945
Randall, R. B. (2021). Vibration-based condition monitoring : industrial, automotive and aerospace applications (2nd ed.). WILEY. https://www.wiley.com/en-sg/Vibration+based+Condition+Monitoring%3A+Industrial%2C+Automotive+and+Aerospace+Applications%2C+2nd+Edition-p-9781119477556
Randall, R. B., & Antoni, J. (2011). Rolling element bearing diagnostics—A tutorial. Mechanical Systems and Signal Processing, 25(2), 485–520. https://doi.org/10.1016/J.YMSSP.2010.07.017
Shalev-Shwartz, S., & Ben-David, S. (2014a). Chapter 19 - Nearest Neighbor. In Understanding Machine Learning: From Theory to Algorithms (Vol. 9781107057135, pp. 258–267). Cambridge University Press. https://doi.org/10.1017/CBO9781107298019
Shalev-Shwartz, S., & Ben-David, S. (2014b). Section 18.3 - Random Forests. Understanding Machine Learning: From Theory to Algorithms, 9781107057135, 255–256. https://doi.org/10.1017/CBO9781107298019
Shalev-Shwartz, S., & Ben-David, S. (2014c). Understanding machine learning: From theory to algorithms. In Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press. https://doi.org/10.1017/CBO9781107298019
Smith, W. A., & Randall, R. B. (2015). Rolling element bearing diagnostics using the Case Western Reserve University data: A benchmark study. Mechanical Systems and Signal Processing, 64–65, 100–131. https://doi.org/10.1016/J.YMSSP.2015.04.021
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
The Prognostic and Health Management Society advocates open-access to scientific data and uses a Creative Commons license for publishing and distributing any papers. A Creative Commons license does not relinquish the author’s copyright; rather it allows them to share some of their rights with any member of the public under certain conditions whilst enjoying full legal protection. By submitting an article to the International Conference of the Prognostics and Health Management Society, the authors agree to be bound by the associated terms and conditions including the following:
As the author, you retain the copyright to your Work. By submitting your Work, you are granting anybody the right to copy, distribute and transmit your Work and to adapt your Work with proper attribution under the terms of the Creative Commons Attribution 3.0 United States license. You assign rights to the Prognostics and Health Management Society to publish and disseminate your Work through electronic and print media if it is accepted for publication. A license note citing the Creative Commons Attribution 3.0 United States License as shown below needs to be placed in the footnote on the first page of the article.
First Author et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.