Improving Semi-Supervised Learning for Remaining Useful Lifetime Estimation Through Self-Supervision
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Jan 14, 2022
Tilman Krokotsch
Mirko Knaak
Clemens G¨uhmann
Abstract
RUL estimation plays a vital role in effectively scheduling maintenance operations. Unfortunately, it suffers from a severe data imbalance where data from machines near their end of life is rare. Additionally, the data produced by a machine can only be labeled after the machine failed. Both of these points make using data-driven methods for RUL estimation difficult. Semi-Supervised Learning (SSL) can incorporate the unlabeled data produced by machines that did not yet fail into data-driven methods. Previous work on SSL evaluated approaches under unrealistic conditions where the data near failure was still available. Even so, only moderate improvements were made. This paper defines more realistic evaluation conditions and proposes a novel SSL approach based on self-supervised pre-training. The method can outperform two competing approaches from the literature and the supervised baseline on the NASA Commercial Modular Aero-Propulsion System Simulation dataset.
##plugins.themes.bootstrap3.article.details##
Keywords
rul estimation, deep learning, semi-supervised learning, predictive maintenance
References
Bachman, P., Alsharif, O., & Precup, D. (2014). Learning with pseudo-ensembles. In Proceedings of the 27th international conference on neural information processing systems-volume 2 (pp. 3365–3373).
Bai, S., Kolter, J. Z., & Koltun, V. (2018). An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv.
Baldi, P., & Chauvin, Y. (1993). Neural networks for fingerprint recognition. neural computation, 5(3), 402–418.
Bromley, J., Bentz, J. W., Bottou, L., Guyon, I., LeCun, Y., Moore, C., . . . Shah, R. (1993). Signature verification using a “siamese” time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7(04), 669–688.
Cheng, C., Zhou, B., Ma, G., Wu, D., & Yuan, Y. (2019). Wasserstein Distance based Deep Adversarial Transfer Learning for Intelligent Fault Diagnosis. arXiv.
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Conference of the north american chapter of the association for computational linguistics: Human language technologies (Vol. 1, pp. 4171–4186).
Doersch, C., Gupta, A., & Efros, A. A. (2015, December). Unsupervised visual representation learning by context prediction. In Proceedings of the ieee international conference on computer vision (iccv).
Franceschi, J.-Y., Dieuleveut, A., & Jaggi, M. (2019). Unsupervised scalable representation learning for multivariate time series. In Advances in neural information processing systems (Vol. 32, pp. 4650–4661). Curran Associates, Inc.
Gebraeel, N., Lawley, M., Liu, R., & Parmeshwaran, V. (2004). Residual life predictions from vibration-based degradation signals: A neural network approach. IEEE Transactions on Industrial Electronics, 51, 694–700. doi: 10.1109/TIE.2004.824875
Gidaris, S., Singh, P., & Komodakis, N. (2018). Unsupervised representation learning by predicting image rotations. In International conference on learning representation.
He, R., Dai, Y., Lu, J., & Mou, C. (2018). Developing ladder network for intelligent evaluation system: Case of remaining useful life prediction for centrifugal pumps. Reliability Engineering & System Safety, 180, 385-393. doi: https://doi.org/10.1016/j.ress.2018.08.010
Heimes, F. O. (2008). Recurrent neural networks for remaining useful life estimation. In International conference on prognostics and health management.
Ioffe, S., & Szegedy, C. (2015, 07–09 Jul). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In F. Bach & D. Blei (Eds.), Proceedings of the 32nd international conference on machine learning (Vol. 37, pp. 448–456). Lille, France: PMLR.
Jiang, J.-R., Lee, J.-E., & Zeng, Y.-M. (2020). Time Series Multiple Channel Convolutional Neural Network with Attention-Based Long Short-Term Memory for Predicting Bearing Remaining Useful Life. Sensors, 20(166).
Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
Kingma, D. P., Rezende, D. J., Mohamed, S., & Welling, M. (2014). Semi-supervised learning with deep generative models. In Proceedings of the 27th international conference on neural information processing systems - volume 2 (pp. 3581–3589).
Krokotsch, T., Knaak, M., & G¨uhmann, C. (2020). A novel evaluation framework for unsupervised domain adaption on remaining useful lifetime estimation. In 2020 ieee international conference on prognostics and health management (icphm). doi: 10.1109/ICPHM49022.2020.9187058
Li, X., Ding, Q., & Sun, J. Q. (2018). Remaining useful life estimation in prognostics using deep convolution neural networks. Reliability Engineering and System Safety, 172. doi: 10.1016/j.ress.2017.11.021
Listou Ellefsen, A., Bjørlykhaug, E., Æsøy, V., Ushakov, S., & Zhang, H. (2019). Remaining useful life predictions for turbofan engine degradation using semi-supervised deep architecture. Reliability Engineering and System Safety, 183, 240–251. doi: 10.1016/j.ress.2018.11.027
McInnes, L., Healy, J., Saul, N., & Grossberger, L. (2018). Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29), 861.
Musgrave, K., Belongie, S., & Lim, S.-N. (2020). A metric learning reality check. In A. ”Vedaldi, H. Bischof, T. Brox, & J.-M. Frahm (Eds.), ”computer vision – eccv 2020” (pp. ”681–699”). ”Springer International Publishing”.
Oliver, A., Odena, A., Raffel, C. A., Cubuk, E. D., & Goodfellow, I. (2018). Realistic evaluation of deep semisupervised learning algorithms. Advances in Neural Information Processing Systems, 31, 3235–3246.
Rasmus, A., Valpola, H., Honkala, M., Berglund, M., & Raiko, T. (2015). Semi-supervised learning with ladder networks. In Proceedings of the 28th international conference on neural information processing systems - volume 2 (pp. 3546–3554).
Sateesh Babu, G., Zhao, P., & Li, X.-L. (2016). Deep Convolutional Neural Network Based Regression Approach for Estimation of Remaining Useful Life. Database Systems for Advanced Applications, 9642, 214–228. doi: 10.1007/978-3-319-32025-0
Saxena, A., & Goebel, K. (2008). Turbofan engine degradation simulation data set. NASA Ames Prognostics Data Repository.
Si, X. S.,Wang,W., Hu, C. H., & Zhou, D. H. (2011). Remaining useful life estimation - A review on the statistical data driven approaches. European Journal of Operational Research, 213. doi: 10.1016/j.ejor.2010.11.018
van Engelen, J. E., & Hoos, H. H. (2020). A survey on semisupervised learning. Machine Learning, 109, 373–440. doi: 10.1007/s10994-019-05855-6
Wu, Y., Yuan, M., Dong, S., Lin, L., & Liu, Y. (2018). Remaining useful life estimation of engineered systems using vanilla LSTM neural networks. Neurocomputing, 275, 167–179. doi: 10.1016/j.neucom.2017.05.063
Yarowsky, D. (1995). Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics (pp. 189–196).
Yoon, A. S., Lee, T., Lim, Y., Jung, D., Kang, P., Kim, D., . . . Choi, Y. (2017). Semi-supervised learning with deep generative models for asset failure prediction. arXiv.
Zheng, S., Ristovski, K., Farahat, A., & Gupta, C. (2017). Long Short-Term Memory Network for Remaining Useful Life estimation. In Ieee international conference on prognostics and health management (pp. 88–95). IEEE. doi: 10.1109/ICPHM.2017.7998311
Zhu, J., Chen, N., & Peng, W. (2019). Estimation of Bearing Remaining Useful Life Based on Multiscale Convolutional Neural Network. IEEE Transactions on Industrial Electronics, 66. doi: 10.1109/TIE.2018.2844856
Bai, S., Kolter, J. Z., & Koltun, V. (2018). An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv.
Baldi, P., & Chauvin, Y. (1993). Neural networks for fingerprint recognition. neural computation, 5(3), 402–418.
Bromley, J., Bentz, J. W., Bottou, L., Guyon, I., LeCun, Y., Moore, C., . . . Shah, R. (1993). Signature verification using a “siamese” time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7(04), 669–688.
Cheng, C., Zhou, B., Ma, G., Wu, D., & Yuan, Y. (2019). Wasserstein Distance based Deep Adversarial Transfer Learning for Intelligent Fault Diagnosis. arXiv.
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Conference of the north american chapter of the association for computational linguistics: Human language technologies (Vol. 1, pp. 4171–4186).
Doersch, C., Gupta, A., & Efros, A. A. (2015, December). Unsupervised visual representation learning by context prediction. In Proceedings of the ieee international conference on computer vision (iccv).
Franceschi, J.-Y., Dieuleveut, A., & Jaggi, M. (2019). Unsupervised scalable representation learning for multivariate time series. In Advances in neural information processing systems (Vol. 32, pp. 4650–4661). Curran Associates, Inc.
Gebraeel, N., Lawley, M., Liu, R., & Parmeshwaran, V. (2004). Residual life predictions from vibration-based degradation signals: A neural network approach. IEEE Transactions on Industrial Electronics, 51, 694–700. doi: 10.1109/TIE.2004.824875
Gidaris, S., Singh, P., & Komodakis, N. (2018). Unsupervised representation learning by predicting image rotations. In International conference on learning representation.
He, R., Dai, Y., Lu, J., & Mou, C. (2018). Developing ladder network for intelligent evaluation system: Case of remaining useful life prediction for centrifugal pumps. Reliability Engineering & System Safety, 180, 385-393. doi: https://doi.org/10.1016/j.ress.2018.08.010
Heimes, F. O. (2008). Recurrent neural networks for remaining useful life estimation. In International conference on prognostics and health management.
Ioffe, S., & Szegedy, C. (2015, 07–09 Jul). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In F. Bach & D. Blei (Eds.), Proceedings of the 32nd international conference on machine learning (Vol. 37, pp. 448–456). Lille, France: PMLR.
Jiang, J.-R., Lee, J.-E., & Zeng, Y.-M. (2020). Time Series Multiple Channel Convolutional Neural Network with Attention-Based Long Short-Term Memory for Predicting Bearing Remaining Useful Life. Sensors, 20(166).
Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
Kingma, D. P., Rezende, D. J., Mohamed, S., & Welling, M. (2014). Semi-supervised learning with deep generative models. In Proceedings of the 27th international conference on neural information processing systems - volume 2 (pp. 3581–3589).
Krokotsch, T., Knaak, M., & G¨uhmann, C. (2020). A novel evaluation framework for unsupervised domain adaption on remaining useful lifetime estimation. In 2020 ieee international conference on prognostics and health management (icphm). doi: 10.1109/ICPHM49022.2020.9187058
Li, X., Ding, Q., & Sun, J. Q. (2018). Remaining useful life estimation in prognostics using deep convolution neural networks. Reliability Engineering and System Safety, 172. doi: 10.1016/j.ress.2017.11.021
Listou Ellefsen, A., Bjørlykhaug, E., Æsøy, V., Ushakov, S., & Zhang, H. (2019). Remaining useful life predictions for turbofan engine degradation using semi-supervised deep architecture. Reliability Engineering and System Safety, 183, 240–251. doi: 10.1016/j.ress.2018.11.027
McInnes, L., Healy, J., Saul, N., & Grossberger, L. (2018). Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29), 861.
Musgrave, K., Belongie, S., & Lim, S.-N. (2020). A metric learning reality check. In A. ”Vedaldi, H. Bischof, T. Brox, & J.-M. Frahm (Eds.), ”computer vision – eccv 2020” (pp. ”681–699”). ”Springer International Publishing”.
Oliver, A., Odena, A., Raffel, C. A., Cubuk, E. D., & Goodfellow, I. (2018). Realistic evaluation of deep semisupervised learning algorithms. Advances in Neural Information Processing Systems, 31, 3235–3246.
Rasmus, A., Valpola, H., Honkala, M., Berglund, M., & Raiko, T. (2015). Semi-supervised learning with ladder networks. In Proceedings of the 28th international conference on neural information processing systems - volume 2 (pp. 3546–3554).
Sateesh Babu, G., Zhao, P., & Li, X.-L. (2016). Deep Convolutional Neural Network Based Regression Approach for Estimation of Remaining Useful Life. Database Systems for Advanced Applications, 9642, 214–228. doi: 10.1007/978-3-319-32025-0
Saxena, A., & Goebel, K. (2008). Turbofan engine degradation simulation data set. NASA Ames Prognostics Data Repository.
Si, X. S.,Wang,W., Hu, C. H., & Zhou, D. H. (2011). Remaining useful life estimation - A review on the statistical data driven approaches. European Journal of Operational Research, 213. doi: 10.1016/j.ejor.2010.11.018
van Engelen, J. E., & Hoos, H. H. (2020). A survey on semisupervised learning. Machine Learning, 109, 373–440. doi: 10.1007/s10994-019-05855-6
Wu, Y., Yuan, M., Dong, S., Lin, L., & Liu, Y. (2018). Remaining useful life estimation of engineered systems using vanilla LSTM neural networks. Neurocomputing, 275, 167–179. doi: 10.1016/j.neucom.2017.05.063
Yarowsky, D. (1995). Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics (pp. 189–196).
Yoon, A. S., Lee, T., Lim, Y., Jung, D., Kang, P., Kim, D., . . . Choi, Y. (2017). Semi-supervised learning with deep generative models for asset failure prediction. arXiv.
Zheng, S., Ristovski, K., Farahat, A., & Gupta, C. (2017). Long Short-Term Memory Network for Remaining Useful Life estimation. In Ieee international conference on prognostics and health management (pp. 88–95). IEEE. doi: 10.1109/ICPHM.2017.7998311
Zhu, J., Chen, N., & Peng, W. (2019). Estimation of Bearing Remaining Useful Life Based on Multiscale Convolutional Neural Network. IEEE Transactions on Industrial Electronics, 66. doi: 10.1109/TIE.2018.2844856
Section
Technical Papers