Predicting Remaining Useful Life using Time Series Embeddings based on Recurrent Neural Networks
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Nov 18, 2020
Narendhar Gugulothu
Vishnu TV
Pankaj Malhotra
Lovekesh Vig
Puneet Agarwal
Gautam Shroff
Abstract
We consider the problem of estimating the remaining useful life (RUL) of a system or a machine from sensor data. Many approaches for RUL estimation based on sensor data make assumptions about how machines degrade. Additionally, sensor data from machines is noisy and often suffers from missing values in many practical settings. We propose Embed-RUL: a novel approach for RUL estimation from sensor data that does not rely on any degradation-trend assumptions, is robust to noise, and handles missing values. Embed-RUL utilizes a sequence-to-sequence model based on Recurrent Neural Networks (RNNs) to generate embeddings for multivariate time series subsequences. The embeddings for normal and degraded machines tend to be different, and are therefore found to be useful for RUL estimation. We show that the embeddings capture the overall pattern in the time series while filtering out the noise, so that the embeddings of two machines with similar operational behavior are close to each other, even when their sensor readings have significant and varying levels of noise content. We perform experiments on publicly available turbofan engine dataset and a proprietary real-world dataset, and demonstrate that Embed-RUL outperforms the previously reported state-of-the-art (Malhotra, TV, et al., 2016) on several metrics.
##plugins.themes.bootstrap3.article.details##
Keywords
Condition Monitoring, Prognostics, Deep Learning, Time Series Embeddings, Recurrent Neural Networks
References
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., et al. (2016). Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467.
Akintayo, A., Lore, K. G., Sarkar, S., & Sarkar, S. (2016). Early Detection of Combustion Instabilities using Deep Convolutional Selective Autoencoders on Hi-speed Flame Video. CoRR, abs/1603.07839. Retrieved from http://arxiv.org/abs/1603.07839
Babu, G. S., Zhao, P., & Li, X.-L. (2016). Deep convolutional neural network based regression approach for estimation of remaining useful life. In International Conference on Database Systems for Advanced Applications (pp. 214–228).
Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Camci, F., Eker, O. F., Bas¸kan, S., & Konur, S. (2016). Comparison of sensors and methodologies for effective prognostics on railway turnout systems. Proceedings of the Institution of Mechanical Engineers, Part F: Journal of Rail and Rapid Transit, 230(1), 24–42.
Che, Z., Purushotham, S., Cho, K., Sontag, D., & Liu, Y. (2016). Recurrent neural networks for multivariate time series with missing values. arXiv preprint arXiv:1606.01865.
Cho, K., Van Merri¨enboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoderdecoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
Croarkin, C., & Tobias, P. (2006). Nist/sematech e-handbook of statistical methods. NIST/SEMATECH.
Dai, A. M., & Le, Q. V. (2015). Semi-supervised sequence learning. In Advances in Neural Information Processing Systems (pp. 3079–3087).
Da Xu, L., He, W., & Li, S. (2014). Internet of things in industries: A survey. IEEE Transactions on industrial informatics, 10(4), 2233–2243.
Eker, O¨ . F., Camci, F., & Jennions, I. K. (2014). A similaritybased prognostics approach for remaining useful life prediction.
Filonov, P., Lavrentyev, A., & Vorontsov, A. (2016). Multivariate Industrial Time Series with Cyber-Attack Simulation: Fault Detection Using an LSTM-based Predictive Data Model. NIPS Time Series Workshop 2016, arXiv preprint arXiv:1612.06676.
Graves, A., Mohamed, A. R., & Hinton, G. (2013). Speech recognition with deep recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2013 (pp. 6645–6649).
Heimes, F. O. (2008). Recurrent neural networks for remaining useful life estimation. In Prognostics and Health Management, 2008. PHM 2008. (pp. 1–6).
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735–1780.
Hu, C., Youn, B. D., Wang, P., & Yoon, J. T. (2012). Ensemble of data-driven prognostic algorithms for robust prediction of remaining useful life. Reliability Engineering & System Safety, 103, 120–135.
Jolliffe, I. (2002). Principal component analysis. Wiley Online Library.
Khelif, R., Chebel-Morello, B., Malinowski, S., Laajili, E., Fnaiech, F., & Zerhouni, N. (2017). Direct remaining useful life estimation based on support vector regression. IEEE Transactions on Industrial Electronics, 64(3), 2276–2285.
Khelif, R., Malinowski, S., Chebel-Morello, B., & Zerhouni, N. (2014). Rul prediction based on a new similarityinstance based approach. In IEEE International Symposium on Industrial Electronics.
Liao, L., & Ahn, H.-i. (2016). A framework of comnbining deep learning and survival analysis for asset health management. 1st ACM SIGKDD Workshop on ML for PHM..
Maaten, L. v. d., & Hinton, G. (2008). Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov), 2579–2605.
Macmann, O. B., Seitz, T. M., Behbahani, A. R., & Cohen, K. (2016). Performing diagnostics and prognostics on simulated engine failures using neural networks. In 52nd AIAA/SAE/ASEE Joint Propulsion Conference (p. 4807).
Malhotra, P., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., & Shroff, G. (2016). LSTMbased Encoder-Decoder for Multi-sensor Anomaly Detection. In Anomaly Detection Workshop at 33rd International Conference on Machine Learn-ing (ICML 2016), CoRR, abs/1607.00148, 2016, https://arxiv.org/abs/1607.00148.
Malhotra, P., TV, V., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., & Shroff, G. (2016). Multi-Sensor Prognostics using an Unsupervised Health Index based on LSTM Encoder-Decoder. 1st ACM SIGKDDWorkshop on ML for PHM. arXiv preprint arXiv:1608.06154.
Malhotra, P., TV, V., Vig, L., Agarwal, P., & Shroff, G. (2017). TimeNet: Pre-trained deep recurrent neural network for time series classification. In 25th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning.
Malhotra, P., Vig, L., Shroff, G., & Agarwal, P. (2015). Long Short Term Memory Networks for Anomaly Detection in Time Series. In ESANN, 23rd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (pp. 89–94).
Mosallam, A., Medjaher, K., & Zerhouni, N. (2014). Data-driven prognostic method based on bayesian approaches for direct remaining useful life prediction. Journal of Intelligent Manufacturing.
Mosallam, A., Medjaher, K., & Zerhouni, N. (2015). Component based data-driven prognostics for complex systems: Methodology and applications. In Reliability systems engineering (icrse), 2015 first international conference on (pp. 1–7).
Ng, A. (2011). Sparse autoencoder. CS294A Lecture notes, 72(2011), 1–19.
Peng, Y., Wang, H., Wang, J., Liu, D., & Peng, X. (2012). A modified echo state network based remaining useful life estimation approach. In IEEE Conference on Prognostics and Health Management (PHM).
Pham, V., Bluche, T., Kermorvant, C., & Louradour, J. (2014). Dropout improves recurrent neural networks for handwriting recognition. In Frontiers in Handwriting Recognition (ICFHR) (pp. 285–290).
Qiu, H., Lee, J., Lin, J., & Yu, G. (2003). Robust performance degradation assessment methods for enhanced rolling element bearing prognostics. Advanced Engineering Informatics, 17(3), 127–140.
Ramasso, E. (2014). Investigating computational geometry for failure prognostics. International Journal of Prognostics and Health Management, 5(1), 005.
Reddy, K. K., Venugopalan, V., & Giering, M. J. (2016). Applying deep learning for prognostic health monitoring of aerospace and building systems. 1st ACM SIGKDD Workshop on ML for PHM..
Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, M. (2008). Metrics for evaluating performance of prognostic techniques. In IEEE International Conference on Prognostics and health management. PHM 2008. (pp. 1–17).
Saxena, A., & Goebel, K. (2008). Turbofan engine degradation simulation data set. NASA Ames Prognostics Data Repository.
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008). Damage propagation modeling for aircraft engine runto- failure simulation. In IEEE International Conference on Prognostics and Health Management, 2008. PHM 2008. (pp. 1–9).
Si, X.-S., Wang, W., Hu, C.-H., & Zhou, D.-H. (2011). Remaining useful life estimation–A review on the statistical data driven approaches. European journal of operational research, 213(1), 1–14.
Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1), 1929–1958.
Srivastava, N., Mansimov, E., & Salakhudinov, R. (2015). Unsupervised learning of video representations using lstms. In International Conference on Machine Learning (pp. 843–852).
Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (pp. 3104– 3112).
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning (pp. 1096–1103).
Wang, T., Yu, J., Siegel, D., & Lee, J. (2008). A similaritybased prognostics approach for remaining useful life estimation of engineered systems. In IEEE International Conference on Prognostics and Health Management, 2008. PHM 2008. (pp. 1–6).
Yan, W., & Yu, L. (2015). On accurate and reliable anomaly detection for gas turbine combustors: A deep learning approach. In Proceedings of the Annual Conference of the Prognostics and Health Management Society.
Zhang, C., Lim, P., Qin, A., & Tan, K. C. (2016). Multiobjective deep belief networks ensemble for remaining useful life estimation in prognostics. IEEE transactions on neural networks and learning systems.
Zhao, R., Wang, J., Yan, R., & Mao, K. (2016). Machine health monitoring with LSTM networks. In IEEE 10th International Conference on Sensing Technology (ICST), 2016 (pp. 1–6).
Zhao, R., Yan, R., Chen, Z., Mao, K., Wang, P., & Gao, R. X. (2016). Deep Learning and Its Applications to Machine Health Monitoring: A Survey. arXiv preprint arXiv:1612.07640.
Zhao, R., Yan, R., Wang, J., & Mao, K. (2017). Learning to monitor machine health with convolutional bidirectional lstm networks. Sensors, 17(2), 273.
Zheng, S., Ristovski, K., Farahat, A., & Gupta, C. (2017). Long Short-Term Memory Network for Remaining Useful Life estimation. In IEEE International Conference on Prognostics and Health Management (ICPHM), 2017 (pp. 88–95).
Akintayo, A., Lore, K. G., Sarkar, S., & Sarkar, S. (2016). Early Detection of Combustion Instabilities using Deep Convolutional Selective Autoencoders on Hi-speed Flame Video. CoRR, abs/1603.07839. Retrieved from http://arxiv.org/abs/1603.07839
Babu, G. S., Zhao, P., & Li, X.-L. (2016). Deep convolutional neural network based regression approach for estimation of remaining useful life. In International Conference on Database Systems for Advanced Applications (pp. 214–228).
Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Camci, F., Eker, O. F., Bas¸kan, S., & Konur, S. (2016). Comparison of sensors and methodologies for effective prognostics on railway turnout systems. Proceedings of the Institution of Mechanical Engineers, Part F: Journal of Rail and Rapid Transit, 230(1), 24–42.
Che, Z., Purushotham, S., Cho, K., Sontag, D., & Liu, Y. (2016). Recurrent neural networks for multivariate time series with missing values. arXiv preprint arXiv:1606.01865.
Cho, K., Van Merri¨enboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoderdecoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
Croarkin, C., & Tobias, P. (2006). Nist/sematech e-handbook of statistical methods. NIST/SEMATECH.
Dai, A. M., & Le, Q. V. (2015). Semi-supervised sequence learning. In Advances in Neural Information Processing Systems (pp. 3079–3087).
Da Xu, L., He, W., & Li, S. (2014). Internet of things in industries: A survey. IEEE Transactions on industrial informatics, 10(4), 2233–2243.
Eker, O¨ . F., Camci, F., & Jennions, I. K. (2014). A similaritybased prognostics approach for remaining useful life prediction.
Filonov, P., Lavrentyev, A., & Vorontsov, A. (2016). Multivariate Industrial Time Series with Cyber-Attack Simulation: Fault Detection Using an LSTM-based Predictive Data Model. NIPS Time Series Workshop 2016, arXiv preprint arXiv:1612.06676.
Graves, A., Mohamed, A. R., & Hinton, G. (2013). Speech recognition with deep recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2013 (pp. 6645–6649).
Heimes, F. O. (2008). Recurrent neural networks for remaining useful life estimation. In Prognostics and Health Management, 2008. PHM 2008. (pp. 1–6).
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735–1780.
Hu, C., Youn, B. D., Wang, P., & Yoon, J. T. (2012). Ensemble of data-driven prognostic algorithms for robust prediction of remaining useful life. Reliability Engineering & System Safety, 103, 120–135.
Jolliffe, I. (2002). Principal component analysis. Wiley Online Library.
Khelif, R., Chebel-Morello, B., Malinowski, S., Laajili, E., Fnaiech, F., & Zerhouni, N. (2017). Direct remaining useful life estimation based on support vector regression. IEEE Transactions on Industrial Electronics, 64(3), 2276–2285.
Khelif, R., Malinowski, S., Chebel-Morello, B., & Zerhouni, N. (2014). Rul prediction based on a new similarityinstance based approach. In IEEE International Symposium on Industrial Electronics.
Liao, L., & Ahn, H.-i. (2016). A framework of comnbining deep learning and survival analysis for asset health management. 1st ACM SIGKDD Workshop on ML for PHM..
Maaten, L. v. d., & Hinton, G. (2008). Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov), 2579–2605.
Macmann, O. B., Seitz, T. M., Behbahani, A. R., & Cohen, K. (2016). Performing diagnostics and prognostics on simulated engine failures using neural networks. In 52nd AIAA/SAE/ASEE Joint Propulsion Conference (p. 4807).
Malhotra, P., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., & Shroff, G. (2016). LSTMbased Encoder-Decoder for Multi-sensor Anomaly Detection. In Anomaly Detection Workshop at 33rd International Conference on Machine Learn-ing (ICML 2016), CoRR, abs/1607.00148, 2016, https://arxiv.org/abs/1607.00148.
Malhotra, P., TV, V., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., & Shroff, G. (2016). Multi-Sensor Prognostics using an Unsupervised Health Index based on LSTM Encoder-Decoder. 1st ACM SIGKDDWorkshop on ML for PHM. arXiv preprint arXiv:1608.06154.
Malhotra, P., TV, V., Vig, L., Agarwal, P., & Shroff, G. (2017). TimeNet: Pre-trained deep recurrent neural network for time series classification. In 25th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning.
Malhotra, P., Vig, L., Shroff, G., & Agarwal, P. (2015). Long Short Term Memory Networks for Anomaly Detection in Time Series. In ESANN, 23rd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (pp. 89–94).
Mosallam, A., Medjaher, K., & Zerhouni, N. (2014). Data-driven prognostic method based on bayesian approaches for direct remaining useful life prediction. Journal of Intelligent Manufacturing.
Mosallam, A., Medjaher, K., & Zerhouni, N. (2015). Component based data-driven prognostics for complex systems: Methodology and applications. In Reliability systems engineering (icrse), 2015 first international conference on (pp. 1–7).
Ng, A. (2011). Sparse autoencoder. CS294A Lecture notes, 72(2011), 1–19.
Peng, Y., Wang, H., Wang, J., Liu, D., & Peng, X. (2012). A modified echo state network based remaining useful life estimation approach. In IEEE Conference on Prognostics and Health Management (PHM).
Pham, V., Bluche, T., Kermorvant, C., & Louradour, J. (2014). Dropout improves recurrent neural networks for handwriting recognition. In Frontiers in Handwriting Recognition (ICFHR) (pp. 285–290).
Qiu, H., Lee, J., Lin, J., & Yu, G. (2003). Robust performance degradation assessment methods for enhanced rolling element bearing prognostics. Advanced Engineering Informatics, 17(3), 127–140.
Ramasso, E. (2014). Investigating computational geometry for failure prognostics. International Journal of Prognostics and Health Management, 5(1), 005.
Reddy, K. K., Venugopalan, V., & Giering, M. J. (2016). Applying deep learning for prognostic health monitoring of aerospace and building systems. 1st ACM SIGKDD Workshop on ML for PHM..
Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, M. (2008). Metrics for evaluating performance of prognostic techniques. In IEEE International Conference on Prognostics and health management. PHM 2008. (pp. 1–17).
Saxena, A., & Goebel, K. (2008). Turbofan engine degradation simulation data set. NASA Ames Prognostics Data Repository.
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008). Damage propagation modeling for aircraft engine runto- failure simulation. In IEEE International Conference on Prognostics and Health Management, 2008. PHM 2008. (pp. 1–9).
Si, X.-S., Wang, W., Hu, C.-H., & Zhou, D.-H. (2011). Remaining useful life estimation–A review on the statistical data driven approaches. European journal of operational research, 213(1), 1–14.
Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1), 1929–1958.
Srivastava, N., Mansimov, E., & Salakhudinov, R. (2015). Unsupervised learning of video representations using lstms. In International Conference on Machine Learning (pp. 843–852).
Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (pp. 3104– 3112).
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning (pp. 1096–1103).
Wang, T., Yu, J., Siegel, D., & Lee, J. (2008). A similaritybased prognostics approach for remaining useful life estimation of engineered systems. In IEEE International Conference on Prognostics and Health Management, 2008. PHM 2008. (pp. 1–6).
Yan, W., & Yu, L. (2015). On accurate and reliable anomaly detection for gas turbine combustors: A deep learning approach. In Proceedings of the Annual Conference of the Prognostics and Health Management Society.
Zhang, C., Lim, P., Qin, A., & Tan, K. C. (2016). Multiobjective deep belief networks ensemble for remaining useful life estimation in prognostics. IEEE transactions on neural networks and learning systems.
Zhao, R., Wang, J., Yan, R., & Mao, K. (2016). Machine health monitoring with LSTM networks. In IEEE 10th International Conference on Sensing Technology (ICST), 2016 (pp. 1–6).
Zhao, R., Yan, R., Chen, Z., Mao, K., Wang, P., & Gao, R. X. (2016). Deep Learning and Its Applications to Machine Health Monitoring: A Survey. arXiv preprint arXiv:1612.07640.
Zhao, R., Yan, R., Wang, J., & Mao, K. (2017). Learning to monitor machine health with convolutional bidirectional lstm networks. Sensors, 17(2), 273.
Zheng, S., Ristovski, K., Farahat, A., & Gupta, C. (2017). Long Short-Term Memory Network for Remaining Useful Life estimation. In IEEE International Conference on Prognostics and Health Management (ICPHM), 2017 (pp. 88–95).
Section
Technical Papers