Remaining Useful Life Estimation Using Neural Ordinary Differential Equations
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Aug 1, 2021
Marco Star
Kristoffer McKee
Abstract
Data-driven machinery prognostics has seen increasing popularity recently, especially with the effectiveness of deep learning methods growing. However, deep learning methods lack useful properties such as the lack of uncertainty quantification of their outputs and have a black-box nature. Neural ordinary differential equations (NODEs) use neural networks to define differential equations that propagate data from the inputs to the outputs. They can be seen as a continuous generalization of a popular network architecture used for image recognition known as the Residual Network (ResNet). This paper compares the performance of each network for machinery prognostics tasks to show the validity of Neural ODEs in machinery prognostics. The comparison is done using NASA’s Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) dataset, which simulates the sensor information of degrading turbofan engines. To compare both architectures, they are set up as convolutional neural networks and the sensors are transformed to the time-frequency domain through the short-time Fourier transform (STFT). The spectrograms from the STFT are the input images to the networks and the output is the estimated RUL; hence, the task is turned into an image recognition task. The results found NODEs can compete with state-of-the-art machinery prognostics methods. While it does not beat the state-of-the-art method, it is close enough that it could warrant further research into using NODEs. The potential benefits of using NODEs instead of other network architectures are also discussed in this work.
##plugins.themes.bootstrap3.article.details##
Keywords
RUL estimation, Deep Learning, machinery prognostics
References
Balandat, M., Karrer, B., Jiang, D. R., Daulton, S., Letham, B., Wilson, A. G., & Bakshy, E. (2019). Botorch: Programmable bayesian optimization in pytorch. CoRR, abs/1910.06403. Retrieved from http://arxiv.org/abs/1910.06403
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015). Weight uncertainty in neural networks. ArXiv, abs/1505.05424.
Chen, T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. (2018). Neural ordinary differential equations. CoRR, abs/1806.07366. Retrieved from http://arxiv.org/abs/1806.07366
Dourado, A., & Viana, F. A. C. (2019). Physics-Informed Neural Networks for Corrosion-Fatigue Prognosis. In Annual conference of the phm society (Vol. 11, pp. 1–12). doi: https://doi.org/10.36001/phmconf.2019.v11i1.814
Dumoulin, V., & Visin, F. (2016). A guide to convolution arithmetic for deep learning. ArXiv, abs/1603.07285. Frazier, P. I. (2018). A tutorial on bayesian optimization.
Gal, Y., & Ghahramani, Z. (2016). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. ArXiv, abs/1506.02142.
Guo, L., Li, N., Jia, F., Lei, Y., & Lin, J. (2017). A recurrent neural network based health indicator for remaining useful life prediction of bearings. Neurocomputing, 240, 98–109. Retrieved from http://dx.doi.org/10.1016/j.neucom.2017.02.045 doi: 10.1016/j.neucom.2017.02.045
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-December, 770–778. doi: 10.1109/CVPR.2016.90
Heimes, F. O. (2008). Recurrent neural networks for remaining useful life estimation. In 2008 international conference on prognostics and health management, phm 2008. doi: 10.1109/PHM.2008.4711422
Hu, Y., & Luo, P. (2013, July.). Performance data prognostics based on relevance vector machine and particle filter. Chemical Engineering Transactions, 33, 349-354. Retrieved from https://www.cetjournal.it/index.php/ cet/article/view/CET1333059 doi: 10.3303/CET1333059
Huang, C. G., Huang, H. Z., & Li, Y. F. (2019). A Bidirectional LSTM Prognostics Method Under Multiple Operational Conditions. IEEE Transactions on Industrial Electronics, 66(11), 8792–8802. doi: 10.1109/TIE.2019.2891463
Lei, Y., Li, N., Guo, L., Li, N., Yan, T., & Lin, J. (2018, May). Machinery health prognostics: A systematic review from data acquisition to RUL prediction. Mechanical Systems and Signal Processing, 104, 799–834. doi: 10.1016/j.ymssp.2017.11.016
Li, X., Ding, Q., & Sun, J. Q. (2018). Remaining useful life estimation in prognostics using deep convolution neural networks. Reliability Engineering and System Safety, 172(December 2017), 1–11. Retrieved from https://doi.org/10.1016/j.ress.2017.11.021 doi: 10.1016/j.ress.2017.11.021
Li, X., Wong, T.-K. L., Chen, R. T. Q., & Duvenaud, D. (2020, August). Scalable gradients for stochastic differential equations. In S. Chiappa & R. Calandra (Eds.), Proceedings of the twenty third international conference on artificial intelligence and statistics (Vol. 108, pp. 3870–3882). PMLR. Retrieved from http://proceedings.mlr.press/v108/li20i.html
Li, X., Zhang, W., & Ding, Q. (2019, Feburary). Deep learning-based remaining useful life estimation of bearings using multi-scale feature extraction. Reliability Engineering and System Safety, 182, 208–218. doi: 10.1016/j.ress.2018.11.011
Listou Ellefsen, A., Bjørlykhaug, E., Æsøy, V., Ushakov, S., & Zhang, H. (2019, March). Remaining useful life predictions for turbofan engine degradation using semisupervised deep architecture. Reliability Engineering and System Safety, 183, 240–251. doi: 10.1016/j.ress.2018.11.027
Nascimento, R. G., & Viana, F. A. C. (2019). Fleet prognosis with physics-informed recurrent neural networks. CoRR, abs/1901.05512. Retrieved from http://arxiv.org/abs/1901.05512
Pasa, G., Medeiros, I., & Yoneyama, T. (2019). Operating Condition-Invariant Neural Network-based Prognostics Methods applied on Turbofan Aircraft Engines. In Annual conference of the phm society (Vol. 11, pp. 1–10). doi: https://doi.org/10.36001/phmconf.2019.v11i1.786
Peng, W., Ye, Z.-S., & Chen, N. (2019). Bayesian Deep Learning based Health Prognostics Towards Prognostics Uncertainty. IEEE Transactions on Industrial Electronics, 1–1. Retrieved from https://ieeexplore.ieee.org/document/8681720/ doi: 10.1109/TIE.2019.2907440
Queiruga, A., Erichson, N., Taylor, D., & Mahoney, M. W. (2020). Continuous-in-depth neural networks. ArXiv, abs/2008.02389.
Saha, B., Goebel, K., Poll, S., & Christophersen, J. (2009). Prognostics methods for battery health monitoring using a Bayesian framework. IEEE Transactions on Instrumentation and Measurement, 58(2), 291–296. doi: 10.1109/TIM.2008.2005965
Sankararaman, S. (2015). Significance, interpretation, and quantification of uncertainty in prognostics and remaining useful life prediction. Mechanical Systems and Signal Processing, 52-53(1), 228–247. doi: 10.1016/j.ymssp.2014.05.029
Saxena, A., & Goebel, K. (n.d.). Turbofan Engine Degradation Simulation Data Set. Moffett Field, CA: NASA Ames Research Center. Retrieved from http://ti.arc.nasa.gov/project/prognostic-data-repository
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008, October). Damage propagation modeling for aircraft engine run-to-failure simulation. In 2008 international conference on prognostics and health management (Vol. 41, pp. 1–9). IEEE. doi: 10.1109/PHM.2008.4711414
Wang, Q., Zhao, B., Ma, H., Chang, J., & Mao, G. (2019). A method for rapidly evaluating reliability and predicting remaining useful life using two-dimensional convolutional neural network with signal conversion. Journal of Mechanical Science and Technology, 33(6), 2561–2571. doi: 10.1007/s12206-019-0504-x
Wu, J., Hu, K., Cheng, Y., Zhu, H., Shao, X., & Wang, Y. (2019). Data-driven remaining useful life prediction via multiple sensor signals and deep long short-term memory neural network. ISA Transactions, 97, 241–250. Retrieved from https://doi.org/10.1016/j.isatra.2019.07.004 doi: 10.1016/j.isatra.2019.07.004
Yang, W., Yao, Q., Ye, K., & Xu, C. Z. (2020). Empirical Mode Decomposition and Temporal Convolutional Networks for Remaining Useful Life Estimation. International Journal of Parallel Programming, 48(1), 61–79. Retrieved from https://doi.org/10.1007/s10766-019-00650-1 doi: 10.1007/s10766-019-00650-1
Yann Lecun. (1989). Generalization and Network Design Strategies.
Yucesan, Y. A., & Viana, F. A. C. (2019). Wind Turbine Main Bearing Fatigue Life Estimation with Physicsinformed Neural Networks. In Annual conference of the prognostics and health management society (Vol. 11, pp. 1–14). doi: 10.36001/PHMCONF.2019.V11I1.807
Zhang,W., Yang, D., &Wang, H. (2019). Data-Driven Methods for Predictive Maintenance of Industrial Equipment: A Survey. IEEE Systems Journal, 13(3), 2213–2227. doi: 10.1109/JSYST.2019.2905565
Zhang, Y., Xiong, R., He, H., & Pecht, M. G. (2018). Long short-term memory recurrent neural network for remaining useful life prediction of lithium-ion batteries. IEEE Transactions on Vehicular Technology, 67(7), 5695–5705. doi: 10.1109/TVT.2018.2805189
Zhao, R., Yan, R., Chen, Z., Mao, K., Wang, P., & Gao, R. X. (2019, January). Deep learning and its applications to machine health monitoring. Mechanical Systems and Signal Processing, 115, 213–237. doi: 10.1016/j.ymssp.2018.05.050
Zhou, Y., Huang, Y., Pang, J., & Wang, K. (2019). Remaining useful life prediction for supercapacitor based on long short-term memory neural network. Journal of Power Sources, 440(March), 227149. Retrieved from https://doi.org/10.1016/j.jpowsour.2019.227149 doi: 10.1016/j.jpowsour.2019.227149
Zhu, J., Chen, N., & Peng, W. (2019, April). Estimation of Bearing Remaining Useful Life Based on Multiscale Convolutional Neural Network. IEEE Transactions on Industrial Electronics, 66(4), 3208–3216. doi: 10.1109/TIE.2018.2844856
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015). Weight uncertainty in neural networks. ArXiv, abs/1505.05424.
Chen, T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. (2018). Neural ordinary differential equations. CoRR, abs/1806.07366. Retrieved from http://arxiv.org/abs/1806.07366
Dourado, A., & Viana, F. A. C. (2019). Physics-Informed Neural Networks for Corrosion-Fatigue Prognosis. In Annual conference of the phm society (Vol. 11, pp. 1–12). doi: https://doi.org/10.36001/phmconf.2019.v11i1.814
Dumoulin, V., & Visin, F. (2016). A guide to convolution arithmetic for deep learning. ArXiv, abs/1603.07285. Frazier, P. I. (2018). A tutorial on bayesian optimization.
Gal, Y., & Ghahramani, Z. (2016). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. ArXiv, abs/1506.02142.
Guo, L., Li, N., Jia, F., Lei, Y., & Lin, J. (2017). A recurrent neural network based health indicator for remaining useful life prediction of bearings. Neurocomputing, 240, 98–109. Retrieved from http://dx.doi.org/10.1016/j.neucom.2017.02.045 doi: 10.1016/j.neucom.2017.02.045
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-December, 770–778. doi: 10.1109/CVPR.2016.90
Heimes, F. O. (2008). Recurrent neural networks for remaining useful life estimation. In 2008 international conference on prognostics and health management, phm 2008. doi: 10.1109/PHM.2008.4711422
Hu, Y., & Luo, P. (2013, July.). Performance data prognostics based on relevance vector machine and particle filter. Chemical Engineering Transactions, 33, 349-354. Retrieved from https://www.cetjournal.it/index.php/ cet/article/view/CET1333059 doi: 10.3303/CET1333059
Huang, C. G., Huang, H. Z., & Li, Y. F. (2019). A Bidirectional LSTM Prognostics Method Under Multiple Operational Conditions. IEEE Transactions on Industrial Electronics, 66(11), 8792–8802. doi: 10.1109/TIE.2019.2891463
Lei, Y., Li, N., Guo, L., Li, N., Yan, T., & Lin, J. (2018, May). Machinery health prognostics: A systematic review from data acquisition to RUL prediction. Mechanical Systems and Signal Processing, 104, 799–834. doi: 10.1016/j.ymssp.2017.11.016
Li, X., Ding, Q., & Sun, J. Q. (2018). Remaining useful life estimation in prognostics using deep convolution neural networks. Reliability Engineering and System Safety, 172(December 2017), 1–11. Retrieved from https://doi.org/10.1016/j.ress.2017.11.021 doi: 10.1016/j.ress.2017.11.021
Li, X., Wong, T.-K. L., Chen, R. T. Q., & Duvenaud, D. (2020, August). Scalable gradients for stochastic differential equations. In S. Chiappa & R. Calandra (Eds.), Proceedings of the twenty third international conference on artificial intelligence and statistics (Vol. 108, pp. 3870–3882). PMLR. Retrieved from http://proceedings.mlr.press/v108/li20i.html
Li, X., Zhang, W., & Ding, Q. (2019, Feburary). Deep learning-based remaining useful life estimation of bearings using multi-scale feature extraction. Reliability Engineering and System Safety, 182, 208–218. doi: 10.1016/j.ress.2018.11.011
Listou Ellefsen, A., Bjørlykhaug, E., Æsøy, V., Ushakov, S., & Zhang, H. (2019, March). Remaining useful life predictions for turbofan engine degradation using semisupervised deep architecture. Reliability Engineering and System Safety, 183, 240–251. doi: 10.1016/j.ress.2018.11.027
Nascimento, R. G., & Viana, F. A. C. (2019). Fleet prognosis with physics-informed recurrent neural networks. CoRR, abs/1901.05512. Retrieved from http://arxiv.org/abs/1901.05512
Pasa, G., Medeiros, I., & Yoneyama, T. (2019). Operating Condition-Invariant Neural Network-based Prognostics Methods applied on Turbofan Aircraft Engines. In Annual conference of the phm society (Vol. 11, pp. 1–10). doi: https://doi.org/10.36001/phmconf.2019.v11i1.786
Peng, W., Ye, Z.-S., & Chen, N. (2019). Bayesian Deep Learning based Health Prognostics Towards Prognostics Uncertainty. IEEE Transactions on Industrial Electronics, 1–1. Retrieved from https://ieeexplore.ieee.org/document/8681720/ doi: 10.1109/TIE.2019.2907440
Queiruga, A., Erichson, N., Taylor, D., & Mahoney, M. W. (2020). Continuous-in-depth neural networks. ArXiv, abs/2008.02389.
Saha, B., Goebel, K., Poll, S., & Christophersen, J. (2009). Prognostics methods for battery health monitoring using a Bayesian framework. IEEE Transactions on Instrumentation and Measurement, 58(2), 291–296. doi: 10.1109/TIM.2008.2005965
Sankararaman, S. (2015). Significance, interpretation, and quantification of uncertainty in prognostics and remaining useful life prediction. Mechanical Systems and Signal Processing, 52-53(1), 228–247. doi: 10.1016/j.ymssp.2014.05.029
Saxena, A., & Goebel, K. (n.d.). Turbofan Engine Degradation Simulation Data Set. Moffett Field, CA: NASA Ames Research Center. Retrieved from http://ti.arc.nasa.gov/project/prognostic-data-repository
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008, October). Damage propagation modeling for aircraft engine run-to-failure simulation. In 2008 international conference on prognostics and health management (Vol. 41, pp. 1–9). IEEE. doi: 10.1109/PHM.2008.4711414
Wang, Q., Zhao, B., Ma, H., Chang, J., & Mao, G. (2019). A method for rapidly evaluating reliability and predicting remaining useful life using two-dimensional convolutional neural network with signal conversion. Journal of Mechanical Science and Technology, 33(6), 2561–2571. doi: 10.1007/s12206-019-0504-x
Wu, J., Hu, K., Cheng, Y., Zhu, H., Shao, X., & Wang, Y. (2019). Data-driven remaining useful life prediction via multiple sensor signals and deep long short-term memory neural network. ISA Transactions, 97, 241–250. Retrieved from https://doi.org/10.1016/j.isatra.2019.07.004 doi: 10.1016/j.isatra.2019.07.004
Yang, W., Yao, Q., Ye, K., & Xu, C. Z. (2020). Empirical Mode Decomposition and Temporal Convolutional Networks for Remaining Useful Life Estimation. International Journal of Parallel Programming, 48(1), 61–79. Retrieved from https://doi.org/10.1007/s10766-019-00650-1 doi: 10.1007/s10766-019-00650-1
Yann Lecun. (1989). Generalization and Network Design Strategies.
Yucesan, Y. A., & Viana, F. A. C. (2019). Wind Turbine Main Bearing Fatigue Life Estimation with Physicsinformed Neural Networks. In Annual conference of the prognostics and health management society (Vol. 11, pp. 1–14). doi: 10.36001/PHMCONF.2019.V11I1.807
Zhang,W., Yang, D., &Wang, H. (2019). Data-Driven Methods for Predictive Maintenance of Industrial Equipment: A Survey. IEEE Systems Journal, 13(3), 2213–2227. doi: 10.1109/JSYST.2019.2905565
Zhang, Y., Xiong, R., He, H., & Pecht, M. G. (2018). Long short-term memory recurrent neural network for remaining useful life prediction of lithium-ion batteries. IEEE Transactions on Vehicular Technology, 67(7), 5695–5705. doi: 10.1109/TVT.2018.2805189
Zhao, R., Yan, R., Chen, Z., Mao, K., Wang, P., & Gao, R. X. (2019, January). Deep learning and its applications to machine health monitoring. Mechanical Systems and Signal Processing, 115, 213–237. doi: 10.1016/j.ymssp.2018.05.050
Zhou, Y., Huang, Y., Pang, J., & Wang, K. (2019). Remaining useful life prediction for supercapacitor based on long short-term memory neural network. Journal of Power Sources, 440(March), 227149. Retrieved from https://doi.org/10.1016/j.jpowsour.2019.227149 doi: 10.1016/j.jpowsour.2019.227149
Zhu, J., Chen, N., & Peng, W. (2019, April). Estimation of Bearing Remaining Useful Life Based on Multiscale Convolutional Neural Network. IEEE Transactions on Industrial Electronics, 66(4), 3208–3216. doi: 10.1109/TIE.2018.2844856
Section
Technical Papers