Data-driven Prognostics with Predictive Uncertainty Estimation using Ensemble of Deep Ordinal Regression Models
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Dec 2, 2019
Vishnu TV
Diksha Chavan
Pankaj Malhotra
Lovekesh Vig
Gautam Shroff
Abstract
Prognostics or Remaining Useful Life (RUL) Estimation from multi-sensor time series data is useful to enable condition-based maintenance and ensure high operational availability of equipment. We propose a novel deep learning based approach for Prognostics with Uncertainty Quantification that is useful in scenarios where: (i) access to labeled failure data is scarce due to rarity of failures (ii) inherent noise ispresent in the sensor readings. The two scenarios mentioned are unavoidable sources of uncertainty in the RUL estimation process, often resulting in unreliable RUL estimates. To address (i), we formulate RUL estimation as an Ordinal Regression (OR) problem and propose LSTM-OR: deep Long Short Term Memory (LSTM) network-based approach to learn the OR function. We show that LSTM-OR naturally allows for the incorporation of censored operational instances in training along with the failed instances, leading to more robust learning. To address (ii), we propose a simple yet effective
approach to quantify predictive uncertainty in the RUL estimation models by training an ensemble of LSTM-OR models. Through empirical evaluation on the publicly available turbofan engine benchmark datasets, we demonstrate that LSTMOR is at par with commonly used deep metric regressionbased approaches for RUL estimation when sufficient failed instances are available for training. Importantly, LSTM-OR outperforms these metric regression-based approaches in the practical scenario where failed training instances are scarce, but sufficient operational (censored) instances are additionally available. Furthermore, our uncertainty quantification approach yields high-quality predictive uncertainty estimates
while also leading to improved RUL estimates compared to single best LSTM-OR models.
##plugins.themes.bootstrap3.article.details##
Keywords
recurrent neural networks, Remaining Useful Life Estimation, uncertainty estimation, deep learning, Ordinal Regression, censored data
References
Babu, G. S., Zhao, P., & Li, X.-L. (2016). Deep convolutional neural network based regression approach for estimation of remaining useful life. In International conference on database systems for advanced applications (pp. 214–228).
Benkedjouh, T., Medjaher, K., Zerhouni, N., & Rechak, S. (2013). Remaining useful life estimation based on nonlinear feature reduction and support vector regression. Engineering Applications of Artificial Intelligence, 26(7), 1751–1760.
Chang, K.-Y., Chen, C.-S., & Hung, Y.-P. (2011). Ordinal hyperplanes ranker with cost sensitivities for age estimation. In Computer vision and pattern recognition (cvpr), 2011 ieee conference on (pp. 585–592).
Cheng, J., Wang, Z., & Pollastri, G. (2008). A neural network approach to ordinal regression. In Neural networks, 2008. ijcnn 2008.(ieee world congress on computational intelligence). ieee international joint conference on (pp. 1279–1284).
Da Xu, L., He, W., & Li, S. (2014). Internet of things in industries: A survey. IEEE Transactions on industrial informatics, 10(4), 2233–2243.
Dong, H., Jin, X., Lou, Y., & Wang, C. (2014). Lithium-ion battery state of health monitoring and remaining useful life prediction based on support vector regressionparticle filter. Journal of power sources, 271, 114–123.
Ellefsen, A. L., Bjørlykhaug, E., Æsøy, V., Ushakov, S., & Zhang, H. (2019). Remaining useful life predictions for turbofan engine degradation using semi-supervised deep architecture. Reliability Engineering & System Safety, 183, 240–251.
Gal, Y., & Ghahramani, Z. (2016). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning (pp. 1050–1059).
Gugulothu, N., Malhotra, P., Vig, L., & Shroff, G. (2018). Sparse Neural Networks for Anomaly Detection in High-Dimensional Time Series. AI4IOT Workshop at International Joint Conference on Artificial Intelligence (IJCAI).
Gugulothu, N., TV, V., Gupta, P., Malhotra, P., Vig, L., Agarwal, P., & Shroff, G. (2018). On practical aspects of using rnns for fault detection in sparsely-labeled multisensor time series..
Gugulothu, N., TV, V., Malhotra, P., Vig, L., Agarwal, P., & Shroff, G. (2017). Predicting remaining useful life using time series embeddings based on recurrent neural networks. International Journal on Prognostics and Health Management, IJPHM, arXiv preprint arXiv:1709.01073.
Harrell, F. E. (2001). Ordinal logistic regression. In Regression modeling strategies (pp. 331–343). Springer.
Heimes, F. O. (2008). Recurrent neural networks for remaining useful life estimation. In Prognostics and Health Management, 2008. PHM 2008. (pp. 1–6).
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735–1780.
Katzman, J. L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., & Kluger, Y. (2018). Deepsurv: Personalized treatment recommender system using a cox proportional hazards deep neural network. BMC medical research methodology, 18(1), 24.
Khan, S., & Yairi, T. (2018). A review on the application of deep learning in system health management. Mechanical Systems and Signal Processing, 107, 241–265.
Khelif, R., Malinowski, S., Chebel-Morello, B., & Zerhouni, N. (2014). Rul prediction based on a new similarityinstance based approach. In IEEE International Symposium on Industrial Electronics.
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in neural information processing systems 30 (pp. 6402–6413).
Lam, J., Sankararaman, S., & Stewart, B. (2014). Enhanced trajectory based similarity prediction with uncertainty quantification. PHM 2014.
Li, X., Ding, Q., & Sun, J.-Q. (2018). Remaining useful life estimation in prognostics using deep convolution neural networks. Reliability Engineering & System Safety, 172, 1–11.
Liu, D., Luo, Y., Peng, Y., Peng, X., & Pecht, M. (2012). Lithium-ion battery remaining useful life estimation based on nonlinear ar model combined with degradation feature. In Annual conference of the prognostics and health management society (Vol. 3, pp. 1803–1836).
Liu, H., Lu, J., Feng, J., & Zhou, J. (2017). Ordinal deep feature learning for facial age estimation. In Automatic face & gesture recognition (fg 2017), 2017 12th ieee international conference on (pp. 157–164).
Luck, M., Sylvain, T., Cardinal, H., Lodi, A., & Bengio, Y. (2017). Deep learning for patient-specific kidney graft survival analysis. arXiv preprint arXiv:1705.10245.
Malhotra, P., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., & Shroff, G. (2016). LSTM-based Encoder- Decoder for Multi-sensor Anomaly Detection. In Anomaly detection workshop at 33rd international conference on machine learning (icml 2016).
Malhotra, P., TV, V., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., & Shroff, G. (2016). Multi-Sensor Prognostics using an Unsupervised Health Index based on LSTM Encoder-Decoder. 1st ACM SIGKDDWorkshop on ML for PHM. arXiv preprint arXiv:1608.06154.
Malhotra, P., Vig, L., Shroff, G., & Agarwal, P. (2015). Long Short Term Memory Networks for Anomaly Detection in Time Series. In ESANN, 23rd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (pp. 89–94).
Niu, Z., Zhou, M., Wang, L., Gao, X., & Hua, G. (2016). Ordinal regression with multiple output cnn for age estimation. In Proceedings of the ieee conference on computer vision and pattern recognition (pp. 4920–4928).
Orozco, B. P., Abbati, G., & Roberts, S. (2018). Mordred: Memory-based ordinal regression deep neural networks for time series forecasting. arXiv preprint arXiv:1803.09704.
Park, L. A., & Simoff, S. (2015). Using entropy as a measure of acceptance for multi-label classification. In International symposium on intelligent data analysis (pp. 217–228).
Pham, V., Bluche, T., Kermorvant, C., & Louradour, J. (2014). Dropout improves recurrent neural networks for handwriting recognition. In Frontiers in Handwriting Recognition (ICFHR) (pp. 285–290).
Sankararaman, S., Daigle, M., Saxena, A., & Goebel, K. (2013, March). Analytical algorithms to quantify the uncertainty in remaining useful life prediction. In 2013 ieee aerospace conference (p. 1-11). doi: 10.1109/AERO.2013.6496971
Sankararaman, S., & Goebel, K. (2013). A novel computational methodology for uncertainty quantification in prognostics using the most probable point concept. In Annual conference of the prognostics and health management society.
Saurav, S., Malhotra, P., TV, V., Gugulothu, N., Vig, L., Agarwal, P., & Shroff, G. (2018). Online anomaly detection with concept drift adaptation using recurrent neural networks. In Proceedings of the acm india joint international conference on data science and management of data (pp. 78–87).
Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, M. (2008). Metrics for evaluating performance of prognostic techniques. In Prognostics and health management, 2008. phm 2008. international conference on (pp. 1–17).
Saxena, A., & Goebel, K. (2008). Turbofan engine degradation simulation data set. NASA Ames Prognostics Data Repository.
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008). Damage propagation modeling for aircraft engine runto-failure simulation. In IEEE International Conference on Prognostics and Health Management, 2008. PHM 2008. (pp. 1–9).
TV, V., Gupta, P., Malhotra, P., Vig, L., & Shroff, G. (2018). Recurrent neural networks for online remaining useful life estimation in ion mill etching system. In Phm society conference (Vol. 10).
Wang, T., Yu, J., Siegel, D., & Lee, J. (2008). A similarity based prognostics approach for remaining useful life estimation of engineered systems. In IEEE International Conference on Prognostics and Health Management, 2008. PHM 2008. (pp. 1–6).
Yang, H.-F., Lin, B.-Y., Chang, K.-Y., & Chen, C.-S. (2013). Automatic age estimation from face images via deep ranking. networks, 35(8), 1872–1886.
Yoon, A. S., Lee, T., Lim, Y., Jung, D., Kang, P., Kim, D., . . . Choi, Y. (2017). Semi-supervised learning with deep generative models for asset failure prediction. arXiv preprint arXiv:1709.00845.
Zaremba, W., Sutskever, I., & Vinyals, O. (2014). Recurrent neural network regularization. arXiv preprint arXiv:1409.2329.
Zhang, C., Lim, P., Qin, A. K., & Tan, K. C. (2016). Multiobjective deep belief networks ensemble for remaining useful life estimation in prognostics. IEEE transactions on neural networks and learning systems, 28(10), 2306–2318.
Zhang, Y., Xiong, R., He, H., & Pecht, M. (2018). Long short-term memory recurrent neural network for remaining useful life prediction of lithium-ion batteries. IEEE Transactions on Vehicular Technology.
Zheng, S., Ristovski, K., Farahat, A., & Gupta, C. (2017). Long short-term memory network for remaining useful life estimation. In Prognostics and health management (icphm), 2017 ieee international conference on (pp. 88–95).
Zhu, L., & Laptev, N. (2017). Deep and confident prediction for time series at uber. In 2017 ieee international conference on data mining workshops (icdmw) (pp. 103–110).
Benkedjouh, T., Medjaher, K., Zerhouni, N., & Rechak, S. (2013). Remaining useful life estimation based on nonlinear feature reduction and support vector regression. Engineering Applications of Artificial Intelligence, 26(7), 1751–1760.
Chang, K.-Y., Chen, C.-S., & Hung, Y.-P. (2011). Ordinal hyperplanes ranker with cost sensitivities for age estimation. In Computer vision and pattern recognition (cvpr), 2011 ieee conference on (pp. 585–592).
Cheng, J., Wang, Z., & Pollastri, G. (2008). A neural network approach to ordinal regression. In Neural networks, 2008. ijcnn 2008.(ieee world congress on computational intelligence). ieee international joint conference on (pp. 1279–1284).
Da Xu, L., He, W., & Li, S. (2014). Internet of things in industries: A survey. IEEE Transactions on industrial informatics, 10(4), 2233–2243.
Dong, H., Jin, X., Lou, Y., & Wang, C. (2014). Lithium-ion battery state of health monitoring and remaining useful life prediction based on support vector regressionparticle filter. Journal of power sources, 271, 114–123.
Ellefsen, A. L., Bjørlykhaug, E., Æsøy, V., Ushakov, S., & Zhang, H. (2019). Remaining useful life predictions for turbofan engine degradation using semi-supervised deep architecture. Reliability Engineering & System Safety, 183, 240–251.
Gal, Y., & Ghahramani, Z. (2016). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning (pp. 1050–1059).
Gugulothu, N., Malhotra, P., Vig, L., & Shroff, G. (2018). Sparse Neural Networks for Anomaly Detection in High-Dimensional Time Series. AI4IOT Workshop at International Joint Conference on Artificial Intelligence (IJCAI).
Gugulothu, N., TV, V., Gupta, P., Malhotra, P., Vig, L., Agarwal, P., & Shroff, G. (2018). On practical aspects of using rnns for fault detection in sparsely-labeled multisensor time series..
Gugulothu, N., TV, V., Malhotra, P., Vig, L., Agarwal, P., & Shroff, G. (2017). Predicting remaining useful life using time series embeddings based on recurrent neural networks. International Journal on Prognostics and Health Management, IJPHM, arXiv preprint arXiv:1709.01073.
Harrell, F. E. (2001). Ordinal logistic regression. In Regression modeling strategies (pp. 331–343). Springer.
Heimes, F. O. (2008). Recurrent neural networks for remaining useful life estimation. In Prognostics and Health Management, 2008. PHM 2008. (pp. 1–6).
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735–1780.
Katzman, J. L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., & Kluger, Y. (2018). Deepsurv: Personalized treatment recommender system using a cox proportional hazards deep neural network. BMC medical research methodology, 18(1), 24.
Khan, S., & Yairi, T. (2018). A review on the application of deep learning in system health management. Mechanical Systems and Signal Processing, 107, 241–265.
Khelif, R., Malinowski, S., Chebel-Morello, B., & Zerhouni, N. (2014). Rul prediction based on a new similarityinstance based approach. In IEEE International Symposium on Industrial Electronics.
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in neural information processing systems 30 (pp. 6402–6413).
Lam, J., Sankararaman, S., & Stewart, B. (2014). Enhanced trajectory based similarity prediction with uncertainty quantification. PHM 2014.
Li, X., Ding, Q., & Sun, J.-Q. (2018). Remaining useful life estimation in prognostics using deep convolution neural networks. Reliability Engineering & System Safety, 172, 1–11.
Liu, D., Luo, Y., Peng, Y., Peng, X., & Pecht, M. (2012). Lithium-ion battery remaining useful life estimation based on nonlinear ar model combined with degradation feature. In Annual conference of the prognostics and health management society (Vol. 3, pp. 1803–1836).
Liu, H., Lu, J., Feng, J., & Zhou, J. (2017). Ordinal deep feature learning for facial age estimation. In Automatic face & gesture recognition (fg 2017), 2017 12th ieee international conference on (pp. 157–164).
Luck, M., Sylvain, T., Cardinal, H., Lodi, A., & Bengio, Y. (2017). Deep learning for patient-specific kidney graft survival analysis. arXiv preprint arXiv:1705.10245.
Malhotra, P., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., & Shroff, G. (2016). LSTM-based Encoder- Decoder for Multi-sensor Anomaly Detection. In Anomaly detection workshop at 33rd international conference on machine learning (icml 2016).
Malhotra, P., TV, V., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., & Shroff, G. (2016). Multi-Sensor Prognostics using an Unsupervised Health Index based on LSTM Encoder-Decoder. 1st ACM SIGKDDWorkshop on ML for PHM. arXiv preprint arXiv:1608.06154.
Malhotra, P., Vig, L., Shroff, G., & Agarwal, P. (2015). Long Short Term Memory Networks for Anomaly Detection in Time Series. In ESANN, 23rd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (pp. 89–94).
Niu, Z., Zhou, M., Wang, L., Gao, X., & Hua, G. (2016). Ordinal regression with multiple output cnn for age estimation. In Proceedings of the ieee conference on computer vision and pattern recognition (pp. 4920–4928).
Orozco, B. P., Abbati, G., & Roberts, S. (2018). Mordred: Memory-based ordinal regression deep neural networks for time series forecasting. arXiv preprint arXiv:1803.09704.
Park, L. A., & Simoff, S. (2015). Using entropy as a measure of acceptance for multi-label classification. In International symposium on intelligent data analysis (pp. 217–228).
Pham, V., Bluche, T., Kermorvant, C., & Louradour, J. (2014). Dropout improves recurrent neural networks for handwriting recognition. In Frontiers in Handwriting Recognition (ICFHR) (pp. 285–290).
Sankararaman, S., Daigle, M., Saxena, A., & Goebel, K. (2013, March). Analytical algorithms to quantify the uncertainty in remaining useful life prediction. In 2013 ieee aerospace conference (p. 1-11). doi: 10.1109/AERO.2013.6496971
Sankararaman, S., & Goebel, K. (2013). A novel computational methodology for uncertainty quantification in prognostics using the most probable point concept. In Annual conference of the prognostics and health management society.
Saurav, S., Malhotra, P., TV, V., Gugulothu, N., Vig, L., Agarwal, P., & Shroff, G. (2018). Online anomaly detection with concept drift adaptation using recurrent neural networks. In Proceedings of the acm india joint international conference on data science and management of data (pp. 78–87).
Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, M. (2008). Metrics for evaluating performance of prognostic techniques. In Prognostics and health management, 2008. phm 2008. international conference on (pp. 1–17).
Saxena, A., & Goebel, K. (2008). Turbofan engine degradation simulation data set. NASA Ames Prognostics Data Repository.
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008). Damage propagation modeling for aircraft engine runto-failure simulation. In IEEE International Conference on Prognostics and Health Management, 2008. PHM 2008. (pp. 1–9).
TV, V., Gupta, P., Malhotra, P., Vig, L., & Shroff, G. (2018). Recurrent neural networks for online remaining useful life estimation in ion mill etching system. In Phm society conference (Vol. 10).
Wang, T., Yu, J., Siegel, D., & Lee, J. (2008). A similarity based prognostics approach for remaining useful life estimation of engineered systems. In IEEE International Conference on Prognostics and Health Management, 2008. PHM 2008. (pp. 1–6).
Yang, H.-F., Lin, B.-Y., Chang, K.-Y., & Chen, C.-S. (2013). Automatic age estimation from face images via deep ranking. networks, 35(8), 1872–1886.
Yoon, A. S., Lee, T., Lim, Y., Jung, D., Kang, P., Kim, D., . . . Choi, Y. (2017). Semi-supervised learning with deep generative models for asset failure prediction. arXiv preprint arXiv:1709.00845.
Zaremba, W., Sutskever, I., & Vinyals, O. (2014). Recurrent neural network regularization. arXiv preprint arXiv:1409.2329.
Zhang, C., Lim, P., Qin, A. K., & Tan, K. C. (2016). Multiobjective deep belief networks ensemble for remaining useful life estimation in prognostics. IEEE transactions on neural networks and learning systems, 28(10), 2306–2318.
Zhang, Y., Xiong, R., He, H., & Pecht, M. (2018). Long short-term memory recurrent neural network for remaining useful life prediction of lithium-ion batteries. IEEE Transactions on Vehicular Technology.
Zheng, S., Ristovski, K., Farahat, A., & Gupta, C. (2017). Long short-term memory network for remaining useful life estimation. In Prognostics and health management (icphm), 2017 ieee international conference on (pp. 88–95).
Zhu, L., & Laptev, N. (2017). Deep and confident prediction for time series at uber. In 2017 ieee international conference on data mining workshops (icdmw) (pp. 103–110).
Issue
Section
Technical Papers