Performance Benchmarking and Analysis of Prognostic Methods for CMAPSS Datasets



Published Nov 1, 2020
Emmanuel Ramasso Abhinav Saxena


Six years and more than seventy publications later this paper looks back and analyzes the development of prognostic algorithms using C-MAPSS datasets generated and disseminated
by the prognostic center of excellence at NASA Ames Research Center. Among those datasets are five run-to-failure CMAPSS datasets that have been popular due to various characteristics
applicable to prognostics. The C-MAPSS datasets pose several challenges that are inherent to general prognostics applications. In particular, management of high variability due to sensor noise, effects of operating conditions, and presence of multiple simultaneous fault modes are some factors that have great impact on the generalization capabilities of prognostics algorithms. More than seventy publications have used the C-MAPSS datasets for developing datadriven prognostic algorithms. However, in the absence of performance benchmarking results and due to common misunderstandings in interpreting the relationships between these datasets, it has been difficult for the users to suitably compare their results. In addition to identifying differentiating characteristics in these datasets, this paper also provides performance results for the PHM’08 data challenge wining entries to serve as performance baseline. This paper summarizes various prognostic modeling efforts that used C-MAPSS datasets and provides guidelines and references to further usage of these datasets in a manner that allows clear and consistent comparison between different approaches.

Abstract 296 | PDF Downloads 373



prognostics, Review, benchmarking, C-MAPSS datasets

Abbas, M. (2010). System level health assessment of complex engineered processes. Unpublished doctoral dissertation, Georgia Institute of Technology.
Al-Salah, T., Zein-Sabatto, S., & Bodruzzaman, M. (2012). Decision fusion software system for turbine engine fault diagnostics. In Southeastcon, 2012 proceedings of ieee (p. 1-6).
Angelov, P., Filev, D., & Kasabov, N. (2010). Evolving intelligent systems: Methodology and applications (J. Willey & Sons, Eds.). IEEE Press Series on Computational Intelligence.
Butcher, J. B., Verstraeten, D., Schrauwen, B., Day, C. R., & Haycock, P. W. (2013). Reservoir computing and extreme learning machines for non-linear time-series data analysis. Neural Network, 38, 76–89.
Coble, J. (2010). Merging data sources to predict remaining useful life - an automated method to identify prognostic parameters. Unpublished doctoral dissertation, University of Tennessee, Knoxville.
Coble, J., & Hines, J. (2008). Prognostic algorithm catego-rization with phm challenge application. In Ieee int. conf. on prognostics and health management.
Coble, J., & Hines, W. (2011). Applying the general path model to estimation of remaining useful life. International Journal of Prognostics and Health Management, 2, 1-13.
El-Koujok, M., Gouriveau, R., & Zerhouni, N. (2011). Reducing arbitrary choices in model building for prognostics: An approach by applying parsimony principle on an evolving neuro-fuzzy system. Microelectronics Reliability, 51(2), 310 - 320.
Feldkamp, L., & Puskorius, G. (1998, Nov). A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification. Proceedings of the IEEE, 86(11), 2259-2277.
Francois, J., Grandvalet, Y., Denoeux, T., & Roger, J.-M. (2003). Resample and combine: An approach to improving uncertainty representation in evidential pattern classification. Information Fusion, 4(2), 75-85.
Frederick, D., DeCastro, J., & Litt, J. (2007). User’s guide for the commercial modular aero-propulsion system simulation (C-MAPSS) (Tech. Rep.). Cleveland, Ohio 44135, USA: National Aeronautics and Space Administration (NASA), Glenn Research Center.
Gouriveau, R., Ramasso, E., & Zerhouni, N. (2013). Strategies to face imbalanced and unlabelled data in PHM applications. Chemical Engineering Transactions, 33, 115-120.
Gouriveau, R., & Zerhouni, N. (2012). Connexionistsystems- based long term prediction approaches for prognostics. IEEE Trans. on Reliability, 61, 909-920.
Heimes, F. (2008). Recurrent neural networks for remaining useful life estimation. In Ieee int. conf. on prognostics and health management.
Hu, C., Youn, B., Wang, P., & Yoon, J. (2012). Ensemble of data-driven prognostic algorithms for robust prediction of remaining useful life. Reliability Engineering and System Safety, 103, 120 - 135.
Huang, G., Zhu, Q., & Siew, C. (2004). Extreme learning machine: A new learning scheme of feedforward neural networks. In International joint conference on neural networks.
Ishibashi, R., & Nascimento Junior, C. (2013). GFRBSPHM: A genetic fuzzy rule-based system for phm with improved interpretability. In Ieee conference on prognostics and health management (phm) (p. 1-7).
Jaeger, H., & Haas, H. (2004). Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science, 304(5667), 78-80.
Javed, K., Gouriveau, R., Zemouri, R., & Zerhouni, N. (2012). Features selection procedure for prognostics: An approach based on predictability. In 8th ifac symposium on fault detection, supervision and safety of technical processes (Vol. 8, p. 25-30).
Javed, K., Gouriveau, R., & Zerhouni, N. (2013). Novel failure prognostics approach with dynamic thresholds for machine degradation. In Ieee industrial electronics conference.
Jianzhong, S., Hongfu, Z., Haibin, Y., & Pecht, M. (2010). Study of ensemble learning-based fusion prognostics. In Prognostics and health management conference, 2010. phm ’10. (p. 1-7).
Kim, H.-E. (2010). Machine prognostics using health state probability estimation. Unpublished doctoral dissertation, School of engineering systems, Faculty of built environmental engineering, Queensland university of technology.
Klir, G., & Wierman, M. (1999). Uncertainty-based information. elements of generalized information theory. In (chap. Studies in fuzzyness and soft computing). Physica-Verlag.
Kuncheva, L. (2004). Classifier ensembles for changing environments. In Int. workshop on multiple classifier systems (Vol. 3077, p. 1-15).
Kuncheva, L., Bezdek, J., & Duin, R. (2001). Decision templates for multiple classifier fusion. Pattern Recognition, 34, 299-314.
Li, X., Qian, J., & Wang, G. (2013). Fault prognostic based on hybrid method of state judgment and regression. Advances in Mechanical Engineering, 2013(149562), 1-10.
Liao, H., & Sun, J. (2011). Nonparametric and semiparametric sensor recovery in multichannel condition monitoring systems. IEEE Transactions on Automation Science and Engineering, 8(4), 744-753.
Lin, Y., Chen, M., & Zhou, D. (2013). Online probabilistic operational safety assessment of multi-mode engineering systems using Bayesian methods. Reliability Engineering & System Safety, 119(0), 150 - 157.
Liu, K., Gebraeel, N. Z., & Shi, J. (2013). A data-level fusion model for developing composite health indices for degradation modeling and prognostic analysis. IEEE Trans. on Automation Science and Engineering.
Moghaddass, R., &Zuo, M. (2014). An integrated framework for online diagnostic and prognostic health monitoring using a multistate deterioration process. Reliability Engineering and System Safety, 124, 92–104.
Monteith, K., Carroll, J., Seppi, K., & Martinez, T. (2011, July). Turning bayesian model averaging into bayesian model combination. In International joint conference on neural networks (p. 2657-2663). doi: 10.1109/IJCNN.2011.6033566
Peel, L. (2008). Data driven prognostics using a Kalman filter ensemble of neural network models. In Int. conf. on prognostics and health management.
Peng, Y., Wang, H., Wang, J., Liu, D., & Peng, X. (2012). A modified echo state network based remaining useful life estimation approach. In Ieee phm conference.
Peng, Y., Xu, Y., Liu, D., & Peng, X. (2012). Sensor selection with grey correlation analysis for remaining useful life evaluation. In Annual conference of the phm society.
Raftery, A., Gneiting, T., Balabdaoui, F., & Polakowski, M. (2003). Using bayesian model averaging to calibrate forecast ensembles. American Meteorological Society, 133(5), 1155-1174.
Ramasso, E. (2009). Contribution of belief functions to hidden Markov models with an application to fault diagnosis. In Machine learning for signal processing.
Ramasso, E. (2014a). Investigating computational geometry for failure prognostics. Int. Journal on Prognostics and Health Management. (submitted)
Ramasso, E. (2014b). Investigating computational geometry for failure prognostics in presence of imprecise health indicator: Results and comparisons on c-mapss datasets. In European conf. on prognostics and health management.
Ramasso, E., & Denoeux, T. (2013). Making use of partial knowledge about hidden states in hidden Markov models: an approach based on belief functions. IEEE Transactions on Fuzzy Systems, 10.1109/TFUZZ.2013.2259496.
Ramasso, E., & Gouriveau, R. (2010). Prognostics in switching systems: Evidential Markovian classification of real-time neuro-fuzzy predictions. In Ieee prognostics and health management conference.
Ramasso, E., & Gouriveau, R. (2013). RUL estimation by classification of predictions: an approach based on a neuro-fuzzy system and theory of belief functions. IEEE Transactions on Reliability, Accepted.
Ramasso, E., Rombaut, M., & Zerhouni, N. (2013). Joint prediction of observations and states in time-series based on belief functions. IEEE Transactions on Systems, Man and Cybernetics - Part B: Cybernetics, 43, 37-50.
Riad, A., Elminir, H., & Elattar, H. (2010). Evaluation of neural networks in the subject of prognostics as compared to linear regression model. International Journal of Engineering & Technology, 10, 52-58.
Richter, H. (2012). Engine models and simulation tools. In Advanced control of turbofan engines (p. 19-33). Springer New York.
Sarkar, S., Jin, X., & Ray, A. (2011). Data-driven fault detection in aircraft engines with noisy sensor measurements. Journal of Engineering for Gas Turbines and Power, 133, 081602.
Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, W. (2008). Metrics for evaluating performance of prognostic techniques. In Int. conf. on prognostics and health management.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2010). Metrics for offline evaluation of prognostic performance. International Journal of Prognostics and Health Management.
Saxena, A., & Goebel, K. (2008). C-mapss data set. NASA Ames Prognostics Data Repository.
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008a). Damage propagation modeling for aircraft engine runto- failure simulation. In Prognostics and health management, 2008. phm 2008. international conference on (pp. 1–9).
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008b). Damage propagation modeling for aircraft engine runto- failure simulation. In Int. conf. on prognostics and health management (p. 1-9). Denver, CO, USA.
Serir, L., Ramasso, E., & Zerhouni, N. (2012). An evidential evolving multimodeling approach for systems behavior prediction. In Annual conference of the phm society.
Siegel, D. (2009). Evaluation of health assessment techniques for rotating machinery. Unpublished master’s thesis, Division of Research and Advanced Studies of the University of Cincinnati.
Siqueira, H., Boccato, L., Attux, R., & Lyra, C. (2012). Echo state networks and extreme learning machines: A comparative study on seasonal streamflow series prediction. In Neural information processing (Vol. 7664, p. 491- 500). Springer Berlin Heidelberg.
Son, K. L., Fouladirad, M., & Barros, A. (2012). Remaining useful life estimation on the non-homogenous gamma with noise deterioration based on gibbs filtering : A case study. In Ieee int. conf. on prognostics and health management.
Son, K. L., Fouladirad, M., Barros, A., Levrat, E., & Iung, B. (2013). Remaining useful life estimation based on stochastic deterioration models: A comparative study. Reliability Engineering and System Safety, 112, 165 - 175.
Sun, J., Zuo, H., Wang, W., & Pecht, M. (2012). Application of a state space modeling technique to system prognostics based on a health index for condition-based maintenance. Mechanical Systems and Signal Processing, 28, 585 - 596.
Tamilselvan, P., & Wang, P. (2013). Failure diagnosis using deep belief learning based health state classification. Reliability Engineering and System Safety, 115(0), 124 - 135.
Wang, P., Youn, B., & Hu, C. (2012). A generic probabilistic framework for structural health prognostics and uncertainty management. Mechanical Systems and Signal Processing, 28, 622 - 637.
Wang, T. (2010). Trajectory similarity based prediction for remaining useful life estimation. Unpublished doctoral dissertation, University of Cincinnati.
Wang, T., Yu, J., Siegel, D., & Lee, J. (2008). A similaritybased prognostics approach for remaining useful life estimation of engineered systems. In Int. conf. on prognostics and health management (p. 1-6).
Xi, Z., Jing, R., Wang, P., & Hu, C. (2013). A copulabased sampling method for data-driven prognostics and health management. In Asme 2013 international design engineering technical conferences and computers and information in engineering conference.
Xue, Y., Williams, D., & Qiu, H. (2011). Classification with imperfect labels for fault prediction. In Proceedings of the first international workshop on data mining for service and maintenance (pp. 12–16). ACM.
Yu, J. (2013). A nonlinear probabilistic method and contribution analysis for machine condition monitoring. Mechanical Systems and Signal Processing, 37, 293-314.
Zein-Sabatto, S., Bodruzzaman, J., & Mikhail, M. (2013). Statistical approach to online prognostics of turbine engine components. In Southeastcon, 2013 proceedings of ieee (p. 1-6).
Zhao, D., P., R. G., & Willett. (2011). Comparison of data reduction techniques based on SVM classifier and SVR performance. In Proc. spie, signal and data processing of small targets (Vol. 8137, p. 1-15).
Technical Papers