Prediction Capability Assessment of Data-Driven Prognostic Methods for Railway Applications



Francesco Di Maio Pietro Turati Enrico Zio


In the development of Prognostics and Health Management (PHM) for industrial applications, the question of which predictive method to use is fundamental. The choice is typically driven by the data and/or the physics-based models available, and the cost-benefit considerations related to PHM implementation, wherein prediction capability plays an important role. By prediction capability of a prognostic method we refer to its ability to provide trustable predictions of the Remaining Useful Life (RUL) of a component or system, with the characteristics required by the given application. A set of Prognostic Performance Indicators (PPIs) is used to guide the choice of the method to be implemented. These PPIs measure different characteristics of a prognostic method and need to be aggregated to enable a final choice of prognostic method, based on its overall performance. We propose an aggregation strategy to identify the prognostic method with the best compromise performance on all PPIs. The strategy is exemplified on a case study with real data taken from industry, whose structure is general and, therefore, applicable to railway industry.

How to Cite

Maio, F. D., Turati, P., & Zio, E. (2016). Prediction Capability Assessment of Data-Driven Prognostic Methods for Railway Applications. PHM Society European Conference, 3(1).
Abstract 79 | PDF Downloads 42



Prognostic Perfomance Indicators (PPI), prediction capability, quality assessment, aggregation strategy

Bechhoefer, E., Bernhard, A., He, D., & Banerjee, P. (2006). Use of Hidden Semi-Markov Models in the Prognostics of Shaft Failure. In AHS International 62nd Annual Forum Proceedings, vol. 2, May 9-11, Phoenix, AZ.
Compare, M., & Zio, E. (2014). Predictive Maintenance by Risk Sensitive Particle Filtering. Reliability, IEEE Transactions on,vol 63(1), pp. 134-143.
Di Maio, F., & Zio, E. (2013). Failure prognostics by a data-driven similarity-based approach. International Journal of Reliability, Quality and Safety Engineering, vol. 20 (2), pp. 1-17, doi: 10.1142/S0218539313500010.
Dong, M., & He, D. (2007). A segmental hidden semi-Markov model (HSMM)-based diagnostics and prognostics framework and methodology. Mechanical Systems and Signal Processing, vol. 21(5), pp. 2248-2266.
Efron, B. (1983). Estimating the error rate of a prediction rule: improvement on cross-validation. Journal of the American Statistical Association, vol. 78(382), pp. 316-331.
Engel, S. J., Gilmartin, B. J., Bongort, K., & Hess, A. (2000). Prognostics, the real issues involved with predicting life remaining. In Aerospace Conference Proceedings, 2000 IEEE, vol. 6, pp. 457-469.
Farrell, M., & Gallagher, R. (2015). The valuation implications of enterprise risk management maturity. Journal of Risk and Insurance, vol. 82(3), pp. 625-657.
Herbsleb, J., Zubrow, D., Goldenson, D., Hayes, W., & Paulk, M. (1997). Software quality and the capability maturity model. Communications of the ACM, vol. 40(6), pp. 30-40.
Huang, N. E., & Wu, Z. (2008). A review on Hilbert‐Huang transform: Method and its applications to geophysical studies. Reviews of Geophysics, vol. 46(2), doi: 10.1029/2007RG000228.
Jardine, A. K., Lin, D., & Banjevic, D. (2006). A review on machinery diagnostics and prognostics implementing condition-based maintenance. Mechanical systems and signal processing, vol 20(7), pp. 1483-1510.
Johnson, S. B., Gormley, T., Kessler, S., Mott, C., Patterson-Hine, A., Reichard, K., & Scandura Jr, P. (Eds.). (2011). System health management: with aerospace applications. John Wiley & Sons.
Kan, M. S., Tan, A. C., & Mathew, J. (2015). A review on prognostic techniques for non-stationary and non-linear rotating systems. Mechanical Systems and Signal Processing, vol. 62,pp. 1-20.
Kan, M. S., Tan, A. C., & Mathew, J. (2015). A review on prognostic techniques for non-stationary and non-linear rotating systems. Mechanical Systems and Signal Processing, vol. 62, pp. 1-20.
Lee, J., Wu, F., Zhao, W., Ghaffari, M., Liao, L., & Siegel, D. (2014). Prognostics and health management design for rotary machinery systems—Reviews, methodology and applications. Mechanical Systems and Signal Processing, vol 42(1),pp. 314-334.
Luo, J., Namburu, M., Pattipati, K., Qiao, L., Kawamoto, M., & Chigusa, S. (2003). Model-based prognostic techniques [maintenance applications]. In AUTOTESTCON 2003. IEEE Systems Readiness Technology Conference. Proceedings, pp. 330-340.
Luo, J., Pattipati, K. R., Qiao, L., & Chigusa, S. (2008). Model-based prognostic techniques applied to a suspension system. Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on, vol. 38(5), pp. 1156-1168.
Mahamad, A. K., Saon, S., & Hiyama, T. (2010). Predicting remaining useful life of rotating machinery based artificial neural network. Computers & Mathematics with Applications, vol.60(4), pp. 1078-1087.
Mendel, J. M. (1995). Fuzzy logic systems for engineering: a tutorial. Proceedings of the IEEE, vol. 83(3), pp. 345-377.
Micea, M. V., Ungurean, L., Cârstoiu, G. N., & Groza, V. (2011). Online state-of-health assessment for battery management systems. Instrumentation and Measurement, IEEE Transactions on, vol. 60(6), pp. 1997-2006.
Paulk, M. (1993). Capability maturity model for software. John Wiley & Sons, Inc.
Pecht, M., & Jaai, R. (2010). A prognostics and health management roadmap for information and electronics-rich systems. Microelectronics Reliability, vol. 50(3), pp. 317-323.
Peng, T., Liu, Y., Saxena, A., & Goebel, K. (2015). In-situ fatigue life prognosis for composite laminates based on stiffness degradation. Composite Structures,132, 155-165.
Polikar, R. (2007). Bootstrap-Inspired Techniques in Computation Intelligence. IEEE Signal Processing Magazine, vol. 4(24), pp. 59-72.
Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, M. (2008). Metrics for evaluating performance of prognostic techniques. In Prognostics and health management, 2008. PHM 2008. international conference on (pp. 1-17). IEEE, October, Denver, CO, USA.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2010). Metrics for offline evaluation of prognostic performance. International Journal of Prognostics and Health Management, vol. 1(1), pp. 4-23.
Saxena, A., Roychoudhury, I., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2012). Requirements Flowdown for Prognostics and Health Management.Proceedings of the AIAA Infotech@ Aerospace, Garden Grove,CA.
Saxena, A., Sankararaman, S., & Goebel, K. (2014). Performance evaluation for fleet-based and unit-based prognostic methods. In Second European conference of the Prognostics and Health Management society, July 8-10, Nantes, FR.
Schwabacher, M. (2005). A survey of data-driven prognostics. In Proceedings of the AIAA Infotech@ Aerospace Conference, pp. 1-5.
Si, X. S., Wang, W., Hu, C. H., & Zhou, D. H. (2011). Remaining useful life estimation–a review on the statistical data driven approaches. European Journal of Operational Research, vol. 213(1), pp. 1-14.
Sikorska, J. Z., Hodkiewicz, M., & Ma, L. (2011). Prognostic modelling options for remaining useful life estimation by industry. Mechanical Systems and Signal Processing, vol. 25(5), pp. 1803-1836.
Tobon-Mejia, D. A., Medjaher, K., Zerhouni, N., & Tripot, G. (2012). A data-driven failure prognostics method based on mixture of gaussians hidden markov models. Reliability, IEEE Transactions on, vol. 61(2), pp. 491-503.
Tsui, K. L., Chen, N., Zhou, Q., Hai, Y., & Wang, W. (2015). Prognostics and health management: A review on data driven approaches. Mathematical Problems in Engineering, 2015.
Vachtsevanos, G., Lewis, F., Roemer, M., Hess, A. & Wu, B. (2006). Intelligent Fault Diagnosis and Prognosis for Engineering Systems, John Wiley & Sons, Inc., Hoboken, NJ, USA. doi: 10.1002/9780470117842.
Vichare, N. M., & Pecht, M. G. (2006). Prognostics and health management of electronics. Components and Packaging Technologies, IEEE Transactions on, vol. 29(1), pp. 222-229.
Walther, B. A., & Moore, J. L. (2005). The concepts of bias, precision and accuracy, and their use in testing the performance of species richness estimators, with a literature review of estimator performance. Ecography, vol. 28(6), pp. 815-829.
Xian, W., Long, B., Li, M., & Wang, H. (2014). Prognostics of lithium-ion batteries based on the verhulst model, particle swarm optimization and particle filter. Instrumentation and Measurement, IEEE Transactions on, vol. 63(1), pp. 2-17.
Yu, H., & Wilamowski, B. M. (2011). Levenberg–marquardt training. Industrial electronics handbook, vol. 5(12), pp. 1-16.
Yu, S. Z. (2010). Hidden semi-Markov models. Artificial Intelligence, vol. 174(2), pp. 215-243.
Zeng, Z., Di Maio, F, Zio, E., & Kang K. (2016). A hierarchical decision making framework for the assessment of the prediction capability of prognostic methods. IEEE Transactions on Reliability. (Submitted).
Zhang, Z., Si, X., Hu, C., & Kong, X. (2015). Degradation modeling–based remaining useful life estimation: A review on approaches for systems with heterogeneity. Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability, vol. 229 (4SI), pp. 343-355, 1748006X15579322.
Zio, E. (2009). Reliability engineering: Old problems and new challenges. Reliability Engineering & System Safety, vol. 94(2), 125-141.
Zio, E., & Compare, M. (2013). Evaluating maintenance policies by quantitative modeling and analysis. Reliability Engineering & System Safety, vol. 109, pp. 53-65.
Zio, E., Di Maio, F. & Stasi, M (2010). A Data-driven Approach for Predicting Failure Scenarios in Nuclear Systems. Annals of Nuclear Energy, vol. 37, pp. 482–491.
Technical Papers