A Modelling Ecosystem for Prognostics



Published Oct 3, 2016
Lachlan Astfalck Melinda Hodkiewicz Adrian Keating Edward Cripps Michael Pecht


This paper evaluates data-driven asset prognostic models from a modelling ecosystem perspective, which includes data description, uncertainty quantification, model selection justification and validation, and application limitations. An easily accessible and comprehensive ecosystem enables efficient reproducibility of previous work to facilitate both the adoption of the models by industry and the development of future scientific methods. The results of this study enable the development of a list of ecosystem elements to accompany the publication of new models. By describing the ecosystem in the communication of new models, researchers can ensure the reproducibility of their models in the wider prognostic community.

How to Cite

Astfalck, L., Hodkiewicz, M., Keating, A., Cripps, E., & Pecht, M. (2016). A Modelling Ecosystem for Prognostics. Annual Conference of the PHM Society, 8(1). https://doi.org/10.36001/phmconf.2016.v8i1.2568
Abstract 222 | PDF Downloads 105



prognostics, Reproducibility, modelling ecosystem

Aizpurua, J. I., & Catterson, V. M. (2015). Towards a methodology for design of prognostic systems. In The Annual Conference of the PHM Society. Coronado, CA, USA.
Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483(7391), 531–533.
Bissel, M. (2013). Reproducibility: The risks of the replication drive. Nature Comment, 503(7476), 333–334.
Casadevall, A., & Fang, F. C. (2010). Reproducible science. Infection and immunity, 78(12), 4972–4975. Center for Open Research. (2016). Open science framework. Retrieved 2016-06-03, from https://osf.io/
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., G¨ulmezoglu, A. M., . . . Oliver, S. (2014). How to increase value and reduce waste when research priorities are set. The Lancet, 383(9912), 156–165.
Chan, A.-W., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gøtzsche, P. C., . . . Van Der Worp, H. B. (2014). Increasing value and reducing waste: addressing inaccessible research. The Lancet, 383(9913), 257–266.
Coble, J. B. (2010). Merging data sources to predict remaining useful life–an automated method to identify prognostic parameters.
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015). The economics of reproducibility in preclinical research. PLoS Biol, 13(6), e1002165.
Glasziou, P., Altman, D. G., Bossuyt, P., Boutron, I., Clarke, M., Julious, S., . . . Wager, E. (2014). Reducing waste from incomplete or unusable reports of biomedical research. The Lancet, 383(9913), 267–276.
Gutzwiller, K. J., & Riffell, S. K. (2014). Rigor and transparency in statistical analyses can help to ensure valid research. Landscape Ecology, 29(7), 1115–1122.
Heng, A., Tan, A. C., Mathew, J., Montgomery, N., Banjevic, D., & Jardine, A. K. (2009). Intelligent conditionbased prediction of machinery reliability. Mechanical Systems and Signal Processing, 23(5), 1600–1614.
Hodkiewicz, M., & Montgomery, N. (2014). Data fitness for purpose: assessing the quality of industrial data for use in mathematical models.
Ioannidis, J. P., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., . . . Tibshirani, R. (2014). Increasing value and reducing waste in research design, conduct, and analysis. The Lancet, 383(9912), 166–175.
Kan, M. S., Tan, A. C., & Mathew, J. (2015). A review on prognostic techniques for non-stationary and nonlinear rotating systems. Mechanical Systems and Signal Processing, 62, 1–20.
Kelly, A. (2006). Maintenance systems and documentation. Butterworth-Heinemann.
Kilkenny, C., Parsons, N., Kadyszewski, E., Festing, M. F., Cuthill, I. C., Fry, D., . . . Altman, D. G. (2009). Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PloS One, 4(11), e7824.
Lee, J., Wu, F., Zhao, W., Ghaffari, M., Liao, L., & Siegel, D. (2014). Prognostics and health management design for rotary machinery systemsreviews, methodology and applications. Mechanical Systems and Signal Processing, 42(1), 314–334.
Leek, J. T., & Peng, R. D. (2015). Opinion: Reproducible research can still be wrong: Adopting a prevention approach. Proceedings of the National Academy of Sciences, 112(6), 1645–1646.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J., . . . Glasziou, P. (2014). Biomedical research: increasing value, reducing waste. The Lancet, 383(9912), 101–104.
McNutt, M. (2014). Reproducibility [editorial]. Science, 343(6168), 229–229. Nature. (2016). Availability of data, material and methods. Retrieved 2016-06-02, from http://www.nature.com/authors/policies/availability.html
Nature Editorial. (2014). Journals unite for reproducibility [editorial]. Nature, 515(6210), 7.
Nosek, B., Alter, G., Banks, G., Borsboom, D., Bowman, S., Breckler, S., . . . Christensen, G. (2015). Promoting an open research culture: Author guidelines for journals could help to promote transparency, openness, and reproducibility. Science, 348(6242), 1422–1425.
Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657–660.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
Pecht, M. (2008). Prognostics and health management of electronics. Wiley.
Peng, R. D. (2009). Reproducible research and biostatistics. Biostatistics, 10(3), 405–408.
Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060), 1226–1227.
Peng, R. D., Dominici, F., & Zeger, S. L. (2006). Reproducible epidemiologic research. American Journal of Epidemiology, 163(9), 783–789.
Prinz, F., Schlange, T., & Asadullah, K. (2011). Believe it or not: how much can we rely on published data on potential drug targets? Nature Reviews, 10(9), 712–712.
Riehle, D. (2007). The economic motivation of open source software: Stakeholder perspectives. Computer, 40(4), 25–32.
Salman, R. A.-S., Beller, E., Kagan, J., Hemminki, E., Phillips, R. S., Savulescu, J., . . . Chalmers, I. (2014). Increasing value and reducing waste in biomedical research regulation and management. The Lancet, 383(9912), 176–185.
Sankararaman, S., Daigle, M. J., & Goebel, K. (2014). Uncertainty quantification in remaining useful life prediction using first-order reliability methods. IEEE Transactions on Reliability, 63(2), 603-619.
Sankararaman, S., & Goebel, K. (2015). Uncertainty in prognostics and systems health management. International Journal of Prognostics and Health Management, 6(10).
Sankararaman, S., Saxena, A., & Goebel, K. (2014). Are current prognostic performance evaluation practices sufficient and meaningful? In The Annual Conference of the PHM Society. Fort Worth, TX, USA.
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2010). Metrics for offline evaluation of prognostic performance. International Journal of Prognostics and Health Management, 1(1), 4–23.
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008). Damage propagation modeling for aircraft engine runto-failure simulation. In The International Conference on Prognostics and Health Management. Denver, CO, USA.
Schwab, M., Karrenbach, M., & Claerbout, J. (2000). Making scientific computations reproducible. Computing in Science & Engineering, 2(6), 61–67.
Sikorska, J., Hodkiewicz, M., DCruz, A., Astfalck, L., & Keating, A. (2016). A collaborative data library for testing prognostic models. In The Third European Conference of the PHM Society. Bilbao, Spain.
Sikorska, J., Hodkiewicz, M., & Ma, L. (2011). Prognostic modelling options for remaining useful life estimation by industry. Mechanical Systems and Signal Processing, 25(5), 1803–1836.
Sonnenburg, S., Braun, M. L., Ong, C. S., Bengio, S., Bottou, L., Holmes, G., . . . Williamson, R. (2007, 10). The need for open source software in machine learning. Journal of Machine Learning Research, 8, 2443-2466.
Steckler, T. (2015). Preclinical data reproducibility for r&dthe challenge for neuroscience [editorial]. Psychopharmacology, 232(2), 317–320.
Sun, J., Zuo, H., Wang, W., & Pecht, M. G. (2012). Application of a state space modeling technique to system prognostics based on a health index for condition-based maintenance. Mechanical Systems and Signal Processing, 28, 585–596.
Sundin, P., Eng, P., Montgomery, N., & Jardine, A. (2007). Pulp mill on-site implementation of cbm decision support software. In ICOMS Asset Management Conference. Melbourne, Australia.
Tian, Z., Wong, L., & Safaei, N. (2010). A neural network approach for remaining useful life prediction utilizing both failure and suspension histories. Mechanical Systems and Signal Processing, 24(5), 1542–1555. US National Institutes of Health. (2014). Principles and guidelines for reporting preclinical research. Retrieved 2016-06-02, from https://www.nih.gov/researchtraining/rigor-reproducibility/principles-guidelines-eportingpreclinical-
Vachtsevanos, G., Lewis, F., Roemer, M., Hess, A., & Wu, B. (2006). Intelligent fault diagnosis and prognosis for engineering systems.
Technical Research Papers

Most read articles by the same author(s)