Are Current Prognostic Performance Evaluation Practices Sufficient and Meaningful?

##plugins.themes.bootstrap3.article.main##

##plugins.themes.bootstrap3.article.sidebar##

Shankar Sankararaman Abhinav Saxena Kai Goebel

Abstract

This paper investigates the shortcomings of performance evaluation for prognostic algorithms, particularly in the presence of uncertainty. To that end, the various elements of a prognostic algorithm (present health state estimation, future load condition, degradation model, and damage threshold) and their effects on prognostics are examined. Each of these elements contribute to overall prediction performance and therefore it is important to distinguish between (1) assessment of the correctness of information regarding these quantities, and (2) the assessment of correctness of the prognostic algorithm. The need for proper accounting for uncertainty in the various associated elements is discussed. Next, the shortcomings of traditional comparisons between ground truth and algorithm prediction is discussed. Several scenarios are pointed out where misleading interpretations about evaluation out-comes are possible. In order to address these shortcomings an “informed evaluation” methodology is being proposed, where the algorithm is informed with future loading/operating conditions before comparing against ground truth. Additionally, the importance of estimating the accuracy of aggregating the different sources of uncertainty using rigorous mathematical procedures is also emphasized. While this discussion does not target developing new metrics, it highlights key criteria for an accurate performance evaluation process under uncertainty and proposes new measures to accomplish this goal.

How to Cite

Sankararaman, S., Saxena, A. ., & Goebel, K. . (2014). Are Current Prognostic Performance Evaluation Practices Sufficient and Meaningful?. Annual Conference of the PHM Society, 6(1). https://doi.org/10.36001/phmconf.2014.v6i1.2520
Abstract 3 | PDF Downloads 1

##plugins.themes.bootstrap3.article.details##

Keywords

prognostics, performance evaluation, uncertainty

References
Calvetti, D., & Somersalo, E. (2007). Introduction to Bayesian scientific computing: ten lectures on subjective computing (Vol. 2). Springer.

Daigle, M., Saxena, A., & Goebel, K. (2012). An efficient deterministic approach to model-based prediction un- certainty estimation. In Annual conference of the prognostics and health management society.

DeCastro, J. A. (2009). Exact nonlinear filtering and pre- diction in process model-based prognostics. In Annual conference of the prognostics and health management society. San Diego, CA..

De Finetti, B., & de Finetti, B. (1977). Theory of probability, volume i. Bull. Amer. Math. Soc, 83, 94–97.

Orchard, M., Kacprzynski, G., Goebel, K., Saha, B., & Vachtsevanos, G. (2008, Oct.). Advances in uncertainty representation and management for particle filtering applied to prognostics. In Prognostics and health management, 2008. phm 2008. international conference on (p. 1 -6). DOI: 10.1109/PHM.2008.4711433.

Roychoudhury, I., Saxena, A., Celaya, J. R., & Goebel, K. (2013). Distilling the verification process for prognostics algorithms. In 2013 annual conference of the prognostics and health management society.

Saha, B., Goebel, K., Poll, S., & Christophersen, J. (2009, Feb.). Prognostics methods for battery health monitoring using a bayesian framework. IEEE Transactions on Instrumentation and Measurement, 58(2), 291 -296. DOI: 10.1109/TIM.2008.2005965.

Sankararaman, S., Daigle, M., & Goebel, K. (2014, June). Uncertainty quantification in remaining useful life pre- diction using first-order reliability methods. Reliability, IEEE Transactions on, 63(2), 603-619. DOI: 10.1109/TR.2014.2313801

Sankararaman, S., & Goebel, K. (2013a). Remaining useful life estimation in prognosis: An uncertainty propagation problem. In 2013 aiaa infotech@ aerospace conference.

Sankararaman, S., & Goebel, K. (2013b). Why is the remaining useful life prediction uncertain? In Annual conference of the prognostics and health management society.

Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, M. (2008). Metrics for evaluating performance of prognostic techniques. In Prognostics and health management, 2008. phm 2008. international conference on (pp. 1–17).

Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2010). Metrics for offline evaluation of prognostic performance. International Journal of Prognostics and Health Management, 1(1), 20.

Saxena, A., Sankararaman, S., & Goebel, K. (2014). Performance evaluation for fleet-based and unit-based prognostic methods. In Second european conference of the prognostics and health management society.

Szabo ́, L. (2007). Objective probability-like things with and without objective indeterminism. Studies In History and Philosophy of Science Part B: Studies In History and Philosophy of Modern Physics, 38(3), 626–634.

Wang, H.-F. (2011, January). Decision of prognostics and health management under uncertainty. International Journal of Computer Applications, 13(4), 1–5. (Published by Foundation of Computer Science)
Section
Technical Papers

Most read articles by the same author(s)

1 2 3 4 5 > >>