Are Current Prognostic Performance Evaluation Practices Sufficient and Meaningful?
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Abstract
This paper investigates the shortcomings of performance evaluation for prognostic algorithms, particularly in the presence of uncertainty. To that end, the various elements of a prognostic algorithm (present health state estimation, future load condition, degradation model, and damage threshold) and their effects on prognostics are examined. Each of these elements contribute to overall prediction performance and therefore it is important to distinguish between (1) assessment of the correctness of information regarding these quantities, and (2) the assessment of correctness of the prognostic algorithm. The need for proper accounting for uncertainty in the various associated elements is discussed. Next, the shortcomings of traditional comparisons between ground truth and algorithm prediction is discussed. Several scenarios are pointed out where misleading interpretations about evaluation out-comes are possible. In order to address these shortcomings an “informed evaluation” methodology is being proposed, where the algorithm is informed with future loading/operating conditions before comparing against ground truth. Additionally, the importance of estimating the accuracy of aggregating the different sources of uncertainty using rigorous mathematical procedures is also emphasized. While this discussion does not target developing new metrics, it highlights key criteria for an accurate performance evaluation process under uncertainty and proposes new measures to accomplish this goal.
How to Cite
##plugins.themes.bootstrap3.article.details##
prognostics, performance evaluation, uncertainty
Daigle, M., Saxena, A., & Goebel, K. (2012). An efficient deterministic approach to model-based prediction un- certainty estimation. In Annual conference of the prognostics and health management society.
DeCastro, J. A. (2009). Exact nonlinear filtering and pre- diction in process model-based prognostics. In Annual conference of the prognostics and health management society. San Diego, CA..
De Finetti, B., & de Finetti, B. (1977). Theory of probability, volume i. Bull. Amer. Math. Soc, 83, 94–97.
Orchard, M., Kacprzynski, G., Goebel, K., Saha, B., & Vachtsevanos, G. (2008, Oct.). Advances in uncertainty representation and management for particle filtering applied to prognostics. In Prognostics and health management, 2008. phm 2008. international conference on (p. 1 -6). DOI: 10.1109/PHM.2008.4711433.
Roychoudhury, I., Saxena, A., Celaya, J. R., & Goebel, K. (2013). Distilling the verification process for prognostics algorithms. In 2013 annual conference of the prognostics and health management society.
Saha, B., Goebel, K., Poll, S., & Christophersen, J. (2009, Feb.). Prognostics methods for battery health monitoring using a bayesian framework. IEEE Transactions on Instrumentation and Measurement, 58(2), 291 -296. DOI: 10.1109/TIM.2008.2005965.
Sankararaman, S., Daigle, M., & Goebel, K. (2014, June). Uncertainty quantification in remaining useful life pre- diction using first-order reliability methods. Reliability, IEEE Transactions on, 63(2), 603-619. DOI: 10.1109/TR.2014.2313801
Sankararaman, S., & Goebel, K. (2013a). Remaining useful life estimation in prognosis: An uncertainty propagation problem. In 2013 aiaa infotech@ aerospace conference.
Sankararaman, S., & Goebel, K. (2013b). Why is the remaining useful life prediction uncertain? In Annual conference of the prognostics and health management society.
Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, M. (2008). Metrics for evaluating performance of prognostic techniques. In Prognostics and health management, 2008. phm 2008. international conference on (pp. 1–17).
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2010). Metrics for offline evaluation of prognostic performance. International Journal of Prognostics and Health Management, 1(1), 20.
Saxena, A., Sankararaman, S., & Goebel, K. (2014). Performance evaluation for fleet-based and unit-based prognostic methods. In Second european conference of the prognostics and health management society.
Szabo ́, L. (2007). Objective probability-like things with and without objective indeterminism. Studies In History and Philosophy of Science Part B: Studies In History and Philosophy of Modern Physics, 38(3), 626–634.
Wang, H.-F. (2011, January). Decision of prognostics and health management under uncertainty. International Journal of Computer Applications, 13(4), 1–5. (Published by Foundation of Computer Science)
The Prognostic and Health Management Society advocates open-access to scientific data and uses a Creative Commons license for publishing and distributing any papers. A Creative Commons license does not relinquish the author’s copyright; rather it allows them to share some of their rights with any member of the public under certain conditions whilst enjoying full legal protection. By submitting an article to the International Conference of the Prognostics and Health Management Society, the authors agree to be bound by the associated terms and conditions including the following:
As the author, you retain the copyright to your Work. By submitting your Work, you are granting anybody the right to copy, distribute and transmit your Work and to adapt your Work with proper attribution under the terms of the Creative Commons Attribution 3.0 United States license. You assign rights to the Prognostics and Health Management Society to publish and disseminate your Work through electronic and print media if it is accepted for publication. A license note citing the Creative Commons Attribution 3.0 United States License as shown below needs to be placed in the footnote on the first page of the article.
First Author et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.