A Framework for Evaluating Analytic Hyperparameters



Published Oct 28, 2022
Shashvat Prakash Antoni Brzoska Sanket Amin


Prognostic models, when feasible, are favored for avoiding unexpected maintenance. There is a need for a common language when discussing prognostic performance and behavior. The approach presented here considers model behavior in terms of two optimizable sub-problems for better performance assessment. The first evaluation construct considers how well the model tracks degradation over time and a second construct considers how effectively it improves operations. The right set of cost functions can determine the suitability to both objectives. The combined construct enables evaluation of a class of models which augment degradation physics with data-driven heuristics, supporting a more explainable recommendation.

How to Cite

Prakash, S., Brzoska, A., & Amin, S. (2022). A Framework for Evaluating Analytic Hyperparameters. Annual Conference of the PHM Society, 14(1). https://doi.org/10.36001/phmconf.2022.v14i1.3199
Abstract 409 | PDF Downloads 368



Evaluation, Business, Cost, ROC, RUL, Threshold

[1] “International Maintenance Review Board Policy Board, Aircraft Health Monitoring (AHM) Integration in MSG 3, Issue Paper 180.” MSG-3: Operator/Manufacturer scheduled Maintenance Development, Vol. 1- Fixed Wing Aircraft Rev. 2018.1, Air Transport Association of America, 2018, https://www.easa.europa.eu/document-library/imrbpb-issue-papers.
[2] L. Zhang, X. Li, J. Yu, “A Review of Fault Prognostics in Condition Based Maintenance,” Proc. SPIE 6357, Sixth International Symposium on Instrumentation and Control Technology (2006).
[3] J. H. Luo, M. Namburu, K. Pattipati, Q. Liu, M. Kawamoto, S. Chigusa, "Model-based prognostic techniques [maintenance applications]," Proceedings of AUTOTESTCON 2003, IEEE Systems Readiness Technology Conference. 22-25, 330-340(2003).
[4] V. Fornlof, D. Galar, A. Syberfeldt and T. Almgren, “Maintenance, Prognostics and Diagnostic Approaches for Aircraft engines,” IEEE Metrology for Aerospace (2016).
[5] J.P. Sprong, X. Jiang, and H. Polinder, Deployment of Prognostics to Optimize Aircraft Maintenance - A Literature Review,” Proceedings of the Annual Conference of the Prognostics and Health Management Society (2019).
[6] K. Pipe, “Practical Prognostics for Condition Based Maintenance,” International Conference on Prognostics and Health Management (2008).
[7] T. Fawcett, “An Introduction to ROC Analysis,” Pattern Recognition Letters. 27 (8): 861–874 (2006).
[8] R. F. Estrada and E. A. Starr, “50 Years of Acoustical Signal Processing for Detection: Coping with the Digital Revolution,” IEEE Annals of the History of Computing 65-78, (2005).
[9] R. B. Abernethy, J. E. Breneman, C. H. Medlin, G. L. Reinman. Weibull Analysis Handbook. West Palm Beach, Pratt and Whitney Government Products Division, Nov. 1983.
[10] S. Pattabhiraman, C. Gogu, N. Kim, R. T. Haftka, and C. Bes, “Skipping unnecessary maintenance using an on-board structural health monitoring system,” Proc IMechE Part O: J Risk and Reliability 226(5) 549-560, (2012).
[11] X. Lei, and P. A. Sandborn. "PHM-based wind turbine maintenance optimization using real options." Int J Progn Health Manag 7.1 (2016): 1-14
[12] X. Lei, P. A. Sandborn, “Maintenance Scheduling Based on Remaining Useful Life Predictions for Wind Farms Managed Using Power Purchase Agreements,” Renewable Energy, vol. 116, Part B, pp.188-198 (2018).
[13] S. Prakash, A. Brzoska, J. Ensberg, ""A Framework for Evaluating PHM Models," IEEE Aerospace Sciences Conference, (2022).
Technical Research Papers