A Three-Dimensional Receiver Operator Characteristic Surface Diagnostic Metric



Donald L. Simon


Receiver Operator Characteristic (ROC) curves are commonly applied as metrics for quantifying the performance of binary fault detection systems. An ROC curve provides a visual representation of a detection system’s True Positive Rate versus False Positive Rate sensitivity as the detection threshold is varied. The area under the curve provides a measure of fault detection performance independent of the applied detection threshold. While the standard ROC curve is well suited for quantifying binary fault detection performance, it is not suitable for quantifying the classification performance of multi-fault classification problems. Furthermore, it does not provide a measure of diagnostic latency. To address these shortcomings, a novel three-dimensional receiver operator characteristic (3D ROC) surface metric has been developed. This is done by generating and applying two separate curves: the standard ROC curve reflecting fault detection performance, and a second curve reflecting fault classification performance. A third dimension, diagnostic latency, is added giving rise to three-dimensional ROC surfaces. Applying numerical integration techniques, the volumes under and between the surfaces are calculated to produce metrics of the diagnostic system’s detection and classification performance. This paper will describe the 3D ROC surface metric in detail, and present an example of its application for quantifying the performance of aircraft engine gas path diagnostic methods. Metric limitations and potential enhancements are also discussed.

How to Cite

L. Simon, D. . (2010). A Three-Dimensional Receiver Operator Characteristic Surface Diagnostic Metric. Annual Conference of the PHM Society, 2(1). https://doi.org/10.36001/phmconf.2010.v2i1.1893
Abstract 347 | PDF Downloads 63



Diagnostic metrics

(Davison and Bird, 2008) C. R. Davison and J. W. Bird. Review of Metrics for Gas Path Diagnostic Health Management Techniques for Gas Turbine Engines, Proceedings of the ASME Turbo Expo 2008, GT2008-50849, 2008.
(Demuth and Beale, 2001) H. Demuth and M. Beale. Neural Network Toolbox for Use with Matlab, 7th printing. Natick, Massachusetts: The MathWorks,
Inc., 2001.
(Fawcett, 2006) T. Fawcett. An Introduction to ROC Analysis, Pattern Recognition Letters, vol. 27, pp.861–874, 2006.
(Fawcett and Flach, 2005) T. Fawcett and P. A. Flach.
A Response to Webb and Ting’s On the Application ROC Analysis to Predict Classification Performance Under Varying Class Distributions, Machine Learning, vol. 58, pp. 33-38, 2006.
(Frederick, DeCastro, and Litt, 2007) D. K. Frederick, J. A. DeCastro, and J. S. Litt. User’s Guide for the Commercial Modular Aero-Propulsion System Simulation (C-MAPSS), NASA Technical Memorandum TM-2007-215026, 2007.
(Gelb, 1974) A. Gelb, Applied Optimal Estimation. Cambridge, Massachusetts: The MIT Press, 1974.
(Hall and Llinas, 2001) D.L. Hall and J. Llinas. Handbook of Multisensor Data Fusion, Boca Raton, Florida: CRC Press LLC, 2001.
(Hanley and McNeil, 1982) J. A. Hanley and B. J. McNeil. The Meaning and Use of the Area Under a Receiver Operating Characteristic (ROC) Curve, Radiology, vol. 143, pp. 29-36, 1982.
(Li, 2002) Y. G. Li. Performance-Analysis-Based Gas Turbine Diagnostics: A Review, Proceedings of the Institution of Mechanical Engineers, Part A: Journal of Power and Energy, vol. 216, No. 5, pp. 363-377, 2002.
(Metz, 1978) C. E. Metz, Basic Principals of ROC Analysis, Seminars in Nuclear Medicine, vol. 8, no. 4, pp. 283-298, 1978.
(SAE, 2008) Health and Usage Monitoring Metrics— Monitoring the Monitor, SAE ARP 5783, 2008.
(Vachtsevanos et al., 2006) G. Vachtsevanos, F. L. Lewis, M. Roemer, A. Hess, and B. Wu. Intelligent Fault Diagnosis and Prognosis for Engineering Systems, 1st ed. Hoboken, New Jersey: John Wiley & Sons, Inc, 2006.
(Volponi and Wood 2005) A. Volponi and B. Wood. Engine Health Management for Aircraft Propulsion Systems, The Forum on Integrated System Health Engineering and Management (ISHEM) in Aerospace, November 7-10, Napa, CA, 2005.
(Von Kalman Institute, 2003) Gas Turbine Condition Monitoring and Fault Diagnostics, Von Karman Institute for Fluid Dynamics Lecture Series 2003- 01, 2003.
(Webb and Ting, 2005) G. I. Webb and K. M. Ting. On the Application ROC Analysis to Predict Classification Performance Under Varying Class Distributions, Machine Learning, vol. 58, pp. 25-32, 2005
Technical Papers