A Three-Dimensional Receiver Operator Characteristic Surface Diagnostic Metric
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Abstract
Receiver Operator Characteristic (ROC) curves are commonly applied as metrics for quantifying the performance of binary fault detection systems. An ROC curve provides a visual representation of a detection system’s True Positive Rate versus False Positive Rate sensitivity as the detection threshold is varied. The area under the curve provides a measure of fault detection performance independent of the applied detection threshold. While the standard ROC curve is well suited for quantifying binary fault detection performance, it is not suitable for quantifying the classification performance of multi-fault classification problems. Furthermore, it does not provide a measure of diagnostic latency. To address these shortcomings, a novel three-dimensional receiver operator characteristic (3D ROC) surface metric has been developed. This is done by generating and applying two separate curves: the standard ROC curve reflecting fault detection performance, and a second curve reflecting fault classification performance. A third dimension, diagnostic latency, is added giving rise to three-dimensional ROC surfaces. Applying numerical integration techniques, the volumes under and between the surfaces are calculated to produce metrics of the diagnostic system’s detection and classification performance. This paper will describe the 3D ROC surface metric in detail, and present an example of its application for quantifying the performance of aircraft engine gas path diagnostic methods. Metric limitations and potential enhancements are also discussed.
How to Cite
##plugins.themes.bootstrap3.article.details##
Diagnostic metrics
(Demuth and Beale, 2001) H. Demuth and M. Beale. Neural Network Toolbox for Use with Matlab, 7th printing. Natick, Massachusetts: The MathWorks,
Inc., 2001.
(Fawcett, 2006) T. Fawcett. An Introduction to ROC Analysis, Pattern Recognition Letters, vol. 27, pp.861–874, 2006.
(Fawcett and Flach, 2005) T. Fawcett and P. A. Flach.
A Response to Webb and Ting’s On the Application ROC Analysis to Predict Classification Performance Under Varying Class Distributions, Machine Learning, vol. 58, pp. 33-38, 2006.
(Frederick, DeCastro, and Litt, 2007) D. K. Frederick, J. A. DeCastro, and J. S. Litt. User’s Guide for the Commercial Modular Aero-Propulsion System Simulation (C-MAPSS), NASA Technical Memorandum TM-2007-215026, 2007.
(Gelb, 1974) A. Gelb, Applied Optimal Estimation. Cambridge, Massachusetts: The MIT Press, 1974.
(Hall and Llinas, 2001) D.L. Hall and J. Llinas. Handbook of Multisensor Data Fusion, Boca Raton, Florida: CRC Press LLC, 2001.
(Hanley and McNeil, 1982) J. A. Hanley and B. J. McNeil. The Meaning and Use of the Area Under a Receiver Operating Characteristic (ROC) Curve, Radiology, vol. 143, pp. 29-36, 1982.
(Li, 2002) Y. G. Li. Performance-Analysis-Based Gas Turbine Diagnostics: A Review, Proceedings of the Institution of Mechanical Engineers, Part A: Journal of Power and Energy, vol. 216, No. 5, pp. 363-377, 2002.
(Metz, 1978) C. E. Metz, Basic Principals of ROC Analysis, Seminars in Nuclear Medicine, vol. 8, no. 4, pp. 283-298, 1978.
(SAE, 2008) Health and Usage Monitoring Metrics— Monitoring the Monitor, SAE ARP 5783, 2008.
(Vachtsevanos et al., 2006) G. Vachtsevanos, F. L. Lewis, M. Roemer, A. Hess, and B. Wu. Intelligent Fault Diagnosis and Prognosis for Engineering Systems, 1st ed. Hoboken, New Jersey: John Wiley & Sons, Inc, 2006.
(Volponi and Wood 2005) A. Volponi and B. Wood. Engine Health Management for Aircraft Propulsion Systems, The Forum on Integrated System Health Engineering and Management (ISHEM) in Aerospace, November 7-10, Napa, CA, 2005.
(Von Kalman Institute, 2003) Gas Turbine Condition Monitoring and Fault Diagnostics, Von Karman Institute for Fluid Dynamics Lecture Series 2003- 01, 2003.
(Webb and Ting, 2005) G. I. Webb and K. M. Ting. On the Application ROC Analysis to Predict Classification Performance Under Varying Class Distributions, Machine Learning, vol. 58, pp. 25-32, 2005
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
The Prognostic and Health Management Society advocates open-access to scientific data and uses a Creative Commons license for publishing and distributing any papers. A Creative Commons license does not relinquish the author’s copyright; rather it allows them to share some of their rights with any member of the public under certain conditions whilst enjoying full legal protection. By submitting an article to the International Conference of the Prognostics and Health Management Society, the authors agree to be bound by the associated terms and conditions including the following:
As the author, you retain the copyright to your Work. By submitting your Work, you are granting anybody the right to copy, distribute and transmit your Work and to adapt your Work with proper attribution under the terms of the Creative Commons Attribution 3.0 United States license. You assign rights to the Prognostics and Health Management Society to publish and disseminate your Work through electronic and print media if it is accepted for publication. A license note citing the Creative Commons Attribution 3.0 United States License as shown below needs to be placed in the footnote on the first page of the article.
First Author et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.