Empirical Evaluation of Diagnostic Algorithm Performance Using a Generic Framework



Published Mar 22, 2021
Alexander Feldman Tolga Kurtoglu Sriram Narasimhan Scott Poll David Garcia Johan de Kleer Lukas Kuhn Arjan van Gemund


A variety of rule-based, model-based and datadriven techniques have been proposed for detection and isolation of faults in physical systems. However, there have been few efforts to comparatively analyze the performance of these approaches on the same system under identical conditions. One reason for this was the lack of a standard framework to perform this comparison. In this paper we introduce a framework, called DXF, that provides a common language to represent the system description, sensor data and the fault diagnosis results; a run-time architecture to execute the diagnosis algorithms under identical conditions and collect the diagnosis results; and an evaluation component that can compute performance metrics from the diagnosis results to compare the algorithms. We have used DXF to perform an empirical evaluation of 13 diagnostic algorithms on a hardware testbed (ADAPT) at NASA Ames Research Center and on a set of synthetic circuits typically used as benchmarks in the model-based diagnosis community. Based on these empirical data we analyze the performance of each algorithm and suggest directions for future development.

Abstract 176 | PDF Downloads 253



diagnosis, diagnostic algorithm, diagnostic performance, fault diagnosis, applications: aviation

Barty´s, M., Patton, R., Syfert, M., de las Heras, S., & Quevedo, J. (2006). Introduction to the DAMADICS actuator FDI benchmark study. Control Engineering Practice, 14, 577–596.
Basseville,M., & Nikiforov, I. V. (1993). Detection of Abrupt Changes: Theory and Application. Prentice-Hall.
Brglez, F., & Fujiwara, H. (1985). A neutral netlist of 10 combinational benchmark circuits and a target translator in Fortran. In Proc. IS-CAS’85, pp. 695–698.
Committee E-32, S. A. P. S. H. M. (2008). Health and usage monitoring metrics, monitoring the monitor. Tech. rep. ARP5783.
Davis, M., Logemann, G., & Loveland, D. (1962). A machine program for theorem-proving. Communications of the ACM, 5(7), 394–397.
de Freitas, N. (2002). Rao-blackwellised particle filtering for fault diagnosis. In Proc. AERO-CONF’02, Vol. 4, pp. 1767–1772.
de Kleer, J. (2009). Minimum cardinality candidate generation. In Proc. DX’09, pp. 397–402.
de Kleer, J., Mackworth, A., & Reiter, R. (1992). Characterizing diagnoses and systems. Arti- ficial Intelligence, 56(2-3), 197–222.
de Kleer, J., & Williams, B. (1987). Diagnosing multiple faults. Artificial Intelligence, 32(1), 97–130.
DePold, H. R., Rajamani, R., Morrison, W. H., & Pattipati, K. R. (2006). A unified metric for fault detection and isolation in engines. In Proc. TURBO’06.
DePold, H. R., Siegel, J., & Hull, J. (2004). Metrics for evaluating the accuracy of diagnostic fault detection systems. In Proc. TURBO’04.
Feldman, A., Provan, G., & van Gemund, A. (2007). Interchange formats and automated benchmark model generators for modelbased diagnostic inference. In Proc. DX’07, pp. 91–98.
Feldman, A., Provan, G., & van Gemund, A. (2008a). Computing minimal diagnoses by greedy stochastic search. In Proc. AAAI’08, pp. 911–918.
Feldman, A., Provan, G., & van Gemund, A. (2008b). Computing observation vectors for max-fault min-cardinality diagnoses. In Proc. AAAI’08, pp. 911–918.
Feldman, A., Provan, G., & van Gemund, A. (2009). The Lydia approach to combinational model-based diagnosis. In Proc. DX’09, pp. 403–408.
Forbus, K., & de Kleer, J. (1993). Building Prob-lem Solvers. MIT Press.
Gertler, J. J. (1998). Fault Detection and Diagno-sis in Engineering Systems. Marcel Dekker.
Grastien, A., & Kan-John, P. (2009). Wizards of Oz – description of the 2009 DXC entry. In Proc. DX’09, pp. 409–413.
Heller, U., & Struss, P. (2001). G+DE - the generalized diagnosis engine. In Proc. DX’01, pp. 79–86.
Hoyle, C., Mehr, A. F., Tumer, I. Y., & Chen, W. (2007). Cost-benefit quantification of ISHM in aerospace systems. In Proc. IDETC/CIE’07.
Iverson, D. L. (2004). Inductive system health monitoring. In Proc. ICAI’04.
Izadi-Zamanabadi, R., & Blanke, M. (1999). A ship propulsion system as a benchmark for fault-tolerant control. Control Engineering Practice, 7(2), 227–240.
Karin, L., Lunde, R., & M¨unker, B. (2006). Modelbased failure analysis with RODON. In Proc. ECAI’06.
Kavčič, M., & Juričić, Ð. (1997). A prototyping tool for fault tree based process diagnosis. In Proc. DX’97, pp. 129–133.
Kurien, J., & Moreno, M. D. R. (2008). Costs and benefits of model-based diagnosis. In Proc. AEROCONF’08.
Kurtoglu, T., Mengshoel, O., & Poll, S. (2008). A framework for systematic benchmarking of monitoring and diagnostic systems. In Proc. PHM’08.
Kurtoglu, T., Narasimhan, S., Poll, S., Garcia, D., Kuhn, L., de Kleer, J., van Gemund, A., & Feldman, A. (2009). First international diagnosis competition - DXC’09. In Proc. DX’09, pp. 383–396.
Larsson, J. E. (1996). Diagnosis based on explicit means-end models. Artificial Intelligence, 80(1).
Lerner, U., Parr, R., Koleer, D., & Biswas, G. (2000). Bayesian fault detection and diagnosis in dynamic systems. In Proc. AAAI’00, pp. 531–537.
Mengshoel, O. (2007). Designing resourcebounded reasoners using Bayesian networks: System health monitoring and diagnosis. In Proc. DX’07, pp. 330–337.
Metz, C. E. (1978). Basic principles of ROC analysis. Nuclear Medicine, 8(4), 283–298.
Narasimhan, S., & Brownston, L. (2007). HyDE- a general framework for stochastic and hybrid modelbased diagnosis. In Proc. DX’07, pp. 162–169.
Orsagh, R. F., Roemer, M. J., Savage, C. J., & Lebold, M. (2002). Development of performance and effectiveness metrics for gas turbine diagnostic technologies. In Proc. AE-ROCONF’02, Vol. 6, pp. 2825–2834.
Poll, S., Patterson-Hine, A., Camisa, J., Garcia, D., Hall, D., Lee, C., Mengshoel, O., Neukom, C., Nishikawa, D., Ossenfort, J., Sweet, A., Yentus, S., Roychoudhury, I., Daigle, M., Biswas, G., & Koutsoukos, X. (2007). Advanced diagnostics and prognostics testbed. In Proc. DX’07, pp. 178–185.
Reiter, R. (1987). A theory of diagnosis from first principles. Artificial Intelligence, 32(1), 57–95.
Roemer, M., Dzakowic, J., Orsagh, R. F., Byington, C. S., & Vachtsevanos, G. (2005). Validation and verification of prognostic health management technologies. In Proc. AERO- CONF’05, pp. 3941–3947.
Roychoudhury, I., Biswas, G., & Koutsoukos, X. (2009). Designing distributed diagnosers for complex continuous systems. IEEE Trans. on Automation Science and Engineering, 6(2), 277–290.
Russell, S., & Norvig, P. (2003). Artificial Intelligence: A Modern Approach. Pearson Education.
Schuster, E. F., & Sype, W. R. (1987). On the negative hypergeometric distribution. International Journal of Mathematical Education in Science and Technology, 18(3), 453–459.
Siddiqi, S., & Huang, J. (2007). Hierarchical diagnosis of multiple faults. In Proc. IJCAI’07, pp. 581–586.
Simon, L., Bird, J., Davison, C., Volponi, A., & Iverson, R. E. (2008). Benchmarking gas path diagnosticmethods: A public approach. In Proc. ASME Turbo Expo 2008.
Sorsa, T., & Koivo, H. (1998). Application of artificial neural networks in process fault diagnosis. Automatica, 29(4), 843–849.
Williams, Z. (2006). Benefits of IVHM: An analytical approach. In Proc. AEROCONF’06, pp. 1–9.
Zymnis, A., Boyd, S., & Gorinevsky, D. (2009). Relaxed maximum a posteriori fault identification. Signal Processing, 89(6), 989–999.
Technical Papers