Assessment of Health Monitoring Trustworthiness of Avionics Systems
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Jul 25, 2021
Vladimir Ulansky
Dr. Igor Machalin
Dr. Iryna Terentyeva
https://orcid.org/0000-0003-1684-4980
Dr. Iryna Terentyeva
Abstract
The article provides a methodology for assessing the trustworthiness of health monitoring the dismounted avionics systems with automated test equipment (ATE). The indicators include the probabilities of false-positive, false-negative, true-positive, and true-negative. For the first time, we introduced into consideration the instability of the source of stimulus signal (SSS), the random and systematic component of the measuring channel error, and the reliability characteristics of the systems themselves. We consider a specific case of an exponential distribution of permanent failures and intermittent faults and derive formulas for calculating the trustworthiness indicators. Numerical calculations illustrate how the probabilities of correct and incorrect decisions depend on accuracy parameters. We show that the probabilities of false-positive and false-negative increase much faster than the probabilities of true-positive and true-negative decrease when the standard deviation of stimulus signal increases. For a Very High-Frequency Omni-Directional Range (VOR) receiver, we demonstrate that even with a zero random error generated by the source of the stimulus signal, the probabilities of false-positive and false-negative are different from zero.
##plugins.themes.bootstrap3.article.details##
Keywords
Health monitoring; False-positive; False-negative; VOR receiver
References
Aeronautical Information Manual (2017). Official Guide to Basic Flight Information and ATC Procedures, U.S. Department of Transportation. Federal Aviation Administration, USA. http://www.faa.gov/atpubs
Aeroflex (2005). Avionics IRIS®2000 automatic test equipment system (2005). Plainview, New York, USA. https://www.avionteq.com/document/IRIS-2000-Specification-Sheet.pdf
Bao, H., Ying, C. L., Shi, Q. W., & Teng, J. F. (2006). Research on portable maintenance aid equipment using ARM-based VXI modules. Proceedings of Sixth International Symposium on Instrumentation and Control Technology. October 13-15, Beijing, China. doi: https://doi.org/10.1117/12.718027
Breitgand, D., Goldstein, M., Henis, E., & Shehory, O. (2011). Efficient сontrol of false negative and false positive errors with separate adaptive thresholds. IEEE Trans. on Network and Service Management, 8(2), 128-140. doi: 10.1109/TNSM.2011.020111.00055
Chen, X., Zhang, Z., & Zhang, Z. (2019). Real-time equipment condition assessment for a class-imbalanced dataset based on heterogeneous ensemble learning. Eksploatacja i Niezawodnosc – Maintenance and Reliability, 21(1), 68–80. doi: http://dx.doi.org/10.17531/ein.2019.1.9
Droste, D. B., & Guilbeaux, G. (2009). Advanced architecture for achieving true vertical testability in next generation ATE. Proceedings of IEEE Autotestcon. 14-17 September, Anaheim, California. doi: 10.1109/AUTEST.2009.5314088
Ebrahimi, N. (2008). Simultaneous control of false positives and false negatives in multiple hypotheses testing. Journal of Multivariate Analysis, 99(3), 437-450. doi: https://doi.org/10.1016/j.jmva.2006.12.006
eCASS electronic consolidated automated support system (2020). Lockheed Martin. Bethesda, Maryland, USA. https://www.lockheedmartin.com/en-us/products/ electronic-consolidated-automated-support-system-ecass.html.
Evlanov, L. G. (1979). Control of dynamic systems. Moscow: Nauka.
Foss, A., & Zaiane, O. R. (2008). Estimating true and false positive rates in higher dimensional problems and its data mining applications. Proceedings of IEEE International Conference on Data Mining Workshops. 15-19 December, Pisa, Italy. doi: 10.1109/ICDMW.2008.38
Hand, D., & Christen, P. (2018). A note on using the F-measure for evaluating record linkage algorithms. Statistics and Computing, 28(3), 539-547. doi: https://doi.org/10.1007/s11222-017-9746-6
Ho, C. Y., Lai, Y. C., Chen, I. W, Wang, F. Y. & Tai, W. H. (2012). Statistical analysis of false positives and false negatives from real traffic with intrusion detection/prevention systems. IEEE Communications Magazine, 50(3), 146-154. doi: 10.1109/MCOM.2012.6163595
Hossin, M., & Sulaiman, M. N. (2015). A review on evaluation metrics for data classification evaluations. Int. J. of Data Mining & Knowledge Management Process, 5(2), 1-11. doi: 10.5121/ijdkp.2015.5201
International Electrotechnical Commission (IEC) (2004). Higher performance protocol for the standard digital interface for programmable instrumentation - Part 1: General. In IEC, IEC-60488-1: 2004. Part 1 (p. 13). Genève, Switzerland: International Electrotechnical Commission.
International Electrotechnical Commission (IEC) (2004). Standard Digital Interface for Programmable Instrumentation - Part 2: Codes, formats, protocols and common commands. In IEC, IEC-60488-2: 2004. Part 2. Genève, Switzerland: International Electrotechnical Commission.
Ilarslan, M., & Ungar, L. Y. (2016). Mitigating the impact of false alarms and no fault found events in military systems. IEEE Instrumentation & Measurement Magazine, 8, 16-22. doi: 10.1109/MIM.2016.7524202
Khan, S, Phillips, P., Hockley, C. & Jennions, I. (2014). No fault found events in maintenance engineering part 2: Root causes, technical developments and future research. Reliability Eng. & System Safety, 123, 196-208. doi: https://doi.org/10.1016/j.ress.2013.10.013
Kudritsky, V. D., Sinitsa, M. A., & Chinaev, P. I. (1977). Automation of health-monitoring of electronic equipment. Moscow: Soviet Radio.
Lisman, J. H. C., & van Zuylen, M. C. A. (1972). Note on the generation of most probable frequency distributions. Statistica Neerlandica, 26(1), 19-23. doi: https://doi.org/10.1111/j.1467-9574.1972.tb00152.x
Ma, L., Zhang, X., Wang , K. & Xu, T. (2013). A general method for module automatic testing in avionics systems. Research Journal of Applied Sciences, Engineering and Technology, 5(20), 4796-4804. doi: 10.19026/rjaset.5.4322
Malesich, M. (2007). Advances in DoD’S ATS framework. Proceedings of IEEE Autotestcon. 17-20 September, Baltimore, MD, USA. doi: 10.1109/AUTEST.2007.4374202
Mane, S., Srivastava, J., Hwang, S. Y., & Vayghan, J. (2004). Estimation of false negatives in classification. Proceedings of Fourth IEEE International Conference on Data Mining. 1-4 November, Brighton, UK. doi: 10.1109/ICDM.2004.10048
Manning, C. D., Raghavan, P., & Schutze, H. (2008). Introduction to information retrieval. Cambridge: Cambridge University Press.
Rausand, M., & Hoyland, A. (2003). System reliability theory: models, statistical methods, and applications. New York: Wiley.
Raza, A. (2018). Maintenance model of digital avionics. Aerospace, 5(2), 1-16. doi: https://doi.org/10.3390/aerospace5020038
Raza, A., Ulanskyi, V., Augustynek, K., & Warwas, K. (2017). Generalized cost functions of avionics breakdown maintenance strategy. Proceedings of IEEE Aerospace Conference. 4-11 March, Big Sky, MT. doi: 10.1109/AERO.2017.7943754
Raza, A., & Ulansky, V. (2018). Modelling of false alarms and intermittent faults and their impact on the maintenance cost of digital avionics. Procedia Manufacturing, 16, 107-114. doi: https://doi.org/10.1016/j.promfg.2018.10.171
Raza, A., & Ulansky, V. (2018). Analyses of warranty losses to avionics suppliers. Proceedings of IEEE Aerospace Conference. 3-10 March, Big Sky, MT. doi: 10.1109/AERO.2018.8396752
Ross, W. A. (2003). The impact of next generation test technology on aviation maintenance. Proceedings of IEEE Autotestcon. 23-25 September, Anaheim, CA, USA. doi: 10.1109/AUTEST.2003.1243547
Scott, C. (2007). Performance measures for Neyman–Pearson classification. IEEE Trans. on Information Theory, 53(8), 2852-2863. doi: 10.1109/TIT.2007.901152
Sokolova, M., Japkowicz, N., & Szpakowicz S. (2006). Beyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation. Proceedings of 19th Australian Joint Conference on Artificial Intelligence: Advances in Artificial Intelligence. 4-8 December, Hobart, TAS, Australia. doi: https://doi.org/10.1007/11941439_114
Stora, M. J., & Droste D. (2003). ATE open system platform. IEEE-Pl552 structured architecture for test systems (SATS). Proceedings of IEEE Autotestcon. 22-25 September, Anaheim, CA, USA. doi: 10.1109/AUTEST.2003.1243559
Spherea (2017). The prepared maintenance solution for airlines and MROS. Hamburg, Germany. https:// www.spherea.de/en/atec-series/
Ulansky, V. (1992). The trustworthiness of multiple-monitoring the operability of non-repairable electronic systems. In Ignatov V. (Ed.), Saving Technologies and Avionics Maintenance of Civil Aviation Aircraft (14-25). Kyiv: KIIGA Press.
Aeroflex (2005). Avionics IRIS®2000 automatic test equipment system (2005). Plainview, New York, USA. https://www.avionteq.com/document/IRIS-2000-Specification-Sheet.pdf
Bao, H., Ying, C. L., Shi, Q. W., & Teng, J. F. (2006). Research on portable maintenance aid equipment using ARM-based VXI modules. Proceedings of Sixth International Symposium on Instrumentation and Control Technology. October 13-15, Beijing, China. doi: https://doi.org/10.1117/12.718027
Breitgand, D., Goldstein, M., Henis, E., & Shehory, O. (2011). Efficient сontrol of false negative and false positive errors with separate adaptive thresholds. IEEE Trans. on Network and Service Management, 8(2), 128-140. doi: 10.1109/TNSM.2011.020111.00055
Chen, X., Zhang, Z., & Zhang, Z. (2019). Real-time equipment condition assessment for a class-imbalanced dataset based on heterogeneous ensemble learning. Eksploatacja i Niezawodnosc – Maintenance and Reliability, 21(1), 68–80. doi: http://dx.doi.org/10.17531/ein.2019.1.9
Droste, D. B., & Guilbeaux, G. (2009). Advanced architecture for achieving true vertical testability in next generation ATE. Proceedings of IEEE Autotestcon. 14-17 September, Anaheim, California. doi: 10.1109/AUTEST.2009.5314088
Ebrahimi, N. (2008). Simultaneous control of false positives and false negatives in multiple hypotheses testing. Journal of Multivariate Analysis, 99(3), 437-450. doi: https://doi.org/10.1016/j.jmva.2006.12.006
eCASS electronic consolidated automated support system (2020). Lockheed Martin. Bethesda, Maryland, USA. https://www.lockheedmartin.com/en-us/products/ electronic-consolidated-automated-support-system-ecass.html.
Evlanov, L. G. (1979). Control of dynamic systems. Moscow: Nauka.
Foss, A., & Zaiane, O. R. (2008). Estimating true and false positive rates in higher dimensional problems and its data mining applications. Proceedings of IEEE International Conference on Data Mining Workshops. 15-19 December, Pisa, Italy. doi: 10.1109/ICDMW.2008.38
Hand, D., & Christen, P. (2018). A note on using the F-measure for evaluating record linkage algorithms. Statistics and Computing, 28(3), 539-547. doi: https://doi.org/10.1007/s11222-017-9746-6
Ho, C. Y., Lai, Y. C., Chen, I. W, Wang, F. Y. & Tai, W. H. (2012). Statistical analysis of false positives and false negatives from real traffic with intrusion detection/prevention systems. IEEE Communications Magazine, 50(3), 146-154. doi: 10.1109/MCOM.2012.6163595
Hossin, M., & Sulaiman, M. N. (2015). A review on evaluation metrics for data classification evaluations. Int. J. of Data Mining & Knowledge Management Process, 5(2), 1-11. doi: 10.5121/ijdkp.2015.5201
International Electrotechnical Commission (IEC) (2004). Higher performance protocol for the standard digital interface for programmable instrumentation - Part 1: General. In IEC, IEC-60488-1: 2004. Part 1 (p. 13). Genève, Switzerland: International Electrotechnical Commission.
International Electrotechnical Commission (IEC) (2004). Standard Digital Interface for Programmable Instrumentation - Part 2: Codes, formats, protocols and common commands. In IEC, IEC-60488-2: 2004. Part 2. Genève, Switzerland: International Electrotechnical Commission.
Ilarslan, M., & Ungar, L. Y. (2016). Mitigating the impact of false alarms and no fault found events in military systems. IEEE Instrumentation & Measurement Magazine, 8, 16-22. doi: 10.1109/MIM.2016.7524202
Khan, S, Phillips, P., Hockley, C. & Jennions, I. (2014). No fault found events in maintenance engineering part 2: Root causes, technical developments and future research. Reliability Eng. & System Safety, 123, 196-208. doi: https://doi.org/10.1016/j.ress.2013.10.013
Kudritsky, V. D., Sinitsa, M. A., & Chinaev, P. I. (1977). Automation of health-monitoring of electronic equipment. Moscow: Soviet Radio.
Lisman, J. H. C., & van Zuylen, M. C. A. (1972). Note on the generation of most probable frequency distributions. Statistica Neerlandica, 26(1), 19-23. doi: https://doi.org/10.1111/j.1467-9574.1972.tb00152.x
Ma, L., Zhang, X., Wang , K. & Xu, T. (2013). A general method for module automatic testing in avionics systems. Research Journal of Applied Sciences, Engineering and Technology, 5(20), 4796-4804. doi: 10.19026/rjaset.5.4322
Malesich, M. (2007). Advances in DoD’S ATS framework. Proceedings of IEEE Autotestcon. 17-20 September, Baltimore, MD, USA. doi: 10.1109/AUTEST.2007.4374202
Mane, S., Srivastava, J., Hwang, S. Y., & Vayghan, J. (2004). Estimation of false negatives in classification. Proceedings of Fourth IEEE International Conference on Data Mining. 1-4 November, Brighton, UK. doi: 10.1109/ICDM.2004.10048
Manning, C. D., Raghavan, P., & Schutze, H. (2008). Introduction to information retrieval. Cambridge: Cambridge University Press.
Rausand, M., & Hoyland, A. (2003). System reliability theory: models, statistical methods, and applications. New York: Wiley.
Raza, A. (2018). Maintenance model of digital avionics. Aerospace, 5(2), 1-16. doi: https://doi.org/10.3390/aerospace5020038
Raza, A., Ulanskyi, V., Augustynek, K., & Warwas, K. (2017). Generalized cost functions of avionics breakdown maintenance strategy. Proceedings of IEEE Aerospace Conference. 4-11 March, Big Sky, MT. doi: 10.1109/AERO.2017.7943754
Raza, A., & Ulansky, V. (2018). Modelling of false alarms and intermittent faults and their impact on the maintenance cost of digital avionics. Procedia Manufacturing, 16, 107-114. doi: https://doi.org/10.1016/j.promfg.2018.10.171
Raza, A., & Ulansky, V. (2018). Analyses of warranty losses to avionics suppliers. Proceedings of IEEE Aerospace Conference. 3-10 March, Big Sky, MT. doi: 10.1109/AERO.2018.8396752
Ross, W. A. (2003). The impact of next generation test technology on aviation maintenance. Proceedings of IEEE Autotestcon. 23-25 September, Anaheim, CA, USA. doi: 10.1109/AUTEST.2003.1243547
Scott, C. (2007). Performance measures for Neyman–Pearson classification. IEEE Trans. on Information Theory, 53(8), 2852-2863. doi: 10.1109/TIT.2007.901152
Sokolova, M., Japkowicz, N., & Szpakowicz S. (2006). Beyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation. Proceedings of 19th Australian Joint Conference on Artificial Intelligence: Advances in Artificial Intelligence. 4-8 December, Hobart, TAS, Australia. doi: https://doi.org/10.1007/11941439_114
Stora, M. J., & Droste D. (2003). ATE open system platform. IEEE-Pl552 structured architecture for test systems (SATS). Proceedings of IEEE Autotestcon. 22-25 September, Anaheim, CA, USA. doi: 10.1109/AUTEST.2003.1243559
Spherea (2017). The prepared maintenance solution for airlines and MROS. Hamburg, Germany. https:// www.spherea.de/en/atec-series/
Ulansky, V. (1992). The trustworthiness of multiple-monitoring the operability of non-repairable electronic systems. In Ignatov V. (Ed.), Saving Technologies and Avionics Maintenance of Civil Aviation Aircraft (14-25). Kyiv: KIIGA Press.
Section
Technical Papers