Statistical Analysis and Runtime Monitoring for an AI-based Autonomous Centerline Tracking System
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Oct 14, 2024
Yuning He
Johann Schumann
Abstract
Autonomous Centerline Tracking (ACT) enables an unmanned aircraft to be guided down the center of the runway, using a camera-based Deep Neural Network (DNN). ACT is safety-critical. The EASA Guidelines for machine-learning based systems list numerous assurance objectives that must be met toward certification and V&V. We extend our statistical analysis framework SYSAI to support meeting assurance objectives for a complex safety-critical system with AI components, and describe a combination with a runtime monitoring architecture that also supports advanced risk mitigation to support safety assurance of a complex AI-based aerospace system.
##plugins.themes.bootstrap3.article.details##
Keywords
Complex safety-critical system, Safe AI, Statistical v&v framework
References
ASTM. (2017). ASTM F3269 - 17 standard practice for methods to safely bound flight behavior of unmanned aircraft systems containing complex functions.
Cohn, D. A. (1996). Neural network exploration using optimal experimental design. Advances in Neural Information Processing Systems, 6(9), 679–686.
EASA, & Daedalean. (2021). Concepts of design assurance for neural networks II (Tech. Rep.).
EASA Concept Paper: First usable guidance for Level 1 machine learning applications (Tech. Rep.). (2021). European Aviation Safety Agency.
Gramacy, R., & Polson, N. (2011). Particle learning of Gaussian process models for sequential design and optimization. Journal of Computational and Graphical Statistics, 20(1), 467–478.
He, Y. (2012). Variable-length functional output prediction and boundary detection for an adaptive flight control simulator (Unpublished doctoral dissertation). University of California at Santa Cruz.
He, Y. (2015). Online Detection and Modeling of safety Boundaries for Aerospace Applications using active learning and Bayesian statistics. In 2015 International Joint Conference on Neural Networks, IJCNN, 2015 (pp. 1–8). IEEE. Retrieved from http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=7256526 doi: 10.1109/IJCNN.2015.7280595
He, Y., & Schumann, J. (2020). A framework for the analysis of Deep Neural Networks in aerospace applications using Bayesian statistics.
He, Y., & Schumann, J. (2023). Statistical analysis and runtime monitoring for an AI-based autonomous Centerline Tracking System. In Proc. phmap.
He, Y., & Schumann, J. (2024). SYSAI for System Health Management - a statistical framework for the analysis of diagnosis systems. In Proc. phm 2024.
He, Y., Schumann, J., & Yu, H. (2022). Toward runtime assurance of complex systems with AI components. In Proc. phme 2022.
Jones, D., Schonlau, M., & Welch, W. J. (1998). Efficient global optimization of expensive black box functions. Journal of Global Optimization, 13, 455–492.
MacKay, D. J. C. (1992). Information–based objective functions for active data selection. Neural Computation, 4(4), 589–603.
Nagarajan, P., Kannan, S. K., Torens, C., Vukas, M. E., & Wilber, G. F. (2021). ASTM F3269 - an industry standard on run time assurance for aircraft systems. In Aiaa scitech 2021 forum. Retrieved from https://arc.aiaa.org/doi/abs/10.2514/6.2021-0525 doi: 10.2514/6.2021-0525
Pike, L., Goodloe, A., Morisset, R., & Niller, S. (2010, November). Copilot: A hard real-time runtime monitor. In Proceedings of the 1st Intl. Conference on Runtime Verification. Springer. (Preprint available at https://leepike.github.io/pub pages/rv2010.html)
Ranjan, P., Bingham, D., & Michailidis, G. (2008). Sequential Experiment Design for Contour Estimation from complex Computer Codes. Technometrics, 50(4), 527–541.
Reinbacher, T., Rozier, K. Y., & Schumann, J. (2014). Temporal-Logic Based Runtime Observer Pairs for System Health Management of Real-Time Systems. In Tools and Algorithms for the Construction and Analysis of Systems - 20th International Conference, TACAS (Vol. 8413, pp. 357–372). Springer.
Rozier, K. Y., & Schumann, J. (2017). R2U2: tool overview. In Proceedings rv-cubes 2017 (pp. 138–156). Society of Automotive Engineers (SAE). (2021). Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles (Tech. Rep. No. J3016 202104). SAE.
Taddy, M. A., Gramacy, R. B., & Polson, N. G. (2011). Dynamic Trees for Learning and Design. Journal of the American Statistical Association, 106(493), 109-123.
Yu, T., & Zhu, H. (2020). Hyper-parameter optimization: A review of algorithms and applications. Retrieved from https://arxiv.org/abs/2003.0568910
Cohn, D. A. (1996). Neural network exploration using optimal experimental design. Advances in Neural Information Processing Systems, 6(9), 679–686.
EASA, & Daedalean. (2021). Concepts of design assurance for neural networks II (Tech. Rep.).
EASA Concept Paper: First usable guidance for Level 1 machine learning applications (Tech. Rep.). (2021). European Aviation Safety Agency.
Gramacy, R., & Polson, N. (2011). Particle learning of Gaussian process models for sequential design and optimization. Journal of Computational and Graphical Statistics, 20(1), 467–478.
He, Y. (2012). Variable-length functional output prediction and boundary detection for an adaptive flight control simulator (Unpublished doctoral dissertation). University of California at Santa Cruz.
He, Y. (2015). Online Detection and Modeling of safety Boundaries for Aerospace Applications using active learning and Bayesian statistics. In 2015 International Joint Conference on Neural Networks, IJCNN, 2015 (pp. 1–8). IEEE. Retrieved from http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=7256526 doi: 10.1109/IJCNN.2015.7280595
He, Y., & Schumann, J. (2020). A framework for the analysis of Deep Neural Networks in aerospace applications using Bayesian statistics.
He, Y., & Schumann, J. (2023). Statistical analysis and runtime monitoring for an AI-based autonomous Centerline Tracking System. In Proc. phmap.
He, Y., & Schumann, J. (2024). SYSAI for System Health Management - a statistical framework for the analysis of diagnosis systems. In Proc. phm 2024.
He, Y., Schumann, J., & Yu, H. (2022). Toward runtime assurance of complex systems with AI components. In Proc. phme 2022.
Jones, D., Schonlau, M., & Welch, W. J. (1998). Efficient global optimization of expensive black box functions. Journal of Global Optimization, 13, 455–492.
MacKay, D. J. C. (1992). Information–based objective functions for active data selection. Neural Computation, 4(4), 589–603.
Nagarajan, P., Kannan, S. K., Torens, C., Vukas, M. E., & Wilber, G. F. (2021). ASTM F3269 - an industry standard on run time assurance for aircraft systems. In Aiaa scitech 2021 forum. Retrieved from https://arc.aiaa.org/doi/abs/10.2514/6.2021-0525 doi: 10.2514/6.2021-0525
Pike, L., Goodloe, A., Morisset, R., & Niller, S. (2010, November). Copilot: A hard real-time runtime monitor. In Proceedings of the 1st Intl. Conference on Runtime Verification. Springer. (Preprint available at https://leepike.github.io/pub pages/rv2010.html)
Ranjan, P., Bingham, D., & Michailidis, G. (2008). Sequential Experiment Design for Contour Estimation from complex Computer Codes. Technometrics, 50(4), 527–541.
Reinbacher, T., Rozier, K. Y., & Schumann, J. (2014). Temporal-Logic Based Runtime Observer Pairs for System Health Management of Real-Time Systems. In Tools and Algorithms for the Construction and Analysis of Systems - 20th International Conference, TACAS (Vol. 8413, pp. 357–372). Springer.
Rozier, K. Y., & Schumann, J. (2017). R2U2: tool overview. In Proceedings rv-cubes 2017 (pp. 138–156). Society of Automotive Engineers (SAE). (2021). Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles (Tech. Rep. No. J3016 202104). SAE.
Taddy, M. A., Gramacy, R. B., & Polson, N. G. (2011). Dynamic Trees for Learning and Design. Journal of the American Statistical Association, 106(493), 109-123.
Yu, T., & Zhu, H. (2020). Hyper-parameter optimization: A review of algorithms and applications. Retrieved from https://arxiv.org/abs/2003.0568910
Section
Technical Papers