Toward Statistical Risk Analysis for Aerospace Systems with AI/ML Components
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Abstract
Aerospace systems increasingly rely on machine learning (ML) and AI components. Such safety-critical applications require careful verification and validation (V&V), and certification. A central task in safety certification is risk assessment: for a large number of off-nominal failure conditions, it must be demonstrated that they do not pose undue risk during operation. Events with major, hazardous, or catastrophic consequences must only occur extremely infrequently.
This paper addresses the statistical modeling and analysis of rare events in AI/ML-based components at both the systemand component-level within the SYSAI analysis framework. Here, we focus on typical perception-related functional tasks that are found in unmanned aerial vehicles (UAVs) or supporting aircraft operations, specifically an Autonomous Centerline Tracking (ACT) system and an Autonomous Emergency Braking System (AEBS). We provide an overview of the SYSAI statistical learning and analysis framework and discuss challenges for the analysis of failure modes and rare events in systems with AI/ML components. We describe how SYSAI can efficiently generate and evaluate scenarios that are likely to contain rare events. These experiments support the estimation of reliable probability distributions for such events and the analysis of temporal dependencies between individual input images.
How to Cite
##plugins.themes.bootstrap3.article.details##
verification and validation, AI/ML systems, Aerospace systems
J., & Man´e, D. (2016). Concrete problems in
ai safety. arXiv preprint arXiv:1606.06565. Retrieved
from https://arxiv.org/abs/1606.06565
Au, S.-K., & Beck, J. L. (2001). Estimation of small failure
probabilities in high dimensions by subset simulation.
In Probabilistic engineering mechanics (Vol. 16,
pp. 263–277). doi: 10.1016/S0266-8920(01)00019-4
Bucklew, J. A. (2004). Introduction to rare event simulation.
Springer. doi: 10.1007/b97424
Cohn, D. A. (1996). Neural network exploration using optimal
experimental design. Advances in Neural Information
Processing Systems, 6(9), 679–686.
Concepts of design assurance for neural networks (CoDANN)
(Tech. Rep.). (2020). European Aviation Safety
Agency.
de Koning, M., Cai, W., Sadigh, B., Oppelstrup, T., Kalos,
M., & Bulatov, V. (2005, 03). Adaptive importance
sampling monte carlo simulation of rare transition
events. The Journal of chemical physics, 122,
074103. doi: 10.1063/1.1844352
Dmitriev, K., Rhein, J., Beller, L., Br¨ocker, J., Huber, E.,
Schumann, J., & Holzapfel, F. (2024). Safety assessment
of a machine learning-based aircraft emergency
braking system: A case study. In 2024 aiaa datc/ieee
43rd digital avionics systems conference (dasc) (p. 1-
10). doi: 10.1109/DASC62030.2024.10749696
Dmitriev, K., Rhein, J., Bender, J., Beller, L., He, Y., Schumann,
J., & Holzapfel, F. (2025). Certifying machine
learning in aviation: An end-to-end dal c case study. In
2025 aiaa datc/ieee 44rd digital avionics systems conference
(dasc).
Do-178c: Software considerations in airborne systems and
equipment certification (No. RTCA DO-178C). (2011).
Washington DC: RTCA, Inc.
EASA concept paper: First usable guidance for level 1&2
machine learning applications (Tech. Rep.). (2023).
European Aviation Safety Agency.
EASA concept paper: First usable guidance for level 1 machine
learning applications (Tech. Rep.). (2021). European Aviation Safety Agency.
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel,
W., Bethge, M., &Wichmann, F. A. (2020). Shortcut
learning in deep neural networks. Nature Machine
Intelligence, 2, 665–673. doi: 10.1038/s42256-020
-00257-z
Goodfellow, I., Shlens, J., & Szegedy, C. (2014, 12). Explaining
and harnessing adversarial examples..
Gramacy, R., & Polson, N. (2011). Particle learning of
Gaussian process models for sequential design and optimization.
Journal of Computational and Graphical
Statistics, 20(1), 467–478.
He, Y. (2012). Variable-length functional output prediction
and boundary detection for an adaptive flight control
simulator (Unpublished doctoral dissertation). University
of California at Santa Cruz.
He, Y. (2015). Online detection and modeling of safety
boundaries for aerospace applications using active
learning and bayesian statistics. In 2015 international
joint conference on neural networks, IJCNN
2015, killarney, ireland, july 12-17, 2015 (pp. 1–
8). IEEE. Retrieved from https://doi.org/
10.1109/IJCNN.2015.7280595 doi: 10.1109/
IJCNN.2015.7280595
He, Y., & Schumann, J. (2020). A framework for the analysis
of deep neural networks in aerospace applications
using bayesian statistics. In Proc. ijcnn, wcci.
He, Y., & Schumann, J. (2024). Statistical analysis and runtime
monitor- ing for an ai-based autonomous centerline
tracking system. IJPHM, 15(3).
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F.,
Dorundo, E., . . . Gilmer, J. (2021). The many faces
of robustness: A critical analysis of out-of-distribution
generalization.. Retrieved from https://arxiv
.org/abs/2006.16241
Janai, J., G¨uney, F., Behl, A., & Geiger, A. (2020). Computer
vision for autonomous vehicles: Problems, datasets
and state of the art. Foundations and Trends in Computer
Graphics and Vision, 12(1–3), 1–308. doi: 10
.1561/0600000079
Jones, D., Schonlau, M., & Welch, W. J. (1998). Efficient
global optimization of expensive black box functions.
Journal of Global Optimization, 13, 455–492.
Juneja, S., & Shahabuddin, P. (2006). Rare-event simulation
techniques: An introduction and recent advances.
In S. Henderson & B. L. Nelson (Eds.), Handbooks in
operations research and management science: Simulation
(Vol. 13, pp. 291–350). Elsevier. doi: 10.1016/
S0927-0507(06)13009-3
Kalra, N., & Paddock, S. (2016). Driving to safety: How
many miles of driving would it take to demonstrate
autonomous vehicle reliability? (Tech. Rep.). RAND
Corporation. doi: 10.7249/RR1478
Leveson, N. G. (2012). Engineering a safer world: Systems
thinking applied to safety. MIT Press.
MacKay, D. J. C. (1992). Information–based objective functions
for active data selection. Neural Computation,
4(4), 589–603.
Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C.
(2020). Hidden stratification causes clinically meaningful
failures in machine learning for medical imaging.
Proceedings of the ACM Conference on Health, Inference,
and Learning (CHIL), 151–159. doi: 10.1145/
3368555.3384468
O’Connor, P. D. T., & Kleyner, A. (2012). Practical reliability
engineering (5th ed.). Wiley.
Ranjan, P., Bingham, D., & Michailidis, G. (2008). Sequential
experiment design for contour estimation from
complex computer codes. Technometrics, 50(4), 527–
541.
Redmon, J., & Farhadi, A. (2018). Yolov3: An incremental
improvement. arXiv preprint arXiv:1804.02767.
Retrieved from https://arxiv.org/abs/1804
.02767
Rubino, G., & Tuffin, B. (2009). Rare event simulation using
monte carlo methods. John Wiley & Sons. doi: 10
.1002/9780470745403
Taddy, M. A., Gramacy, R. B., & Polson, N. G. (2011). Dynamic
trees for learning and design. Journal of the
American Statistical Association, 106(493), 109-123.
Vesely, W. (2002). Fault tree handbook with aerospace applications.
Washington, D. C.: NASA Office of Safety
and Mission Assurance.
Yadron, D., & Tynan, D. (2016). Tesla driver dies
in first fatal crash while using autopilot mode.
The Guardian. Retrieved from https://
www.theguardian.com/technology/2016/
jun/30/tesla-autopilot-death-self
-driving-car-elon-musk
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
The Prognostic and Health Management Society advocates open-access to scientific data and uses a Creative Commons license for publishing and distributing any papers. A Creative Commons license does not relinquish the author’s copyright; rather it allows them to share some of their rights with any member of the public under certain conditions whilst enjoying full legal protection. By submitting an article to the International Conference of the Prognostics and Health Management Society, the authors agree to be bound by the associated terms and conditions including the following:
As the author, you retain the copyright to your Work. By submitting your Work, you are granting anybody the right to copy, distribute and transmit your Work and to adapt your Work with proper attribution under the terms of the Creative Commons Attribution 3.0 United States license. You assign rights to the Prognostics and Health Management Society to publish and disseminate your Work through electronic and print media if it is accepted for publication. A license note citing the Creative Commons Attribution 3.0 United States License as shown below needs to be placed in the footnote on the first page of the article.
First Author et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.