Assessment of Health Monitoring Trustworthiness of Avionics Systems

The article provides a methodology for assessing the trustworthiness of health monitoring the dismounted avionics systems with automated test equipment (ATE). The indicators include the probabilities of false-positive, false-negative, true-positive, and true-negative. For the first time, we introduced into consideration the instability of the source of stimulus signal (SSS), the random and systematic component of the measuring channel error, and the reliability characteristics of the systems themselves. We consider a specific case of an exponential distribution of permanent failures and intermittent faults and derive formulas for calculating the trustworthiness indicators. Numerical calculations illustrate how the probabilities of correct and incorrect decisions depend on accuracy parameters. We show that the probabilities of false-positive and false-negative increase much faster than the probabilities of true-positive and true-negative decrease when the standard deviation of stimulus signal increases. For a Very High-Frequency Omni-Directional Range (VOR) receiver, we demonstrate that even with a zero random error generated by the source of the stimulus signal, the probabilities of false-positive and false-negative are different from zero.


INTRODUCTION
Currently, many airlines and air-force bases worldwide use ATE for monitoring and diagnostics of avionics systems. Examples of such systems are ATEC Series 7 manufactured by Spherea (2017), eCASS -electronic Consolidated Automated Support System made by Lockheed Martin (2020), ATE IRIS 2000/IRS 1200 system manufactured by Aeroflex (2005), and many others. The main goal of using ATE is to reduce maintenance costs over the lifetime of avionics, which depends on solving the following major tasks: increasing ATE versatility (an increase in the number of types of tested avionics systems) and improves the validity of operability checking and troubleshooting. For instance, the eCASS is compatible with more than 550 sets of test programs that test avionics systems on multiple platforms. Modern ATE are built on a modular principle using interface PXI bus and standard digital interface for programmable instrumentation IEC- 60488-1 (2004) and IEC-60488-2 (2004). There are two primary directions of ATE development: generality and openness (Ma et al., 2013); the main aspects of these directions investigated by (Droste & Guilbeaux, 2009), (Evlanov, 1979), and (Stora & Droste, 2003).
At the stages of the design and operation of ATE for avionics systems, the task of assessing the trustworthiness indicators of health monitoring is highly topical. Indeed, inspection errors such as false-positive and false-negative can lead to economic losses and flight safety reduction.
Let us look at how false-positive and false-negative events affect the cost of avionics maintenance and flight safety. Modern digital avionics systems present modular units with high requirements for testability and maintainability (eCASS, 2020). An avionics system usually consists of one or several line-replaceable units (LRU) or line-replaceable modules (LRM). Each LRU or LRM comprises a set of shop replaceable units (SRU) representing a printed circuit board assemblies (PCB). Based on the three-component level of avionics systems (LRU, SRU, nonrepairable element), the following three levels of maintenance are composed: organizational maintenance (O-level), intermediate maintenance (I-level), and depot maintenance (D-level). The O-level maintenance targets the isolation of defected LRU at the aircraft parking. I-level maintenance performs isolation of the LRU failure with depth to SRU. At I-level, specialized ATE automates most of the test procedures. D-level maintenance performs isolation of failure with depth to nonrepairable elements. The specialized repair centers or original equipment manufacturers typically conduct D-level maintenance. As has been shown (Bao et al., 2006;Raza et al., 2017;Ross, 2003), the maintenance system may comprise all three levels or consist of the two if the only O-and I-levels or O-and D-levels are applicable.
As is well known (Raza, 2018;Raza & Ulansky, 2020), any two or three-level maintenance options must check the operability of the dismantled LRU or LRM. We illustrate this statement by the diagram presented in Figure 1 for two-level maintenance, including O-and I-level. As we can see in Figure 1, the dismounted LRU can be judged as operable or inoperable by the results of testing with the help of ATE. Since the flow of dismounted LRU includes both inoperable and operable units, one of four incompatible events may occur according to the checking results: true-positive, falsepositive, true-negative, or false-negative. When a truepositive or false-negative event occurs, the unit is shipped to the warehouse of spare LRU. On the other hand, when a truenegative or false-positive event occurs, the unit is delivered to the manufacturer or outsourcing company for repair. Thus, from the diagram in Figure 1, the occurrence of a falsenegative (undetected failure) when checking the LRU will ultimately lead to installing a faulty unit onboard the aircraft that may have an impact on flight safety. An example of such an onboard safety-related avionics system is the Instrument Landing System (ILS), which usually comprises three and sometimes even more identical receivers with a k-out-of-n reliability structure (Rausand & Hoyland, 2003). However, the appearance of a false-positive event will lead to shipping the operable LRU to the manufacturer or outsourcing company for repair, which we associate with economic losses.
Let us consider the most significant publications in this area. (Breitgand et al., 2011) developed a specific algorithm for controlling the rate of false-positive and false-negative. (Ho et al., 2012) proposed a false-positive and false-negative assessment mechanism that collects corresponding errors from real-world traffic and statistically analyzes these cases. (Mane et al., 2004) considered a capture-recapture-based method to estimate false-negatives when using two or more independent classifiers. (Foss & Zaiane, 2008) proposed an algorithm for computation true-positive and false-positive rates using a statistical error rate algorithm. (Ebrahimi, 2008) considered the problem of determining thresholds controlling both false-positives and false-negatives by using a specific risk function. (Scott, 2007) proposed performance measures to evaluate and compare classifiers concerning minimizing the probability of false-negative whereas restricting the probability of a false positive. (Evlanov, 1979;Kudritsky et al., 1977;Ulansky, 1992) considered analytical methods for calculating the probabilities of false-positive and falsenegative. The measurement result includes the actual value of the monitoring parameter and additive random noise in these studies.
Note that all the cited studies do not consider the specifics of health monitoring the avionics systems. We should also note that the metric F1 score, widely used in binary classification and statistical analysis (Chen, 2019;Hossin & Sulaiman, 2015;Manning et al., 2008;Sokolova et al., 2006), is impractical to use for assessing the trustworthiness of monitoring avionics systems for two reasons. Firstly, it gives equal importance to precision and recall, but in practice, different types of classification errors lead to various losses (Hand & Christen, 2018). Secondly, there is only a statistical formula for calculating the F1 score. Hence, to assess the increase or decrease in trustworthiness when changing the testing procedure is possible only through numerous tests, which leads to high costs. Therefore, this study provides a methodology for calculating the trustworthiness indicators of the health monitoring of avionics systems on the example of navigation and landing systems; we consider the instability of the SSS, the accuracy characteristics, and the reliability of the systems themselves. We examine the primary sources of measurement errors in detail when testing VOR receivers and formulate the events that lead to the correct and incorrect decisions. We derive the generalized expressions of trustworthiness indicators and specific formulas for the case of an exponential distribution of permanent failures and intermittent faults. Next, we illustrate the dependence of the probabilities of correct and incorrect decisions when testing VOR receivers versus different accuracy parameters. Finally, we consider an example of assessing the trustworthiness indicators when testing ILS with ATE.

TRUSTWORTHINESS ASSESSMENT METHODOLOGY
The following section outlines the general methodology of assessing the health monitoring trustworthiness of avionics systems on the example of a navigation system.

Block Diagram of VOR Receiver Health Monitoring
Let us determine the trustworthiness indicators of the health monitoring of the onboard VOR radio receiver, which represents a typical avionics LRU. Onboard equipment VOR provides aircraft navigation on land lighthouses VOR. This equipment allows to solve the following navigation tasks: • Determine the magnetic bearing of the VOR ground-based radio beacon; • Determine the location of the aircraft using the magnetic bearings of two VOR radio beacons; • Determine the drift angle in flight.
According to Aeronautical Information Manual (2017), an error of ± 1° usually characterizes the accuracy of determining VOR radio beacons' bearings using onboard equipment. When checking the VOR receiver's operability with the help of ATE, the tested parameter is the error of azimuth measurement. Figure 2 shows a block diagram of monitoring the health of an airborne VOR receiver. A specific stimulus signal of magnitude A must be applied to the VOR receiver to monitor azimuth measurement error. Therefore, the controller applies a control signal to the SSS, which, acting on the corresponding controls, sets the required shape and magnitude of the stimulus signal at its output. Since the SSS has finite stability, instead of the required value of the stimulus signal A, it applies a signal of magnitude A + Γ to the input of the VOR receiver, where Γ is the random component of the SSS error. Further, we assume that the systematic component of the SSS error was eliminated as a result of planned calibrations; therefore, we do not consider it. In turn, the VOR receiver introduces the error Θ + Ξ into the signal A applied to its input, where Θ and Ξ are, respectively, the systematic and random components of the error of azimuth measurement. Then, by the signal from the controller, the memory feeds the value of signal A to the subtractor. Thus, we can represent the difference signal at the output of the subtractor as follows: Signal Y feeds into the comparator that uses the following decision rule. If |y| ≤ |Δ|, then it makes the decision "the error of the azimuth measurement in the tolerance" (comparator output 1 in Figure 2), where Δ is the limit of the permissible error of azimuth measurement by the VOR receiver and y is the realization of the random variable Y. If |y| ˃ |Δ|, then it makes the decision "the error of the azimuth measurement is out of the tolerance" (comparator output 2 in Figure 2). Since both the subtractor and comparator are microprocessor devices, we can neglect the errors of the subtraction and comparison operations.
The random errors in Equation (1) are not correlated with each other. Therefore, between the random variables Γ, Θ, and Ξ, there is an additive relationship. Indeed, the random error Г is not dependent on Θ and Ξ because it is generated by the external source of stimulus signal. Further, for a specific VOR receiver, the systematic measurement error Θ is not a random variable, but a constant value, which depends on the accuracy of the initial setup of the measuring path of the VOR receiver at the manufacturer. The measuring channel of the VOR receiver receives, filters, amplifies and demodulates high-frequency signals from the antenna, as well as converts analog low-frequency signals into a digital code, which is transmitted to the onboard computer. In the process of converting a high-frequency signal into a digital code, a significant number of different operations (analog and digital) are performed, which cause the appearance of a random component of the azimuth measurement error Ξ. The random component Ξ fluctuates around some constant value, which is the systematic component of the azimuth measurement error. Moreover, for different VOR receivers with varying values of the systematic component, the standard deviations of these fluctuations are practically the same. Due to these reasons, it is assumed that there is an additive relationship between the random variables Θ and Ξ.

Formalization of Incompatible Events Based on Decision-making when Monitoring the Health of VOR Receiver
Due to the presence of a random component Z = Ξ + Γ in the difference signal Y, the decisions made by the comparator may turn out to be erroneous. In this case, incorrect decisions such as false-positive (SFP) and false-negative (SFN) take place accordingly when the following complex events occur: where ∩ is the symbol of the intersection of different events.
Event (2) means that the systematic error of azimuth measurement is within the tolerance, but its measured value is out of tolerance. Event (3) implies that the systematic error of azimuth measurement is out of tolerance, but its measured value is within the tolerance.
On the other hand, the comparator can make the correct decisions, such as true-positive (STP) and true-negative (STN).
Event (4) means that the systematic error of azimuth measurement and its measured value is within the tolerance. Event (5) implies that the systematic error of azimuth measurement and its measured value is out of tolerance.
As we can see from relations (2)-(5), events SFP, SFN, STP, and STN represent a group of mutually exclusive events; the sum of their probabilities is unity. Figure 3 shows the graph of decision-making when checking the VOR receiver, where P is the a priori probability of the VOR receiver operability and q(θ) is the probability density function (PDF) of the systematic component of the azimuth measurement error over the set of the same VOR receivers. According to the graph of decision-making in Figure 3, we can write the following obvious equations:

Determination of Probabilities of Correct and Incorrect Decisions
We show the expressions for the prior probabilities P and 1 -P on the edges of the decision-making graph in Figure 3.
Since the random variable Z is the sum of two independent continuous random variables Ξ and Γ, its PDF g(z) can be represented as a composition of the PDF of these random variables, i.e., where f(ξ) and φ(γ) are PDF of random variables Ξ and Γ, respectively.
Using the theorems of addition and multiplication of probabilities and considering Equation (8), we determine the probabilities of events (2)-(5) as follows: -the probability of false-positive -the probability of true-positive

Case of Sudden Failures
ATE checks the health of VOR units rejected during operation by built-in test equipment (BITE) and dismounted from the boards of the aircraft fleet. In the flow of dismounted units, we can observe VOR receivers with permanent failures and receivers removed from the aircraft boards due to intermittent faults registered by BITE. Indeed, according to (Ilarslan & Ungar, 2016;Khan et al., 2014), the rate of intermittent faults for avionics is from 20 % to 50 % of the total percentage of removals. Due to the presence in the flow of dismounted units, both operable and failed (i.e., with permanent failures) VOR receivers, all the formulated events (2)-(5) are possible.
At the initial stage of equipment operation, the laws of degradation of monitoring parameters are generally unknown. Therefore, the calculation of reliability is carried out for the exponential distribution law of permanent failures. So, further, we will assume that only sudden failures occur in VOR receivers.
The systematic component of the azimuth measurement error depends on the accuracy of the initial setup of the measuring path of the VOR receiver and the presence of defects in it. By defects, we mean breaks, short circuits in the electrical circuitry of the dismounted VOR receiver, and other quantitative changes in the properties of the components, leading to a sudden exit of the systematic part of the azimuth measurement error beyond tolerance.
Thus, we can present the systematic error of azimuth measurement as follows: 0 1 Θ , if an intermittent fault occurs in the VOR receiver, resulting in dismounting it from the aircraft board Θ , if a permanent defect occurs in the VOR receiver, resulting in a reduc = 2 tion in systematic error below the permissible limit Θ , if a permanent defect occurs in the VOR receiver, leading to an increase in systematic error over the permissible limit The following apparent conditions are met: |Θ0| ≤ |Δ|, |Θ1| ˃ |Δ|, and |Θ2| ˃ |Δ|. These conditions mean that in the absence of permanent failure in the VOR receiver, all possible values of the systematic error lie within tolerance. In the presence of permanent failure, the value of the systematic component is out of tolerance. Figure 4 shows the graph of transitions of the VOR receiver to states leading to dismounting from the aircraft. In Figure  4, λ0 is the rate of intermittent faults transferring the VOR receiver to the state D0, in which it is dismounted from the aircraft board not having any permanent failure, λ1 and λ2 are the rates of permanent failures that transfer the VOR receiver from the operable state to inoperable states D1 and D2, corresponding to a decrease and increase in systematic error by an amount higher than permissible. Thus, the VOR receiver that does not have a permanent failure can be dismounted from an aircraft due to one or more intermittent faults recorded by the BITE during flight (state D0). According to Figure 1, ATE will test the dismounted unit at the I-level maintenance and with probability P(STP) does not confirm the existence of a permanent failure. Moreover, conventional ATE will not also detect the root of intermittent faults (Anderson, 2014).
Inoperable states D1 and D2 correspond to situations when a permanent failure occurs during flight and BITE detects it.
As we can see in Figure 4, at the exponential failure distribution, the total transition rate is 0 1 2 λ λ λ = + + Based on the above reasoning, we can present the PDF of the systematic error in measuring the azimuth of the VOR receivers coming to the health monitoring as follows: where q0(θ), q1(θ), and q2(θ) are, respectively, the PDF of the systematic component of the azimuth measurement error in the absence and presence of permanent failures.
Considering Equation (14) and the fact that random variables Θ0, Θ1, and Θ2 have non-overlapping intervals of existence, we transform the probabilities (9)-(12) to the following form: -the probability of false-positive We can assume that the random variable Θ0 has a normal PDF since the distribution of Θ0 depends on the accuracy of tuning the channel for measuring the azimuth of the VOR receiver at the manufacturer. When setting up the measuring channels of electronic devices, the rule of three sigmas usually has a place. Since |Θ0| ≤ |Δ|, it is evident that the random variable Θ0 has a truncated normal distribution with the mean square deviation of σθ = |Δ|/3 and mathematical expectation E(Θ0) = 0.
In the design and early stages of VOR receives' operation, engineers do not know the mathematical expectation and the standard deviation of the systematic component of the error in measuring the azimuth by the failed VOR receivers. However, it is usually possible to determine the boundary values of this error from the operational algorithm of the VOR receiver. A uniform distribution has the maximum entropy with the known boundaries of the systematic measurement error (Lisman & van Zuylen, 1972). Therefore, we assume that the random variables Θ1 and Θ2 have a uniform distribution. (14) the values of the PDF of random variables Θ0, Θ1, and Θ2, we obtain

Substituting into Equation
where Δl and Δh are, respectively, the lower and upper boundaries of the systematic component of the error in azimuth measurement in the presence of defects in the VOR receiver, and c is the normalization constant.

RESULTS AND DISCUSSION
Example 1. Let us consider an example of calculating the trustworthiness indicators when monitoring the health of dismounted VOR receivers. The tested VOR equipment operates in the frequency range 108.0-117.975 MHz. There are 160 fixed frequencies (channels) for work with VOR beacons. Azimuth measurement error is no more than 0.5°. We use the following initial data: |Δ| = 0.00873 rad, Δh = -Δl = 0.04365 rad, σξ = σθ = 0.00291 rad, and σγ = 0.00175 rad. Table 1 illustrates the dependence of trustworthiness indicators on the fraction of VOR receivers with intermittent faults arriving for health monitoring using ATE when λ1/Λ = λ2/Λ = (1 -λ0/Λ)/2. The fraction of VOR receivers dismounted from the aircraft due to intermittent faults (λ0/Λ) corresponds to the operable units that do not have permanent failures.
As we can see in Table 1, the probability of false-positive increases, and the probability of false-negative decreases with an increase in the fraction of operable VOR receivers. At the same time, the probability of true-positive increases, and true-negative decreases.
To explain this behavior of trustworthiness indicators, we should consider their dependence on the ratio λ0/Λ determined by the following equations: As we can see in Equations (25) and (26), with an increase in the ratio λ0/Λ, the sum of the probabilities of false-positive and true-positive increases and the sum of the probabilities of false-negative and true-negative decreases. Since we did not change the accuracy characteristics of the measuring channel when making calculations, the dependencies in Table 1 are linear.
Let us investigate the dependence of the trustworthiness indicators of health monitoring on the accuracy characteristics of the measuring channel when λ0/Λ = 0.4 and λ1/Λ = λ2/Λ = 0.3.  Analyzing Figures 7 and 8, we can make the following conclusions. The probability of false-positive is very much dependent on the standard deviation of the random error generated by the SSS. Indeed, when σγ changes from 0 to 0.01 rad, the probability P(SFP) increases from 2 % to 17 %.
The probability of true-positive decreases when the standard deviation of the random error generated by the SSS increases. When σγ changes from 0 to 0.01 rad, the probability P(STP) decreases from 38 % to 23 %. From Figures 9 and 10, we can conclude that the probability of false-negative increases and the probability of truenegative decreases with a rise in the standard deviation σγ from 0 to 0.01 rad. However, the impact of the standard deviation of the random error generated by the SSS on probabilities P(SFN) and P(STN) is less than on probabilities P(SFP) and P(STP). Indeed, when σγ changes from 0 to 0.01 rad, the probability P(SFN) increases from 2 % to 7.5 %, and the probability P(STN) decreases from 57.8 % to 53 %.   Table 2 shows the calculated values of the trustworthiness indicators. As we can see in Table 2, the trustworthiness of health monitoring ILS using ATE is relatively high. The probability of a false-positive and a false-negative does not exceed 1.5 % and 0.67 %, respectively.
The probability of correct testing of the ILS receiver by two parameters, we calculate by the formula of Kudritsky et al. (1977): where P(SFP)i and P(SFN)i are, respectively, the probabilities of false-positive and false-negative when testing i-th parameter.

CONCLUSION
This paper has developed a new mathematical model for assessing the trustworthiness indicators of health monitoring the dismounted avionics systems, which include the probabilities of false-positive, false-negative, true-positive, and true-negative. Using the designed block diagram of the VOR receiver health monitoring, we have formulated the corresponding decision rule. Based on the decision rule, we have derived general equations for computing the probabilities of correct and incorrect decisions when monitoring the health of dismounted avionics systems by ATE. The proposed equations are applicable at arbitrary distributions of monitoring parameters and measurement errors.
We specifically considered the case of an exponential distribution of permanent failures and intermittent faults; then, we derived formulas for calculating the trustworthiness indicators since usually there is statistical information for this distribution. By numerical calculations, we have shown that the probability of false-positive increases, and the probability of true-positive decreases when the standard deviation of the stimulus signal increases. Besides that, the probability of false-positive has a much stronger dependence than that of true-positive. Indeed, the first one rises from 2 % to 17 % when the standard deviation increases from 0 to 0.01 rad, whereas the second one decreases from 38 % to 23 %. We have shown that the probability of false-negative increases and the probability of true-negative decreases with a rise in the standard deviation of the stimulus signal. However, this dependence is not as strict as for probabilities of falsepositive and true-positive. We also demonstrated that even with a zero error generated by the source of the stimulus signal, the probabilities of false-positive and false-negative are different from zero; this is due to the presence of a random component of the azimuth measurement error created by the VOR receiver. Numerical calculations have shown that the trustworthiness of health monitoring the ILS LRU is high enough because a false-positive and a false-negative probability does not exceed 1.5 % and 0.67 %, respectively.
Our future work will include developing effectiveness criteria of health monitoring the dismounted avionics systems that affect flight safety or flight regularity. We also plan to elaborate methods for increasing the trustworthiness of health monitoring through repeated measurements and control tolerances.

NOMENCLATURE
A actual value of the stimulus signal Г deviation of the stimulus signal Θ systematic error of azimuth measurement Ξ the random component of the error of azimuth measurement Y random input signal of the comparator Δ limit of the permissible error of azimuth measurement by the VOR receiver y realization of the random variable Y SFP false-positive event SFN false-negative event STP true-positive event STN true-negative event Z sum of two independent random variables Ξ and Γ g(z) probability density function of random variable Z f(ξ) probability density function of random variable Ξ φ(γ) probability density function of random variable Γ q(θ) probability density function of random variable Θ P(SFP) probability of false-positive P (SFN) probability of false-negative P(STP) probability of true-positive P (STN) probability of true-negative Θ0 systematic error of azimuth measurement Θ1 systematic error of azimuth measurement when a permanent failure occurs, resulting in a reduction of systematic error below the permissible limit Θ2 systematic error of azimuth measurement when a permanent failure occurs, resulting in an increase of systematic error over the permissible limit λ0 rate of intermittent faults transferring the VOR receiver to the state in which it is dismounted from the aircraft board, not having any permanent failure λ1 rate of permanent failures that transfer VOR receiver from the operable state to inoperable state corresponding to a decrease in systematic error by an amount higher than permissible λ2 rate of permanent failures that transfer VOR receiver from the operable state to inoperable state corresponding to an increase in systematic error by an amount higher than permissible Λ total transition rate q(θ) probability density function of systematic component of the azimuth measurement error over the set of the same VOR receivers q0(θ) probability density function of systematic component of the azimuth measurement error in the absence of permanent failures q1(θ) probability density function of systematic component of the azimuth measurement error when a permanent failure occurs, resulting in a reduction of systematic error below the permissible limit q2(θ) probability density function of systematic component of the azimuth measurement error when a permanent failure occurs, resulting in an increase of systematic error over the permissible limit σξ standard deviation of random variable Ξ σγ standard deviation of random variable Γ σθ standard deviation of random variable Θ D0 state of VOR receiver in which it is dismounted from the aircraft board not having any permanent failure D1 state of VOR receiver in which it is dismounted from the aircraft board due to a permanent failure corresponding to a decrease in systematic error by an amount higher than permissible D2 state of VOR receiver in which it is dismounted from the aircraft board due to a permanent failure corresponding to an increase in systematic error by an amount higher than permissible Δh higher boundary of the systematic component of the error in azimuth measurement in the presence of defects in the VOR receiver Δl lower boundary of the systematic component of the error in azimuth measurement in the presence of defects in the VOR receiver c normalization constant