Impact of Prognostic Uncertainty in System Health Monitoring

Across many industries, systems are exceeding their intended design lives, whether they are ships, bridges or military aircraft. As a result failure rates can increase and unanticipated wear or failure conditions can arise. Health monitoring research and application has the potential to more safely lengthen the service life of a range of systems through utilization of sensor data and knowledge of failure mechanisms to predict component life remaining. A further benefit of health monitoring when combined across an entire platform is system health management. System health management is an enabler of condition based maintenance, which allows repair or replacement based on material condition, not a set time. Replacement of components based on condition can enable cost savings through fewer parts being used and the associated maintenance costs. The goal of this research is to show the management of system health can provide savings in maintenance and logistics cost while increasing vehicle availability through the approach of condition based maintenance. This work examines the impact of prediction accuracy uncertainty in remaining useful life prognostics for a squadron of 12 aircraft. The uncertainty in this research is introduced in the system through an uncertainty factor applied to the useful life prediction. An ARENA discrete event simulation is utilized to explore the effect of prediction error on availability, reliability, and maintenance and logistics processes. Aircraft are processed through preflight, flight, and post-flight operations, as well as maintenance and logistics activities. A baseline case with traditional time driven maintenance is performed for comparison to the condition based maintenance approach of this research. This research does not consider cost or decision making processes, instead focusing on utilization parameters of both aircraft and manpower. The occurrence and impact of false alarms on system performance is examined. The results show the potential availability, reliability, and maintenance benefits of a health monitoring system and explore the diagnostic uncertainty.

This work examines the impact of prediction accuracy uncertainty in remaining useful life prognostics for a squadron of 12 aircraft. The uncertainty in this research is introduced in the system through an uncertainty factor applied to the useful life prediction. An ARENA discrete event simulation is utilized to explore the effect of prediction error on availability, reliability, and maintenance and logistics processes. Aircraft are processed through preflight, flight, and post-flight operations, as well as maintenance and logistics activities. A baseline case with traditional time driven maintenance is performed for comparison to the condition based maintenance approach of this research.
This research does not consider cost or decision making processes, instead focusing on utilization parameters of both aircraft and manpower. The occurrence and impact of false alarms on system performance is examined. The results show the potential availability, reliability, and maintenance benefits of a health monitoring system and explore the diagnostic uncertainty.

BACKGROUND
Across military and commercial fleets, aircraft are an example where lengthening service lives and budget constraints can adversely affect safety. As a result, more frequent inspections are required as service life increases to ensure safety of the users and the environment. However, the cost of large scale modifications or replacement in the case of hundreds of aircraft is a significant hurdle to overcome in most instances (Shoup, Donohue, & Lang, 2011). The impact of shrinking budgets can also reduce inspection frequency or delay needed repairs in favor of only performing mission critical tasks (Roach, 2009). Maintenance strategies must change to meet the extended in-service requirements and the constraints imposed by shrinking government and industry budgets.
Condition based maintenance (CBM) is an evolving maintenance concept with a goal of reducing maintenance and thus life cycle costs while increasing operational availability made possible, in part, by leveraging health monitoring techniques. Department of Defense Instruction (DoDI) 4151.22 defines CBM as "the application and integration of appropriate processes, technologies, and knowledge-based capabilities to achieve the target availability, reliability, and operation and support costs of DoD systems and components across their life cycle," (Under Secretary of Defense (AT&L), May 2008, p. 1-1). Integrated system health management and its impact on performance, cost, supply chain as well as traditional maintenance inspections and practices are the focus of this research. With the F-35 maintenance and logistics alone projected to cost $1.1 trillion over the 55 year life span amid shrinking defense budgets, the need to reduce the life cycle cost (LCC) of military aircraft is paramount (Shalal-Esa, 2013). Additionally, legacy aircraft may not be fitted with the proper sensors to fully implement health assessment leading to costly inspections, in both time and maintenance dollars. This reduces operational availability (Ao) and the funds available for other needs.
CBM is a demand driven maintenance process based on indications of stresses or impending failure of a component or system. When appropriately applied, CBM has the potential to reduce lifecycle cost and increase mission reliability by eliminating unnecessary maintenance actions (Butcher, 2000). Ellis (2008) argues that cost-effective systems monitoring allows repair actions based on system condition rather than costly time-based maintenance. Additionally, maintenance may be forecast for completion that minimizes impact on the operational mission of the system. Secondary failures, where one component's failure causes adverse performance or accelerated degradation of interrelated components, may also be reduced by implementing CBM as a result of prompt repair or replacement of the primary cause of fault.
CBM compares data collected from vehicle systems and their components and compares that information with a predetermined threshold prior to failure, or to failure for some non-critical components, then dictates repairs or replacement of parts. Additionally, interim time based inspections required under the baseline preventive maintenance (PM) approach are forgone, or significantly reduced in frequency, in lieu of continuous analysis of the aircraft via the integrated systems health management (ISHM) system. CBM requires sensor or inspection data to accurately diagnose the condition of a component. Manual inspections can prove costly in terms of time to perform if the part requires disassembly or removal of other components to observe its condition. Technology exists for some, and is under development for other components, to determine wear or impending failure conditions in lieu of manual inspections (Glaser, Li, Wang, Ou, & Lynch, 2007;Speckmann, 2007). The data from these health monitoring sensors may then be compiled to predict remaining useful life. Certainty is not 100%, be it in the interpretation of data collected on component condition or in prediction of remaining life based on that sensor data. This uncertainty has the potential to lead to poor estimation of component condition, which can result in false conclusions about safety of flight decisions and ultimately to critical failures.

Integrated Systems Health Management Enabler
The benefits of ISHM are the abilities to reduce inspection length, defer maintenance and migrate to maintenance on demand with the end goal to increase operational availability through reduced maintenance time (Speckmann, 2007). Applying ISHM enables CBM as opposed to preprogrammed periodic maintenance practices; that is, maintaining only when required instead of when prescribed by schedules, thus optimizing maintenance labor (Roach, 2009). SHM technologies and resulting modified maintenance programs serve to reduce the total life-cycle cost of a system and increase availability. While this may drive increased acquisition cost of a weapon system or aircraft due to the inclusion of health monitoring systems, the goal is to offset the increase with reduced operations and maintenance costs over the life of the program. Published literature shows the savings potential of ISHM enabled condition based maintenance on aircraft life cycle cost:  40% for vehicle maintenance (Walls, Thomas, & Brady, 1999)  30% to 50% for fuselage panels (Pattabhiraman, Kim, & Haftka, 2010)  10% electrical components (Scanff et al., 2007)  50-80% for the Boeing 777 (Gorinevsky, Gordon, Beard, Kumar, & Chang, 2005).
In general, an application project could choose to increase the detection capability, accepting a higher acquisition cost with the goal of lowering the overall system life cycle cost through more efficient operations and maintenance. For a given detection system, however, increasing the detection capability (e.g., lowering a threshold) will come at the expense of a degraded false alarm rate; the two are competing objectives. Ultimately, the value of the prognostic system will depend on the achievable balance between detectability for safety concerns and acceptable false alarm rates to avoid unnecessary and expensive maintenance actions. Aircraft, or other vehicle, availability is linked to the balance of sensor reliability and detectability and the capability of the system to decrease maintenance duration (Hoyle, Mehr, Turner, & Chen, 2007).
It is important to understand that uncertainty will exist in the diagnosis and prognosis of system health. Numerous points of entry exist for uncertainty to work its way into remaining useful life (RUL) prediction. Component performance data is dependent on sensor health and accuracy. It is also difficult to anticipate the exact conditions, load, environment, etc, that the vehicle or machine will undergo during operation or storage. Quantifying and compiling these uncertainties is a difficult task individually and made harder by potential amplifying effects on each other. Sankararaman and Goebel (2013) discuss factors of uncertainty in RUL prediction and lay out methods to quantify and interpret the sources. They also stress the need to accurately determine the uncertainty in the prediction for the prediction to be of use. The goal is that the probabilistic estimates of RUL based on real time monitoring allow increased time to accumulate on parts, thus increasing the MTBF for the ISHM aircraft and generating savings through fewer spares procurements or repair actions.
Determining the effectiveness of system health monitoring approaches requires a method for comparison of techniques.
The remainder of this paper discusses modeling approaches, evaluation techniques and results of this research.

MODELING APPROACHES
Research into the effects of prognostics on integrated logistics, maintenance and aircraft systems frequently neglects the impact of uncertainty on HM model outcomes.
Rebulanan utilizes a discrete event simulation to represent the F-35 autonomic logistic system (ALS) system with a health management system, LRUs, communication system, supply, and maintenance systems (Rebulanan, 2000). Rebulanan further evaluates performance with aircraft availability, mission capable and non-mission capable rates, and mission reliability. Rebulanan's model shows sensitivity of the supply wait time to the detection lead time for an impending failure and the supply stock levels. This outcome is somewhat intuitive in that as the prognosis of an impending failure is detected earlier and with greater accuracy, the supply system can plan further in advance, ensuring parts are available when required. Rodrigues and Yoneyama (2012; explore the effect of prognostics on spare parts inventories for both repairable and non-repairable systems compared with conventional supply processes. Both studies, simulated over 15 years each, show cost savings for the ISHM enabled system over the conventional one. In their work on non-repairable items they discuss uncertainty in failures and their impact on supply policy, but they do not include the impact of prognostic uncertainty on maintenance operations for false alarm adjudication or aircraft operational availability. Similarly, while they do address prognostic error in repairable systems they focus on the impact of sparing to account for fleet availability without addressing false alarms and how they might drive costs. Both works provide an excellent analysis of the cost impact of sparing decisions based upon health monitoring information. Out of stock costs are difficult to quantify but do impact down time for supply, which is where the impact is captured in our research model. A limitation of the nor-repairable study is that only one item is investigated, leaving interactions of multiple components in question. Kählert, Giljohanan and Klingauf (2014) utilize a MATLAB discrete event simulation to analyze one Lufthansa A320 component with 100% unscheduled replacement. They utilize process times, reliability, prognostic accuracy, and cost to evaluate PHM system performance. Additionally, the use of historic Lufthansa maintenance data provides added realism in the research. The research focus only extends for two weeks around a replacement, thus leaving out some potential for a false alarm condition to exist prematurely. One of their final conclusions is a realistic PHM system could save approximately 20% of annual fleet operation costs.

MODEL DESCRIPTION
In this research, an Arena discrete event simulation is utilized to represent a squadron of 12 aircraft and their associated mission, maintenance and supply processes over a 15 year duration. This model explores the impacts to this squadron in analyzing a model containing elements not addressed in the works of section 2. The authors add uncertainty not found in Rebulanan's work with an interaction of multiple components missing from Rodriques and Yoneyama.

Model Components and Architecture
The initial component failure properties were randomly generated from a uniform(250,1000) distribution for parts A-T. These times are then utilized for component replacements in the model. Each aircraft is generated and assigned 20 components with a failure time randomly sampled from an exponential distribution, with mean time between failure (MTBF) given in The sampled failure times are considered "truth" in terms of component failure times. That is, if the line replaceable unit (LRU) incurs more than the associated failure time in hours without being repaired or preemptively replaced as a result of scheduled preventive maintenance, overhaul in the baseline case or ISHM indicated replacement in the prognostic case, a failure occurs. Aircraft flow through preflight processing and mission preparation prior to actually flying an assigned mission. The ISHM system performs a scan to determine if the aircraft is anticipated to have enough useful life to complete the mission. Each component decreases its life only during engine running operations: taxi, take-off, flying, landing and parking. In this work, it is assumed that LRUs operate until failure. These processes are visually depicted in Figure 1.
After sortie completion, diagnostics are again performed and in the baseline case, maintenance is performed as well. ISHM aircraft perform post flight scan and if acceptable are released for next flight. Baseline aircraft are inspected and checked for LRU preventive maintenance time. If PM is not required, routine maintenance and inspections are performed and the aircraft released for next mission. Aircraft are then either parked until their next mission or turned for another flight.
In the maintenance module, the number of indicated failures is recorded and the maintenance clock starts. A detailed inspection is performed for both the ISHM and baseline cases, though shorter for the ISHM case. False alarms are recorded and in the ISHM case if a false alarm threshold over the lifetime of the part is reached, the ISHM system undergoes maintenance. The model indicates a false alarm condition if the predicted component RUL is less than the "truth" remaining time minus a safety factor and the anticipated sortie duration. In the baseline case supply stock is reduced and if not in stock the aircraft is grounded until the part arrives. Parts are processed by supply (occurs simultaneously with other aircraft operations in the ISHM case) and transferred from supply to maintenance. Aircraft are maintained and LRU(s) life characteristics are resampled from the failure distribution(s) in Table 1. The aircraft repair is checked and the vehicle is routed back into the mission queue. In the ISHM case, if the standby time until the next mission is greater than the mean time to perform any outstanding maintenance actions, the aircraft is routed to be maintained so as not to impact mission operations. In the baseline case, unless the part is scheduled for preventive maintenance the condition is not known thus the need for repair or replacement is unanticipated and the aircraft continues normal mission operations. Maintenance actions are performed serially on each aircraft, that is, only one inspection or maintenance action at a time, continuing until all required actions are complete. This assumption likely over constrains maintenance personnel actions, leading to slightly higher maintenance delays, but is done for model simplicity and has the same effect on the baseline and health monitoring cases. It is assumed that all component inspection times for indicated or actual failures are triangularly distributed (20, 30, 45) minutes and LRU replacements triangularly distributed (60, 90, 240) minutes. These times were chosen to represent a range of repairs and inspections while not portraying items which may require multiple days to maintain. Additionally, in this research required personnel for maintenance actions are always considered available. LRUs are always replaced when they are serviced.
Supplies are input into the model at an initial stock level and a reorder point. In both the baseline and ISHM cases, the stock level and reorder points are fixed for the simulation. The levels are discussed further in section 3.2. Once reorder point is reached, the difference between stock level and reorder point is ordered. Time between order and delivery is log-normally distributed (2,1) days for all parts. Additionally, a processing time upon receipt is incurred.
If RUL is within a 10 hour safety factor from failure the aircraft is routed to maintenance. If RUL is within a prescribed lead time window, a supply check is performed and if parts aren't in stock they are ordered to meet predicted maintenance activities. If RUL is within a defined maintenance window, component service can occur if the parts are in stock or the aircraft can continue flying missions if there is sufficient RUL.

Sensor and Prognostics Process
The ISHM routine begins by computing the remaining useful life (RUL) of each component. The RUL prognosis has two components, the diagnosis from the HM system and the prediction uncertainty. In this research component diagnostics is taken as perfect, i.e. sensor always knows exact health. In new components sensor diagnostics can have difficulty detecting the health state, thus providing data that may not be useful. As failure becomes more imminent, sensor diagnostics can provide a more exact condition diagnosis. The resulting determination leads to component RUL being predicted as: ( 1) Where Diagnosis is the log mean and equivalent to the true remaining life and, uncertainty is the log standard deviation defined in Eq. (2).
Uncertainty is varied in this research to determine the impact of uncertain prognostics on Ao and sortie rates. Uncertainty is calculated as: ( 2) Where Part RUL is the previous RUL prediction for that part and, uncertainty factor is a design variable.
This information is sent to the CBM module where maintenance predictions are performed. While no specific RUL prognostic technique is used, the technique above is utilized to represent compounded error or uncertainty built up in the system. Initially, RUL estimation is chiefly impacted by the uncertainty factor, but in section 4.2, additional degradation to the system is added to account for sensor diagnostic losses. Eqs.
(1) and (2) are representative equations developed by the authors to portray the behavior of health monitoring systems. They are not intended to mimic the performance of a particular system, but to represent the functionality of a monitoring system. The uncertainty factor is a representation of the accumulated variability in the prognostics for remaining useful life. This work ranges the uncertainty factor from a low of 0, to represent perfect prognosis, to a high of 100, which approaches half the MTBF of some parts. Examining a range of variability between these end points allows system designers to quantify how much uncertainty is acceptable in a health monitoring system before selecting one for inclusion on an aircraft.
The system then enters a decision node where the RUL is compared to a set safety factor, which would be a policy decision based on mission requirements. If there is RUL above the safety factor and the projected sortie length does not encroach on the safety factor, the aircraft is cleared for flight. If the RUL is below the safety factor, the component(s) are flagged and sent to maintenance. If RUL is sufficient, the aircraft is cleared for the next process. In all, the aircraft is checked prior to mission preparation (fuelling and cargo loading), prior to take-off, during flight, and upon landing. If all of these checks are satisfactory the aircraft continues through missions and standby time until a maintenance action is required.
The CBM system preorders parts to meet demands as described above. If the part is not in stock, the aircraft is placed in a non-mission capable supply hold until the part arrives. Upon maintenance completion, the ISHM equipped aircraft bypasses additional check-outs normally performed to inspect work, instead relying on the ISHM system to perform them. The aircraft is then released for the next mission tasking.

Evaluation Parameters
Establishment of useful performance measures to evaluate the model is essential. To that end, metrics currently used to determine aircraft and system performance are preferred as a means of comparison. Three categories of metrics, although interwoven, are laid out below and are used when discussing the results of this research: availability; reliability; and maintenance.

Availability
To understand operational availability and why it is a good measure of system performance for this model, it is useful to be familiar with achieved and inherent availabilities as well.
Inherent availability (A i ) is the availability of a system operating under an ideal support system. This means delays for logistics, administrative delays and preventive maintenance time are excluded, leaving only operating time and corrective maintenance.
Achieved Availability (A a ) adds preventive maintenance to A i in addition to corrective maintenance. Logistics, supply and administrative delays are ignored and those assets are assumed to be instantaneously available when required. Achieved availability is determined examining the mean time between maintenance, MTBM, and the mean maintenance time (MMT).
Operational availability (A o ) adds the final piece to the downtime portion of the equation. A o includes logistics, supply and administrative delays to the PM and CM for the system resulting in the mean down time for the system. Operational availability is the system availability the user of a system realizes, (ReliaSoft, 2007). Mathematically, operational availability is: Where MLDT is the mean logistics delay time.
Eq. (3) is not the only way to define operational availability. Pryor (2008) discusses methods to calculate A o seen in Eq.
(3), but the definition is slightly different.
Where OT is the operational time, ST is the standby time, TPM is the total preventive maintenance time, TCM is the total corrective maintenance time, and TALDT is the total administrative and logistics delay time, equivalent to MLDT. Figure 2 shows the components of up and down times. This is by no means an exhaustive list and further breakdowns are possible, especially in the administrative and logistics delay blocks, but for this research these components define the temporal parameters. A function of a system's operational availability, average daily flying hours is a measurement of the ability of the squadron as a whole to perform the assigned missions. Further, the number of sorties flown per day is a function of the mission requirements, but also the performance of the aircraft as well as maintenance and logistics systems.

Reliability
In the commercial environment, up and down times can also be assigned costs as the systems impact revenue generation. Kählert, Giljohanan and Klingauf discuss dispatch reliability, or the "ratio of revenue departures without delay or cancellations compared to all flights," (2014, p.1). They go on to summarize commercial aircraft cost accounting for delays and cancellations. Downtime has an associated cost beyond maintenance labor in lost revenue. Similarly, uptime has the potential to generate revenue, when not in a standby capacity. For military systems, assigning costs to up and downtime is problematic as there is no profit to generate and supporting national security is difficult to assign a value to.
In essence, military aircraft are consumptive, always operating at a loss. Policy and research can, however, strive to reduce these consumption costs.
False alarms diagnosed or predicted by the ISHM system drive unnecessary maintenance and supply actions as well as placing an otherwise mission capable aircraft into a NMC state. These maintenance and supply actions increase the overall cost impact of the ISHM system as they are not free.
A key requirement for successful deployment of an ISHM architecture enabling CBM is a low false alarm rate with reliable detection (Ellis, 2008;Van Horenbeek, Van Ostaeyen, Duflou, & Pintelon, 2013). False alarms in the baseline model result from CND and RTOK discussed previously. Totals for each of the models will be recorded for comparison. Additionally, an increase in false alarms, above a predetermined threshold, on an aircraft with an ISHM system will trigger an inspection of the ISHM system sensors providing erroneous data and potentially of the ISHM system logic itself.
The ability to tolerate false alarms is a two-fold evaluation. First, the cost associated with each false alarm shrinks any cost benefit of the ISHM system over the baseline system. Second, too many false alarms can trigger a "cry wolf" attitude towards the system or result in wasted time maintaining, or checking the system thus decreasing the operational availability of the aircraft and the reliability of the ISHM system. For an ISHM architecture to be effective it cannot trigger excessive false alarms which, in turn, trigger maintenance actions on the system.

Maintenance and Logistics
Inspection intervals are time driven processes under the baseline aircraft case and are prescribed to monitor systems for indications of damage. They are generally based on historic or predicted failure data and are conducted to ensure early indications of failure are discovered before they catastrophically fail the system or adjacent components. An assumption for this research is that all systems of interest on the aircraft are monitored in the ISHM model. If that were not the case time based, but informed through ISHM inferences, inspections would still be required. In this Accounting for the required time to repair and inspect aircraft is critical in determining the impacts of system changes to downtime and manpower costs. In addition to the repair of malfunctioning components, inspections based upon fault indications, either in performance or indicated by the ISHM system, drive mission unavailability and decrease system performance metrics. A common metric is to measure the required maintenance man hours per aircraft flight hour or MMH/FH. This factor can then be utilized in forecasting manpower requirements and required downtime based on mission requirements. Similarly, mean down time (MDT), the average amount of time it takes to return an aircraft to flying status once a fault is indicated, is a commonly used maintenance performance metric.
Supply delay is the time between actual part need and when the supply system delivers the part to maintenance and will impact both the baseline and ISHM/CBM cases. Nonmission capable supply (NMCS) is the common measure of this supply delay. The prognostic CBM case will anticipate failure and sparing requirements further out from maintenance demand and allow for advanced ordering if stock levels are inadequate. The current baseline process relies on anticipating failures and providing stock levels at individual bases or in some cases a central location that can be tasked to deliver spares when required. This process increases the logistic footprint by requiring storage facilities for materiel that may not be needed for upwards of a year. Managing these spares and the facility requires additional resources, manpower and money. "Logistics response time, a measure of supportability and an indirect measure of readiness," (Deputy Under Secretary of Defense for Logistics and Materiel Readiness, May, 2008, p. 6-4), drives shorter maintenance times and as such impacts supply and maintenance downtime.
While maintenance policy and cost decisions impact LRU replacement decisions, the prognostics capability plays an important role in determining when to repair or exchange components. Confidence in the performance of the diagnostics and prognostics systems could lead to a decreasing safety factor as to when maintenance occurs. This resulting increase in useable time of each part saves money through extended service life for the components and reduces the amount of supplies consumed. Capturing the amount of useful life lost for the components can quantify the gains that may be achievable.

Model Variables
This research explores the impact of RUL prediction uncertainty on the availability, reliability, and maintenance and logistics categories above. Evaluation of the model is accomplished through simulation of 15 years of aircraft utilization. Further, two design cases are initially utilized in the simulations. The remaining useful life uncertainty factor is varied at 14 levels with two false alarm limits at 0 and 10000 and the model assessed at each increment. The levels for the FA limit is meant to indicate that at 0, the ISHM system is always maintained after a false alarm and at 10000, policy allows nearly unlimited false alarms by the ISHM system before requiring repair. These levels are found in . At each uncertainty factor 100 simulations are run to establish confidence in the results, and the means of these data are presented. Sensitivity to values of FA limit greater than 0 is presented later in this paper once sensor and prognostics degradation are considered. Additionally, two simulations of the baseline case with no prognostics are run where component stock levels are varied.
Stock levels for the ISHM case are held to 1 nominally and ordered as predicted by the system. In the baseline case, two comparisons are examined, one where the stock levels are kept the same as the ISHM case. The other stock level case holds 4 parts in stock and reorders when the level drops to 2. This variance of stock level for the baseline case makes the process comparable to minimal levels as in the ISHM case and robust levels when failure is somewhat uncertain.

No ISHM Degradation Results
Daily flying hour averages for all simulation runs are located in . It is noted in these data that a decrease of 19.04 flying hours per day occurs over the range of uncertainty factors for a FA limit of 0. This decrease is smaller when the FA limit is 10000, reaching 3.45 hours. This reduction corresponds to 6949 and 1261 hours respectively in lost flying each year, the equivalent of removing more than 1 aircraft's missions from the flight taskings in the unlimited case and over 5 aircraft in the 0 FA limit case. The last two rows in contain performance results of the baseline model where the numbers in parentheses represent the stock level and reorder point respectively. For the baseline model, the (1,0) supply case yields only 18.36 daily flying hours while the (4,2) case achieves 27.94 hours. The chief cause of this difference is attributed to the (1,0) case waiting for supplies to be delivered as they are only ordered as needed and only 1 item is held in stock. The ISHM cases all benefit from the prognostic capability of the ISHM system in ordering supplies to meet requirements.
A typical measure when examining the maintenance demand of an aircraft is maintenance man hours per flying hour. Figure 3 examines MMH/FH for the case where all false alarms trigger ISHM system maintenance and the case where FAs in the system do not incur ISHM maintenance, merely downtime to adjudicate the alarm does not require maintenance. As shown in Figure 3, the 0 FA limit case MMH/FH increases linearly as the uncertainty factor increases. This growth results from the number of maintenance actions on the ISHM system as every FA triggers ISHM maintenance. Maintaining the ISHM system takes more time than merely adjudicating a false alarm by the ISHM system thus the increase in maintenance hours. In the case where FAs do not trigger ISHM repair, the MMH/FH grow slowly reaching a maximum of 0.268 vs. 4.198 for the 0 FA case. Inspection and maintenance times drive the maintenance hours and if inspection times were to increase significantly, the number of false alarms shown in Figure 4 could change the behavior of Figure 3. Additionally, as the uncertainty factor increases more false alarms occur as shown in Figure 4 as does the resulting downtime associated with the false alarms observed in Figure 5. For comparison, the baseline cases have MMH/FH ratios of 0.546 and 0.549 for the (1,0) and (4,2)    year. The quantity increases from 0 for the 0 uncertainty factor, perfect prognosis, case to 101.52 and 196.65 for the 0 and unlimited FA cases respectively at the 100 uncertainty factor case. As observed in the figure, since the amount of time spent in maintenance repairing the ISHM system for every FA in the 0 limit case increases as the uncertainty factor increases the number of false alarms is lower. It should be noted that this is not a reduction in the FA rate, as the prognosis accuracy is not degrading over time for this initial investigation. This mostly results from the maintenance time taking away time when the aircraft could be flying and, as noted in , the mean daily flying hours are nearly double for the unlimited FA case.

Figure 4: False Alarms per Aircraft per Year
While the number of false alarms per aircraft per year is nearly doubled in the 0 limit case versus the no limit case, FA downtime increases at a considerably higher rate. As shown in Figure 5, the average downtime each aircraft experiences per year due to FA increases from 0 for the perfect prognosis case to 111.31 hours for the unlimited FA case and over 2000 hours for the FA limit 0 case. The increase is attributed to the additional maintenance required to maintain the ISHM system at the lower FA trigger.

Figure 5: False Alarm Downtime per Aircraft per Year
Compiling all components of downtime and the number of times the aircraft is down for maintenance leads to the mean down time for an occurrence. As shown in Figure 6, mean down time decreases from 3.24 hours when the uncertainty prognosis is perfect to a low 1.09 hours when the uncertainty factor is 100 and FA limit is unlimited. This decrease is attributed to the fact that while the aircraft is being removed from service more often to adjudicate false alarms as the uncertainty factor increases, the inspections do not take as long as the aircraft is quickly returned to operation. MDT for the 0 FA limit case grows as the uncertainty factor rises, mostly due to all components requiring inspection and sensor repair for each time down. As uncertainty rises, the aircraft is brought down more frequently, but more often for a false alarm than maintenance actions. Adjudicating a false alarm through inspection takes less time than a repair, thus the down time is smaller. For the baseline (1,0) case, MDT is 171.82 hours, and for the (4,2) case 17.47 hours. The MDT for the (1,0) case is high mainly due to NMCS as there is only a stock level of 1 LRU and parts are ordered on demand, not schedule. The other major driver for the baseline MDT is the PM process.

Figure 6: Mean Down Time
At the low end of the uncertainty factor range, the fixed 10 hour safety factor imposed on each part accounts for a majority of the lost life each LRU, with the remainder mostly coming from the component not being able to safely cover the projected sortie duration. As the uncertainty factor increases, the mean life lost per component increases as well due to the uncertainty in the RUL prediction necessitating replacement before LRU failure. Additionally, the between mission maintenance window check forwards aircraft for LRU replacement or repair if the RUL prediction is within the designated maintenance window and parts are in stock. Figure 7 depicts the simulation outcome described above, growing from 15.63 hours to 35.40 hours for the uncertainty factor 100 case for each FA limit. Taken over the 15 years, the total life lost ranges from a low of 144506 hours for the perfect prognostics condition to 321236 hours for the case where uncertainty factor is 100 and FA limit is unlimited. This translates to 36.67 years of part life lost for the latter case. The mean life lost for each FA limit case is approximately equal at each point, thus they are collocated in the figure. This results from the fact that while the ISHM system may require more maintenance, the LRU components are only replaced as required. Of note is the max total life lost for the 0 FA limit case is 207901, occurring at an uncertainty factor of 20. The total life lost then continues to drop off as the uncertainty factor rises. This is due to the number of hours being flown by the aircraft declining as the uncertainty factor increases, thus not requiring LRU replacement as frequently. The lost utilization and cost implications of this figure could provide justification for system implementation. Component life lost in the baseline case is driven by the time based preventive maintenance (PM) cycle. In this research, the PM cycle is set at 400 hours whereby all components with less than 400 hours remaining, by time accounting, are replaced, yielding a mean life lost of 376.49 and 377.06 hours for the (1,0) and (4,2) cases respectively. False alarms and maintenance hours are important when determining cost, support requirements, and system confidence, but users, whether they are military or commercial, want to know how often their aircraft are available and when tasked if they can complete the mission. Utilizing Eq. (6) to calculate Ao, Figure 8 shows the impact of uncertainty factor and FA limit. Operational availability drops from 0.983 for both FA limit levels at an uncertainty factor of 0 to 0.754 for the uncertainty factor 100, FA limit 0 case and 0.969 for the unlimited FA case. The increase in downtime to repair the ISHM system in the 0 FA limit case is the driving factor in the decrease in Ao over the uncertainty levels. In the baseline cases, Ao is 0.618 and 0.941 for the (1,0) and (4,2) cases respectively. Ao is low in the (1,0) case again for the NMCS condition.

Sensor and Prognostic Degradation Results
A further examination of the impact of a degrading prognostics capability is examined as well. The Eq.
(2) becomes: This degradation factor places an additional uncertainty on the RUL prediction given as: Where growth factor is either 50 or 200 to provide different rates of degradation. Referring to Table 1, it is shown that component MTBF is bounded between 250 and 1000 hours. Therefore, the impact on RUL uncertainty could grow to nearly the component life in the case of part P if left unchecked. The Part ISHM timer is the accumulated life on the ISHM components associated with a specific component. The timer is reset upon component replacement or when a false alarm limit is reached thereby initiating maintenance on the ISHM system. Degradation factor increases as a function of the accumulated time on the Part ISHM timer. Thus, the longer the ISHM system is in operation, the higher the degradation factor becomes adding to the uncertainty in the system. As with Eqs. (1) and (2), Eqs. (5) and (6) are representative equations developed by the authors to portray the behavior of health monitoring systems.
Including the degradation factor in the model as in Eq. (5) shows a false alarm limit may be useful in actual aircraft operation. Fixing the error factor at 20, towards the lower end of the range, an exploration of the impact of false alarm limits is made. The growth factors of 50 and 200, utilized in Eq. (6), are hereafter referred to as high and low respectively. These factors correspond to a growth rate of 20 and 5 per hundred hours of accumulated time on the ISHM system respectively. The degradation factor adds additional uncertainty to the RUL prediction to examine the effect of degrading sensor or prognostics capability through use of the aircraft. In the analysis of degradation factor, FA limit is the variable of change and is varied from 0 to 100.
Examining the impact of FA limit on mean daily flying hours for the squadron shows that the 0 FA limit case, for which every false alarm triggers ISHM maintenance, dramatically reduces the flying hours. This results from the amount of maintenance required on the ISHM system depleting available hours to fly missions. These results are shown in Figure 9 and indicate that the low degradation growth rate reduces the flying hours from 32.67 at a FA limit of 2 to 31.15 at 100. In contrast, the high growth rate drops the daily hours from 32.16 at FA limit 2 to 27.49 at FA limit 100. The difference in the magnitude of the declines lies in the fact that the high degradation rate increases uncertainty in the RUL prediction, thus driving false alarm occurrence up. That is, when the FA limit is 0 and there is a false alarm, the ISHM system is always repaired. When the FA limit increases to 2, this allows flights to continue until 2 false alarms are incurred, thus allowing increased flying hours for the aircraft. The degradation factor, slow deterioration of prognostics system, accounts for the remaining decline in daily flying hours. This results from compounded error in the system increasing as the time between service lengthens due to the FA limit being raised. Figure 9: Daily Hours Flown Figure 3 shows that for a static uncertainty factor of 20 the MMH/FH was 0.165 and 0.743 for the FA limit 10000 and 0 cases respectively. Figure 10 below shows that the high degradation rate reaches 0.74 at a FA limit of 100 and the low rate 0.378. The graph does not show the FA limit 0 MMH/FH data of 2.361 for the low and 2.468 for the high to allow better visualization of the remaining data. It is observed in Figure 11 that the impact of the high growth rate greatly increases the number of false alarms, thus increasing the maintenance hours required per aircraft flight hour shown in Figure 10. As previously mentioned, Figure 11 is perhaps the best indicator of the impact of degradation growth rates on aircraft operations. The high growth rate proves true to its name as the rate of increase in false alarms per aircraft per year remains higher than the low growth rate over the range of FA limits. The number of false alarms increases as a result of the degradation factor continually increasing as the ISHM system is not being maintained at the shorter intervals a lower FA limit brings. The impact of the increase in false alarms, and thus downtime, is a decrease in operational availability, Ao, as the FA limit increases. Shown in Figure 12, the Ao trend follows that of the daily flying hours and inversely the trends of false alarms and MMH/FH. Operational availability peaks at a FA limit of 4 for both the high and low growth rates. The low growth rate levels off around 0.96 at FA limit 60 while the high rate continues a decline to 0.93 at FA limit 100 without leveling off. While Figure 11 shows the increased growth in number of false alarms, the true utility of the model is in determining the "sweet spot" across the performance curves. This is the location where a peak or trough in the curves indicates performance drops off on either side and thus this set of factors should be considered for system design. In this paper, examining Figure 10 and Figure 12 show a performance drop off at a FA limit of 4. shows that mission reliability declines across all FA limits, indicating that keeping the FA limit as low as possible is desirable. These results are specific to the set of inputs used in the model. If the time for inspection of a failure condition or to repair the ISHM system were changed, the potential for a different outcome in FA limits exists. Therein lies the utility of the model in being able to change input characteristics and policies to determine system level performance metrics.
In comparing the sensor degradation case in Figure 9 with the baseline case, daily flying hours remain higher than the baseline case across the FA limit range. The MMH/FH for the degradation case with low growth rate remains below that of the baseline cases, while the high growth rate case is higher than the baseline cases for FA limits above 40. As previously discussed, the FA limit "sweet spot" in this model is 4 thus MMH/FH would be approximately 0.3 and less than the baseline cases. Comparing Ao between the baseline and degradation models shows that around the 4 FA limit results, the degradation cases are above 0.96 while the baseline cases are 0.618 and 0.941 for the (1,0) and (4,2) cases respectively. This again shows the ISHM system to provide higher performance. Finally, the mission reliability for the baseline cases of 84.85% is higher than the ISHM cases, which are below 70% at the 4 FA limit case. Across the model metrics the ISHM case with degradation tends towards higher performance than the baseline. Depending on the desired performance levels desired for the aircraft program managers are left to weigh the performance metrics.
In the model case where degradation is present, for the uncertainty factor chosen it is generally best to set the false alarm limit low. Programmatic policy of cost, availability and reliability will drive towards the selection of a proper limit. Additionally, changes to degradation factor, i.e. ISHM sensor and prognostic characteristics, and RUL uncertainty, prediction algorithm accuracy, can change model outcomes.
Cost to implement a certain health monitoring technology on the aircraft may outweigh the benefit of its inclusion if it drives too many false alarms or too much repair time.
Absent the cost impacts of manpower and component replacement, the decision as to how much uncertainty in prognostics is an easier proposition. It is shown in the model with no degradation that as the RUL uncertainty increases, most performance characteristics are adversely impacted. The comparison of baseline to ISHM cases shows the potential advantages implementation of health monitoring and condition based maintenance. The test for program managers then becomes selecting the appropriate system characteristics to meet overall aircraft fleet performance and cost metrics.

CONCLUSION
This research shows employment of an ISHM system supporting CBM can produce system performance greater than baseline systems. The main contribution of this effort is as a simulation tool to compare sensing options and examine their impact on desired performance factors. The ability to input ISHM system and aircraft characteristics and investigate alternative approaches to monitoring and maintenance makes this tool useful in program decisions on whether or not to implement monitoring techniques. While determining causes of system uncertainty is outside the scope of this research, quantifying the impact of the uncertainty is demonstrated. As a system designer it is important to note, as this research shows, the amount of uncertainty in your system, particularly in the prognostics. This uncertainty could be mitigated with better sensors, techniques or processing algorithms. Further, the designer should seek to minimize either the number or false alarms the prognostic system produces or set an appropriate limit on false alarms to minimize the impact of additional inspection time to adjudicate system condition.
As cost is not included in this work making a true comparison among options is difficult. A program manager must weigh the technology costs to achieve the performance observed in the model and compare those with system objectives. This task becomes easier if these variables can be explored across a range of scenarios as this research provides.
Future work in this research will explore the impact of cost, supply factors and manpower requirements.