Reaction Wheels Fault Isolation Onboard 3-Axis Controlled Satellite using Enhanced Random Forest with Multidomain Features

As the number of satellite launches increases each year, it is only natural that an interest in the safety and monitoring of these systems would increase as well. However, as a system becomes more complex, generating a high-fidelity model that accurately describes the system becomes complicated. Therefore, imploring a data-driven method can provide to be more beneficial for such applications. This research proposes a novel approach for data-driven machine learning techniques on the detection and isolation of nonlinear systems, with a case-study for an in-orbit closed loop-controlled satellite with reaction wheels as actuators. High-fidelity models of the 3axis controlled satellite are employed to generate data for both nominal and faulty conditions of the reaction wheels. The generated simulation data is used as input for the isolation method, after which the data is pre-processed through feature extraction from a temporal, statistical, and spectral domain. The pre-processed features are then fed into various machine learning classifiers. Isolation results are validated with cross-validation, and model parameters are tuned using hyperparameter optimization. To validate the robustness of the proposed method, it is tested on three characterized datasets and three reaction wheel configurations, including standard four-wheel, three-orthogonal, and pyramid. The results prove superior performance isolation accuracy for the system under study compared to previous studies using alternative methods (Rahimi & Saadat, 2019, 2020).


INTRODUCTION
With the ever-growing number of satellites being launched into space, it is crucial that the health monitoring and safety of these systems be advanced enough to compensate for the lack of redundant components and their decreasing size. The attitude determination and control subsystem (ADCS), which employs reaction wheels (RWs) as actuators, is one of the most critical components of a satellite subsystem. By accelerating or decelerating the flywheels attached to the electric motor, RWs can correct the satellite's orientation or perform maneuvers under disturbances (Rahimi et al., 2017). Smaller satellites require less cost for design and mass-production, meaning multiple can be launched into space at a time. Therefore, to ensure reliability and mission success, the health and maintenance of these systems are essential. As a result, fault detection and isolation (FDI) methods for ADCS has developed great incentive to be improved and advanced.
The two main categories for FDI approaches are model-based and data-driven. In model-based approaches, the continuous comparison between the actual system state and nominal state is conducted and in turn, generates residuals. This is achieved by using a mathematical model of the system with ideal behavior. One major advantage of this method is that it has the ability to provide a description of the dynamic behavior and physical understanding of the system (Tidriri et al., 2016). However, developing an accurate mathematical model that considers uncertainties and modeling errors is difficult due to some uncertainties being unquantifiable. As previously mentioned, model-based methods compare the available data with prior information and generate residuals. One method of residual generation is through observer-based techniques. If the system is observable (i.e., it is possible to determine the entire system behavior from the outputs) and the process parameters are known, then it is possible to estimate the output of a process with an observer and the residuals. In (Jia et al., 2019), two Non-Linear Observers (NLO) are developed to robustly reconstruct bias faults and effectiveness factors of RWs, and a systematic observer design method is obtained. Fault-tolerant attitude control for the spacecraft system was not considered in this study, however. Four classifiers, Random Forest (RF), SVM, Partial Least Squares (PLS), and Naïve Bayes (NB), are combined to create a fusion framework for FDI in RWs (Nozari et al., 2019). In (Nemati et al., 2019), nonlinear observers are used for fault diagnosis of spacecraft. The observers are designed to ensure convergence of the estimation error to zero for the nominal nonlinear system. Fault isolation is achieved using a General Observer Scheme (GOS).
However, when dealing with model-based approaches, there are certain limiting factors such as system complexity, high dimensionality, nonlinearity processing.
Data-driven approaches focus on analyzing system outputs and rely immensely on large amounts of data, allowing these methods to perform well with large-scale and complex systems, as well as reduce time and costs since the development of models is unnecessary. This approach can be beneficial if no mathematical model or expert knowledge about a system is available.
A highly nonlinear dynamic system is the result of modelling the complexity of a single RW. More RWs can be placed into the assembly combined with the attitude and dynamics of the satellite in orbit. If the parameters of interest are non-measurable to the FDI scheme, it becomes difficult to detect faults and isolate their root causes. When conducting data-driven FDI, there are two critical factors any scheme should have, 1) generation of a feature subset, 2) selection of the most optimal classifier. Research conducted by (ElDali & Kumar, 2021) proposed a growing neural network for aircraft engine fault diagnosis, detection, and prognostics. The model optimized the Long-Short Term Memory (LSTM) algorithm and was used to detect failure for RWs in the pyramid configuration. The model tried to detect failures in the four RWs by calculating a Health Index value. A value between 0 and 1, with the latter representing a greater chance of failure. The values were acquired from the residuals of the acquired speed values and the predicted ones. The study (Lee et al., 2020) addresses fault detection and identification in the RWs of nanosatellites based on a Deep Learning (DL) algorithm. Fault detection is accomplished using the residuals between the measured attitude and the estimated attitude. Trial and error were used to optimize the learning and hyperparameters of the LSTM model used within this study. However, this technique did not consider the temporal elements of the attitude information and was more likely to misjudge because only the instantaneous values were used.
Most of the research above also involves instances where residual generation is applied through analytical models. This method, however, does not do well in explaining the facility point behind the detected faults. Mixed learning models are limited to the performance of the best classifier used within the ensemble. If other classifiers have limitations, it can hinder the overall accuracy of the method. Furthermore, most methods deal with only one RW and do not consider multiple faults or different RW assemblies. Based on the open literature of past studies, a research objective was established. Which is the design and development of a data-driven FDI scheme that is capable of autonomously isolating the location of faults within a nonlinear system, with a case-study for an in-orbit closed-loop controlled satellite with reaction wheels as actuators This method is to be developed to be applicable to other systems, however it is being designed for this specific application. To achieve the above research objective, Time Series Feature Extraction Library (TSFEL) transforms the time-domain data into a feature-space state. The new feature space is then fed as input into multiple machine learning models (MLMS). Sensitivity analysis of missing sensors, missing values and noise analysis was also conducted on various datasets. The robustness of this method was also tested on three different configurations of the Reaction Wheel Assemblies (RWA). Lastly, k-fold cross-validation is used to validate the results. The choice for using less computationally expensive methods versus more advanced and highly computational methods was made to ensure applicability and accessibility of the proposed in this study on a broader range of satellite units with lower-end computational capacities. This is particularly important as smaller satellites with limited capacity may not house supercomputers onboard, and computational power and energy consumption can be limiting factors in deploying such algorithms for the FDI of such units.
Past research on data-driven approaches shows that these methods are known to have some limitations. The proposed method can deal with the extensive consumption of resources, memory, and algorithmic complexity by applying a hierarchical approach to enhance the isolation scheme and only utilizing the appropriate resources when required.
The novelties of this approach are: 1. Based on previous assessments, common feature subsets such as statistical features do not result in the best results. Therefore, an automated feature extraction library is used to obtain the best feature subset from a temporal, statistical, and spectral domain which has not been explored for this specific application, to represent the characteristics of the multiple in-phase faults from different points of view and where most literature only consider one space-state transformation. 2. Consideration of three different fault time characteristics: Abrupt, Transient, and a general case. Where most literature mainly considers abrupt faults. 3. Use of variable inception time and duration of the faults for different cases to tackle the applicability range of data-driven methods and broaden the scope, where other literature only considers constants. 4. Classification of multiple in-phase faults where other literature primarily considers either single faults or multiple out-of-phase faults. The remainder of this paper is organized as follows: Section 2 presents the problem definition. Section 3 explains the methodology employed in the study. Section 4 presents the case studies used to evaluate the proposed method. Section 5 offers insight into the results and discussion of the study. Section 6 concludes the paper with final remarks.

PROBLEM DEFINITION
A nonlinear system in discrete-time state space can be represented in Eq. (1) where at time step , ! ∈ ℝ & is the state vector, ∈ ℝ ' depicts the control input vector, ! ∈ ℝ ( , ! ∈ ℝ ' , ! $ ∈ ℝ & , ! % ∈ ( and ∈ ℝ ' represent the system parameter vector, measurement vector, additive process noise for states, additive process noise for parameters, and additive measurement noise, respectively. (•) is a nonlinear process model, and (•) is a nonlinear measurement model. Full state measurement ! = ! + ! is considered. In this study, it is under the assumption that the system component faults are reflected as changes in the physical system parameters (Sobhani-Tehrani et al., 2014). Eq. (2) describes the faulty system, also known as a multi-parameter fault model, with: where ) ∈ ℝ ( is the nominal parameter vector representation and the fault parameter vector containing L fault elements is represented by ! ∈ ℝ * . By using the fault model given by Eq. (2), one can transform the problem of nonlinear fault diagnosis into the form of an on-line nonlinear parameter tracking problem. Furthermore, for fault isolation, one can extract single-parameter models Ωi = 1, … , . This is shown in Eq. (3) below: The mission of the data-driven algorithm is set to classify the current state of the system as one of the possible faulty cases. In this study, multiple fault scenarios are considered.

METHODOLOGY
A hierarchical approach is used within this study for the methodology when tackling the FDI problem. This ensures a reduction in computation time by allocating the right resources to be used only when required. A data-driven fault diagnosis model is created to classify multiple in-phase faults of a satellite RW. The data is simulated using RW data and then extracted for features. The features are calculated using TSFEL and then reduced using Recursive Feature Elimination (RFE) (Bahl et al., 2019) to produce a feature subset ranking. The features are fed as input for the machine learning model (MLM) as inputs for training and testing. The MLMs implemented within this study include Gradient Boosting (GB), Decision Tree (DT), Random Forest (RF), and Multi-Layer Perceptron. The final step includes employing tenfold -cross-validation to provide a more robust estimation of feature selection.

Data Preprocessing
It requires much time and effort for data science experts to model the Machine Learning Models (MLM) and tune its hyperparameters. For some spacecraft systems, historical data is not always available for training because the data is being collected in real-time. This puts a limitation on the model's time to tune and train the algorithm. It also makes onboard health monitoring a difficult task since the model would have to be created after the data is transmitted to Earth. Auto-machine learning can compensate for the drawbacks of normal machine learning techniques by automating the data pre-processing, model selection, hyperparameter optimization and the interpretation of the results. To assist the MLM in accurately distinguishing between nominal and fault scenarios, the time-series data is transformed into a feature space state. The features are acquired from raw time series data through Python package TSFEL, which extracts over 60 different features across a spectral, temporal, and statistical domain (Barandas et al., 2020). The new representation of the data is what is used as input for the training set. The data is also filtered for missing values, highly correlated features, and low variance features. Finally, the data is normalized before being fed into the MLMs.

Feature Reduction
Less sensitive and inaccurate features can reduce the model's ability to predict faults accurately. That is why Recursive Feature Elimination Cross-Validation (RFECV) was implemented in the pre-processing stages. RFECV is an iterative backward selection method that uses a classification model to produce a feature subset ranking instead of a feature ranking. All the features are fitted to the model, and at each iteration, the feature with the least importance is eliminated (Ramachandran & Siddique, 2019). For this study, RF is used to train the model, and for each tree, the model estimates the variable importance by recording the out-of-bag prediction accuracy for every predictor variable premutation. The difference between the prior and the altered model accuracy is averaged over all trees and normalized by the standard error at each iteration. The results for RFECV determined that the optimal number of features was 22, as shown in Figure 1.

Machine Model Selection
In this section, the various machine learning models implemented during the study are briefly discussed. Gradient Boosting (GB) (Bentéjac et al., 2021) is a machine learning technique utilized for regression or classification. This is done by constructing a prediction model comprising an ensemble of weaker models (i.e., Decision Tree). The objective of the GB is to minimize the loss between the actual class value and the predicted class value. The Random Forest (RF) (Probst et al., 2019) classifier consists of many individual decision trees that operate as an ensemble. Each tree will return a class, and the class with the most votes becomes the model's prediction. The low correlation between the trees makes RF a helpful classifier. This protects each tree from errors of other individual trees. A Decision Tree (DT) learns from data to try and approximate a sine curve (Charbuty & Abdulazeez, 2021). This curve has a set of if-then-else decision rules. It breaks down data into smaller subsets while in the same notion incrementally making an associated DT. To enhance the RF and DT classifiers, a meta-classifier Ada-Boost is used. It begins by fitting the classifier on the original data set and continues to fit additional copies of the classifier on the original data. However, in successive iterations, the weights of the incorrectly classified instances are adjusted to achieve better classification. The Multi-Layer Perceptron (MLP) is a feed-forward Artificial Neural Network (ANN) (Mishra & Huhtala, 2019). It can be referred to as a network with multiple layers of perceptron (nodes), each with a threshold activation function. There are at least three layers of nodes in an MLP: input layer, hidden layer, and output layer. Besides the input layer, each node uses a non-linear activation function. Backpropagation is used for training. The hyperparameters for the MLP were tuned using GridsearchCV to run each model with different hyperparameters and return the one with the optimal hyperparameters.

Training, Testing, and Validation
The input data was trained and tested at various splits. After a series of runs, an 80:20 train-test split resulted in the best performance for the MLMs. The K-fold cross-validation technique is also implemented to evaluate predictive models by partitioning the original samples into a training set and test set. The benefit of this method is that all observations are used for both training and validation, and each observation is used for validation only once.
In the following section, details on a case-study are provided to evaluate the performance of the proposed methodology.

CASE STUDY: FAULT ISOLATION OF SATELLITE RWS
To assess the performance of the proposed FDI method, the faults for RWs onboard a 3-axis controlled satellite are employed. The simulation consists of a high-fidelity nonlinear model of the RW, nonlinear satellite attitude dynamics, and a sliding mode controller (SMC), depicted in Figure 2. It should be noted that in this study, the simulation setup is assumed to have a perfect sensor, and there is no estimator, so the noise system output feeds directly back into the sum junction before the controller. In the following sections, the effect of noise in system measurement is studied under sensitivity analysis discussions for the proposed method. Each component for this case study can be detailed as follows:

Satellite Dynamics
A fully actuated rigid body spacecraft with RWs as actuators under external and internal torques can be expressed as: is the spacecraft's moment of inertia including the RWs and is defined as = . − 1 5 , 1 ∈ ℝ 646 = ([ 1#, 18 , 13 , 16 ]) stands for the axial moment of inertia of each RW. the mapping matrix ∈ 346 captures the influence of the actuator's toques to the principal axes of the spacecraft. Angular velocity of the spacecraft relative to the inertial frame expressed in the body frame is denoted by ,-, . The external torque is represented by 2 ∈ ℝ 34# and the torque generated by the RW is /0 . Using quaternions, the kinematic equation for the spacecraft can be formulated as: where the unit quaternion is represented by O w identity matrix is I ∈ ℝ 343 and the skew-symmetric matrix of the quaternion vector : × is expressed as: :

Controller
A simplified nonlinear sliding mode controller with quaternion tracking error is defined as (Kumar et al., 2018): where <: ∈ 34# and <6 ∈ and are the desired attitude, and 2 5 2 + 62 8 = 1. The rotation matrix is given by: Here 2 5 2 = 1 , 2 = − 2 × 2 . Relative angular velocity 2 ∈ 34# is formulated as follows: where < ∈ 34# denotes the desired angular velocity. The required control command is defined as: where @ and # are positive constants and ∈ A4# = [ : , ,* , ] 5 . The simplified required input can now be calculated for the actuator given actuator dynamics and required control command as (Kumar, 2018): is the motor torque constant for the RWs. All control parameters including , @ and # are set to the value of 1 for the required simulations in this case study. The full derivation of this controller can be found in (Godard, 2010) on pages 113 to 115.

Actuator
The RW used in this study is an ITHACO "' type A'" by Goodrich. A high fidelity RW nonlinear model was obtained from Bialke (Bialke, 1998) and was integrated into the ACS dynamics. The RW nonlinear model, including discontinuous functions approximated by sinusoidal functions, can be formulated as: where # and 8 represent the motor disturbance, 3 denotes the EMF torque-limiting block. The variables 6 , F and B)'' represent the analytical approximation of the sign function in the Coulomb friction block, speed limiter block and torque command voltage, respectively.

Reaction Wheel
As previously mentioned, the performance of three reaction wheel assembly configurations was studied. The proposed methodology was evaluated on the following configurations: (a) Standard four-wheel with three orthogonal RWs and one redundant oblique RW, (b) Pyramid configuration consisting of four RWs, and (c) three-orthogonal RWs (see Figure 3). The actuator's torque contribution to each principal axis of the spacecraft body frame is mapped as follows (Ismail & Varatharajoo, 2010): where 4 , G , H are the torques applied to the satellite in the x, y, and z direction and 1+ is the torque generated by each RW on its respective axis of rotation. The mapping matrix (A) is as follows: where /0+ is the mapping matrix for configuration and = 1 refers to the standard 4-wheel configuration, = 2 refers to the pyramid configuration and = 3 refers to the 3orthogonal configuration. The math functions sine and cosine are abbreviated as c (·) and s (·), and α is the in-plane angle while β is the out-of-plane angle (Kök, 2012). For the three-orthogonal wheel configuration, a separate dataset had to be generated to account for the proper number of RWs engaged. Cases 1-3 shown in Table 2 referenced in section 4.1.5 are identical to the datasets constructed for the standard and pyramid configurations except for fewer reactions wheels, resulting in a lower number of fault combinations.

Fault Formation
The simulation data is obtained from a closed-loop ACS simulation of a three-axis stabilized low earth orbit (LEO) satellite. Abrupt and transient time-varying fault cases, as well as a generalized fault case, are added into two of the RW components as variations via the bus voltage ( ) and motor torque back electromotive force (BEMF) constant ( ). and kt are modelled as the voltage drop of the power bus and the variations in the torque gain, respectively. The multi-parametrized fault model is obtained by replacing , with , 0 + 1 and , with , 0 + 2 , where j is the index for the RW unit among the units considered and I + are unknown fault parameters, indicating a possible fault in either the bus voltage or motor current of each wheel. The value for I + will be zero at any given time due to the additive form of the fault parameters. Any deviation from zero for any fault parameter would indicate the fault size and its severity. Within this study, only system satellite outputs, namely and are observed for faulty behavior. Abrupt faults can be viewed as instantaneous faults and can lead to complete failure of the system. Transient faults are more temporary faults within the system and may return to normal parameters given a certain amount of time. This study investigates the effect of fault duration, inception of the fault, and severity of faults during a mission. The initial moment the fault is introduced into the system can be described as the inception time. Duration is the length of time the fault remains within the system. Lastly, the severity denotes the magnitude by which the faulty parameter has deviated from its nominal values during the fault period. With these definitions in mind, three datasets are created when evaluating the proposed methods: 1. Case 1 (abrupt fault): The inception time for the faults in both and are set at a random value + ∈ (0,10) seconds and the duration of these faults is defined from the inception time to the end of the simulation. 2. Case 2 (transient bounded fault): Inception time for faults in both and are set at a random value + ∈ (0,10) seconds and the duration of these faults are set randomly within < ∈ (10,20) second range.

Case 3 (transient unbounded fault): The inception time
for faults in and are set at a random value + ∈ (5,55) seconds and the duration of each fault is determined randomly within < ∈ (5,60 − + ) to not exceed the total simulation of 60 seconds.
All cases include a random combination of D and JK. faults, with a random severity for each with a set fault deviation range presented in Table 1. The difference in the fault definitions for Cases 2 and 3 lie within the definition of their fault inception time and duration. The larger inception time for Case 3 creates a broader scope or window for fault inception, which is a better representation of real-world scenarios. This distinction for cases 2 and 3 makes them more complex.

Dataset Structure
This section provides details of the dataset used for this study.

Raw Data
As previously mentioned, it is assumed that the faults injected into the RW units can be detected and isolated by evaluating the satellite output parameters. The satellite attitude parameters include quaternions, # to 6 , and angular speed # to 3 . Based on the desired conditions for the case scenarios presented in section 4.5, input values are fed into the MATLAB simulator, which returns the raw data used as input for the FDI algorithm. For this study, 5,000 simulations per scenario were run to create a data set, and since there are 16 scenarios, 80,000 simulations were collected to create one dataset. Since the data pertaining to 6 is not independent of # to 3 they were omitted when fed into the FDI algorithm during the pre-processing stage. Of the 80,000 CSV files stored, each one contains a data set of simulation time step of 0.1 seconds, and each simulation has a time length of 60 seconds; therefore, 600 rows are generated. The number of columns is determined as 2 × 8 = 16 for two sets of 4 parameters related to the nominal and faulty parameters. This raw data is later transformed into the three different domains through TSFEL, which extracts features from a spectral, statistical, and temporal domain as previously mentioned, and ultimately is fed into the MLMs. Combination theory is used to calculate the total number of combinations for a four-wheel and three-wheel RW assembly, which can be seen in Eq.
where is the number of faulty units at time step , and is the number of available units that could become faulty. To be able to refer to each combination (fault scenario), all possible cases are assigned a number in Table 2 for each RW wheel assembly configuration. The fault scenarios for the standard four-wheel and pyramid configurations are the same, while because of the number of wheels involved in the assembly, the scenario numbers and combinations are different for the 3-orthogonal assembly. Once the simulations are executed via the inputs for fault scenarios in Table 1 and the satellite ACS setup in Figure 2, the satellite measurables q and ω are stored in comma-separated value (CVS) files for each simulation separately in the format presented in Table 2 where FW refers to faulty wheels. All the CSV files are used as input for training and testing the MLMs to evaluate their performance under the proposed method and compare their merits.

Configuration
No

RESULTS AND DISCUSSION
The accuracy of classification methods, as discussed in Section 3, which is employed for the FDI problem discussed in Section 2 for the data acquired in the case study detailed in Section 4, is presented in, Table 4, Table 5, and Table 6.
The simulation is tested at a size of 20%. For the cross-validation, the random forest classifier was employed to save on computational costs, the splitting strategy was set to 10-folds, and the parameter that was varied in this analysis was the max depth of the trees. From the results listed in the tables above, the classifier's ability to predict is significantly affected by the fault severity and duration definitions given with the scope of the simulated time series. The case definition described in Section 4.1.5 shows that case 3 is below adequate for any real-world applications. This can be anticipated due to a combination of 1) the fault inception time initiating nearing the end of the simulation time and 2) the fault duration being relatively short. These two problems stem from the influence of the controller. Since the controller compensates for the system when detecting a fault, if the inception time and duration of the fault are not distinct enough, then the system has a hard time classifying the faults. In practical applications of in-orbit satellites, cases 1 and 2 are more practical. If a commanded orientation change causes a fault to occur, the fault will most likely persist to the following command, representing the cases outlined in section 4.1.5. If the MLM fails to identify the fault during a maneuver, the fault will be present in the following command, leading to a true-positive prediction. The learning curves, validation curves, and precision-recall graphs can be seen in Appendix A.

Method Case 1 (%) Case 2 (%) Case 3 (%)
It should also be mentioned that the configuration of the RW assembly influences the accuracy of the proposed method. It is expected that the overall accuracy of the cases for the threeorthogonal wheel, standard four-wheel, and pyramid go from best to worst, respectively. This is due to the placement of the RWs or the lack of a redundant RW component. Since the three-orthogonal wheel configuration only has three wheels on individual axes, it is not affected by the moments generated by the other RWs. Therefore, if there is a sudden change in the torque generated, there should be a more direct effect on the satellite's behavior. However, this is not the case for the other configurations since the torque generated by these RWs is proportionally changing or contributing to the x, y, and z axes, which results in some direct effect from the three RWs on the x, y and z axes and some indirect contributions from the fourth wheel. The pyramid configuration has a more challenging time with this proposed method due to its assembly's symmetry. If there is an abnormality in the first two RWs and the same discrepancy occurs in the other two RWs the system will have difficulty distinguishing between the two faults.

Confusion Matrices
The model's performance is evaluated using the test data and used to construct the con-fusion matrices with more details per each scenario. To normalize the results, the number of instances tested per class is used. The results for the confusion matrices are demonstrated in Table 7, Table 8, and Table  8.  Table 7. Confusion matrix for standard 4-wheel Along the diagonal, the values that depict the percentage of instances correctly predicted can be seen. Scenario 0 refers to the conditions where all RW units are healthy, and the 100% accuracy establishes the learner's ability to distinguish between nominal and faulty conditions. Scenario 0-4 represents the single faults of the RW where only one unit can fail. Within these sections of the confusion matrices shown above, there is at most only one misclassification amongst the classes. This provides great confidence in the methodology to identify single faults at the very least correctly. The remainder of the scenarios pertain to the definitions described in Table 2. As more units fail simultaneously, the performance of the classifier slightly degrades. This can be explained due to the impact of multiple faulty units on the overall system behavior. As the predictions are made based on the systemlevel measurements (i.e., satellite quaternions and angular velocities), it is expected for the predictors to identify similar behavior as contributed to by similar actuator faults. Therefore, in any fault scenario, the system-level impact from actuator faults is similar, the predictors will identify them as the same scenario and this can become more evident as more actuator units become faculty simultaneously.  Table 9. Confusion matrix for 3-axis orthogonal

Sensitivity Analysis
Although many MLMs were tested against the proposed method to simplify analysis results, only one machine learning classifier was used for sensitivity analysis. The model's sensitivity is evaluated for missing values, missing sensors, and noise.

Noise Analysis
To study the effects of noisy raw data on the model's performance, Gaussian noise was added at multiple signal-to-noise ratios. Gaussian noise with a zero mean was added during the study.

Missing Sensor Analysis
For this missing sensor analysis, the fault isolation method was tested under the assumption that not all sensors provide full measurement during the system's operation. Therefore, measurements from some of the sensors are missing due to one or multiple failed sensors. Sensor readings are represented by the satellite quaternions ( # , 8 , 3 ) and angular velocities ( # , 8 , 3 ) and the results for the analysis can be seen in Table 13, Table 14, and Table 15, respectively. In each of these tables, various combinations of working sensors (available measurements) are provided along with the isolation method accuracy for the various severity cases introduced in Section 4.1.5 (i.e., Cases 1, 2, and 3). As can be seen from the results in Table 13, Table 14, and  Table 15, when more measurements are available, the isolation accuracy is higher. As the number of failed sensors increases and there are fewer available measurements for the isolation method to work with, the accuracy decreases. Additionally, it can be observed that the accuracy of the isolation method decreases with the increase in case complexity (Cases

Missing Value Analysis
Sensory data containing missing values are not uncommon due to miscommunication between channels or sensor components. In this section, a complete analysis of the model's performance with missing values is evaluated. The simulated data does not initially contain any missing values. Therefore, missing values had to be generated manually at different percentages. Sklearn's Iterative Imputer was used for interpolation during the data pre-processing stage to ensure the proper calculation of missing values. To accomplish this, the iterative imputer modelled each feature with missing values as a function of other features using the round-robin method. To initialize the missing values, the mean along each column was used to replace the missing values. The results for the missing value analysis are presented in Table 16, Table 17,  and Table 18, where MMP refers to missing measurement percentage ranging from 0% for full measurements availability and up to 50% of missing measurements in the available data for the isolation method.  Table 16. Missing value analysis for Case 1 (abrupt fault) As can be seen from Table 16, Table 17, and Table 18, the accuracy of the isolation method decreases with the increase in the number of missing measurements.  Additionally, it can be observed that the accuracy of the isolation method decreases with the increase in complexity of the testing case (Cases 1, 2, and 3), where Case 1 is the least complex case, and Case 3 is the most complex case.

Validation and Learning Curve
As previously mentioned in Chapter 3, validation testing was conducted on the methodology. It is sometimes useful to plot the influences of a single hyperparameter instead of selecting multiple to find out whether the estimator selected is overfitting or underfitting hyperparameter values. Based on the training and validation score plot, if both are low, the estimator is underfitting the data. If the training score remains high and the validation score is low, the estimator is overfitting the data. Otherwise, if both are high, then the estimator is working well. Due to the random forest classifier being computational less expensive and having good accuracy results for the more practical case study scenario. It was selected as the estimator to plot the validation curves shown in Figure 4

Precision and Recall Curve
The precision and recall curve can summarize the tradeoff between the true positive rate and positive predictive value for a predictive model using various probability thresholds. Precision can be defined as a ratio of the number of true positives divided by the sum of the true positives and false positives and describes how well a model predicts positive classes using Precision = True Positives True Positives + False Positives (16) The recall is the ratio of true positives divided by the sum of the true positive and false negatives. It is the same as sensitivity as

Recall=
True Positives True Positives + False Negatives (17) A high area value under the curve represents high precision and recall, where a high precision translates to a low false positive rate, and a high recall relates to a low false negative rate. Thus, high results determine that the classifier is returning accurate results. The precision and recall curves for the three configurations can be seen in Figure 10, Figure 11, and Figure 12. These figures show the precision-recall curves for each class (fault scenario) that was tested under this study.

CONCLUSION
In this paper, the development of a data-driven fault isolation machine learning algorithm for nonlinear systems was investigated. The performance of the proposed method was evaluated for a case study involving FDI of an ADCS system using RWs as actuators onboard an in-orbit satellite. An automated feature extraction approach was implemented to transform the data into three different domains, after which the feature reduction was utilized to find the optimal number of features for training. The proposed fault isolation scheme was evaluated against various MLMs and with a thorough sensitivity analysis. The Random Forest classifier resulted in the overall highest accuracy achieved amongst the more applicable cases and the three configurations considered in this study. This provides enough confidence that the proposed method could be a successful candidate in such applications. The future scope of this work includes implementing a change point Figure 11. Precision and recall for standard 4-wheel configuration Figure 10. Precision and recall for 3-orthogonal wheel configuration detection algorithm on the more complex testing conditions, Case 3, to help better present any underlying patterns in the dataset. Expanding upon dataset cases is also essential, as this study has determined that the parameters used to develop the faulty dataset significantly impact the ability of the machine learning algorithm to detect a faulty condition correctly. Additionally, empirical data from in-orbit satellites can further enhance the applicability of the proposed method for real-life applications.