Anomaly Detection in Time Series Data A Novel Approach using Transformer Neural Networks for Reconstruction and Residual Analysis
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
https://orcid.org/0000-0002-1612-9840
Erik Vanem
Knut Erik Knutsen Vilmar Æsøy Houxiang Zhang
Abstract
This paper presents a novel unsupervised approach for anomaly detection in marine diesel engines using a Transformer Neural Network based autoencoder (TAE) and residual analysis with Sequential Probability Ratio Test (SPRT) and Sum of Squares of Normalized Residuals (SSNR). This approach adeptly captures temporal dependencies within normal time-series data without the necessity for labeled failure data. To assess the performance of the proposed methodology, a dataset containing faulty data is generated under the same operational profile as the normal training data. The model undergoes training using normal data, after which the faulty data is reconstructed utilizing the trained model. Subsequently, SPRT and SSNR are used to analyze the residuals from the observed and reconstructed faulty data. Significant deviations surpassing a predefined threshold are identified as anomalous behavior. Additionally, this study explores various architectures of Transformer neural networks and other types of neural networks to conduct a comprehensive comparative analysis of the performance of the proposed approach. Insights and recommendations derived from the performance analysis are also presented, offering valuable information for potential users to leverage. The experimental results demonstrate that the proposed approach can accurately and efficiently detect anomalies in marine diesel engines. Therefore, this approach can be considered as a promising solution for early anomaly detection, leading to timely maintenance and repair, and preventing costly downtime. Also, avoid accidents with probability severe consequences.
##plugins.themes.bootstrap3.article.details##
Marine Diesel Engine, PHM, Deep Learning, Machine Learning
Brandsæter, A., Manno, G., Vanem, E., & Glad, I. K. (2016). An application of sensor-based anomaly detection in the maritime industry. In 2016 ieee international conference on prognostics and health management (icphm) (pp. 1–8).
Brandsæter, A., Vanem, E., & Glad, I. K. (2019). Efficient online anomaly detection for ship systems in operation. Expert Systems with Applications, 121, 418–437.
Ellefsen, A. L., Bjørlykhaug, E., Æsøy, V., Ushakov, S., & Zhang, H. (2019). Remaining useful life predictions for turbofan engine degradation using semi-supervised deep architecture. Reliability Engineering & System Safety, 183, 240–251.
Han, P., Ellefsen, A. L., Li, G., Holmeset, F. T., & Zhang, H. (2021). Fault detection with lstm-based variational autoencoder for maritime components. IEEE Sensors Journal, 21(19), 21903-21912. doi: 10.1109/JSEN.2021.3105226
Han, P., Li, G., Skulstad, R., Skjong, S., & Zhang, H. (2020). A deep learning approach to detect and isolate thruster failures for dynamically positioned vessels using motion data. IEEE Transactions on Instrumentation and Measurement, 70, 1–11.
Hemmer, M., Klausen, A., Khang, H. V., Robbersmyr, K. G., & Waag, T. I. (2020). Health indicator for low-speed axial bearings using variational autoencoders. IEEE Access, 8, 35842-35852. doi: 10.1109/ACCESS.2020.2974942
Hongchun, Y., Fanlun, X., & Xiaoyong, H. (2001). A method for deciding the number of hidden neurons of the feedforward neural networks. IFAC Proceedings Volumes, 34(26), 31-35. (4h IFAC/CIGR Workshop on Artificial Intelligence in Agriculture 2001, Budapest, Hungary, 6-8 June 2001) doi: https://doi.org/10.1016/S1474-6670(17)33628-5
Hu, K., Cheng, Y., Wu, J., Zhu, H., & Shao, X. (2021). Deep bidirectional recurrent neural networks ensemble for remaining useful life prediction of aircraft engine. IEEE Transactions on Cybernetics.
Knutsen, K. E., Liang, Q., Karandikar, N., Ibrahim, I. H. B., Tong, X. G. T., & Tam, J. J. H. (2022). Containerized immutable maritime data sharing utilizing distributed ledger technologies. In Journal of physics: Conference series (Vol. 2311, p. 012006).
Kriegeskorte, N., & Golan, T. (2019). Neural network models and deep learning. Current Biology, 29(7), R231–R236.
Liang, Q., Knutsen, K. E., Vanem, E., Zhang, H., & Æsøy, V. (2023). Unsupervised anomaly detection in marine diesel engines using transformer neural networks and residual analysis. In Phm society asia-pacific conference (Vol. 4).
Liang, Q., Knutsen, K. E., Vanem, E., Æsøy, V., & Zhang, H. (2024). A review of maritime equipment prognostics health management from a classification society perspective. Ocean Engineering, 301, 117619. doi: https://doi.org/10.1016/j.oceaneng.2024.117619
Liang, Q., Tvete, H., & Brinks, H. (2020). Prediction of vessel propulsion power from machine learning models based on synchronized ais-, ship performance measurements and ecmwf weather data. In Iop conference series: Materials science and engineering (Vol. 929, p. 012012).
Liang, Q., Tvete, H. A., & Brinks, H. W. (2019). Prediction of vessel propulsion power using machine learning on ais data, ship performance measurements and weather data. In Journal of physics: Conference series (Vol. 1357, p. 012038).
Listou Ellefsen, A., Han, P., Cheng, X., Holmeset, F. T., Æsøy, V., & Zhang, H. (2020). Online fault detection in autonomous ferries: Using fault-type independent spectral anomaly detection. IEEE Transactions on Instrumentation and Measurement, 69(10), 8216-8225. doi: 10.1109/TIM.2020.2994012
Massoudi, M., Verma, S., & Jain, R. (2021). Urban sound classification using cnn. In 2021 6th international conference on inventive computation technologies (icict) (pp. 583–589).
Pukelsheim, F. (1994). The three sigma rule. The American Statistician, 48(2), 88–91.
Stalk, P. (2021). Review of maritime transport. UNITED NATIONS CONFERENCE ON TRADE AND DEVELOPMENT, 1-177.
Tuli, S., Casale, G., & Jennings, N. R. (2022). Tranad: Deep transformer networks for anomaly detection in multivariate time series data. arXiv preprint arXiv:2201.07284.
Vanem, E., & Brandsæter, A. (2021). Unsupervised anomaly detection based on clustering methods and sensor data on a marine diesel engine. Journal of Marine Engineering & Technology, 20(4), 217–234.
Vanem, E., & Storvik, G. O. (2017). Anomaly detection using dynamical linear models and sequential testing on a marine engine system. In Annual conference of the phm society (Vol. 9).
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., . . . Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
Wald, A. (1992). Sequential tests of statistical hypotheses. Springer.
Zhang, Z., Song, W., & Li, Q. (2022). Dual-aspect self-attention based on transformer for remaining useful life prediction. IEEE Transactions on Instrumentation and Measurement, 71, 1-11. doi: 10.1109/TIM.2022.3160561