A Generic Machine Learning Framework for Fully-Unsupervised Anomaly Detection with Contaminated Data
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Abstract
Anomaly detection (AD) tasks have been solved using machine learning algorithms in various domains and applications. The great majority of these algorithms use normal data to train a residual-based model, and assign anomaly scores to unseen samples based on their dissimilarity with the learned normal regime. The underlying assumption of these approaches is that anomaly-free data is available for training. This is, however, often not the case in real-world operational settings, where the training data may be contaminated with a certain fraction of abnormal samples. Training with contaminated data, in turn, inevitably leads to a deteriorated AD performance of the residual-based algorithms.
In this paper we introduce a framework for a fully unsupervised refinement of contaminated training data for AD tasks. The framework is generic and can be applied to any residual-based machine learning model. We demonstrate the application of the framework to two public datasets of multivariate time series machine data from different application fields. We show its clear superiority over the naive approach of training with contaminated data without refinement. Moreover, we compare it to the ideal, unrealistic reference in which anomaly-free data would be available for training. Since the approach exploits information from the anomalies, and not only from the normal regime, it is comparable and often outperforms the ideal baseline as well.
##plugins.themes.bootstrap3.article.details##
Fully Unsupervised Learning, Deep Learning, Machine Learning, Anomaly Detection, Contaminated Data, Data Refinement, Time Series, Fault Detection, Turbofan Engine, Acoustic Sensor Data
Audibert, J., Michiardi, P., Guyard, F., Marti, S., & Zuluaga, M. A. (2020). Usad: Unsupervised anomaly detection on multivariate time series. In Proceedings of the 26th acm sigkdd international conference on knowledge discovery & data mining (pp. 3395–3404).
Beggel, L., Pfeiffer, M., & Bischl, B. (2020). Robust anomaly detection in images using adversarial autoencoders. In Machine learning and knowledge discovery in databases: European conference, ecml pkdd 2019, würzburg, germany, september 16–20, 2019, proceedings, part i (pp. 206–222).
Berg, A., Ahlberg, J., & Felsberg, M. (2019). Unsupervised learning of anomaly detection from contaminated image data using simultaneous encoder training. arXiv preprint arXiv:1905.11034.
Bergman, L., & Hoshen, Y. (2020). Classification-based anomaly detection for general data. arXiv preprint arXiv:2005.02359.
Chao, M. A., Kulkarni, C., Goebel, K., & Fink, O. (2022). Fusing physics-based and deep learning models for prognostics. Reliability Engineering & System Safety, 217, 107961.
Frederick, D. K., DeCastro, J. A., & Litt, J. S. (2007). User’s guide for the commercial modular aero-propulsion system simulation (c-mapss) (Tech. Rep.).
Golan, I., & El-Yaniv, R. (2018). Deep anomaly detection using geometric transformations. Advances in neural information processing systems, 31.
Hendrycks, D., Mazeika, M., & Dietterich, T. (2018). Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606.
Latecki, L. J., Lazarevic, A., & Pokrajac, D. (2007). Outlier detection with kernel density functions. In Mldm (Vol. 7, pp. 61–75).
Li, X., Ding, Q., & Sun, J.-Q. (2018). Remaining useful life estimation in prognostics using deep convolution neural networks. Reliability Engineering & System Safety, 172, 1–11.
Michau, G., Frusque, G., & Fink, O. (2022). Fully learnable deep wavelet transform for unsupervised monitoring of high-frequency time series. Proceedings of the National Academy of Sciences, 119(8), e2106598119.
Munir, M., Siddiqui, S. A., Dengel, A., & Ahmed, S. (2018). Deepant: A deep learning approach for unsupervised anomaly detection in time series. Ieee Access, 7, 1991–2005.
Purohit, H., Tanabe, R., Ichige, K., Endo, T., Nikaido, Y., Suefusa, K., & Kawaguchi, Y. (2019). Mimii dataset: Sound dataset for malfunctioning industrial machine investigation and inspection. arXiv preprint arXiv:1909.09347.
Qiu, C., Li, A., Kloft, M., Rudolph, M., & Mandt, S. (2022). Latent outlier exposure for anomaly detection with contaminated data. In International conference on machine learning (pp. 18153–18167).
Saxena, A., & Goebel, K. (2008). Turbofan engine degradation simulation data set. NASA Prognostics Data Repository. Moffett Field, CA.
Schneider, T., Qiu, C., Kloft, M., Latif, D. A., Staab, S., Mandt, S., & Rudolph, M. (2022). Detecting anomalies within time series using local neural transformations. arXiv preprint arXiv:2202.03944.
Schölkopf, B., Williamson, R. C., Smola, A., Shawe-Taylor, J., & Platt, J. (1999). Support vector method for novelty detection. Advances in neural information processing systems, 12.
Shenkar, T., & Wolf, L. (2022). Anomaly detection for tabular data with internal contrastive learning. In International conference on learning representations.
Sohn, K., Li, C.-L., Yoon, J., Jin, M., & Pfister, T. (2020). Learning and evaluating representations for deep one-class classification. arXiv preprint arXiv:2011.02578.
Tax, D. M., & Duin, R. P. (2004). Support vector data description. Machine learning, 54, 45–66.
Wang, G., Zhan, Y., Wang, X., Song, M., & Nahrstedt, K. (2022). Hierarchical semi-supervised contrastive learning for contamination-resistant anomaly detection. In European conference on computer vision (pp. 110–128).
Yoon, J., Sohn, K., Li, C.-L., Arik, S. O., Lee, C.-Y., & Pfister, T. (2021). Self-supervise, refine, repeat: Improving unsupervised anomaly detection. arXiv preprint arXiv:2106.06115.
Zgraggen, J., Guo, Y., Notaristefano, A., & Goren-Huber, L. (2023). Fully unsupervised fault detection in solar power plants using physics-informed deep learning. (To be published)
Zhang, C., Song, D., Chen, Y., Feng, X., Lumezanu, C., Cheng, W., . . . Chawla, N. V. (2019). A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. In Proceedings of the aaai conference on artificial intelligence (Vol. 33, pp. 1409–1416).
Zhou, C., & Paffenroth, R. C. (2017). Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining (pp. 665–674).
Zong, B., Song, Q., Min, M. R., Cheng, W., Lumezanu, C., Cho, D., & Chen, H. (2018). Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International conference on learning representations.