Unsupervised Feature Learning Using Domain Knowledge Based Autoencoder
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Jul 14, 2017
Hyunjae Kim
Byeng D. Youn
Abstract
Autoencoder is an unsupervised feature engineering technique, which is an emerging technique in the prognostics and health management (PHM) domain. However, since neural network based techniques such as auto-encoder have many hyper-parameters belonging to the number of hidden
units, the number of layers and activation functions, substantial efforts are required in a heuristic way to make the autoencoder learn proper features. In this paper, we propose a novel method to regularize an auto-encoder by exploiting domain knowledge, such as mechanical engineering
expertise. The proposed autoencoder learns robust features in a fast and efficient manner, therefore resulting in minimal consideration for the hyper-parameters. In the proposed method, some of the hidden units of the autoencoder are forcibly pre-trained by back-bone signals, such as 1X
sinusoidal wave of vibration, and the remaining hidden units efficiently learn the features of fault signals by minimizing the redundancy of learned features. The domain knowledge based regularization reduces the degree of freedom (DOF) of the autoencoder model as well as guide the model to learn the more physically reasonable features. Various fault data measured from a journal bearing rotor testbed are used for demonstration of the proposed method.
##plugins.themes.bootstrap3.article.details##
Keywords
PHM
References
Alain, G., & Bengio, Y. (2014). What regularized autoencoders learn from the data-generating distribution. Journal of Machine Learning Research, 15(1), 3563-3593.
Ben-David, S., Blitzer, J., Crammer, K., & Pereira, F. (2007). Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 137.
Bengio, Y. (2012). Deep learning of representations for unsupervised and transfer learning. ICML Unsupervised and Transfer Learning, 27, 17-36.
Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Transactions on pattern analysis and machine intelligence, 35(8), 1798-1828.
Bengio, Y., Paiement, J.-F., Vincent, P., Delalleau, O., Le Roux, N., & Ouimet, M. (2003). Out-of-sample extensions for lle, isomap, mds, eigenmaps, and
spectral clustering. Mij, 1, 2.
Bengio, Y., Yao, L., Alain, G., & Vincent, P. (2013). Generalized denoising auto-encoders as generative models. Paper presented at the Advances in neural information processing systems.
Bergstra, J. S., Bardenet, R., Bengio, Y., & Kégl, B. (2011). Algorithms for hyper-parameter optimization. Paper presented at the Advances in neural information processing systems.
Chen, M., Weinberger, K. Q., & Blitzer, J. (2011). Cotraining for domain adaptation. Paper presented at the Advances in neural information processing
systems.
Chen, M., Xu, Z., Weinberger, K., & Sha, F. (2012). Marginalized denoising autoencoders for domain adaptation. arXiv preprint arXiv:1206.4683.
Dauphin, Y. N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. (2014). Identifying and attacking the saddle point problem in high-dimensional nonconvex optimization. Paper presented at the Advances in neural information processing systems.
Doersch, C. (2016). Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908.
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., . . . Lempitsky, V. (2016). Domain-adversarial training of neural networks.
Journal of Machine Learning Research, 17(59), 1-35.
Glorot, X., Bordes, A., & Bengio, Y. (2011a). Deep Sparse Rectifier Neural Networks. Paper presented at the Aistats.
Glorot, X., Bordes, A., & Bengio, Y. (2011b). Domain adaptation for large-scale sentiment classification: A deep learning approach. Paper presented at the Proceedings of the 28th international conference on machine learning (ICML-11).
Goodfellow, I., Lee, H., Le, Q. V., Saxe, A., & Ng, A. Y. (2009). Measuring invariances in deep networks. Paper presented at the Advances in neural
information processing systems.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde- Farley, D., Ozair, S., . . . Bengio, Y. (2014). Generative adversarial nets. Paper presented at the Advances in neural information processing systems.
Gopalan, R., Li, R., & Chellappa, R. (2011). Domain adaptation for object recognition: An unsupervised approach. Paper presented at the Computer Vision (ICCV), 2011 IEEE International Conference on.
Goroshin, R., & LeCun, Y. (2013). Saturating auto-encoders. arXiv preprint arXiv:1301.3577. Im, D. J., Kim, C. D., Jiang, H., & Memisevic, R. (2016). Generating images with recurrent adversarial networks. arXiv preprint arXiv:1602.05110.
Kingma, D. P., Mohamed, S., Rezende, D. J., & Welling, M. (2014). Semi-supervised learning with deep generative models. Paper presented at the Advances in neural information processing systems.
LeCun, Y. A., Bottou, L., Orr, G. B., & Müller, K.-R. (2012). Efficient backprop Neural networks: Tricks of the trade (pp. 9-48): Springer.
Makhzani, A., & Frey, B. (2013). K-sparse autoencoders. arXiv preprint arXiv:1312.5663.
Makhzani, A., & Frey, B. J. (2015). Winner-take-all autoencoders. Paper presented at the Advances in neural information processing systems.
Ming Harry Hsu, T., Yu Chen, W., Hou, C.-A., Hubert Tsai, Y.-H., Yeh, Y.-R., & Frank Wang, Y.-C. (2015). Unsupervised Domain Adaptation with Imbalanced Cross-Domain Data. Paper presented at the Proceedings of the IEEE International Conference on Computer Vision.
Pan, S. J., & Yang, Q. (2010). A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10), 1345-1359.
Raina, R., Battle, A., Lee, H., Packer, B., & Ng, A. Y. (2007). Self-taught learning: transfer learning from unlabeled data. Paper presented at the Proceedings of the 24th international conference on Machine learning.
Rifai, S., Mesnil, G., Vincent, P., Muller, X., Bengio, Y., Dauphin, Y., & Glorot, X. (2011). Higher order contractive auto-encoder. Machine Learning and
Knowledge Discovery in Databases, 645-660.
Rifai, S., Vincent, P., Muller, X., Glorot, X., & Bengio, Y. (2011). Contractive auto-encoders: Explicit invariance during feature extraction. Paper
presented at the Proceedings of the 28th international conference on machine learning (ICML-11).
Sanguinetti, G. (2008). Dimensionality reduction of clustered data sets. IEEE Transactions on pattern analysis and machine intelligence, 30(3), 535-540.
Skolidis, G. (2012). Transfer learning with Gaussian processes.
Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1), 1929-1958.
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015). Unsupervised Learning of Video Representations using LSTMs. Paper presented at the ICML.
Van Der Maaten, L., Postma, E., & Van den Herik, J. (2009). Dimensionality reduction: a comparative. J Mach Learn Res, 10, 66-71.
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. Paper presented at the Proceedings of the 25th international conference on Machine learning.
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P.-A. (2010). Stacked denoising autoencoders: Learning useful representations in a
deep network with a local denoising criterion. Journal of Machine Learning Research, 11(Dec), 3371-3408.
Yu, D., & Deng, L. (2011). Deep Learning and Its Applications to Signal and Information Processing [Exploratory DSP. IEEE Signal Processing Magazine, 28(1), 145-154. doi:10.1109/msp.2010.939038
Zhang, C., Zhang, L., & Ye, J. (2012). Generalization bounds for domain adaptation. Paper presented at the Advances in neural information processing systems.
Ben-David, S., Blitzer, J., Crammer, K., & Pereira, F. (2007). Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 137.
Bengio, Y. (2012). Deep learning of representations for unsupervised and transfer learning. ICML Unsupervised and Transfer Learning, 27, 17-36.
Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Transactions on pattern analysis and machine intelligence, 35(8), 1798-1828.
Bengio, Y., Paiement, J.-F., Vincent, P., Delalleau, O., Le Roux, N., & Ouimet, M. (2003). Out-of-sample extensions for lle, isomap, mds, eigenmaps, and
spectral clustering. Mij, 1, 2.
Bengio, Y., Yao, L., Alain, G., & Vincent, P. (2013). Generalized denoising auto-encoders as generative models. Paper presented at the Advances in neural information processing systems.
Bergstra, J. S., Bardenet, R., Bengio, Y., & Kégl, B. (2011). Algorithms for hyper-parameter optimization. Paper presented at the Advances in neural information processing systems.
Chen, M., Weinberger, K. Q., & Blitzer, J. (2011). Cotraining for domain adaptation. Paper presented at the Advances in neural information processing
systems.
Chen, M., Xu, Z., Weinberger, K., & Sha, F. (2012). Marginalized denoising autoencoders for domain adaptation. arXiv preprint arXiv:1206.4683.
Dauphin, Y. N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. (2014). Identifying and attacking the saddle point problem in high-dimensional nonconvex optimization. Paper presented at the Advances in neural information processing systems.
Doersch, C. (2016). Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908.
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., . . . Lempitsky, V. (2016). Domain-adversarial training of neural networks.
Journal of Machine Learning Research, 17(59), 1-35.
Glorot, X., Bordes, A., & Bengio, Y. (2011a). Deep Sparse Rectifier Neural Networks. Paper presented at the Aistats.
Glorot, X., Bordes, A., & Bengio, Y. (2011b). Domain adaptation for large-scale sentiment classification: A deep learning approach. Paper presented at the Proceedings of the 28th international conference on machine learning (ICML-11).
Goodfellow, I., Lee, H., Le, Q. V., Saxe, A., & Ng, A. Y. (2009). Measuring invariances in deep networks. Paper presented at the Advances in neural
information processing systems.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde- Farley, D., Ozair, S., . . . Bengio, Y. (2014). Generative adversarial nets. Paper presented at the Advances in neural information processing systems.
Gopalan, R., Li, R., & Chellappa, R. (2011). Domain adaptation for object recognition: An unsupervised approach. Paper presented at the Computer Vision (ICCV), 2011 IEEE International Conference on.
Goroshin, R., & LeCun, Y. (2013). Saturating auto-encoders. arXiv preprint arXiv:1301.3577. Im, D. J., Kim, C. D., Jiang, H., & Memisevic, R. (2016). Generating images with recurrent adversarial networks. arXiv preprint arXiv:1602.05110.
Kingma, D. P., Mohamed, S., Rezende, D. J., & Welling, M. (2014). Semi-supervised learning with deep generative models. Paper presented at the Advances in neural information processing systems.
LeCun, Y. A., Bottou, L., Orr, G. B., & Müller, K.-R. (2012). Efficient backprop Neural networks: Tricks of the trade (pp. 9-48): Springer.
Makhzani, A., & Frey, B. (2013). K-sparse autoencoders. arXiv preprint arXiv:1312.5663.
Makhzani, A., & Frey, B. J. (2015). Winner-take-all autoencoders. Paper presented at the Advances in neural information processing systems.
Ming Harry Hsu, T., Yu Chen, W., Hou, C.-A., Hubert Tsai, Y.-H., Yeh, Y.-R., & Frank Wang, Y.-C. (2015). Unsupervised Domain Adaptation with Imbalanced Cross-Domain Data. Paper presented at the Proceedings of the IEEE International Conference on Computer Vision.
Pan, S. J., & Yang, Q. (2010). A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10), 1345-1359.
Raina, R., Battle, A., Lee, H., Packer, B., & Ng, A. Y. (2007). Self-taught learning: transfer learning from unlabeled data. Paper presented at the Proceedings of the 24th international conference on Machine learning.
Rifai, S., Mesnil, G., Vincent, P., Muller, X., Bengio, Y., Dauphin, Y., & Glorot, X. (2011). Higher order contractive auto-encoder. Machine Learning and
Knowledge Discovery in Databases, 645-660.
Rifai, S., Vincent, P., Muller, X., Glorot, X., & Bengio, Y. (2011). Contractive auto-encoders: Explicit invariance during feature extraction. Paper
presented at the Proceedings of the 28th international conference on machine learning (ICML-11).
Sanguinetti, G. (2008). Dimensionality reduction of clustered data sets. IEEE Transactions on pattern analysis and machine intelligence, 30(3), 535-540.
Skolidis, G. (2012). Transfer learning with Gaussian processes.
Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1), 1929-1958.
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015). Unsupervised Learning of Video Representations using LSTMs. Paper presented at the ICML.
Van Der Maaten, L., Postma, E., & Van den Herik, J. (2009). Dimensionality reduction: a comparative. J Mach Learn Res, 10, 66-71.
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. Paper presented at the Proceedings of the 25th international conference on Machine learning.
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P.-A. (2010). Stacked denoising autoencoders: Learning useful representations in a
deep network with a local denoising criterion. Journal of Machine Learning Research, 11(Dec), 3371-3408.
Yu, D., & Deng, L. (2011). Deep Learning and Its Applications to Signal and Information Processing [Exploratory DSP. IEEE Signal Processing Magazine, 28(1), 145-154. doi:10.1109/msp.2010.939038
Zhang, C., Zhang, L., & Ye, J. (2012). Generalization bounds for domain adaptation. Paper presented at the Advances in neural information processing systems.
Section
Special Session Papers