Regularizing Deep Neural Networks with an Ensemble-based Decorrelation Method

Regularizing Deep Neural Networks with an Ensemble-based Decorrelation Method

Shuqin Gu, Yuexian Hou, Lipeng Zhang, Yazhou Zhang

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
Main track. Pages 2177-2183. https://doi.org/10.24963/ijcai.2018/301

Although Deep Neural Networks (DNNs) have achieved excellent performance in many tasks, improving the generalization capacity of DNNs still remains a challenge. In this work, we propose a novel regularizer named Ensemble-based Decorrelation Method (EDM), which is motivated by the idea of the ensemble learning to improve generalization capacity of DNNs. EDM can be applied to hidden layers in fully connected neural networks or convolutional neural networks. We treat each hidden layer as an ensemble of several base learners through dividing all the hidden units into several non-overlap groups, and each group will be viewed as a base learner. EDM encourages DNNs to learn more diverse representations by minimizing the covariance between all base learners during the training step. Experimental results on MNIST and CIFAR datasets demonstrate that EDM can effectively reduce the overfitting and improve the generalization capacity of DNNs  
Keywords:
Machine Learning: Machine Learning
Machine Learning: Neural Networks
Machine Learning: Deep Learning