Diverse Neuron Type Selection for Convolutional Neural Networks

Diverse Neuron Type Selection for Convolutional Neural Networks

Guibo Zhu, Zhaoxiang Zhang, Xu-Yao Zhang, Cheng-Lin Liu

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
Main track. Pages 3560-3566. https://doi.org/10.24963/ijcai.2017/498

The activation function for neurons is a prominent element in the deep learning architecture for obtaining high performance. Inspired by neuroscience findings, we introduce and define two types of neurons with different activation functions for artificial neural networks: excitatory and inhibitory neurons, which can be adaptively selected by self-learning. Based on the definition of neurons, in the paper we not only unify the mainstream activation functions, but also discuss the complementariness among these types of neurons. In addition, through the cooperation of excitatory and inhibitory neurons, we present a compositional activation function that leads to new state-of-the-art performance comparing to rectifier linear units. Finally, we hope that our framework not only gives a basic unified framework of the existing activation neurons to provide guidance for future design, but also contributes neurobiological explanations which can be treated as a window to bridge the gap between biology and computer science.
Keywords:
Machine Learning: Neural Networks
Machine Learning: Deep Learning