A Genetic Programming Approach to Designing Convolutional Neural Network Architectures

1 Introduction

Deep neural networks (DNNs) have shown excellent performance on many challenging machine learning tasks, such as image recognition, speech recognition, and reinforcement learning tasks. Convolutional neural networks (CNNs) [LeCun et al., 1998], the DNN model often used for computer vision tasks, have seen huge success, particularly in image recognition tasks in the past few years. A standard CNN architecture consists of several convolutions, pooling, and fully connected layers. Recent studies have proposed a new CNN architecture that achieves higher performance, e.g., GoogLeNet [Szegedy et al., 2015] and ResNet [He et al., 2016]. These widely-used networks are designed by humans, but designing such architectures requires expert knowledge and trial-and-error. Hence, the automatic design of CNN architectures has much attention.

Recent studies for the automatic design of CNN require many weight-parameter optimizations; the weights of candidate CNN architectures are trained by a stochastic gradient descent (SGD), and the architectures are evaluated with trained weights on a validation dataset 1 . Evolutionary computation (EC) or reinforcement learning (RL) is employed as an architecture search method. The EC based architecture search methods [Miikkulainen et al., 2017; Real et al., 2017; Xie and Yuille, 2017] generate a new architecture by recombination and mutation operators, and evaluate it on a validation dataset after the weight training. The EC algorithms are expected to find the architecture that maximizes the performance (e.g., classification accuracy). The RL based architecture search methods [Baker et al., 2017; Zoph and Le, 2017] optimize the policy that generates the CNN architectures by such as the policy gradient and Qlearning. Also, the reward used in the RL algorithms is the performance on a validation dataset. Reducing the computational cost for the architecture search is crucial topic because these methods require much computational cost to optimize neural network architectures.

This paper is an abridged version of the paper [Suganuma et al., 2017a] that won a best-paper award at the evolutionary machine learning (EML) track of GECCO 2017 conference.

In this work, we attempt to design CNN architectures based on genetic programming. We use the Cartesian genetic programming (CGP) [Miller and Thomson, 2000] encoding scheme to represent the CNN architecture, where the architecture is represented by a directed acyclic graph. The advantage of this representation is its flexibility; it can represent variable-length network architectures and skip connections. In addition, we adopt relatively highly-functional modules such as convolutional blocks and tensor concatenation as the node functions in the graph to reduce the search space of architectures. To evaluate the architecture represented by CGP, we train the network by SGD on a training dataset. The performance on a validation dataset is then assigned as the fitness of the architecture. Based on this fitness evaluation, the architecture is optimized to maximize the fitness (i.e., the validation accuracy) by evolutionary algorithms. To verify the performance of the proposed approach, we experimented involving constructing a CNN architecture for the image classification task with the CIFAR-10 and CIFAR-100 datasets. The experimental result shows that the proposed method can be used to find competitive CNN architectures compared with state-of-the-art models automatically.

1 In contrast, the traditional evolutionary neural networks, socalled neuroevolution [Stanley and Miikkulainen, 2002], typically optimizes both of the weights and architecture at the same time by an evolutionary computation method.

2 CNNArchitecture Design Using Cartesian Genetic Programming

2.1 Representation of CNNArchitectures

We represent the architecture of CNNs by a directed acyclic graph defined on a two-dimensional grid of M rows and N columns. This graph is optimized by an evolutionary algorithm, where the graph called phenotype is encoded by a list of integers called genotype. Figure 1 shows an example of a genotype (top), a phenotype (middle) and the corresponding CNN architecture (bottom).

A genotype consists of MN + 1 genes, and each gene has information regarding the type T and connections C of a node. The node corresponds to a highly-functional module in CNN, e.g., the type T specifies one of the convolution or residual blocks, pooling layers, and tensor operations described later in detail. The connections C, specifying the node number in the anterior columns than the target node, represent which nodes are the input to the target node. This connection restriction ensures the feed-forward network structure. The last (MN + 1)-st gene, having only connection information C, represents the output layer, and its type is fixed based on the task.

In the CGP encoding scheme, there may be the nodes that do not connect to the output node; we call these nodes nonactive nodes. In contrast, we define the nodes that connect to the output node as active nodes. Note that nodes depicted in the neighboring two columns are not necessarily connected. Thus, the resulting CNN architecture can have a different number of modules (layers) depending on the node connections, i.e., the number of layers is decided by evolutionary algorithms. Note that the maximum depth of a network is N. To control how the number of layers will be chosen, we introduce a hyper-parameter called level-back L, such that nodes given in the n-th column are allowed to be connected from nodes given in the columns ranging from max(0, n - L) to n-1, where the zeroth column indicates the inputs. If we use smaller L, then the resulting CNNs will tend to be deeper.

2.2 Node Functions

Referring to the modern CNN architectures, we select the highly-functional modules as node functions. We prepare the six types of node functions called ConvBlock, ResBlock, max pooling, average pooling, concatenation, and summation.

The ConvBlock consists of standard convolution processing with a stride of 1 followed by batch normalization [Ioffe and Szegedy, 2015] and rectified linear units (ReLU) [Nair and Hinton, 2010]. The ResBlock is composed of a convolution processing, batch normalization, ReLU, and tensor summation. The ResBlock performs an identity mapping by shortcut connections as described in [He et al., 2016]. For the number F and receptive field size k of filters of ConvBlock and ResBlock, we chose them from F ∈ {32, 64, 128} and k ∈ {3 × 3, 5 × 5}, respectively.

The max and average poolings perform a max and average operation, respectively, over the local neighbors of the feature Genotype maps. We use the pooling with the 2 × 2 receptive field size and the stride of 2.

The concatenation function concatenates two feature maps in the channel dimension. If the input feature maps to be concatenated have different numbers of rows or columns, we down-sample the larger feature maps by max pooling so that they become the same sizes of the inputs. The summation performs the element-wise addition of two feature maps, channel by channel. In the same way as the concatenation, if the input feature maps to be added have different numbers of rows or columns, we down-sample the larger feature maps by max pooling. In addition, if the inputs have different numbers of channels, we pad the smaller feature maps with zeros for increasing channels. In Figure 1, the summation node performs max pooling to the first input so as to get the same input tensor sizes.

The output node represents the softmax function with the number of classes. The outputs fully connect to all elements of the input.

2.3 Evolutionary Algorithm

We use a point mutation as the genetic operator in the same way as the standard CGP. The type and connections of each node randomly change to valid values according to a mutation rate. The standard CGP mutation has the possibility of affecting only non-active nodes. In that case, the phenotype does not change by the mutation and does not require a fitness evaluation again. The fitness evaluation of the CNN architectures is so expensive because it requires the training of the CNN. To use the computational resource efficiently, we apply the mutation operator until at least one active node changes for reproducing the candidate solution. We call this mutation a forced mutation. Moreover, to maintain a neutral drift, which is effective for CGP evolution [Miller and Smith, 2006; Miller and Thomson, 2000], we change only the non-active nodes of a parent by the mutation if the fitnesses of the offsprings do not improve. We call this mutation a neutral mutation.

We use the modified (1 + λ) evolutionary strategy (with λ = 2 in our experiments). The procedure of our modified algorithm is as follows:

1. Generate an initial individual at random as parent P , and train the CNN represented by P followed by assigning the validation accuracy as the fitness.
2. Generate a set of λ offsprings O by applying the forced mutation to P .
3. Train the λ CNNs represented by offsprings O in parallel, and assign the validation accuracies as the fitness.
4. Apply the neutral mutation to parent P.
5. Select an elite individual from the set of P and O, and then replace P with the elite individual.
6. Return to step 2 until a stopping criterion is satisfied.

3 Experiments and Results

3.1 Dataset

We test our method on the image classification task using the CIFAR-10 and CIFAR-100 datasets in which the number of classes is 10 and 100, respectively. The numbers of training and test images are 50, 000 and 10, 000, respectively, and the size of images is 32 × 32. We randomly sample 45, 000 images from the training set to train the CNN, and we use the remaining 5, 000 images for the validation set of the CGP fitness evaluation.

We preprocess the data with the per-pixel mean subtraction. To prevent overfitting, we use a weight decay with the coefficient 1.0×10-4 and data augmentation. We use the data augmentation method based on [He et al., 2016]: padding 4 pixels on each side followed by choosing a random 32 × 32 crop from the padded image, and random horizontal flips on the cropped 32 × 32 image.

3.2 Experimental Setting

To assign the fitness to the candidate CNN architectures, we train the CNN by SGD with a mini-batch size of 128. The softmax cross-entropy loss is used as the loss function. We initialize the weights by the He's method [He et al., 2015] and use the Adam optimizer [Kingma and Ba, 2015] with β1 = 0.9, β2 = 0.999, ε = 1.0 × 10-8 , and an initial learning rate of 0.01. We train each CNN for 50 epochs and reduce the learning rate by a factor of 10 at 30th epoch.

For the parameter setting for CGP, we chose the mutation rate as r = 0.05, the graph with M = 5 and N = 30, and level-back L = 10. The offspring size of λ is set to two; that is the same number of GPUs in our experimental machines. We test two node function sets called ConvSet and ResSet for our method. The ConvSet contains ConvBlock, max pooling, average pooling, concatenation, and summation, and the ResSet contains ResBlock, max pooling, average pooling, concatenation, and summation. The difference between these two function sets is whether we adopt ConvBlock or ResBlock. The numbers of generations are 500 for ConvSet, 300 for ResSet.

After the CGP process, we re-train the best CNN architecture using 50, 000 training images, and we calculate the classification accuracy for the 10, 000 test images to evaluate the designed CNN architectures.

In this re-training phase, we optimize the weights of the obtained architecture for 500 epochs with a different training procedure; we use SGD with a momentum of 0.9, a minibatch size of 128, and a weight decay of 5.0 × 10-4 . We start a learning rate of 0.01 and set it to 0.1 at 5th epoch, then we reduce it to 0.01 and 0.001 at 250th and 375th epochs, respectively. This learning rate schedule is based on the reference [He et al., 2016]. We run the proposed method on the machine with 3.2GHz CPU, 32GB RAM, and two NVIDIA GeForce GTX 1080 (or two GTX 1080 Ti) GPUs.

3.3 Results

We run our method for three times on each dataset and report the classification performance. We compare the classification performance of our method with the state-of-the-art methods and summarize the classification error rates in Table 1. We refer to the architectures designed by our method as CGP-CNN. For instance, CGP-CNN (ConvSet) means the proposed method with the node function set of ConvSet. The models, Network in Network, VGG, ResNet, FractalNet, and Wide ResNet are hand-crafted CNN architectures, whereas the CoDeepNEAT, MetaQNN, Genetic CNN, Largescale Evolution, and Neural Architecture Search are the models constructed by the architecture search method.

As can be seen in Table 1, the error rates of our methods are competitive with the state-of-the-art methods. In particular, CGP-CNN (ResSet) shows good performance, and the architectures constructed by using our method have a good balance between classification errors, the numbers of parameters and GPUs. The Neural Architecture Search achieved the best error rate on the CIFAR-10 dataset, but this method used 800 GPUs for the architecture search. Our method can find a competitive architecture with a reasonable machine resource.

Examples of CNN architectures designed by our method are shown in [Suganuma et al., 2017a; 2017b]. We can observe that these architectures are quite different; the concatenation and summation nodes are not frequently used in CGPCNN (ResSet), whereas these nodes are frequently used in CGP-CNN (ConvSet). These nodes lead a wide network; therefore, the network of CGP-CNN (ConvSet) is a wider structure than that of CGP-CNN (ResSet).

4 RelatedWork

Some architecture search methods based on either EC [Miikkulainen et al., 2017; Real et al., 2017; Xie and Yuille, 2017] or RL [Baker et al., 2017; Zoph and Le, 2017] for DNNs have been proposed at nearly the same time with our work. Since these approaches require much computational cost in general, the recent works concentrate on the reduction of computational cost and improvement of architecture search efficiency [Cai et al., 2018; Liu et al., 2017; 2018; Pham et al., 2018; Real et al., 2018; Zhong et al., 2017; Zoph et al., 2017]. For instance, Cai et al. [Cai et al., 2018] specify an initial architecture by an existing one, and the actions, deepening the architecture by adding a layer or widening an existing layer by replacing an existing layer with a wider layer, are applied to the initial architecture to generate a new architecture. Pham et al. [Pham et al., 2018] have introduced the parameter sharing across models during the architecture search and achieved a test error of 2.89% on the CIFAR-10 dataset with one GPU.

Most methods of the CNN architecture search are applied to image classification tasks such as using CIFAR-10. They, however, can be naturally applied to other computer vision tasks and DNNs. The architecture search method explained in this paper has applied to automatically construct the autoencoder typed CNNs for the image denoising and inpainting tasks [Suganuma et al., 2018]. The result shows that the designed architectures outperform the existing state-of-the-art hand-crafted architectures on both denoising and inpainting tasks.

5 Conclusion

In this work, we have attempted to take a GP-based approach for designing the CNN architectures and have verified its potential. The proposed method designs the CNN architectures based on CGP and adopts the highly-functional modules, such as convolutional blocks and tensor operations, for searching for the adequate architectures efficiently. We have constructed the CNN architecture for the image classification task with the CIFAR-10 and CIFAR-100 datasets. The experimental result showed that the proposed method can automatically find competitive CNN architectures compared with the state-of-the-art models.

One direction of future work is to develop the evolutionary algorithm to reduce the computational cost of the architecture design, e.g., increasing the training data for the neural network as the generation progresses. Another future work is to apply the proposed method to other image datasets and tasks.