Improving CNN Performance with Min-Max Objective

1 Introduction

Recent years have witnessed the bloom of convolutional neural networks (CNNs) in many computer vision and pattern recognition applications, including object recognition [Krizhevsky et al., 2012], object detection [Girshick et al., 2014], semantic segmentation [Girshick et al., 2014], object tracking [Wang and Yeung, 2013], image retrieval [H¨orster and Lienhart, 2008], etc. These great successes can be attributed to the following three main factors: (1) the rapid progress of modern computing technologies represented by GPGPUs and CPU clusters has allowed researchers to dramatically increase the scale and complexity of neural networks, and to train and run them within a reasonable time frame, (2) the availability of large-scale datasets with millions of labeled training samples has made it possible to train deep CNNs without a severe overfitting, and (3) the introduction of many training strategies, such as different types of activation functions [Krizhevsky et al., 2012; Goodfellow et al., 2013b; He et al., 2015b; Jin et al., 2015], different types of pooling [He et al., 2014;

⇤Corresponding author: Yihong Gong(ygong@mail.xjtu.edu.cn)

Gong et al., 2014], dropout [Srivastava et al., 2014], dropconnect [Wan et al., 2013], pre-training [Dahl et al., 2012], batch normalization [Ioffe and Szegedy, 2015], etc, has helped to generate better deep network models from the BP-based training process.

We argue that the strategies to improve object recognition accuracies by developing deeper, more complex network structures and employing ultra large scale training data are unsustainable, and are approaching their limits. This is because very deep models, such as GoogLeNet [Szegedy et al., 2015], BN-Inception [Ioffe and Szegedy, 2015], ResNet [He et al., 2015a] not only become very difficult to train and use, but also require large-scale CPU/GPU clusters and complex distributed computing platforms. These requirements are out of reach of many research groups with limited research budgets, as well as many real applications. If this trend continues, research and development in this field will soon become the privilege of a handful of large Internet companies.

The crux of object recognition is the invariant features. When an object undergoes identity-preserving image transformations, such as shift in position, change in illumination, shape, viewing angle, etc, the feature vector describing the object also changes accordingly. If we project these feature vectors into a high dimensional feature space (with the same dimension as that of the feature vectors), they form a low dimensional manifold in the space. The invariance is accomplished when each manifold belonging to a specific object category becomes compact, and the margin between two manifolds becomes large.

Based on the above observations, in this paper, we propose a novel objective, called Min-Max objective, to improve object recognition accuracies of CNN models. The Min-Max objective enforces the following properties for the features learned by a CNN model: (1) each object manifold is as compact as possible, and (2) the margin between two different object manifolds is as large as possible. The margin between two manifolds is defined as the Euclidian distance between the nearest neighbors of the two manifolds. In principle, the proposed Min-Max objective is independent of any CNN structures, and can be applied to any layers of a CNN model. Our experimental evaluations show that applying the Min-Max objective to the top layer is most effective for improving the model's object recognition accuracies.

To summarize, The main contributions of this paper are as

follows:

• ÷ We propose the Min-Max objective to enforce the compactness of each manifold and the large margin between different manifolds for object features learned by a CNN model.
• ÷ Rather than directly modulating model weights and connecting structures as in typical deep learning algorithms, in our framework, the Min-Max objective directly modulates learned feature maps of the layer to which the Min-Max objective is applied.
• ÷ Experiments with different CNN models on four benchmark datasets demonstrate that CNN models trained with the Min-Max objective achieve remarkable performance improvements compared to the corresponding baseline models.

The remaining of this paper is organized as follows: Section 2 reviews related works. Section 3 describes our method, including the general framework, the formulation of the MinMax objective and its detailed implementation. Section 4 presents the experimental evaluations and analysis. Section 5 provides discussions about the proposed method, and Section 6 concludes our work.

2 RelatedWork

Methods to improve object recognition accuracies of CNNs can be broadly divided into the following three categories: (1) increasing the model complexity, (2) increasing the training samples, and (3) improving training strategies. This section reviews representative works for each category.

Increasing the model complexity includes adding either the network depth or the number of feature maps in each layer.

Increasing the number of training samples often leads to improvement of recognition accuracies. Data augmentation [Krizhevsky et al., 2012] is a low cost way of enlarging training samples by using label-preserving transformations, such as cropping and random horizontal flipping of patches, random scaling and rotation, etc. Another way to utilize more samples is to apply unsupervised learning [Wang and Gupta, 2015]. In practise, performance improvement by simply adding more training samples will gradually saturate. For example, it is reported that increasing the number of training samples from 1M to 14M from the ImageNet dataset only improved the image classification accuracy by 1% 1 .

There are also methods that aim to improve training of CNN models to enhance performances. For instance, different types of activation functions, such as ReLU, LReLU, PReLU, APL, maxout, SReLU, etc, have been used to handle the gradient exploding and vanishing effect in BP algorithm. The dropout [Srivastava et al., 2014] and dropconnect [Wan et al., 2013] are proven to be effective at preventing neural networks from overfitting. Spatial pyramid pooling [He et al., 2014] is used to eliminate the requirement that input images must be a fixed-size (e.g., 224 ⇥ 224) for CNN with fully connected layers. Batch normalization proposed by Ioffe and Szegedy [2015] can be used to accelerate deep network training by reducing internal covariate shift. Lin et al. [2014] proposed the Network-In-Network (NIN) structure in which the convolution operation was substituted by a micro multi-layer perceptron (MLP) that slides over the input feature maps in a similar manner to the conventional convolution operation. The technique can be regarded as applying an additional 1 ⇥ 1 convolutional layer followed by ReLU activation, and therefore can be easily integrated into the current CNN pipelines. The Deeply-Supervised Nets (DSN) proposed by Lee et al. [2014] used a classifier, such as SVM or soft-max, at each layer during training to minimize both the final output classification error and the prediction error at each hidden layer.

1 http://image-net.org/challenges/LSVRC/2014/results

3 The ProposedMethod

3.1 General Framework

Let ¶Xi↪ ci♦ni=1 be the set of input training data, where Xi denotes the ith raw input data, ci 2 ¶1↪ 2↪ · · · ↪ C♦ denotes the corresponding ground-truth label, C is the number of classes, and n is the number of training samples. The goal of training CNN is to learn filter weights and biases that minimize the classification error from the output layer. A recursive function for an M-layer CNN model can be defined as follows:

where, W(m) denotes the filter weights of the mth layer to be learned, b(m) refers to the corresponding biases, ⇤ denotes the convolution operation, f(·) is an element-wise nonlinear activation function such as ReLU, and X(m) i represents the feature maps generated at layer m for sample Xi. The total parameters of the CNN model can be denoted as W= ¶W(1) ↪ · · · ↪W(M) ; b(1) ↪ · · · ↪ b(M) ♦ for simplicity.

As described in Section 1, we aim to improve object recognition accuracies of a CNN model by embedding the MinMax objective into certain layer of the model during the training process. Embedding this objective into the kth layer is equivalent to using the following cost function to train the model:

where `(W↪Xi↪ ci) is the classification error for sample Xi, L(X (k) ↪ c) denotes the Min-Max objective. The input to it includes X (k) = ¶X(k) 1 ↪ · · · ↪X(k) n ♦ which denotes the set of produced feature maps at layer k for all the training samples, and c = ¶ci♦ni=1 which is the set of corresponding labels. Parameter λ controls the balance between the classification error and the Min-Max objective.

Note that X (k) depends on W(1) ↪ · · · ↪W(k) . Hence directly constraining X (k) will modulate the filter weights from 1th to kth layers (i.e. W(1) ↪ · · · ↪W(k) ) by feedback propagation during the training phase.

3.2 Min-Max Objective

For X (k) = ¶X(k) 1 ↪ · · · ↪X(k) n ♦, we denote by xi the column expansion of X(k) i . The goal of the proposedMin-Max objective is to enforce both the compactness of each object manifold, and the max margin between different manifolds. Inspired by the Marginal Fisher Analysis research from [Yan et al., 2007], we construct an intrinsic and a penalty graph to characterize the within-manifold compactness and the margin between the different manifolds, respectively, as shown in Figure 1. The intrinsic graph shows the node adjacency relationships for all the object manifolds, where each node is connected to its k1-nearest neighbors within the same manifold. Meanwhile, the penalty graph shows the betweenmanifold marginal node adjacency relationships, where the marginal node pairs from different manifolds are connected. The marginal node pairs of the cth (c 2 ¶1↪ 2↪ · · · ↪ C♦) manifold are the k2-nearest node pairs between manifold c and other manifolds.

Then, from the intrinsic graph, the within-manifold compactness can be characterized as:

where G(I) ij refers to element (i↪ j) of the intrinsic graph adjacency matrix G(I) = (G(I) ij )n⇥n, and ⌧k1 (i) indicates the index set of the k1-nearest neighbors of xi in the same manifold as xi.

From the penalty graph, the between-manifold margin can be characterized as:

where G(P) ij denotes element (i↪ j) of the penalty graph adjacency matrix G(P) = (G(P) ij )n⇥n, ⇣k2 (c) is a set of index pairs that are the k2-nearest pairs among the set ¶(i↪ j)♣i 2 ⇡c↪ j ◁2 ⇡c♦, and ⇡c denotes the index set of the samples belonging to the cth manifold.

Based on the above descriptions, We propose the Min-Max objective which can be expressed as:

Obviously, minimizing the Min-Max objective is equivalent to enforcing the learned features to form compact object manifolds and large margins between different manifolds simultaneously. The objective results in the following cost function:

In the next subsection, we will derive the optimization algorithm for Eq. (9).

3.3 Implementation

We use the back-propagation method to train the CNN model, which is carried out using mini-batch. Therefore, we need to calculate the gradients of the cost function with respect to the features of the corresponding layers.

Let G = (Gij)n⇥n = G(I) -G(P) , then the Min-Max objective can be written as:

where H = [x1↪ · · · ↪ xn], = D - G, D = diag(d11↪ · · · ↪ dnn), dii = Pnj=1↪j6=i Gij, i = 1↪ 2↪ · · · ↪ n, i.e. is the Laplacian matrix of G, and tr(·) denotes the trace of a matrix.

The gradients of L with respect to xi is

where, (:↪i) denotes the ith column of matrix .

To further improve the effectiveness of the proposed MinMax objective, we also consider using the kernel trick to define the adjacency matrix, that is, G(I) ij and G(P) ij can be defined as:

Then, the gradients @L @xi for the kernel version of theMin-Max objective can be derived as follows.

where, Φ denotes the Laplacian matrix of V = (Vij)n⇥n, Vij = - Gij σ2 kxi -xjk2 .

The total gradient with respect to xi is simply the combination of the gradient from the conventional CNN model and the above gradient from the Min-Max objective.

4 Experiments and Analysis

4.1 Overall Settings

To evaluate the effectiveness of the proposed Min-Max objective for improving object recognition accuracies of CNN models, we conduct experimental evaluations using shallow and deep models, respectively. Since our experiments show that embedding the proposed Min-Max objective to the top layer of a CNN model is most effective for improving its object recognition performance, in all experiments, we only embed the Min-Max objective into the top layer of the models and keep the other parts of networks unchanged. For the settings of those hyper parameters (such as the learning rate, weight decay, drop ratio, etc), we follow the published configurations of the original networks. All the models are implemented using the Caffe platform [Jia et al., 2014] from scratch without pre-training. During the training phase, some parameters of the Min-Max objective need to be determined. For simplicity, we set k1 = 5, k2 = 10 for all the experiments, and it is possible that better results can be obtained by tuning k1 and k2. σ2 is empirically selected from ¶0▷1↪ 0▷5♦, and λ 2 [10-6 ↪ 10-9 ].

4.2 Datasets

We conduct performance evaluations using four benchmark datasets, i.e. CIFAR-10, CIFAR-100, MNIST and SVHN. The reason for choosing these datasets is because they contain a large amount of small images (about 32 ⇥ 32 pixels), so that models can be trained using computers with moderate configurations within reasonable time frames. Because of this, the four datasets have become very popular choices for deep network performance evaluations in the computer vision and pattern recognition research communities.

CIFAR-10 Dataset. The CIFAR-10 dataset [Krizhevsky and Hinton, 2009] contains 10 classes of natural images, 50↪ 000 for training and 10↪ 000 for testing. Each image is a 32 ⇥ 32 RGB image.

CIFAR-100 Dataset. The CIFAR-100 dataset [Krizhevsky and Hinton, 2009] is the same in size and format as the CIFAR-10 dataset, except that it has 100 classes. The number of images per class is only one tenth of the CIFAR-10 dataset.

MNIST Dataset. The MNIST dataset [LeCun et al., 1998] consists of hand-written digits 0-9 which are 28 ⇥ 28 gray images. There are 60↪ 000 images for training and 10↪ 000 images for testing.

SVHN Dataset. The Street View House Numbers (SVHN) dataset [Netzer et al., 2011] consists of 630↪ 420 color images of 32 ⇥ 32 pixels in size, which are divided into the training set, testing set and an extra set with 73↪ 257, 26↪ 032 and 531↪ 131 images, respectively. Multiple digits may exist in the same image, and the task of this dataset is to classify the digit located at the center of an image.

4.3 Experiments with ShallowModel

In this experiment, the CNN "quick" model from the Caffe package 2 (named Quick-CNN) is selected as the baseline model. This model consists of 3 convolution layers and 1 fully connected layers. Experimental results of test error rates on the CIFAR-10 test set are shown in Table 1. In this Table, Min-Max and kMin-Max correspond to the models trained with the Min-Max objective and the kernel version of the Min-Max objective respectively. From Table 1 it can be observed that, when no data augmentation is used, compared with the baseline model, the proposed Min-Max objective and its kernel version can remarkably reduce the test error rates by 5▷41% and 5▷88%, respectively.

We also evaluated the Quick-CNN model using the CIFAR-100 and SVHN datasets. The MNIST dataset can't be used to test the model because images in the dataset are 28 ⇥ 28 in size, and the model only takes 32 ⇥ 32 images as its input. Table 2 and Table 3 list the respective comparison results. As can be seen, when no data augmentation is used, compared with the corresponding baseline model, the proposed Min-Max objective and its kernel version can remarkably reduce the test error rates by 4▷49%, 5▷04% respectively on CIFAR-100, and by 3▷50%, 4▷07% respectively on SVHN. The improvements on SVHN in terms of absolute percentage are not as large as those on the other two datasets, because the baseline Quick-CNN model already achieves a single-digit error rate of 8▷92%. However, in terms of relative reductions of test error rates, the numbers have reached 39% and 46%, respectively, which are quite significant. These remarkable performance improvements demonstrate the effectiveness of the proposed Min-Max objective.

4.4 Experiments with DeepModel

Next, we apply the proposed Min-Max objective to the wellknown NIN models [Lin et al., 2014]. NIN consists of 9 convolution layers and no fully connected layer. Indeed, it is a very deep model, with 6 more convolution layers than that of the Quick-CNN model. Four benchmark datasets are used in the evaluation, including CIFAR-10, CIFAR-100, MNIST and SVHN.

The CIFAR-10 and CIFAR-100 datasets are preprocessed by the global contrast normalization and ZCA whitening as in [Lin et al., 2014; Lee et al., 2014]. To be consistent with the previous works, for CIFAR-10, we also augmented the training data by zero-padding 4 pixels on each side, then corner-cropping and random horizontal flipping. In the test phase, no model averaging was applied, and we only cropped the center of a test sample for evaluation. No data whitening or data augmentation was applied for the MNIST dataset. For SVHN, we followed the training and testing protocols in [Lin et al., 2014; Lee et al., 2014], i.e., 400 samples per class selected from the training set and 200 samples per class from the extra set were used for validation, while the remaining 598↪ 388 images of the training and the extra sets were used for training. The validation set was only used for tuning hyper-parameters and was not used for training the model. Preprocessing of the dataset again follows [Lin et al., 2014; Lee et al., 2014], and no data augmentation was used.

2 The model is available from Caffe package [Jia et al., 2014]

Table 4, 5, 6 and 7 show the evaluation results on the four benchmark datasets, respectively, in terms of test error rates. For fairness, for the baseline NIN model, we list the test results from both the original paper [Lin et al., 2014] and our own experiments. In these tables, we also include the evaluation results of some representative methods, including Stochastic Pooling [Zeiler and Fergus,

2013], CNN + Spearmint [Snoek et al., 2012], Maxout Networks [Goodfellow et al., 2013b], Prob. Maxout [Springenberg and Riedmiller, 2014], DroptConnect [Wan et al., 2013], Learned Pooling [Malinowski and Fritz, 2013], Tree based priors [Srivastava and Salakhutdinov, 2013], Multistage Architecture [Jarrett et al., 2009], Multi-digit Recognition [Goodfellow et al., 2013a], and DSN [Lee et al., 2014], which is the state-of-the-art model on these four datasets. It is worth mentioning that DSN is also based on the NIN structure with layer-wise supervisions.

From these tables, we can see that our proposed Min-Max objective achieves the best performance against all the compared methods. The evaluation results shown in the four tables can be summarized as follows.

• ÷ Compared with the corresponding baseline models, the proposed Min-Max objective and its kernel version can remarkably reduce the test error rates on the four benchmark datasets.
• ÷ On CIFAR-100, the improvement is most significant: NIN+kMin-Max achieved 33▷12% test error rate which is 2▷38% better than the baseline NIN model, and 1▷45% better than the state-of-the-art DSN model.
• ÷ Although the test error rates are almost saturated on the MNIST and SVHN datasets, the models trained by the proposed Min-Max objectives still achieved noticeable improvements on the two datasets compared to the baseline NIN model and the state-of-the-art DSN model. In terms of relative reductions of test error rates, compared with the corresponding baseline models, the numbers have reached 36% and 28% respectively on these two datasets, which are significant.

These results once again substantiate the effectiveness of the proposed Min-Max objective.

4.5 Visualization

To get more insights of the proposed Min-Max objective, we visualize the top layer (i.e. the embedded layer) features obtained from two distinct models on the CIFAR-10 test set. The Quick-CNN and NIN models are selected as the respective baseline models. Figure 2 and Figure 3 show the feature visualizations for the two models respectively. It is clearly seen that, in both cases, the proposed Min-Max objective can help making the learned features with better within-manifold compactness and between-manifold separability as compared to the corresponding baseline models.

4.6 Additional Computational Cost Analysis

CNNmodel is trained via stochastic gradient descent (SGD), where in each iteration the gradients can only be calculated from a mini-batch of training samples. In our method, during training phase, the additional computational cost is to calculate the gradients of the Min-Max objective with respect to the features of the embedded layer. According to Section 3.3, the gradients have the closed-form computational formula, which make the additional computation cost very low based on a mini-batch. In practise, the additional computational cost is negligible compared to that of the corresponding baseline CNN model.

5 Discussions

To investigate the effect of the embedded position of the Min-Max objective on the performance, we conducted experiments with Quick-CNN on the CIFAR-10 dataset by embedding the Min-Max objective into different layers of the model, and the results of test error rates are shown in Table 8. We can clearly see that the higher the embedded layer, the better the performance. Interestingly, this concurs with the object recognition mechanism of the ventral stream in the human visual cortex. In recent years, research studies in the fields of physiology, biology, neuroscience, etc [DiCarlo et al., 2012] have revealed that the human ventral stream consists of layered structures, where the lower layers strive to extract useful patterns or features that can well represent various types of objects/scenes in the natural environment, while higher layers serve to gradually increase the invariance degree of the low-level features to achieve accurate object/scene recognitions. These research discoveries suggest to us that it is more effective and appropriate to embed the invariance objectives into higher rather than lower layers of a deep CNN model. In the experiments, we also found that, for a given embedded layer, as long as the order of magnitude of λ is appropriate, the performance of our method does not change much, and the higher the embedded layer, the higher the order of magnitude of the optimal λ.

Moreover, it is clear from Figure 2(a) and 3(a) that through many layers of nonlinear transformations, a deeper network is able to learn good invariant features that can sufficiently separate objects of different classes in a high dimensional feature space. In contrast, features learned by a shallower network are not powerful enough to appropriately "untangle" those different object classes. This explains why the proposed Min-Max objectives are more effective for improving object recognition accuracies when applied to shallower networks.

6 Conclusion

In this paper, we propose a novel and general framework to improve the performance of CNN by embedding the proposed Min-Max objective into the training procedure. The Min-Max objective can explicitly enforce the learned object feature maps with better within-manifold compactness and between-manifold separability, and can be universally applied to different CNN models. Experiments with shallow and deep models on four benchmark datasets including CIFAR-10, CIFAR-100, SVHN and MNIST demonstrate that CNN models trained with the Min-Max objective achieve remarkable performance improvements compared to the corresponding baseline models. Experimental results substantiate the effectiveness of the proposed method.

Acknowledgments

This work is supported by National Basic Research Program of China (973 Program) under Grant No. 2015CB351705, the National Natural Science Foundation of China (NSFC) under Grant No. 61332018, and the Media Technology Laboratory, Central Research Institute, Huawei Technologies Company, Ltd.