Group Reconstruction and Max-Pooling Residual Capsule Network

1 Introduction

In recent years, convolutional neural network (CNN) has been a main deep learning model for computer vision tasks [Krizhevsky et al., 2012]. However, internal data representation of a convolutional neural network does not take into account important spatial hierarchies between simple and complex objects [Hinton et al., 2018]. Especially, viewpoint invariance is achieved by pooling [Simonyan and Zisserman, 2014] so that the small perturbations in the input can not effect the output too much [Cohen and Welling, 2016], [Cohen et al., 2018]. This results in the loss of information about the internal properties of present entities (e.g location, orientation, shape and pose) in an image and relationships between them. Capsule network is a new type network introduced in [Sabour et al., 2017] for image classification. There are two key features distinguishing them from CNNs.

∗Correspondingauthor:NannanWang(nnwang@xidian.edu.cn)

On one hand, rather than using a scalar, a capsule is a group of neurons which can denote many more properties such as pose, viewpoint, velocity, grain or regions, etc. In this way, the network can not only extract features, but also extract variants of them. The introduction of capsules provides a new method of feature learning, which is represented by the parameterization of the entity vector (named capsule) to represent a wide variety of attributes, so that better robustness can be obtained.

On the other hand, pooling is replaced with a routing scheme to send lower-level capsule (e.g nose, mouth, ears etc.) outputs as input to parent capsule (e.g face) that represent part-whole relationships to achieve translation equivariance and untangles the coordinate frame of an entity through linear transformations. Such a scheme can be seen as a feature clustering and be optimized by coordinate descent through several iterations. However, the computational complexity in the routing scheme is fairly high [Chen and Crandall, 2018]. The capsule network has shown its potential by achieving a state-of-the-art result of 0.25% test error on MNIST , better than the previous baseline of 0.39%. However, its performance on complex data, such as CIFAR-10 [Krizhevsky and Hinton, 2009], seems not so good, only 68.93% [Xi et al., 2017].

Compared with pooling, routing will take every lower capsule learned into account when mapping from lower capsules to higher counterparts. This scheme works well on MNIST [LeCun, 1998] according to handwritten digits hardly have noise so that every capsule in lower layer represents the desired features. However, normal images like CIFAR-10 [Krizhevsky and Hinton, 2009] have much more noise which can cause that the higher capsules achieve too much useless information such as background. This will disrupt useful lower capsule routing to its higher capsules. The computational complexity becomes too high while dealing with large datasets [Ben-Israel and Greville, 2003], [Li et al., 2018].

To tackle these drawbacks, we propose a group reconstruction and max-pooling residual capsule network (GRMRCapsNet). It can significantly reduce computational complexity and improve the recognition accuracy for complex data such as images in CIFAR-10/100. There are three main contributions in our paper:

• Different from the method of multiplying the low-level capsule dimension to the high-level capsule by multiplying the viewing angle invariant matrix in [Sabour et al., 2017], [Hinton et al., 2018], we use a convolutional layer named Convolutional Transformation Layer to compute capsules within a local receptive field so as to reduce the complexity of computation.
• We propose a new routing scheme named GroupReconstruction Routing Algorithm to map from lowlevel capsules to high-level capsules in a supervised way. Our method firstly divides all capsules in the lower layer into different groups and then routes them separately in each group. By doing this, some capsules that are divided into the same group will be given a very low weight (almost zero) in the route of the next layer, while other capsule groups important can achieve high weight. For all capsules in one group share the parameters in routing, the computational complexity can also be reduced. For all capsules in one group share the parameters in routing, the computational complexity can also be reduced.
• We design a Capsule Max-Pooling Residual Block which can adaptively choose the routing or max-pooling. Specifically, we add capsule-based max-pooling as a skip connection into each block. Formally, denoting the routing mapping as R(x). We can get the higher level capsules u from the lower level capsules v by the mapping of u = R(v). Specifically, we let M(x) denote the max-pooling, the original mapping is changed into u = R(v)+M(v). It is conceivable that the mapping of the network tends to u ≈ M(v) when the pooling effect is better, and the mapping of that tends to u ≈ M(v) when the routing effect is better.

We firstly conduct experiments on CIFAR-10 to show that our proposed group capsule routing have explicit improvement over original routing algorithm. Then, we present comprehensive experiments on CIFAR-10/100 and SVHN to evaluate our network, which shows that our method have a positively improvement against state-of-the-arts.

2 RelatedWork

2.1 Capsules

[Sabour et al., 2017] presented a vector consisting of neurons, called capsule, whose orientation represents the properties of the entity and whose length represents the probability of it. In [Hinton et al., 2018], a capsule was introduced by a 4x4 pose matrix and an activation probability which can respectively evaluate the properties and existence of entities.

2.2 Routing

According to [Sabour et al., 2017], let ui, vj denote the lower and higher capsules in a layer respectively, where i, j indicate the index of capsules. Use d1, d2, n1, n2 to represent the dimensions and number of capsules in the lower and higher layers, i.e., {ui ∈ Rd1 }n1 i=1 , {vj ∈ Rd2 }n2 j=1 . The first step is mapping from the dimension of ui to that of vj by νj|i = wij · ui where wij ∈ Rd2×d1 is a transform matrix. The second step is an agreement routing method to cluster all lower capsules into higher ones. The transformed capsules are multiplied by a routing coefficient:

The process of dynamically updating cij could be deemed as voting. Firstly, given b(r) ij ← 0, r ← 0, then:

where v(r) is computed from s(r) via squash(·):

The coefficients in the routing algorithm is updated by an unsupervised approach iteratively which finds the best voting c [Sabour et al., 2017]. In [Hinton et al., 2018], the routing algorithm was replaced by a modified EM-algorithm to fit a gaussian mixture model (GMM). Given Rij ← 1/n2, the EM routing clusters n1 capsules the dimensions of d1 into n2 capsules the dimensions of d2 iteratively:

where αi represents the activation probability of low-level capsules uj. M - step produces the activation αj together with the mean µj and std standard deviation σj in high layer. Then, the co-efficients Rij is updated by the the E - step.

In [Lenssen et al., 2018], they proposed a group equivariant capsule network (GECN) which can guarantee equivariance and invariance properties just like the group convolution. That paper also introduced a scheme for routing by agreement algorithms, a spatial aggregation method, and the ability to integrate group convolutions. In [Zhang et al., 2018a], they proposed a orthogonal projections onto capsule subspaces method to capture features. Compared with introducing some brand new architectures for capsule nets,that paper tends to learn a group of capsule subspaces to represent a set of entity classes. In [Zhang et al., 2018b], they tried to extract multi-labeled relation with the capsule network by add attention scheme into routing-by-agreement algorithm.

3 Methodology

3.1 Convolutional Transformation Layer

As mentioned in Section 2, before they are fed into routingby-agreement algorithm, capsules in low layer u with the dimension of d1 should be mapped to ν with the dimension of d2 by a transformation matrix wij ∈ Rd2×d1 . However, the number of parameters would become very large with an increasing dimension d1 and d2. According to [Hinton et al.,

2011], [Sabour et al., 2017], We can summarize the two main roles of thematrixwij. Firstly, thematrixwij encodes important spatial and other relationships between ui which represent lower level feature (eyes, mouth and nose) and vj higher level feature (face). Secondly, matrices attempt to learn to transform low-layer capsules into high-layer capsules from a different viewpoint.

Based on the above analysis that only information between adjacent capsules is beneficial, we propose Convolutional Transformation Layer (CTL) to extract the local perspective and relationship of all capsules. Specifically, the voting νt|m is obtained by um fed into convolutional capsule layer with 32 group convolution [Krizhevsky et al., 2012] which indicates 32 capsule types, a receptive field of 3×3, a padding of 1 and a stride of 1 × 1.

3.2 Group Reconstruction Routing Algorithm

In this section, we will detail the proposed Group Reconstruction Routing Algorithm which can effectively avoid information confusion during the mapping from the low-level capsule to the high-level capsule. The low-level capsules firstly are fed into Group Capsule Layer, in which capsules are divided into several groups. Following is the Reconstruction Feedback Routing which can use back propagation to update the coefficients compared with iterative ones.

Group Capsule Layer

According to [Sabour et al., 2017] , the routing-by-agreement algorithm could find a potential relationship between the capsules in lower layers and those in higher layers, that is, if the two capsules are more similar, their cosine similarity will be higher. Nevertheless, for a capsule, it is not directly related to all other capsules, so it is not reasonable to route the entire capsules in lower layers. What's worse, all capsules learned can hardly be used to represent features, because a significant portion of the capsules are noise ones. Capsules in higher layers achieving the noisy-mixed capsules in lower layers, capsule networks works not very well on complex data.

Taking this into account, we propose a method which divides capsules into several groups {Gn}, trying to learn the capsules in each group are as similar as possible. As shown in Figure 1, we divide capsules into N groups of which M capsules make up. The length of each capsule is L. Capsules in different groups are present by different colors. For example, the first capsule group is shown by blue color and the second one is present by green. In each group n, we aggregate M capsules into T capsules with the length of K by the Group Reconstruction Routing algorithm which shares the same variables, so the computational complexity in routing could be reduced. Totally, we can obtain N × T capsules in the higher layer. For example, for a face, we should learn capsule groups that represent the eyes, nose, ears, mouth and so on respectively.

Reconstruction Feedback Routing

In each group which is described in the above section, what we get are capsules with similar information. In [Hinton et al., 2018], Gaussian Mixed Model (GMM) plays the role of the routing-by-agreement algorithm. Formally, let νij, Rij, αij denote voting, the pose entity and the activation, then we

where µj and σ2 j is the the mean and std standard deviation of vj .

In this paper, we directly learn µj and σ2 j by back propagation algorithm. Similarly to [Kingma and Welling, 2013], let us consider capsules in high layers V = {vt}Tt=1 which are generated by capsules in low layers U = {um}Mm=1 . Given the p(V |U) which represents the posterior probability of V for U and p(U) the distribution of U, the distribution p(V ) can easily be obtained by the corresponding formula:

Therefore, if {um}Mi=m and {vt}Tt=1 are i.i.d. and obey the Gaussian distribution, we can know that p(vt|um) ∼ N (vt; um, σ2m).

Based on the above analysis, we build two neural networks: µt = fµ(νt|m), σ2 t = fσ(νt|m). so as to express p(V ) directly by formula:vt = N( ˆ vt; ut|m, σ2t|m), (10)

where ˆ vt is computed by a neural network fv(νt|m). We can also get the reconstructed U through a generator ˆU = g(V ). Let us define D(ˆum, um)2 as the distance of the ˆum and um, then we can get the Reconstruction Loss:

We use cosine similarity to describe the distance between the reconstructed capsules and the original capsules.

Then, in each group, capsules are aggregated into higher layer capsules by Group Reconstruction Feedback Routing Algorithm with LRecon restricted shown in Algorithm1. Firstly, the transformed capsules in lower layers are are

n

Algorithm 1 Group Reconstruction Routing Algorithm.

Input: low-level capsules um Output: high-level capsules vt

• 1: for all capsules um: νt|m ←CTLt(um)
• 2: divide all capsule um into groups Gn;
• 3: for all groups Gn do
• 4: for all capsule νt|m ∈ Gn: µt|m ← fµ(νt|m), σ2 t|m ← fσ(νt|m), ˆ vt ← fv(νt|m)
• 5: for all capsules ˆ vt: vt ←N(ˆ vt; ut|m, σ2t|m)
• 6: for all capsules vt: ˆum ← gˆu(vt)
• 7: end for
• 8: for all capsules ˆum: LRecon = D(ˆum, um)2
• 9: fµ, fσ, fv, gˆu ←BP(LRecon)
• 10: return vt

evenly divided into N groups, each group corresponding to the same type of capsules, which will avoid higher layer capsules mixing in useless information in the routing. In order to reduce the number of parameters to build a deeper network, we improve the original method [Hinton et al., 2018] which iteratively obtains higher layer capsules using the EM algorithm. Instead, we design three stacked layers: fµ, fσ and fv to directly fit the corresponding mean µ, variance σ2 and capsules ˆ v. Then, the higher-layer capsules are obtained by the Equation 10. Finally, based on the hypothesis that the higher layer capsules can reconstruct the lower layer capsules, we design generation layers gˆu to reconstruct the lower layer capsules, and compare their similarity by Equation 11, and add this similarity loss to the BP algorithm to update fµ, fσ and fv and gˆu in a feedback manner. Compared to [Sabour et al., 2017][Hinton et al., 2018], our method can have a better performance in a supervised way.

3.3 Capsule Max-Pooling Residual Block

As we know, once the pooling is proposed, it has been widely used because of its good performance on the reduction of parameters to avoid overfitting. Considering its advantages that routing does not have, we propose a capsule-based maxpooling and add it to every few stacked layers which is shown in Figure 3. Formally, we define the block named Capsule Max-Pooling Residual Block (CMPR-Block) as:

here ν are the capsules transformed by CTL in the lower layer and v are higher layer capsules and the dimensions of M(x) and R(x) are equal. The function R(ν) represents the RFRouting which is described in section 3.2 and the function M(ν) indicates the capsule max-pooling. As Figure 3 shows, in each group, we firstly compute the l2 norm for all capsules: ‖v‖2, then we choose the max value of these l2 norms with the kernel size of k and stride of s. The whole process can be simply described as: vres = max(‖v‖)2. The operation R+M is performed by element-wise addition just like [He et al., 2016]. The shortcut connections shown in Equation 12 will not bring extra calculations and parameters, which will be shown in Section 4.3.

3.4 Network Architecture

The architecture of Group Reconstruction and Max-Pooling Residual Capsule Network (GRMR-CapsNet) is shown in Figure 3. The input to the network is a 3×32×32 image. The network begins with the four CMPR-Block described in Section 3.3 . Each block contains 32 groups in which the number of capsules is equal. In the first CMPR-Block, each group has 32×32 capsules with a dimension of 2. In the following layers, the number of capsules in each group is gradually halved, while the dimensions of the capsule are gradually doubled (ig, the second layer has 16 × 16 capsules with a dimension of 4, and the third layer has 8 × 8 capsules with a dimension of 8 .....). In the end, we will get 32 groups of 4×4 capsules each with a dimensions of 16 and these capsules are fed into the Group Capsule Layer. After routing by G-Caps Layer, there is only one capsule left in each group. Then, the capsules of these groups are sent to the Class Caps Layer, similarly to the primary caps layer in [Sabour et al., 2017], which is a fully connected capsule layer. Capsules in different groups are clustered into n types of capsules by the dynamic routing algorithm, and the length of each obtained capsule represents the activation. In this paper, the loss we used is the margin loss proposed by [Sabour et al., 2017].

3.5 Margin Loss

Just like [Sabour et al., 2017], we use the length of the capsule to represent the probability of its entity existence. We want the capsule in the top layer for image class n to have the most significant length if and only if that the n class image is fed into the network. To allow for multiple classes, a separate margin loss Ln follows, for each image capsule, n:

where Tn = 1 if a image of class n is present. m+ = 0.9 means that the length of the vector vn is at least 0.9 to be activated, and m- = 0.1 means that the length of the vector should be less than 0.1 without the presence of class n images. The λ down-weighting of the loss for absent image classes stops the initial learning from shrinking the lengths of the activity vectors of all the image capsules. We use λ = 0.5. The total loss is simply the sum of the losses of all image capsules.

4 Experiments

4.1 Datesets

We do experiments on both CIFAR [Krizhevsky and Hinton, 2009] and SVHN [Netzer et al., 2011] datasets to evaluate the performance of our network. CIFAR-10 and CIFAR-100 are labeled as a subset of 80 million tiny image datasets. The Street View House Number dataset has 73257 colored digits for training, 26032 digits for testing, with an additional 531131 training images available. We adapt the standard data augmentation: horizontal flipping and shifting to the datasets.

4.2 Implementation Details

Convolutional Transformation Layer. As methioned in 3.1, we use the convolutional capsule layer with 64 group convolution [Krizhevsky et al., 2012] which indicates 64 capsule types, a receptive field of 3×3, a padding of 1 and a stride of 1 × 1.

GroupReconstructionRoutingAlgorithm. We use group convolution to implement group capsule. We define three convolutional layers as a function layer and use three funtion layers to learn the fµ, fσ, fv and gˆu. Instead directly computing the v by Equation 10, we let α, β and γ as the coefficient to the µ, σ and ˆu so that the network can coverage better. Contrary to the function layer, we use three deconvolutional layers each of which has a kernel size of 3 × 3, a padding of 1 and a stride of 1.

CapsuleMax-PoolingResidual Block. We use themethod introduced in 3.3 with a kernel of 2×2 and a stride of 2 in each group. After the operation M + R performed by elementwise addition, we use the squash expressed by Equation 3 to normalize the capsules cascaded with ReLu [Nair and Hinton, 2010].

4.3 Results

We perform experiments with the capsule network [Sabour et al., 2017] named CapsNetI as baseline for comparison with the proposed G-CapsNet. In particular, the baseline model is shallow with only two convolutional layers and one fully connected layer. Conv1 has 256, 9 × 9 convolution kernels with a stride of 1 and ReLU activation. The second layer (PrimaryCapsules) is a convolutional capsule layer. In the third layer (DigitsCapsules) with the matrix Wij the size of 8 × 16, using routing to cluster the class capsules. In our first experiment which is shown in Table 1, we replace DigitsCapsules Layer with our method. Specifically, dividing the capsules into 16, 32, 64 groups respectively, we obtain three models: CapsuleNet-16, CapsuleNet-32 and CapsuleNet-64. We conduct experiments on CIFAR-10 datasets to evaluate the proposed G-CapsNets compared with the capsule network [Sabour et al., 2017]. For CapsNetI, the number of parameters is about 8.2M and the accuracy is 68.93%. Compared with the baseline model, we proposed model G-CapsNet16 which contains 16 groups has the 1.5M numbers of parameters and the accuracy of 70.45% as shown in Table 1. At the same time, when the capsule is divided into 32 and 64 groups, the number of parameters decreases to 1.5M and 1.4M and the accuracy improves to 71.77% and 72.87% respectively. This result indicates that the more the number of groups divided, the denser the capsule relationship within the group, and the more able to distinguish the discrimination of different groups of capsules. Then, we add Capsule Max-Pooling to the G-CapsNet-64, scilicet CMPRBlock proposeed in Section 3.3. The result shows that Capsule Max-Pooling can slightly improve the performance without enduring the increasing computation complexity. Lastly, considering LRecon, we will get the top 1 accuracy of 75.62% which indicates that the reconstruction loss will indeed benefit the routing.

Comparison with State-of-the-arts

In order to demonstrate the superiority of our model, CRMRCapsNet presented in 3.4 , we also conduct a experiment compared with the state-of-art network: ResNet110 [He et al., 2016], VGG16 [Simonyan and Zisserman, 2014], GoodInit [Mishkin and Matas, 2015], Batch NIN [Chang and Chen, 2015], Maxout [Goodfellow et al., 2013] and CapsNetII. In this paper, our purpose is to improve the performance of Capsule Networks. We just choose the works which are widely used to compare with. Instead the CapsNetI, we use six layers to extract features to build capsules before the PrimaryCapsules Layer. Each layer contains two convolutional layers with the kernel of 2×2, the padding of 1×1 and the stride of 1. Following is Batch Normalization and ReLu. Initial learning rate is set to 0.0001 and maximum epoch is 80. Adam [Kingma and Ba, 2014] is used with momentum 0.9. Batch size [Ioffe and Szegedy, 2015] is 128. We conduct the experiments on CIFAR-10/100 and SVHN. It is clear from Table 2 that the best top 1 accuracy of state-of-the-arts on three datesets are GoodInit's 94.16%, ResNet110's 72.78% and GoodInit's 98.21% respectively. Our proposed method achieves the top 1 accuracy of 94.91%, 76.21% and 98.64%, which is better than theirs on all three datasets. And we have a 7.57%, 15.77% and 2.94% increasing of the basic CapsNet which just uses dynamic routing or EM routing.

5 Conclusion

In this paper, we propose a Group Reconstruction and MaxPooling Residual Capsule Network (GRMR-CapsNet) that consists of Capsule Max-Pooling Residual Block (CMPRBlock). In each block, all capsules transformed by Convolutional Transformation Layer (CTL) are fed into Group Capsule Layer (GCL) in which the capsules are devided into N groups. To reduce the computational complexity and mapping many more useful information to higher layer capsules, we introduce Reconstruction Feedback Routing that construct three funtion to obtain µ, σ2 and ˆ v and define a LRecon as a feedback to the block. The proposed network takes a image as input and predicts its classification just like the capsule network. The class capsules the length of which represents the probability of the entity can be obtained through Class Caps Layer. We do experiments to prove that the proposed CMPRBlock with LRecon could greatly improve the performance of capssule network and reduce its computational complexity. Simultaneously, we find different numbers of group can have an effect on the performance. Experiment results on CIFAR-10/100 and SVHN show that our network is competitive compared with state-of-the-arts.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (under Grant 61876142, 61432014, U1605252, 61772402, and 61671339), in part by the National Key Research and Development Program of China under Grant 2016QY01W0200, in part by the National HighLevel Talents Special Support Program of China under Grant CS31117200001, in part by the Young Elite Scientists Sponsorship Program by CAST under Grant 2016QNRC001, in part by the Young Talent fund of University Association for Science and Technology in Shaanxi, China, in part by the CCF-Tencent Open Research Fund (No. RAGR20180105) and Tencent AI Lab Rhino-Bird Focused Research Program (No. JR201923), and in part by the Xidian UniversityIntellifusion Joint Innovation Laboratory of Artificial Intelligence, in part by the Fundamental Research Funds for the Central Universities under Grant JB190117.