Decomposable-Net: Scalable Low-Rank Compression for Neural Networks

1 Introduction

Deep neural networks (DNNs) have achieved higher performance on various machine-learning tasks. However, since computational resources are limited on edge devices such as smartphones, it is desirable to optimize the model size suited to a specific device. Various methods for compressing DNNs have been proposed, including factorized convolutions [Howard et al., 2017], low-rank approximations [Jaderberg et al., 2014], pruning [Liu et al., 2017], quantization [Han et al., 2016], and neural architecture search [Tan et al., 2019]. However, changing the model size is generally difficult for those methods once training and compression are completed; even when the same base model is reconfigured on different devices, retraining from scratch or fine-tuning according to the resources of the target devices is still required. In addition, for multi-task recognition systems operating on devices such as robots, it is preferable that the computational load of each task can be dynamically changed according to resource usage. One option is to store multiple models with different sizes in advance and switch those models, but doing so requires additional storage. Therefore, it is desirable that the model size can be flexibly changed without retraining, which we refer to as scalability in this paper.

To that end, [Yu et al., 2019] introduced slimmable networks whose width can be changed to predefined ones after training. Moreover, [Yu and Huang, 2019] proposed universally slimmable networks (US-Nets) that extend slimmable networks to arbitrary widths. However, since these methods directly reduce the width (i.e., dimensionality) in each layer, the principal components are not taken into account. In addition, they uniformly reduce the width across all layers, which ignores differences in the importance of different layers.

In this paper, we propose Decomposable-Net as a novel scheme that allows DNNs to flexibly change their size after training. A weight matrix in each layer is decomposed into two low-rank matrices via singular value decomposition (SVD). By changing the rank in each layer, DecomposableNet can scale the model to an arbitrary size (Figure 1(a)). In contrast to [Yu and Huang, 2019], we reduce the rank (i.e., the redundant basis in the principal subspace), for which the importance of each basis is naturally characterized by a singular value. This prevents the weight matrix in each layer from losing important directions, as illustrated in Figure 1(c). Moreover, our approach is essentially different from the existing low-rank compression methods [Kim et al., 2016; Xu et al., 2020] that apply low-rank approximation during or after learning and optimize the model for a specific size. Although Decomposable-Net is a single model and is not necessary to retrain, we show that it performs significantly better than those methods in a wide range of model sizes.

Our technical contributions are summarized as follows.

• We propose a novel backpropagation scheme that jointly minimizes losses for both of full- and low-rank networks (Figure 1(b)). It is designed not only to maintain the performance of a full-rank network but also to improve mul-

tiple low-rank networks (to be used for the inference).

• We introduce a simple criterion for the rank selection that utilizes a singular value for each basis, which effectively suppresses approximation error.
• The basic performace with SVD-based (e.g., channeland spatial-wise) decompositions are enhanced by our methods, in addition to the scalability without retraining.

2 Methods

2.1 Overview

For the ℓ-th network layer, let yℓ = W ⊤ ℓ xℓ ∈ Rnℓ be an output vector given by the linear transformation of an input vector xℓ = φ(yℓ-1) ∈ Rmℓ with the weight matrix Wℓ ∈ Rmℓ×nℓ , where φ(·) is the activation function, and mℓ and nℓ are the numbers of input and output nodes, respectively. Let Rℓ = min(mℓ, nℓ) be the full rank of the weight matrix. Given Uℓ ∈ Rmℓ×Rℓ and Vℓ ∈ Rnℓ×Rℓ as matrices that have left and right singular vectors (i.e., bases) as columns, and Sℓ ∈ RRℓ×Rℓ as a diagonal matrix composed of singular values (σℓ), we can formulate the SVD as Wℓ = UℓSℓV ⊤ ℓ .

An example of Decomposable-Net with fully connected layers is shown in Figure 1. Each weight matrix in the network is decomposed into two matrices of rank Rℓ via SVD, and we control the rank to change the model size. This can be viewed as inserting a sub-layer between the original layers and changing its width from Rℓ. The number of parameters in each layer becomes (mℓ +nℓ)Rℓ by this decomposition. Thus, we can compress the network to an arbitrary size by changing the rank rℓ (≤ Rℓ) within the range 1 ≤ rℓ < mℓnℓ/ (mℓ +nℓ). For CNNs, we transform convolutional tensors to the matrix form and apply the low-rank approximation to every convolutional and fully connected layer. Note that our methods do not rely on a specific decomposition form, and the SVD-based methods, including channel- and spatial-wise [Zhang et al., 2016; Tai et al., 2016] decompositions, can be adopted. However, these low-rank approximations solely deteriorate the performance. Thus, we propose a learning method to alleviate it.

2.2 Learning

© 2020 Toshiba Corporation For the L-layer network, let W = {Wℓ}Lℓ=1 be a set of weight matrices and let ˜ W = {˜ Wℓ}Lℓ=1 be a set of their low-rank approximations, respectively. As shown in Figure 1(b), we additionally propagate each mini-batch to a low-rank network whose weight ˜ W is generated fromW via SVD. Other trainable parameters Θ (e.g., biases) are shared between the fulland low-rank networks 1 . Therefore, the number of total parameters is the same as in normal learning.

For the purpose to maximize the performance of smaller subnetworks, we propose to jointly minimize losses for both full- and low-rank networks as follows:

Here, L(·) is a loss function, Db is a training sample in a mini-batch, B is the batch size, and λ ∈ [0, 1] is a hyperparameter for balancing between the two losses. R(·) is a regularization function, for which we adopt the sum of squared Frobenius norms R(W) = 12 ∑Lℓ=1 ‖Wℓ‖2F , and η is a hyperparameter to control its strength. In order to learn models in an end-to-end manner, backpropagation through SVD is involved in computing ∇ℓ := ∂L(Db, ˜ W,Θ)/∂Wℓ. The detailed derivation of the gradient and a practical technique for improving the numerical stability are given in appendix A of the extended version of this paper [Yaguchi et al., 2019]. The gradients for Θ are simply computed from the λ-weighted average of both networks.

Since we need a single model that performs well at multiple sizes, we aim to optimize all the low-rank networks to be selected at the inference phase. However, it is computationally intractable to do so in each iteration step. Thus, we select one low-rank network with a random size from a predifined range. Specifically, a ratio Z of the number of bases

1 For the batch normalization (BN) layer, γ and β are shared. µ and σ2 are not shared and are not tracked by moving average during training. These are computed on the whole training set for each model after training, as in US-Nets.

1

(referred to as rank ratio) is sampled from uniform distribution U(αl, αu) with 0 < αl < αu ≤ 1. Then, we reduce d = (1 - Z) ∑Lℓ=1 Rℓ bases across the entire network using a criterion described in the next subsection. The pseudo-code for learning Decomposable-Net is given in Algorithm 1.

2.3 Rank Selection

At the stage of compression, it is critically important to select an appropriate rank for maximizing the performance (as we see in Figure 2(c)). Given a target size for a model, we select the rank of each layer by reference to a simple criterion: C(ℓ, i) = σℓ,i (a singular value), where i ∈ [1,Rℓ] is a basis index. In practice, we globally sort the singular values of all bases in the entire network in ascending order and store them as a single list. The only step necessary for selection is to reduce the first d bases in the list. We apply the same criterion to learning and inference. d is determined by the rank ratio Z during learning and the target model size for inference.

The impact of removing a weight w ∈ W on a loss L(W) can be approximated via the first-order Taylor expansion: L(W|w=0) - L(W) ≈ -w · (∂L(W)/∂w), and its absolute value can be used as an importance measure for pruning [Lubana and Dick, 2021]. In pruning during learning, [Lubana and Dick, 2021] showed that keeping large weights results in faster convergence, and using only the magnitude |w| works well as the importance. According to that, we can characterize the importance of each basis by σℓ,i · |∂L(Wℓ)/∂σℓ,i|, which is proportional to the magnitude of singular value σℓ,i. Therefore, it is expected that our criterion simply using singular values can supress the impact on the loss (i.e., approximation error). Note that our criterion is different from the energy-based criterion [Alvarez and Salzmann, 2017; Xu et al., 2020]: C ′ (ℓ, i) = ∑ik=1 σ2 ℓ,k / ∑Rℓ j=1 σ2 ℓ,j . Because it evaluates the accumulated energy ratio for each matrix, the importance characterized by the magnitude of singular value is ignored. In section 4, we experimentally show that our criterion outperforms it.

2.4 Analysis

Here, we describe the mathematical interpretation for our methods and show that Decomposable-Net possesses favorable properties as a scalable model. Proofs of the propositions described below are given in appendix B and C of [Yaguchi et al., 2019].

Proposition 1. Let ˜ Wℓ(rℓ) be a rank-rℓ approximation for Wℓ via SVD, and let ˆ yℓ(rℓ) = ˜ Wℓ(rℓ)⊤xℓ. Then, the squared error between an original yℓ and ˆ yℓ(rℓ) satisfies ‖yℓ - ˆ yℓ(1)‖2 ≥ · · · ≥ ‖yℓ - ˆ yℓ(rℓ)‖2 ≥ ‖yℓ - ˆ yℓ(rℓ + 1)‖2 ≥ · · · ≥ ‖yℓ - ˆ yℓ(Rℓ)‖2 = 0.

Therefore, the errors between the original output yℓ and its approximation ˆ yℓ monotonically decrease as the rank increases. Although this is a natural property of SVD, it is desirable for changing the model size with maintaining the performance. It does not exactly hold for the entire network, but we can show that changes of the upper bound depends on the layer-wise error ‖yℓ - ˆ yℓ‖2 :

Algorithm 1 Learning Decomposable-Net

Require: Training set S, loss function L, neural network N with a set of weight matrices W = {Wℓ}Lℓ=1 in which Wℓ has rank Rℓ, and other trainable parameters Θ. Regularization parameter η, balancing parameter λ ∈ [0, 1], lower and upper rank ratios of αl and αu (0 < αl < αu ≤ 1), batch size B, and criterion C for rank selection

Ensure: Return N with learned W and Θ

• 1: InitializeW and Θ
• 2: repeat
• 3: Draw Z from the uniform distribution U(αl, αu)
• 4: Select the ranks of the weight matrices {rℓ}Lℓ=1 by reducing d = (1 - Z) ∑Lℓ=1 Rℓ bases across the entire network, according to the criterion C
• 5: Build a low-rank networkwith a set of weight matrices ˜ W, via rank-rℓ approximation of Wℓ for ℓ ∈ [1, L]
• 6: Draw a batch of samples {Db}Bb=1 from S
• 7: Compute the gradients of the objective function in Eq. (1), with respect to Wℓ ∈ W and θ ∈ Θ
• 8: UpdateWℓ ∈W and θ ∈ Θwith the gradients
• 9: until stopping criterion is met

Proposition 2. For a fully connected network with 1Lipschitz activation function (e.g., ReLU), let p and ˜ p be K-class softmax outputs from the full- and low-rank networks, respectively. Then, KL divergence between them can be bounded as KL(p|| ˜ p) ≤

where ‖ · ‖2 indicates the spectral norm.

Hence, it can be expected that the performance of entire network changes in conjunction with the rank in each layer. Regarding the Lipschitz constant ∏Lj=ℓ+1 ‖Wj‖22 , we experimentally verified that the empirical value is much smaller than the theoretical one [Wei and Ma, 2019] (see appendix E of [Yaguchi et al., 2019]).

Under the condition that a trained network has near lowrank weight matrices, [Arora et al., 2018; Suzuki et al., 2020a; Suzuki et al., 2020b] proved that the condition contributes not only to compressing the network efficiently, but also to yielding a better generalization error bound for the non-compressed (full-rank) network. That motivates our approach, namely, a learning scheme that aims to facilitate the performance of low-rank networks as well as that of a fullrank network.

3 RelatedWork

3.1 Low-Rank Approximations

Compression methods based on low-rank approximation have been proposed in the literature, and those can be divided into three categories. (1) Before learning, [Ioannou et al., 2016] factorized a convolutional kernel into horizontal and vertical kernels. However, the rank in each layer needs to be manually selected. (2) After learning, [Jaderberg et al., 2014;

Zhang et al., 2016] applied data-dependent optimization with the low-rank constraint. Alternatively, tensor methods (e.g., CP and Tucker decompositions) that directly approximate four-way convolutional tensors have been adopted [Lebedev et al., 2015; Kim et al., 2016]. However, since the approximation is applied after learning the model in each of these methods, the low-rankness is not necessarily ensured. Moreover, they optimize networks under a given target size, and thus optionally require retraining or fine-tuning to reconfigure the models for different devices. (3) During learning, [Alvarez and Salzmann, 2017; Xu et al., 2020] utilized trace-norm regularization as a low-rank constraint. The performance of these methods depends on a hyperparameter for adjusting the strength of regularization. It is difficult to decide on an appropriate range for the hyperparameter in advance, meaning that selection requires trial and error to achieve a particular model size.

Our methods are related to (3), in particular to Trained Rank Pruning (TRP) [Xu et al., 2020]. TRP minimizes the loss only for the low-rank network produced by SVD. Since it applies trace-norm regularization and truncates small singular values during learning, the resulting ranks in the low-rank network are fixed. Thus, the network does not perform well with other ranks, as shown in section 4.

3.2 Scalable Networks

[Yu et al., 2019; Yu and Huang, 2019] proposed slimmable networks and US-Nets, which are scalable in the width direction. These works are closely related to ours, but there are differences in some aspects. First, since these methods directly and uniformly reduce the width for every layer, the principal components are not taken into account, and the relative importance of each layer is not considered. Second, although the US-Nets allow arbitrary changes in width, monotonicity of error is not guaranteed when changing widths in each layer.

Recent works further acquired scalability in depth [Zhang et al., 2019], resolution [Cai et al., 2020; Yu et al., 2020; Yang et al., 2020], and kernel size [Cai et al., 2020; Yu et al., 2020]. The proposed methods are focused on improving the width scalability, but can be complementarily combined with those directions. This topic remains as our future works.

4 Experiments

We evaluate our methods on image-classification tasks of CIFAR-10/100 [Krizhevsky, 2009] and ImageNet [Deng et al., 2009] datasets using deep CNNs. For the CIFAR-10 dataset, we adopt a VGG-like network with fifteen layers (VGG-15) [Zagoruyko, 2015; Liu et al., 2017]. For the CIFAR-100 and ImageNet datasets, we adopt ResNet-34 and ResNet-50 [He et al., 2016], respectively. We follow the same baseline setup for the CIFAR datasets as used by [Zagoruyko and Komodakis, 2016], and the setup of [Yu et al., 2019] for the ImageNet dataset. The additional details of experiments are described in appendix D of [Yaguchi et al., 2019].

For all experiments, we report the results of the models after the last epoch. The results are averaged over multiple runs with different random seeds, specifically, with five seeds (0 ∼ 4) on the CIFAR datasets and three seeds (0 ∼ 2) on the ImageNet dataset. All methods are evaluated in terms of the tradeoff between validation (top-1) accuracy and the number of multiply-accumulate operations (MACs). The number of MACs is defined as ∑Lℓ=1 PℓHℓWℓ, where Pℓ, Hℓ, and Wℓ are the number of parameters, and the height and width of an output feature map, respectively. We compare channel- and spatial-wise decompositions for convolutional layers, but use the channel-wise one for 1×1 kernels, since they have no spatial redundancy. We experimentally tuned hyperparameters, and set αu = 0.25, 0.5, and 0.8, respectively for CIFAR-10, CIFAR-100, and ImageNet datasets, while fixing αl = 0.01 for all datasets. Except for the results shown in Figure 2(a), we set λ = 0.5 to balance the performance of a full- and lowrank models. We implemented other methods in the following experiments and evaluated them under our settings.

4.1 Ablation Study

We verify each component in our methods with ResNet-34 on the CIFAR-100 dataset. Firstly, we evaluate our learning method with various values for parameters λ ∈ [0, 1] that balance between two losses. Each model learned with a specific λ is evaluated for several sizes of the rank ratio Z ∈ {0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.75, 1} without retraining. The results are illustrated in Figure 2(a). Each curve in the figure shows the performance changes in a specific model size (e.g., 0.05 and 1 correspond to 5% and the full model, respectively) against λ, where points at λ = 0 correspond to results from normal learning. We can see that the performances of compressed models improve as λ increases, while that of the full model is almost flat. This implies that our learning method not only maintains the performance of full-rank network, but also improves multiple low-rank networks in a single model. Notably, the model with λ = 0.3 achieves 77.65% at full size, which is better than the 77.49% under normal learning, demonstrating that the compressible network generalizes well [Suzuki et al., 2020b]. The effect of our method is also verified in Figure 2(b), where ours (decomp-net) is consistently better than normal learning (base), with both channel- (ch) and spatial-wise (sp) decompositions.

Secondly, we evaluate the criterion for rank selection with fixed learning method. A method that uniformly reduce the rank across all layers (uni) and the energy-based method [Alvarez and Salzmann, 2017; Xu et al., 2020] (eng) are compared with ours (sv) in Figure 2(c). While eng is better than uni, sv successfully maintains the performance and is the best among them. The difference in selected ranks is illustrated in Figure 2(d). It can be observed that the rank distribution of sv is sharper than that of eng, which is nearuniform. We consider that eng does not fully characterize the importance of each basis, since it ignores the differences of the magnitude of singular values accross layers.

4.2 Comparison with Low-Rank Networks

First, we compare Decomposable-Net with TRP [Xu et al., 2020]. We implemented TRP referencing to their code 2 . TRP has three parameters, namely, the strength τ of trace norm regularization, a threshold e for truncating singular values, and a truncation interval T. TRP optimizes a model of a specific size, but the size of an obtained model depends on those parameters. Thus, we compare several models individually trained with τ ∈ {0.0001, 0.0003} and e ∈ {0.995, 0.999, 0.9995, 0.9999} for the CIFAR datasets, and τ ∈ {0.0001, 0.0003, 0.0005} and e ∈ {0.999995, 0.999999} for the ImageNet dataset. We report the results with T = 1 since it was the best among {1, 2, 4, 8} for all datasets.

2 https://github.com/yuhuixu1993/Trained-Rank-Pruning

The results with spatial-wise decomposition are summarized in Figure 3(a) (the additional results with channel-wise decomposition are illustrated in Figure 6 in the appendix of [Yaguchi et al., 2019]). TRP adopts the energy-based criterion for rank selection, and thus we compress their models with it at several sizes after training. Since TRP truncates small singular values during learning, the resulting ranks in the low-rank network are fixed. Thus, the network does not perform well at sizes other than obtained ranks, as shown in the figure. In contrast, although Decomposable-Net is a single model, it is basically better at almost all sizes. Moreover, our methods do not require exhaustive parameter searches to yield a good model at a specific size, since they require training only one time and changing size. Figure 2(e) shows an example of singular values obtained from each learning method. The singular values with TRP rapidly decrease to zero, because TRP applies trace-norm regularization. The bases with a singular value of zero can be reduced without inducing errors. However, we consider that this limits the representational power of larger models, as we can see the flat lines for TRP in Figure 3(a). On the other hand, our methods reduce singular values in comparison with normal learning, but do not push them to zero. We see that this contributes to improving the performance of low-rank networks while maintaining that of the full-rank network.

Next, we compare with retraining-based methods in which low-rank approximation and fine-tuning are applied after normal learning. Specifically, we adopt a method by [Kim et al., 2016] that directly approximates convolutional tensors with Tucker decomposition, where variational Bayesian matrix factorization (VBMF) [Nakajima et al., 2012] is applied to determine the ranks. We used publicly available code 3 for VBMF. For Tucker decomposition, we apply higher-order orthogonal iteration [De Lathauwer et al., 2000] under given ranks. Then, we perform fine-tuning with a grid search for selecting training parameters. The ranges for the search and the obtained parameters are described in appendix D of [Yaguchi et al., 2019].

The results with the best parameters are shown in Figure 3(b). We additionally evaluate channel- (ch) and spatialwise (sp) decompositions by switching them and individually optimizing the parameters. Note that sp vbmf ft corresponds to the method by [Tai et al., 2016], in which the rank selection method is changed from manual to VBMF. It can be observed that the performance significantly drops from base after decomposition, but can be recovered by retraining (dashed vertical lines). This is because the low-rankness is not ensured in the models trained with normal learning. Tucker decomposition is more efficient than other ones and is comparable to ours at a specific size on the CIFAR datasets. However, the performance drops when the size is changed, and retraining is required to recover the performance again. For practical use, such methods are not suited to reconfigure the same model on other devices with different resources. In this paper, we adopt the matrix (channel- and spatial-wise) decompositions, but it is possible to extend our methods with Tucker decomposition since it is a generalization of SVD to higher orders.

3 https://sites.google.com/site/shinnkj23/downloads

4.3 Comparison with Slimmable Networks

We compare Decomposable-Net with slimmable networks [Yu et al., 2019] and US-Nets [Yu and Huang, 2019]. We implemented them by referencing their code 4 . Slimmable networks are trained at four fixed widths of ratios {0.25, 0.5, 0.75, 1}. US-Nets are trained with a lower width of ratio 0.25, upper width of ratio 1, and two random widths between them. In-place distillation [Yu and Huang, 2019] is applied to US-Nets. The original loss of slimmable networks is an un-weighted sum of all training losses with different widths. Although it works well with ResNet-50 on the ImageNet dataset, the performance on the CIFAR datasets is unsatisfactory especially when comparing the accuracy (76.6%) of a full-width ResNet-34 with the baseline (77.5%) on the CIFAR-100 dataset. Therefore, we use the average instead of the sum on the CIFAR datasets.

The results are shown in Figure 3(c). For the CIFAR datasets, all methods have performance with the full model comparable to that of the baseline (base). However, we can see that the performances of slimmable networks and USNets decrease as MACs reduce, which are slightly faster than ours. This can be interpreted as those methods directly reducing the width (i.e., dimensionality) in each layer, and thus losing important directions in the weight space. To better understand this property, we illustrate an example of layer-wise error in Figure 2(f). It can be seen that the error for US-Nets rapidly increases as the number of parameters decreases in the layer, and this increase is not monotonic. In comparison, the error for Decomposable-Net increases slower than that for US-Nets, and monotonicity is ensured as expected. We see this is due to taking the principal components into account for Decomposable-Net, which contributes to maintaining performance over a wide range of model sizes. Moreover, we believe our criterion, which ununiformly select ranks in the network (as we see in Figure 2(d)) contributes too, in contrast to slimmable networks and US-Nets, which reduce the width uniformly across all layers.

4 https://github.com/JiahuiYu/slimmable networks

For the ImageNet dataset, we can see that the accuracy of Decomposable-Net is maintained for the full model better than under slimmable networks and US-Nets. Specifically, the accuracy of the full model is 76.48% for the baseline and 76.01% for Decomposable-Net, which is better than 75.31% for US-Nets. Moreover, Decomposable-Net achieves better accuracy than the baselines individually trained with 0.75× and 0.5× number of channels, while slimmable networks and US-Nets show lower accuracies than the baselines. It can be also observed that Decomposable-Net is better up to the model with around 0.2×MACs, but lower than slimmable networks and US-Nets when the compression rate is extremely high. Our methods use SVD and reduce the bases, which means it does not change the number of inputs and outputs (i.e., the input and output dimensionalities). The number of parameters in each layer is (mℓ + nℓ)rℓ, thus decreasing linearly with respect to the rank. US-Nets reduces both input and output dimensionality, meaning the number of parameters decreases at a quadratic rate. This makes it easier for US-Nets to achieve extremely high compression. Therefore, it can be worthwhile to combine our methods with pruning to reduce the number of input or output channels, by incorporating a channel-level sparseness constraint during learning [Alvarez and Salzmann, 2017]. We will consider this for future improvements.

4.4 Remarks

The results on the ImageNet dataset are summarized in Table 1, where we picked up results from the methods compared in this paper, for around 0.25×MACs models. To further validate our learning method, we decomposed ResNet-50 with the same ranks (i.e., exactly same MACs) as DecomposableNet in Table 1. Then, we trained it normally from scratch, and obtained the top-1 accuracy of 71.55±0.17%. Therefore, we see that the performance of our method (73.15±0.06%) is not easily achieved with the normal training, even by specializing the model to a specific size.

The wall-clock time for training Decomposable-Net with ResNet-50 on the ImageNet dataset was 6.9 days with eight NVIDIA V100 GPUs, whereas TRP (that also applies SVD during training) and US-Nets took 5.7 and 6.9 days, respectively. Therefore, the training cost of Decomposable-Net is not particularly high and comparable to them. Additionally, we measured the actual inference time per image with Intel Core i7-8700K CPU on the CIFAR-100 dataset. We used approximately 0.25×MACs model for Decomposable-Net with ResNet-34. For the other methods, we chose the models closest to ours in terms of MACs or accuracy. As shown in Table 2, we can see that Decomposable-Net is as fast as Tucker decomposition with the accuracy comparable to US-Nets.

5 Conclusions

We proposed a novel method that allows DNNs to flexibly change their size after training, by means of the rank of lowrank matrices in each layer. To simultaneously maintain the performance of a full-rank network and improve multiple low-rank networks in a single model, we proposed a novel learning method that explicitly minimizes losses for both of full- and low-rank networks in a differentiable way. In experiments on multiple image-classification tasks using deep CNNs, Decomposable-Net exhibited favorable performance at several sizes of compressed models, relative to that of the low-rank compression methods and slimmable networks.

Acknowledgements

TS was partially supported by JSPS KAKENHI (18K19793, 18H03201, and 20H00576), and JST CREST.