Learning Shared Knowledge for Deep Lifelong Learning using Deconvolutional Networks

1 Introduction

Much of the success of deep discriminative learning, including visual recognition [Krizhevsky et al., 2012; Simonyan and Zisserman, 2015; He et al., 2016] and image segmentation, stems from the remarkable ability of neural networks to generalize to unseen data that is drawn from the same underlying data generating process as the training set. Comparatively less success has been achieved when deep networks are deployed in an online multi-task learning (MTL) or lifelong learning fashion [Chen and Liu, 2016]. In these settings, the system faces the more formidable challenge of repeatedly generalizing to new data distributions (i.e., tasks), by leveraging knowledge of previously encountered tasks. Such a system should improve performance on new tasks via transfer, without compromising performance (i.e., catastrophic forgetting [Kirkpatrick et al., 2017]) on previously encountered tasks, and additionally permit new knowledge to benefit previously learned tasks (i.e., reverse transfer [Ruvolo and Eaton, 2013]). Therefore, in contrast to single-task learning, lifelong learning systems must exploit the relationships that exist between the tasks to facilitate transfer.

A natural approach to the lifelong learning problem is to exploit the compositional nature of neural networks. In general, we expect a deep neural net to learn hierarchical features whose level of abstraction correlates closely with depth in the network. Many methods for MTL and lifelong learning with deep nets exploit this property, sharing lower layers of the network between tasks while enforcing separate topmost layer(s) for each task. Such hard parameter sharing (HPS) methods typically train the shared lower layers in a multi-task setting, under the hope that they will acquire universal features suitable for multiple related tasks [Baxter, 2000]. These methods support limited transfer to new tasks by training task-specific classifiers on top of these pre-trained lower layers.

Rather than imposing rigid task relationships by explicit weight sharing, it is desirable to learn task relationships organically from data. Non-deep MTL methods [Kumar and Daume, 2012; Maurer et al., 2013; Ruvolo and Eaton, 2013] have shown success with this notion, enabling transfer between tasks by factorizing the model parameter space via a shared latent knowledge base. Each task-specific model is then reconstructed as a linear combination of components from the shared knowledge base. Although effective in providing flexible transfer, this approach is currently limited to learning with shallow models. Deep MTL methods such as tensor factorization [Yang and Hospedales, 2017], progressive neural networks [Rusu et al., 2016], and dynamic filters [Jia et al., 2016; Ha et al., 2017] relate multiple deep networks and permit transfer, but do not provide sufficiently flexible transfer mechanisms nor support all characteristics (such as reverse transfer) needed for lifelong learning.

Such a factorized knowledge base used for shallow MTL, however, could be adapted to connect the weight matrices between multiple deep networks since a deep network can be thought of as a stack of shallow models. This factorized form of the knowledge base would permit flexible transfer between multiple deep networks, adapting the transfer based on the similarity of each network's respective task.

This idea is the focus of this paper, in which we propose a new shared-knowledge neural architecture that exploits the structure of convolutional neural networks (CNNs). Inspired by the success of deconvolutional networks for image segmentation [Noh et al., 2015], we introduce a deconvolutional approach to share latent knowledge across multiple CNNs. Our approach, deconvolutional factorization, can be viewed Fully-connected layers

256 64

Tensor contraction

Deconvolutional

mappings

Shared

knowledge base

Deconvolutional

mapping

Tensor contraction

as conceptually analogous to using a sparse factorization of the linear model parameter space for transfer in shallowMTL [Kumar and Daume, 2012; Maurer et al., 2013] and lifelong learning [Ruvolo and Eaton, 2013] methods, but generalized to deep networks. In our adaptation of factorized transfer to deep convolutional networks, the model parameters to be factored are the filters of the convolutions on each layer, and the sparsity condition follows from the fact that the deconvolution operator maps from a compact latent representation to a higher-dimensional feature space. This contrasts with existing approaches which utilize tensor factorization [Yang and Hospedales, 2017] with regularization to achieve sparsity.

We evaluate our proposed method on image recognition in challenging lifelong learning settings, where the system learns tasks consecutively. Our results showcase the strong benefits of the flexible transfer process provided by deconvolutional factorization over HPS, progressive networks, and other current methods. We also show that our approach converges rapidly to a high-performance model by utilizing knowledge transfer, and that it resists catastrophic forgetting.

2 The Deconvolutional Factorized CNN

A lifelong learning system faces a series of consecutive tasks Z1, . . . ,ZTmax , and must learn a model (i.e., a classifier) for each task [Chen and Liu, 2016]. The system has no a priori information about the task distribution, order, or total number of tasks Tmax. This paper focuses on the classification setting, where each task Zt has an associated data feature space Xt and label space Yt, from which labeled examples are drawn. Here, each learning task Zt admits an associated convolutional neural network CNNt : Xt ↦→ Yt trained on labeled data for that task. Each CNNt has d layers and is trained consecutively, possibly with transfer from any previously learned tasks Z1, . . . , Zt-1.

Our architecture, called a deconvolutional factorized CNN (DF-CNN), seeks to address this lifelong learning problem using deep convolutional networks with a shared knowledge base to enable transfer between tasks (Fig. 1). To facilitate transfer, our architecture maintains a shared latent knowledge base that connects the various layers across the task-specific CNNs. Recall that a CNN is composed of multiple layers of stacked filters, each of which is parameterized. The filters of the CNNs are generated from the learned latent knowledge base by the deconvolution operator (transposed convolution), followed by a tensor contraction. The remainder of this section explains this process. Unlike previous methods that involve tensor factorization to achieve sparsity, our proposal is naturally sparse by virtue of the deconvolution operator.

nL(l)o

2.1 Factorized Transfer

To enable transfer among the different CNNt task models, we draw inspiration from the use of factorized transfer in shallow MTL [Kumar and Daume, 2012; Maurer et al., 2013] and lifelong learning [Ruvolo and Eaton, 2013] methods. These shallow methods learn a set of T task-specific linear models parameterized by W = [θ1, . . . , θT ] ∈ Rd×T and assume that these model parameters admit a rank-constrained matrix factorization of the form W = LS, where L ∈ Rd×k is a basis over the model parameter space, S ∈ Rk×T are the coefficients over this basis to reconstruct the parameters, and k is the dimension of the latent space. In effect, these approaches learn a knowledge base L that represents a shared subspace for the model parameters, and facilitate transfer to new tasks by learning models within this subspace.

Since these methods only operate on linear task models, we cannot adopt them directly for use on deep nets. However, we show next that we can adapt this notion of a shared knowledge base in combination with a novel type of factorized transfer via deconvolution to operate effectively on CNNs.

2.2 Factorized Transfer via Deconvolution

For each convolutional layer l ∈ {1, . . . , d} of each taskspecific CNNt, let W (l) t ∈ Rh×w×cin×cout denote its corresponding filters where h and w are the filter height and width, and cin and cout are the numbers of input and output channels.

To enable transfer between the convolutional layers of different CNNt task models, we introduce a task-independent layer-dependent shared knowledge base L(l) for each layer l, which is shared across all tasks. Following similar assumptions used in factorized transfer for shallow models, we assume that each W (l) t is derived from the corresponding shared latent knowledge base L(l) , enabling connections between filters at the l-th layer of different task models.

Specifically, we utilize a deconvolutional mapping and a tensor contraction to factorize the filters {W (l) t }Tt=1 into the shared knowledge base L(l) , which we take to be a 3rd-order tensor L(l) ∈ Rˆh× ˆw׈ c . We first deconvolve L(l) into

where D(l) t is a 3rd-order tensor of size h×w× c, V (l) t ∈ Rp×p׈ c×c is the filter of the task-dependent deconvolutional mapping, and p is the spatial size of the deconvolutional filters. The deconvolutional mapping learns to generate a basis of convolutional filters within L(l) . We then apply the tensor contraction to reconstruct each W (l) t based on D(l) t :

where U (l) t is a 3rd-order tensor of size c × cin × cout, and both subscripts (k, ·, ·) and (·, ·, k) express the elements' index in the tensor. Similar to channel-wise convolution, the tensor contraction expresses the filter as a linear combination of the basis vectors, transforming D(l) t to reconstruct the convolution filter W (l) t by changing the size of channels. Note

Algorithm 1 DF-CNN (λ, kbSize, transformSize)

L(1:d) ← randInit(kbSize) while isAnotherTaskAvailable() do (Xt, yt, t) ← getNextTaskTrainingData() if isNewTask(t) then (V (1:d) t , U (1:d) t ) ← randomInit(transformSize) end if while continueBatchTraining() do // reconstruct NN from shared KB for l = 1 to d do D(l) t ← deconv(L(l) , V (l) t ) W (l) t ← tensorDot(D(l) t ,U (l) t ) end for taskNet t ← buildNeuralNet (W (1:d) t ) // update shared KB and task-specific transforms Xb, yb ← drawMiniBatch(Xt, yt) (L(1:d) , V (1:d) t , U (1:d) t ) ← gradientOptimizer(Xb, yb, taskNet t, λ) end while end while

that V (l) t and U (l) t are task-specific, and serve to transform the shared knowledge bases into a model specific to task Zt.

Similar to the work of dynamic filter generation [Jia et al., 2016; Ha et al., 2017], a single operation can be employed to expand the knowledge base into a large task-specific filter. However, in our proposal, deconvolution and tensor contraction are used instead as a two-staged expansion, distinguishing between the transfer process along the spatial axis of the images and along the channels of the images. We also explore two alternate formulations of factorized transfer in the Online Appendix 1 as ablated versions of our approach.

2.3 Training DF-CNNArchitecture

Our learning approach must update both the shared knowledge bases and task-specific knowledge transformations while training on each task in a lifelong setting. The knowledge bases {L(l) }dl=1 and task-specific knowledge transformations {(V (l) t ,U (l) t )}dl=1 can be trained end-to-end via gradient-based optimization. The process of training the DF-CNN is described in Algorithm 1. As hyperparameters, the algorithm requires the learning rate λ of the optimizer, the size of the knowledge bases (kbSize ∈ R3d ), and the dimensions of the task-specific tensors V (l) t and U (l) t (transformSize ∈ R(4d+3d) ).

The shared knowledge bases {L(l) }dl=1 are randomly initialized prior to learning the first task. Upon receiving the labeled training data for each new task Zt, the task-specific knowledge transformations {(V (l) t ,U (l) t )}dl=1 are first initialized randomly, and then these parameters along with the knowledge bases {L(l) }dl=1 are updated according to the observed training instances (Xt, yt). During training on task Zt, the knowledge transformations for all tasks except t are held unchanged, while the shared knowledge bases are updated. Since the convolutional filters for each CNNt are generated dynamically from the shared knowledge bases, changes to {L(l) }dl=1 can affect previously trained networks CNN1, . . . , CNNt-1 without retraining those networks; this phenomenon is known as reverse transfer [Ruvolo and Eaton, 2013]. Despite the lack of explicit mechanisms to prevent catastrophic forgetting [Rusu et al., 2016] (i.e., severe negative reverse transfer) in these previously learned models, we show empirically that deconvolutional factorization of the model parameter space resists catastrophic forgetting and indeed exhibits positive reverse transfer-these results mirror similar results on lifelong learning of shallow linear models via factorized transfer [Ruvolo and Eaton, 2013].

1 The online appendix is available on the third author's website at http://www.seas.upenn.edu/∼eeaton/papers/Lee2019Learning.pdf

3 Alternative Approaches to DeepMulti-Task and Lifelong Learning

Now that we have introduced the DF-CNN, we briefly compare it to alternative approaches for sharing inter-task knowledge in deep networks (see Fig. 2).

Explicit weight-sharing. Hard parameter sharing (HPS) is widely used for MTL and lifelong learning in neural networks [Caruana, 1993; Yim et al., 2015; Ranjan et al., 2017; Huang et al., 2013; Bell and Renals, 2015], sharing lower network layers for feature extraction with task-specific output layers. The lowest layers can also be specific to the input domain to allow adaptation across different feature spaces. The explicit sharing of layers forces them to learn universal features for multiple tasks, with the task-specific layers mapping from the universal features to the output appropriate to each task. One variant of this architecture correlates task-specific fully-connected layers by a tensor normal distribution to learn the relations between tasks [Long et al., 2017]. Moreover, there are deep lifelong learning methods that automatically modify the architecture and size of the neural network [Lu et al., 2017; Yoon et al., 2018]. This type of approach can add hidden units as necessary, split them into disjoint groups for different feature spaces, and consolidate groups of hidden units to avoid over-fitting and enforce transfer between tasks. Although these methods are more flexible than HPS for learning task relationships, they still explicitly share lower layers across all or partial sets of tasks. In contrast, our approach uses a shared knowledge base to flexibly relate layers across the task CNNs instead of explicitly sharing them.

Pipelined transfer. Instead of learning a monolithic network with explicit weight sharing, another approach to deep MTL and lifelong learning constructs task-specific subnetworks that learn each task, but make additional lateral connections to utilize learned representations from other tasks [Misra et al., 2016; Rusu et al., 2016; Gao et al., 2019; Pinto and Gupta, 2017; Liu et al., 2017b]. This architecture enables the network to learn and maintain task-dependent low- and high-level features, so it is more robust to handling diverse tasks and to avoiding catastrophic forgetting than explicit weight sharing. However, these approaches need to train the cross-task connections, so the size of the network increases at most quadratically with the number of tasks. The quantity of task-specific parameters in our DF-CNN is linearly proportional to the number of tasks Tmax , so the total network size grows more slowly with Tmax while maintaining the flexibility of sharing representations at any level.

Shared knowledge base. Approaches that transform shared knowledge to construct task-specific networks have a clear connection to our architecture. Sharable detectors for face alignment [Liu et al., 2017a] have used the sparse representation of a shared basis to express a 2nd-order tensor of regression sub-networks for facial landmarks. As another example, deep MTL via tensor factorization [Yang and Hospedales, 2017] employed tensor decomposition to define a sharable basis and task-specific mappings of both 2ndorder tensors for fully-connected layers and 4th-order tensors for convolutional layers. Despite similar underlying ideas, our method exploits a deconvolution operation for mapping between the knowledge base and the task-specific filter parameters, thereby enabling the network to extract and share more abstract knowledge across tasks. The key advantage of our deconvolutional factorization is that it provides a flexible mechanism for sharing knowledge between task CNNs, allowing the architecture to flexibly reuse knowledge based on actual task relationships.

Dynamic filter generation. Our proposed idea is also related to architectures that employ dynamic filters [Jia et al., 2016; Ha et al., 2017], which dynamically generate taskspecific CNN filters conditioned upon each task's input. The filter-generating network of these dynamic filters can serve to facilitate transfer across tasks. These works focus primarily upon learning domain-specific transformations (e.g., steerable filters) to relate rather similar tasks, whereas our approach learns shared representations to facilitate flexible transfer between tasks at multiple levels of abstraction. Additionally, the filter-generating networks typically employ a general-purpose neural network architecture (i.e., a multilayer perceptron or a convolutional network), which takes only the input space and desired output spaces into consideration. Consequently, the filter-generating networks and resulting filters may not differentiate between spatial patterns and patterns across channels-key properties of convolutional filters [Chen et al., 2017; Hu et al., 2018]-which is distinguished in the DF-CNN by separate deconvolution and tensor contraction steps.

4 Evaluation on Lifelong Learning Scenarios

We evaluated our proposed approach against a variety of alternative methods on lifelong learning scenarios using two visual recognition data sets. For completeness, an empirical evaluation on MTL scenarios is included in the Appendix 1 .

4.1 Baseline Approaches

Our experiments compare our proposed approach to singletask learning and three baselines that are representative of the different approaches described in Section 3:

Single-task learning. STL trains a separate and isolated network for each task. STL has a clear disadvantage against transfer-based methods in the few-data or noisy-data regime.

Hard parameter sharing. HPS [Caruana, 1993] involves sharing the lower layers across tasks, with separate taskspecific output layers. A useful heuristic which we follow is that all convolution layers are shared while all fullyconnected layers are task-specific. HPS is expected to perform well when tasks share a common set of useful representations, but may fail when tasks are sufficiently dissimilar.

Progressive neural networks. ProgNN [Rusu et al., 2016] allows each task model to build upon its predecessors. Note that their paper applied the architecture to reinforcement learning, while our evaluation focuses on supervised settings. For the construction of the ProgNNs, we reduced the dimension of the previous task models' features by a factor of two (for details, see [Rusu et al., 2016, Section 2: Adapters]).

Dynamically expandable network. DEN [Yoon et al., 2018] grows the size of the neural network according to the performance on the current task. The DEN adapts to a new task by a combination of selective retraining, expansion of the network to improve performance, and splitting to avoid catastrophic forgetting. We used the DEN implementation provided by the original authors.

We also evaluated the performance of tensor factorization [Yang and Hospedales, 2017] on MTL scenarios (see the Appendix). However, the method showed reasonable performance only when it was initialized by trained STL models, so we omitted the tensor factorization method from the lifelong learning experiments reported here in the main paper.

4.2 Experimental Setup

We evaluated all approaches in a lifelong learning setting, in which tasks were presented sequentially.

Data sets and tasks. We generated two lifelong learning problems using the CIFAR-100 [Krizhevsky and Hinton, 2009] and Office-Home [Venkateswara et al., 2017] data sets. For CIFAR-100, we created a series of 10 image classification tasks, where each task consists of ten distinct classes. For each task, we sampled only 4% of the available CIFAR100 data following a lifelong learning assumption of limited per-task training data [Chen and Liu, 2016], and split it into training and validation sets in the ratio 5.6:1 (170 training and 30 validation instances per task). We used all CIFAR100 test images for the test set (1,000 instances per task). The Office-Home dataset is naturally split into multiple domains, and we focus on two of those domains: Product images and Real-World images. We created 5 image classification tasks from each of these two domains, resulting in 10 tasks with 13 image classes per task. There is no pre-specified training/validation/test split in the original data set, so we randomly split the data into those with a 60%, 10%, and 30% ratio, respectively. This results in approximately 550 training, 90 validation, and 250 test instances.

Methodology. All models were trained end-to-end on only one task at any moment, and the task was switched to the next one after every 2,000 (CIFAR-100) and 1,000 (Office-Home) training epochs, regardless of the model's convergence. The optimal hyper-parameters for each model were determined by performance on the validation sets. In addition to the lifelong learning baseline models, we compared two versions of STL: one with 3.28M (CIFAR-100) or 26.8M (Office-Home) parameters total (328k or 2.68M per individual task model) and the other with 9.35M (CIFAR-100) or 129.3M (Office-Home) parameters total (935k or 12.9M per task model). Our full DF-CNN has 7.96M (CIFAR-100) or 201.8M (Office-Home) parameters total, so we also examined a reduced-size DFCNN with 2.8M parameters total in order to better match the total number of parameters of the baseline approaches in the CIFAR-100 experiments. Details of training process, architecture of the networks and hyper-parameters used for each data set are described in Appendix B. Note that, since the purpose of the evaluation is to compare the approaches for knowledge transfer across tasks in a lifelong learning setting, the experiment design (and consequently individual model performance) may differ from other single-task-focused publications on these methods or data sets.

Metrics. We assessed performance of our DF-CNN as well as aforementioned approaches by measuring accuracy on the held-out test set for all learned tasks at each timestep. To aggregate performance across tasks, we also computed the following metrics:

• Peak Per-Task Accuracy: The best test accuracy of each task during its training phase. This metric focuses on the approach's peak performance on the current task.
• Catastrophic Forgetting Ratio: The ratio of a task's test accuracy after training on subsequent tasks to its peak per-task accuracy. This ratio shows how much the approach can maintain its performance on older tasks.
• Convergence: We measure the convergence of training on a task as the number of epochs over the training set needed for the test accuracy to reach 98% of its peak pertask accuracy. The number of training epochs till convergence shows the effect of knowledge transfer from previously learned tasks.

4.3 Results on Lifelong Learning

The performance of all approaches is summarized in Figures 3 (CIFAR-100) and 5 (Office-Home). For the CIFAR100 experiments, we also depict the lifelong learning process by visualizing the dynamic test accuracy of each task model over time, averaged over 5 trials, in Fig. 4. Once a task has been learned, we repeatedly evaluated its model's performance as the system learns more tasks, exploring the effect of learning subsequent tasks on previous task models. Significant decreases in task performance after training indicate catastrophic forgetting [Kirkpatrick et al., 2017]; increases in performance when training on other tasks indicate (positive) reverse transfer. The counterpart to Fig. 4 for the OfficeHome experiments is in Appendix D.

First, we can observe that HPS and DEN suffer from catastrophic forgetting as the shared layers were adapted to new tasks, as shown by the rapid decline in performance once learning on each task finishes (Fig. 4a as well as Fig. 3b and Fig. 5b). Additionally, both models could not achieve a peak per-task accuracy comparable to or better than that of STL consistently in the the CIFAR-100 experiments, and even HPS did not converge to peak per-task accuracy faster than STL. This means that the adaptation of the knowledge in these explicit weight-sharing models is not guaranteed to have a positive effect on training, even after the neural network shifts its attention to focus more on current tasks and forget knowledge of previous tasks.

In contrast to HPS and DEN, ProgNN is able to retain its performance on previous tasks after learning new tasks, because it is designed not to update the parameters for previous tasks. The lateral connections of ProgNN improved test accuracy in a few tasks (e.g., the 7-9th tasks of the CIFAR100 experiment), but the benefit of transfer was marginal in comparison with the single-task learners. Moreover, ProgNN requires approximately twice as much training time as others.

DF-CNN showed significant improvement in peak per-task accuracy over STL, HPS, and ProgNN for the CIFAR-100 ex-

(a) Peak per-task accuracy and training

time with 95% confidence intervals

periments, and peak per-task accuracy better than STL for the Office-Home experiments. Moreover, DF-CNN converges to 98% of peak per-task accuracy more than twice as fast as other approaches do. The improvement in peak per-task accuracy compared to STL, together with the improved convergence speed relative to STL, demonstrate positive knowledge transfer from older tasks within the DF-CNN framework.

The performance of previous task models within the DFCNN deteriorated slightly as the system observed new tasks because the shared knowledge base was updated by the new tasks without consideration to previous tasks (Fig. 3b and 5b). However, the rate of losing performance is much slower than the degradation of HPS and DEN in both lifelong learning experiments, and we can even observe performance recovering over time (Fig. 4c). Especially, in the CIFAR100 experiments (Fig. 4c), the performance of the model on the earliest tasks appears to have the most degradation, and it maintains almost constant post-training performance once the shared knowledge base became mature (e.g., the 4-10th tasks). Moreover, we can find positive reverse transfer of knowledge from new tasks to older tasks, starting at the 8th task of the CIFAR-100 experiments, which had not occurred during the training of other baseline models.

The reduced-size DF-CNN (∼2.8M trainable parameters) catastrophically lost its performance on the first task because of its reduced capacity of the shared knowledge. Even with the limited capacity of both shared knowledge and task-specific knowledge transformation, the reduced-size DFCNN still showed improvement in accuracy, speed of convergence and robust retention of performance on previous tasks as compared to the baselines. These results support the benefit of knowledge transfer between tasks through the shared knowledge and deconvolutional mapping.

In summary, the alternative approaches to lifelong learning achieved improvement in peak per-task accuracy in comparison with STL on one of two data sets, but they slowly converged to their peak performance and showed other weaknesses such as catastrophic forgetting or large training time. Contrary to this, our DF-CNN achieved good peak per-task accuracy, fast convergence, knowledge retention and reasonable training time simultaneously. This result supports that our DF-CNN is able to extract and transfer knowledge between tasks more flexibly than competing methods while resisting catastrophic forgetting in the lifelong setting.

5 Conclusion

Deconvolutional factorization provides an effective means of knowledge transfer between CNNs in lifelong learning settings. Even though our DF-CNN architecture must train more parameters as compared to other approaches with the same base model (i.e., the individual task CNNs), it converges faster while providing comparable or better accuracy than competing approaches. Critically, the use of deconvolutional factorization and tensor contraction provides for flexible transfer between tasks, enabling the DF-CNN to resist catastrophic forgetting.

Acknowledgements

We would like to thank Sima Behpour, Jorge Mendez, Boyu Wang, and the anonymous reviewers for their helpful feedback on this work. The research presented in this paper was partially supported by the Lifelong Learning Machines program from DARPA/MTO under grant #FA8750-18-2-0117 and by AFRL grant #FA8750-16-1-0109.