Tri-net for Semi-Supervised Deep Learning

1 Introduction

Deep neural networks (DNNs) have become a hot wave during the past few years, and great successes have been achieved in various real applications, such as image classification [Krizhevsky et al., 2012], object detection [Girshick et al., 2014], scene labeling [Shelhamer et al., 2017], etc. DNNs always learn a large number of parameters requiring a large amount of labeled data to alleviate overfitting. It is well-known that collecting tremendous high-quality labeled data is expensive, yet we could easily collect abundant unlabeled data in many real applications. Hence, it is desirable to use unlabeled data to improve the performance of DNNs when training with limited labeled data.

A natural idea is to combine semi-supervised learning [Chapelle et al., 2006; Zhu, 2007; Zhou and Li, 2010] with deep learning. The disagreement-based learning [Zhou and Li, 2010] plays an important role in semi-supervised learning, in which co-training [Blum and Mitchell, 1998] and tri-training [Zhou and Li, 2005b] are two representatives. The basic idea of disagreement-based semi-supervised learning is to train multiple learners for the task and exploit the disagreements during the learning process. The disagreement in cotraining is based on different views, while tri-training uses bootstrap sampling to get diverse training sets. Co-training has been combined with deep model for the tasks which have two views [Cheng et al., 2016; Ardehaly and Culotta, 2017]. Nevertheless, in real applications, we always confront the task with one-view data, and tri-training can be utilized no matter whether there are one or more views.

In this paper, we propose tri-net which combines tritraining with deep model. We first learn three initial modules, and each module is then used to predict a pool of unlabeled data, where two modules label some unlabeled instances for another module. Later, three modules are refined by using the newly labeled examples. We consider three key techniques in tri-net, i.e., model initialization, diversity augmentation and pseudo-label editing, which can be summarized as follows: we use output smearing [Breiman, 2000] to help generate diverse and accurate initial modules; we finetune the modules in some specific rounds on labeled data to augment the diversity among them; we propose a data editing method named DES based on the intuition that stable pseudolabels are more reliable. Experiments are conducted on three benchmark datasets, i.e., MNIST, SVHN and CIFAR-10, and the results demonstrate that our tri-net has good performance on all datasets. In particular, it achieves 8.45% error rate on CIFAR-10 by using only 4,000 labeled examples. With more sophisticated initialization methods, tri-net can get even better performance. For example, when we use the semisupervised deep learning method Π model [Laine and Aila, 2016] to initialize our tri-net, we can achieve 8.30% error rate on CIFAR-10 by using only 4,000 labeled examples.

The rest of this paper is organized as follows: we introduce related work in Section 2 and present our tri-net in Section 3. Experimental results are given in Section 4. Finally, we make a conclusion in Section 5.

2 RelatedWork

Many methods have been proposed to tackle semi-supervised learning, we only introduce the most related ones. For more information of semi-supervised learning, see [Chapelle et al., 2006; Zhu, 2007; Zhou and Li, 2010].

Disagreement-based semi-supervised learning started from the seminal paper of Blum and Mitchell [1998] on cotraining. Co-training first learns two classifiers from two views and then lets them label unlabeled data for each other to improve performance. However, in most real applications the data sets have only one view rather than two. Some methods employed different learning algorithms or different parameter configurations to learn two different classifiers [Goldman and Zhou, 2000; Zhou and Li, 2005a]. Although these methods do not rely on the existence of two views, they require special learning algorithms to construct classifiers. Zhou and Li [2005b] proposed tri-training, which utilizes bootstrap sampling to get three different training sets and generates three classifiers from these three training sets respectively. Tritraining requires neither the existence of multiple views nor special learning algorithms, thus it can be applied to more real applications. For these algorithms, there have been some theoretical studies to explain why unlabeled data can improve the learning performance [Blum and Mitchell, 1998; Balcan et al., 2004; Wang and Zhou, 2010; Balcan and Blum, 2010].

With the fast development of deep learning, disagreementbased semi-supervised learning has been combined with deep model for some applications. Cheng et al. [2016] developed a semi-supervised multimodal deep learning framework based on co-training to deal with the RGB-D object-recognition task. They utilized each view (i.e., RGB and depth) to learn a DNN and the two DNNs labeled unlabeled data to augment the training set. Ardehaly and Culotta [2017] combined cotraining with deep model to address the demographic classification task. They generated two DNNs from two views (i.e., image and text) respectively and let them provide pseudolabels for each other. Nevertheless, many tasks have only one view in real applications. It is more desirable to develop the disagreement-based deep models for one-view data.

There are many other methods in semi-supervised deep learning. Some of them were based on generative models. These methods paid efforts to learn the input distribution p(x). Variational auto-encoder (VAE) combined variational methods with DNNs to help estimate p(x) [Kingma et al., 2014; Maaløe et al., 2016] while generative adversarial networks (GANs) aimed to leverage a generator to detect the low-density boundaries [Salimans et al., 2016; Dai et al., 2017]. In contrast to the generative nature, our tri-net is a discriminative model and does not need to estimate p(x). Some combined graph-based methods with deep neural networks [Weston et al., 2012; Luo et al., 2017]. They enforced smoothness of the predictions with respect to the graph structure while we do not need to construct the graph. Some were perturbation-based discriminative methods. They utilized local variations of the input to regularize the output to be smooth [Bachman et al., 2014; Rasmus et al., 2015; Laine and Aila, 2016; Sajjadi et al., 2016]. VAT [Miyato et al., 2017] and VAdD [Park et al., 2018] introduced adversarial training [Goodfellow et al., 2014] into these methods while temporal ensembling [Laine and Aila, 2016] and mean teacher [Tarvainen and Valpola, 2017] introduced ensemble learning [Zhou, 2012] into them. Compared with these state-of-the-art methods, our method can achieve better performance.

3 Our Approach

3.1 Overview

In semi-supervised learning, we have a small labeled data set L = {(xl, yl)|l = 1, 2, . . . , L} with L labeled examples and a large-scale unlabeled data set U = {(xu)|u = 1, 2, . . . , U} with U unlabeled instances. Suppose the data have C classes and yl = (yl1, yl2, . . . , ylC), where ylc = 1 if the example belongs to the c-th class otherwise ylc = 0, for c = 1, 2, . . . , C. Our goal is to learn a model from the training set L ∪ U to classify unseen instances. In this paper, we propose tri-net by combining tri-training with deep neural network. Our tri-net has three phases which are described as follows.

Initialization. The first step in tri-net is to generate three accurate and diverse modules. Instead of training three networks separately, tri-net is one DNN which is composed of a shared module MS and three different modules M1, M2 and M3. Here, M1, M2 and M3 classify the shared features generated by MS. This network structure is inspired by Saito et al. [2017] and is efficient for implementation. In order to get three accurate and diverse modules, we use output smearing (Section 3.2) to generate three different labeled data sets, i.e., L1os , L2os and L3os . We train MS, M1, M2 and M3 simultaneously on the three data sets. Specifically, MS and Mv are trained on Lvos (v = 1, 2, 3).

Training. In the training process, some unlabeled data will be labeled and added into the labeled training sets. In order not to change the distribution of labeled training sets, we assume that the unlabeled data are selected from a pool of U. We use N to denote the size of the pool. This strategy is widely used in semi-supervised learning [Blum and Mitchell, 1998; Zhou and Li, 2005a; Saito et al., 2017]. With three modules, if two modules agree on the prediction of the unlabeled instance from the pool and the prediction is confident and stable, the two modules will teach the third module on this instance. The instance with the pseudo-label predicted by the two modules is added into the training sets of the third module. Then the third module is refined with the augmented training set. Here, confident prediction means that the average maximum posterior probability of the two modules is

Algorithm 1 Tri-net

Input:

Labeled set L and unlabeled set U

Labeling: the methods of labeling when the predictions of two classifiers are confident and agree with each other

DES: the methods of pseudo-label editing

σ0: the initial threshold parameter for filtrating the unconfident pseudo-labels

σos: the value to decrease σ if output smearing is used in this learning round

Output:

Tri-net: the model composed of MS, M1, M2 andM3

• 1: Initialization:

2: Generate {L1os, L2os, L3os} by using output smearing on L 3: Train MS,M1,M2,M3 with mini-batch from training set L1os, L2os, L3os 4: flagos = 1; σ = σ0 5: Training: 6: for t = 1 → T do 7: Nt =min(1000 ×2t ,U) 8: if Nt = U then 9: if mod(t, 4) = 0 then 10: Train MS,M1,M2,M3 with mini-batch from training set L1os, L2os, L3os 11: f lagos = 1 ; σ = σ - 0.05 12: continue 13: if flagos = 1 then 14: f lagos = 0 ; σt = σ - σos 15: else 16: σt = σ 17: for v = 1 → 3 do 18: PLv ← ∅ 19: PLv ←Labeling(MS,Mj,Mh, U,Nt, σt)(j, h ̸= v) 20: PLv ←DES(MS,PLv,Mj,Mh) 21: ˆ Lv ← L ∪ PLv 22: if v = 1 then 23: TrainMS,Mv with mini-batch from training set ˆ Lv 24: else 25: TrainMv with mini-batch from training set ˆ Lv 26: return MS,M1,M2 andM3

larger than the threshold σ. Stable prediction means that the pseudo-label should not change much when the modules predict the instance repeatedly and the details will be presented in Section 3.4. Three modules will be more and more similar since they augment the training sets of one another [Wang and Zhou, 2017]. To tackle this problem, we fine-tune the modules on labeled data to augment the diversity among them in some specific rounds. The whole training process is shown in Algorithm 1.

Inference. Given an unseen instance x, we use the average of the posterior probability of the three modules as the posterior probability of our method. The unseen instance x is classified with maximum posterior probability shown in Eq. 1, where MS denotes the shared module and Mv (MS(x)) denotes the label predicted by Mv (v = 1, 2, 3) on x.

3.2 Output Smearing

Output smearing was proposed by Breiman [2000]. It constructs diverse training sets by injecting random noise into true labels and generates modules from the diverse training sets respectively. Injecting noise into true labels can also regularize the modules by smoothing the labels [Szegedy et al., 2016]. We apply this technique to initialize our modules M1, M2 and M3. For an example {xl, yl} (l = 1, 2, . . . , L), where yl = (yl1, yl2, . . . , ylC), ylc = 1 if the example belongs to the c-th class otherwise ylc = 0. In output smearing, we add noise into every component of yl.

where zlc is sampled independently from the standard normal distribution, std is the standard deviation, ReLU is a function

Here, we use ReLU function to ensure ˆ ylc non-negative and normalize ˆ ylc according to Eq. 4.

With output smearing, we construct three diverse training sets L1os ,L2os and L3os from the initial labeled data set L, where Lvos = {(xl, ˆ yv l )|1 ≤ l ≤ L} (v = 1, 2, 3) is constructed by output smearing and ˆ yv l is calculated according to Eq. 4. Then we initialize tri-net with L1os , L2os and L3os by minimizing Loss shown in Eq. 5.

Here, Ly denotes the standard softmax cross-entropy loss function, MS denotes the shared module, M1,M2 and M3 denote the three modules in tri-net, Mv (MS(xl)) denotes the output of Mv on xl where Mv classifies the features generated byMS on xl (v = 1, 2, 3).

3.3 Diversity Augmentation

Diversity among three modules in tri-net plays an important role in the training process. When three modules label unlabeled data to augment the training sets of one another, they become more and more similar. In order to maintain the diversity, we fine-tune three modules M1, M2 and M3 on the diverse training sets L1os ,L2os and L3os in some specific rounds. In the experiments, the fine-tuning is executed every 3 rounds, which will be described in Section 4.

3.4 Pseudo-Label Editing

The pseudo-labels of the newly labeled examples may be incorrect, and these incorrect pseudo-labels will degenerate the performance. Data editing which can deal with the suspicious pseudo-labels is important and there have been some data-editing methods in semi-supervised learning [Zhang and Zhou, 2011]. However, these existing methods are usually based on graph and are difficult to be used in DNNs due to the high dimension. Here, we propose a new data-editing method for DNNs with dropout [Srivastava et al., 2014]. Generally, dropout works in two modes: at training mode, the connections of the network are different in every forward pass; at test mode, the connections are fixed. This means that the prediction for dropout working in training mode may change. For each (xi, yi), yi is the pseudo-label predicted by the modules working in test mode. We use dropout working in train mode to measure the stability of the pseudo-labeled data, i.e., we use the modules to predict the label of xi for K times in training mode and record the frequency k that the prediction is different from yi. If k > K3 , we regard the pseudo-label yi of xi as an unstable pseudo-label. For these unstable pseudolabels, we will eliminate them. We set K = 9 in all experiments.

4 Experiments

4.1 Setup

Datasets. We run experiments on three widely used benchmark datasets, i.e., MNIST, SVHN, and CIFAR-10. We randomly sample 100, 1,000, and 4,000 labeled examples from MNIST, SVHN and CIFAR-10 as the initial labeled data set L respectively and use the standard data split for testing as that in previous work.

Network Architectures. The network architecture of trinet for CIFAR-10 is shown in Figure 2, which is derived from the popular architecture [Laine and Aila, 2016] used in semi-supervised deep learning. In order to get more diversity among three modules, we use different convolution kernel sizes, different network structures (with/without residual block) and different depths for M1, M2 and M3. The network architectures for MNIST and SVHN are similar to that in Figure 2 but in a smaller size.

Parameters. In order to prevent the network from overfitting, we gradually increase the pool size N = 1000 × 2t up to the size of unlabeled data U [Saito et al., 2017], where t denotes the learning round. The maximal learning round T is set to be 30 in all experiments. We gradually decrease the confidence threshold σ after N = U to make more unlabeled data to be labeled (line 11, Algorithm 1). In the training process, we respectively fine-tune three modules M1, M2 and M3 on the diverse training sets L1os , L2os and L3os every 3 rounds after N = U to maintain the diversity (line 10, Algorithm 1). Since L1os , L2os and L3os are injected into random noise, the confidence threshold σ is decreased by σos (line 14, Algorithm 1). We set σ0 = 0.999 and σos = 0.01 in MNIST; σ0 = 0.95 and σos = 0.25 in SVHN and CIFAR10. We use dropout (p = 0.5) after each max-pooling layer, use Leaky-ReLU (α = 0.1) as activate function except the FC layer, and use soft-max for FC layer. We also use BatchNormalization [Ioffe and Szegedy, 2015] for all layers except the FC layer. We use SGD with a mini-batch size of 16. The learning rate starts from 0.1 in initialization (from 0.02 in training) and is divided by 10 when the error plateaus. In initialization, three modules M1, M2 and M3 are trained for up to 300 epochs in SVHN and CIFAR-10 (100 in MNIST). In training, three modules M1, M2 and M3 are trained for up to 90 epochs in SVHN and CIFAR-10 (60 in MNIST). We set std = 0.05 in SVHN and CIFAR-10 (0.001 in MNSIT). We use a weight decay of 0.0001 and a momentum of 0.9. Following the setting in Laine and Aila [2016], we use ZCA, random crop and horizon flipping for CIFAR-10, zero-mean normalization and random crop for SVHN.

4.2 Results

We compare our tri-net with state-of-the-art methods shown in Table 1. Recently, Abbasnejad et al. [2017] exploited a pretrained model in their infinite Variational Autoencoder (infinite VAE) method, however, the state-of-the-art methods did not use the pre-trained model. To make a fair comparison, we do not exploit the pre-trained model as that in state-ofthe-art methods. The results in Table 1 indicate that tri-net has good performance. It achieves the error rate of 0.53% on MNIST with 100 labeled examples and 8.45% error rate on CIFAR-10 with 4000 labeled examples, which are much better than state-of-the-art methods. Since tri-net exploits three modules while the state-of-the-art methods exploit one or two modules, the time cost of tri-net is more than that of these methods.

There is an initialization in tri-net, with more sophisticated initialization methods, tri-net could have better performance. Π model [Laine and Aila, 2016] is a rising semi-supervised deep learning method. It evaluates each input twice based on the neural network and calculates the loss between the two predictions to regularize the neural network. We also use Π model to initialize three modules M1, M2 and M3 in tri-net and call it tri-net + Π model. The results are also shown in Table 1. From Table 1, we can find that tri-net + Π model performs better than tri-net and achieves the error rate of 3.45% on SVHN with 1000 labeled examples.

Tri-net is a semi-supervised learning method by using unlabeled data to improve learning performance. It has been reported that semi-supervised learning with the exploitation of unlabeled data might deteriorate learning performance [Balcan and Blum, 2010; Chapelle et al., 2006]. Now, we demonstrate whether the performance of tri-net will be deteriorated by keeping on using unlabeled data. As tri-net labels more and more unlabeled data, we depict the error rates of three modules M1, M2, M3 and tri-net in every learning round in Figure 3, which shows that except very few learning rounds, the performance is not deteriorated by keeping on using unlabeled data.

4.3 Further Discussion

In order to generate three accurate and diverse modules M1, M2 and M3, we introduce output smearing in initialization. We record the error rates of ensemble of three modules M1, M2, M3 and their agreement in the initialization with/without output smearing. The results are shown in Table 2. Table 2 indicates that on all three datasets the error rates of ensemble of M1, M2 and M3 in initialization with output smearing are lower than that without output smearing. Three modules M1, M2 and M3 generated with output smearing also have large diversity (low agreement means large diversity). As trinet goes on, M1, M2 and M3 become similar, and then finetuning is introduced to augment the diversity among them. Some pseudo-labels may be incorrect, pseudo-label editing is used to alleviate the influence of suspicious pseudo-labels.

To show that whether these techniques are helpful to trinet, we run experiments with/without them in tri-net, and the results are shown in Figure 4. Figure 4 indicates that when all three techniques are used, tri-net has the best performance. It implies that these techniques are very necessary for tri-net and each of them makes a contribution to the good performance of tri-net.

Different network structures are used to get three diverse modules M1, M2 and M3, we conduct the experiments with the same network structure for three modules M1, M2 and M3 as a comparison. The results shown in Table 3 indicate that different structures bring better performance. The parameter σos controls the confidence threshold when output smearing is used in the training process. We conduct the experiments with different σos ∈ [0.01, 0.25], and the results shown in Table 4 indicate that tri-net is not very sensitive to the parameter σos.

5 Conclusion

In this paper, we propose tri-net for semi-supervised deep learning, in which we generate three modules to exploit unlabeled data by considering model initialization, diversity augmentation and pseudo-label editing simultaneously. Experiments on several benchmarks demonstrate that our method is superior to state-of-the-art semi-supervised deep learning methods. In particular, it can achieve the error rate of 8.30% on CIFAR-10 by using only 4000 labeled examples. Extending tri-net with more modules could exploit the power of ensemble in labeling the unlabeled data confidently. In this situation, one important issue is to maintain the diversity among these modules, which will be an interesting research direction in semi-supervised deep learning.

Acknowledgments

This work was supported by the NSFC (61751306, 61673202, 61503179), the Jiangsu Science Foundation (BK20150586) and the Fundamental Research Funds for the Central Universities.