Perturb, Predict & Paraphrase: Semi-Supervised Learning using Noisy Student for Image Captioning

1 Introduction

Semi-supervised learning (SSL) has been a long-standing problem of interest, with the potential to leverage large volumes of unlabeled data in conjunction with relatively smaller amounts of labeled data. SSL techniques have seen a resurgence in recent years by incorporating data augmentation into the training pipeline, along with loss functions that are robust to the label noise introduced by these augmentations. These techniques have been shown to yield impressive performance improvements even with using small amounts of labeled data [Berthelot et al., 2019; Xie et al., 2020a; Sohn et al., 2020]. While prior work has predominantly focused on image classification, the application of SSL techniques to sequence prediction tasks (e.g. image captioning)

has been far less explored. In this work, we present the first detailed investigation of a popular SSL technique for the task of image captioning and propose new enhancements that help significantly outperform existing state-of-the-art approaches on image captioning in limited labeled data settings.

Our proposed techniques are based on the popular SSL paradigm of self-training within a teacher-student framework. This involves a teacher model that is trained on a small amount of labeled data and subsequently used to annotate unlabeled data. A student model is then trained on the combination of labeled data and unlabeled data (with pseudo labels generated by the teacher model). Recent work introduced Noisy Student Training [Xie et al., 2020b] that augments this training regime by introducing noise in the student during training and iterating the process. This approach has led to consistent improvements in performance across diverse tasks including image classification [Xie et al., 2020b], object detection [Zoph et al., 2020], machine translation [He et al., 2020] and speech recognition [Park et al., 2020].

In this work, we adapt and improve noisy student training for image captioning. We present two main improvements to the existing noisy student training paradigm:

• We propose "Object Dropout" that directly perturbs object features in an image for noisy student training. This technique performs at par with standard data augmentation techniques while being significantly faster than the latter to implement.
• We propose the use of label augmentation in selftraining for sequential outputs using a paraphrase generator that is trained on unpaired caption text data. This module also fixes errors and improves the quality of low-confidence predictions of unlabeled images. This is a very effective augmentation step that helps improve CIDEr scores on the test set of MS-COCO over previous state-of-the-art approaches by 11.5 points.

Code, models, and datasets will be made publicly available at https://github.com/csalt-research/perturb-predict-paraphrase.

2 RelatedWork

2.1 Semi-Supervised Learning (SSL)

SSL is an established area of machine learning, with numerous survey articles outlining the main techniques and approaches in SSL [Zhu and Goldberg, 2009; Chapelle et al.,

2010]. Self-training is a very popular SSL paradigm that uses model predictions on unlabeled data as pseudo-labels (after suitable filtering based on confidence thresholds) [hyun Lee, 2013]. Self-training has seen success in various tasks such as domain adaptation [Zou et al., 2018], speech recognition [Kahn et al., 2020; Park et al., 2020] and object detection [Rosenberg et al., 2005; Zoph et al., 2020]. A promising new technique that uses self-training and teacher-student learning is the Noisy Student Training framework [Xie et al., 2020b]. Ours is the first work to use this framework for the task of image captioning.

A key idea in SSL, that noisy student also exploits, is that of consistency regularization. Consistency regularization utilizes unlabeled data by assuming that perturbations in inputs and model weights should not affect model predictions. This idea was initially brought to the forefront by [Sajjadi et al., 2016; Tarvainen and Valpola, 2017; Laine and Aila, 2017] and has since been utilized in several recent SSL methods [Sohn et al., 2020; Xie et al., 2020a; Berthelot et al., 2019] to obtain state-of-the-art SSL results. The perturbations involved could be of various forms including stochastic weight transformations like dropout [Srivastava et al., 2014] and/or data augmentations like AutoAugment [Cubuk et al., 2019]. We explore the use of feature augmentations which has not been sufficiently explored.

2.2 Image Captioning

A wide array of approaches have been employed to leverage partially annotated images and unannotated text for image captioning. [Feng et al., 2019; Kim et al., 2019; Gu et al., 2019; Laina et al., 2019] propose GAN-based techniques, [Liu et al., 2018] describes a self-retrieval module and [Guo et al., 2020] proposes visual concept to caption generation. Most of these techniques try to map images and captions to the same latent space by utilizing complex task-specific architectures. Orthogonal to these techniques, our method does not rely on complex task-specific modules, it is model agnostic and hence easy-to-use.

To the best of our knowledge, Self Distillation [Chen et al.,

2021] is most closely related to our work; they utilize selftraining and share our characteristics of model agnosticism. The key novelties of our method compared to Self Distillation include a) the use of noisy student training, b) our proposal of a paraphrasing model to fix and diversify pseudo labels c) consistency-based training through novel and efficient feature augmentation techniques such as object dropout. Moreover, we significantly outperform Self Distillation on all metrics in the limited labeled data setting.

3 Our Noisy Student Training Approach

Figure 1 illustrates our approach with a schematic diagram and Algorithm 1 outlines our noisy student training algorithm for image captioning. Consider a set of labeled examples L = {(x1, y1), (x2, y2), . . . , (xm, ym)}, a set of unlabeled images UI = {xm+1, xm+2, ..., xm+n}, and a set of unpaired captions UC = {y1, y2, ..., yn} where x refers to an image and y refers to a caption. The first step in noisy student training is to train the teacher model T to minimize cross-entropy loss on the labeled dataset L; this is denoted by ℓ(yi, T (xi)) for the ith instance, where T (xi) refers to the caption predicted by T for xi. This step is followed by N self-training iterations. In each iteration, the model T is used to predict captions for each of the unlabeled examples in UI using a sequential decoder. A student model S is trained from scratch on the combined datasets L and UI (with its corresponding pseudo labels ˆ ym+1, . . . , ˆ yn) to optimize a joint cross-entropy loss function LS. At the end of each iteration, the newly trained student becomes the teacher for the next iteration.

Introducing stochasticity into student training helps it generalize better than the teacher. The sources of stochasticity can be broadly classified as input noise and model noise. In [Xie et al., 2020b], dropout and stochastic depth are used for model noise and data augmentation via RandAugment [Cubuk et al., 2019] is used for input noise. In [He et al., 2020] that uses noisy student training for machine translation and text summarization, input word tokens are corrupted for input noise and dropout contributes to model noise. Sim- Algorithm 1 Noisy Student Training for Captioning.

Input: N, L, UI, UC, Paraphraser P, Student model S, Teacher model T

• 1: Train T on L to minimize LT = 1m ∑mi=1 ℓ(yi, T (xi))
• 2: Fine Tune P using captions in UC
• 3: for t ← 1 to N do
• 4: {ˆ ym+1, . . . , ˆ yn} ← T (UI )
• 5: Add noise to the student model S
• 6: Train S to minimize LS

ilar to prior work, we also use dropout as a source of model noise. For input noise, we do not use data augmentation in the typical sense where image transformations (e.g. rotate, posterize, etc.) are applied to raw images. Instead, we directly perturb image features via a simple technique that we refer to as object dropout. Additionally, we introduce stochasticity in the label space with the help of a paraphrase module, that is independently trained on caption text.

3.1 Object Dropout

One of the most popular feature representations for image captioning makes use of the bottom-up attention model [Anderson et al., 2018]. This feature extractor identifies image regions, that typically coincides with objects in the image, and produces corresponding feature vectors. The number of feature vectors per image can thus be variable, depending on the number of salient image regions that are identified. Randomly dropping feature vectors per image would have the effect of restricting information only to a subset of objects. More concretely, we define an object dropout probability pobj, such that an object-based feature vector within an image can be entirely dropped (i.e. zeroed out) with probability 1-pobj. We use object dropout as a source of input noise. Unlike standard data augmentation techniques that require transformations of raw images followed by bottom-up feature extraction, object dropout is a feature augmentation technique that is directly applied to the bottom-up features making it much more computationally efficient.

3.2 Label Augmentation via Paraphrasing

Unlike classification tasks where class labels can be easily augmented using simple techniques like linear interpolation [Zhang et al., 2018a], augmenting sequence labels is a non-trivial task. Our solution to this problem is to use a paraphrase module trained on caption data and use it to transform predicted captions for unlabeled images into multiple paraphrased forms. For our paraphraser, we start with a powerful pretrained text-to-text model and further finetune it on a paraphrase task involving caption text. To train the paraphraser, one would need access to groups of similar captions which can serve as paraphrases of each other. Identifying these groups and its impact on performance is discussed further in the experiments in Section 5.5.

Apart from providing different ways in which the same caption can be expressed, the paraphraser module also acts as a language model and fixes syntactic issues in the predicted captions which are otherwise non-trivial to fix. We will elaborate on this further, along with providing illustrative examples, in Section 5.5.

4 Experimental Setup

Datasets. We conduct experiments on the MSCOCO dataset [Lin et al., 2014], the standard benchmark used for image captioning. This dataset contains 123k images with up to 5 captions per image. We adopt the standard "Karpathy" split used in all prior work, with 113k images used in training, and 5k images each used for validation and testing. For unlabeled data, we use the "Unlabeled COCO" split from the official MSCOCO Caption challenge. (This allows for consistent usage of unlabeled data across different amounts of labeled data, as in Table 3.) This unlabeled dataset contains 123k images with no corresponding captions.

Evaluation Metrics. We use the standard metrics used to evaluate image captioning systems, namely BLEU, METEOR, ROUGE, SPICE, WMD and CIDEr. Consistent with prior work on image captioning, we select the model checkpoint with the best CIDEr score on the validation set to evaluate on the test set. Beam decoding is used for evaluation with the beam width set to 5.

Implementation Details. We use the Attention on Attention Network (AoANet) [Huang et al., 2019] as our base model for the teacher and student. In the fully supervised setting, AoANet was recently shown to achieve state-of-the-art performance on the MSCOCO image captioning dataset, thus making it a good choice for us to adopt as our teacher/student models. A Faster-RCNN model, pretrained on ImageNet and Visual Genome, is used to extract bottom-up feature vectors of images, as described in Section 3.1. Unless specified otherwise, we use beam decoding to generate pseudo labels with a beam width of 2. For the teacher model, we use model dropout with probability p = 0.3, no object dropout and label smoothing with probability 0.1. The student model is randomly initialized, and trained from scratch. The student model uses the same AoANet architecture as the teacher model.The labeled batch size is 16, with 5 captions per image, and the unlabeled batch size is 96 with 1 caption per image. The number of noisy student iterations N = 1.For the paraphraser, a pre-trained sequence-to-sequence transformerbased model, BART [Lewis et al., 2020] is fine-tuned to paraphrase captions. More details about the data used to train the paraphraser is mentioned in Section 5.5.

5 Experiments and Results

Unless specified otherwise, we mainly focus on the low labeled data setting i.e., using 1% of MSCOCO labeled data. This is similar to the setup adopted in both prior works on SSL for image captioning [Kim et al., 2019; Chen et al., 2021]. We also provide results on varying amounts of labeled data, including 100% of the labeled data in MSCOCO.

5.1 Comparisons with PriorWork

Table 1 presents a comparison of our best numbers using 1% labeled data with previous state-of-the-art approaches in both semi-supervised and unsupervised image captioning. For a fair comparison, we utilize the base AoANet model without fusion or ensembling, which is similar in performance to the Up-Down architecture utilized in Self Distillation [Chen et al., 2021]. We observe that noisy student training, both with and without paraphrasing, performs significantly better across all metrics. Particularly notable is the large improvement in performance, especially on CIDEr scores, by using the paraphraser for label augmentation.

5.2 Proportions of Labeled and Unlabeled Data

The ratio of labeled to unlabeled data used during training is an important factor that influences the performance of SSL algorithms. We analyze the impact on performance in both directions. We first fix the amount of labeled data and vary the amount of unlabeled data. Next, we fix the amount of unlabeled data and study the impact on performance by varying the amount of labeled data.

Table 2 shows the performance of the student model on using a fixed 1% of labeled data and varying the amount of unlabeled data from 1% to 100%. There is a clear trend of improvement in performance as we increase the amount of unlabeled data. A key observation here is that when the amount of unlabeled data is comparable to the amount of labeled data, the performance of the student is worse than the teacher. However, as the amount of unlabeled data becomes an order of magnitude higher than labeled data, the student begins to outperform the teacher. (Similar observations were made in [Xie et al., 2020b] as well.)

From Table 3, we find that the performance improvements of the student compared to the teacher diminish as the ratio of unlabeled to labeled data decreases. In fact in the 100% labeled data setting, we observe that the student performs worse than the teacher. (In Section 5.6, we show that having more unlabeled data results in the student outperforming the teacher, even in the high labeled data regime.)

Related to the overall proportions of labeled and unlabeled data, ratio between unlabeled and labeled batch sizes is also an important factor to be considered. More so in our setting, where we have multiple captions per image in our labeled dataset, which affects the labeled batch sizes.

5.3 Student Initialization

An important part of the noisy student algorithm is that the student be trained from scratch. Here, we compare training a student model from scratch, i.e. the student parameters are randomly initialized, with training a student model with a warm start, i.e. the trained teacher parameters are used to initialize the student.

Table 4 shows that both random initialization, and teacher initialization result in student models that outperform the teacher. However, the student trained from scratch outperforms the student with warm start. Similar observations were made for noisy student training used with image classification tasks [Xie et al., 2020b].

5.4 Sources of Stochasticity

We use model dropout probability p to control model noise, and object dropout probability pobj to control input noise.

In Table 5, we report scores on both the test set and validation set to show that both sources of noise appear to be useful. While the best test scores were obtained by setting p = 0.3, the best validation scores were obtained by setting pobj = 0.3. Here, we only considered object dropout on the unlabeled data.

Table 6 compares image augmentation as a source of stochasticity for student training. With color jittering as the primary augmentation, we notice that inclusion of data augmentation does not provide significant gains, if any, over our approach. We also consider strengthening the baseline augmentation system by using a larger student model with more number of attention heads, including random flips and rotations for augmentation, and using object dropout. Out of these systems, student using color jittering with object dropout performs the best. Finally, we note that our approach is 12× faster than noisy student training with data augmentation, and achieves comparable performance.

5.5 Paraphrasing

As motivated in Section 3.2, label augmentation via paraphrasing can be extremely effective in low resource settings with sequential outputs. The paraphraser is trained on the set of unpaired captions UC. To train a paraphraser, we first need to create groups of similar captions. When available, multiple captions present for each image can be used to create the respective groups. Otherwise, we can perform clustering on the available captions in a meaningful embedding space and use the resulting clusters.

Table 7 shows the impact of paraphrasers trained using both the above-mentioned grouping techniques ("Multiple Captions" and "Clustering") on student training. The number of available captions per image is 5; hence, we set the number of clusters such that each cluster has at most 5 captions. A pretrained BERT model [Devlin et al., 2019] is used to compute caption embeddings, and k-means clustering [Johnson et al., 2019] is used to compute clusters in this embedding space. It is clear that the use of the paraphraser leads to large improvements in performance compared to not using a paraphraser at all (as in Table 5). Interestingly, we observe that the BLEU-4 scores with clustering captions are higher than annotated caption groups, while CIDEr scores are lower. We hypothesize that the BERT-based clustering paraphraser maintains sentence semantics such as n-grams better than human annotated text and hence yields high BLEU-4 scores. However, since it is not conditioned on image objects and the clusters might tend to bias the paraphraser more towards general words (rather than content-specific words), the BERT-based clustering paraphraser does not do as well as the "Multiple Captions" paraphraser on CIDEr scores that rewards retention of content words. "No Finetuning" refers to using the BART model as-is, without any finetuning on caption text. As expected, finetuning the paraphraser on caption text is critical to derive performance improvements.

Figure 2 shows test predictions using a student model trained with and without paraphrasing. It is clear that the use of the paraphraser in the student model helps produce captions that are superior in quality compared to the student model without the paraphraser. Since the paraphraser scaffolds on BART that has a strong in-built language model, the paraphraser also has the additional benefit of fixing poorquality caption predictions containing repeated words and illformed sentences. This allows us to make effective use of the unlabeled data and not have to filter out low-confidence pseudo labels. Instead, our approach using the paraphraser attempts to correct the errors in the pseudo labels. Some examples are listed in Figure 3 where the errors in red are fixed by the paraphraser.

Figure 3: Examples of low-quality pseudo labels with errors (in red) by the teacher and the paraphrased model correcting these errors.

We also do an ablation on the benefits of using multiple paraphrased captions for each unlabeled image. We see clear improvements on CIDEr scores going from 1 caption per image (82.4) to 5 paraphrased captions per image (84.2); the BLEU-4 scores stayed roughly the same.

5.6 Noisy Student and High Labeled Data

As described in Section 5.2, our approach works well in the low labeled data regime where the amount of unlabeled data is much larger than the labeled data. However, when the sizes of the unlabeled and labeled data are comparable, we find that the student fails to outperform the teacher. We identify two strategies that mitigate this issue: (1) Decaying the weight of unlabeled data during student training and (2) Using a larger unlabeled dataset. Decaying the weight of the unlabeled data in the training objective smoothly interpolates separate-training [He et al., 2020] and joint-training [Xie et al., 2020b]. For a larger unlabeled dataset, we consider the Conceptual Captions Dataset [Sharma et al., 2018] that contains roughly 3.3M image-caption pairs.

Table 8 compares the effect of the two above-mentioned strategies on student training using 100% of the MSCOCO labeled data. We use only the images from the Conceptual Captions dataset as unlabeled data. Due to computational constraints, we restricted ourselves to using 1M images. We implement the student with weight decay by linearly decaying the contribution of the unlabeled data to the training loss LS from 1 to 0 in 10 epochs. We observe that both of these strategies successfully extend our noisy student training to be effective even in the high labeled data regime.

6 Conclusion

This is the first work to present a comprehensive analysis of the noisy student framework, a state-of-the-art approach in semi-supervised learning, for image captioning. We show the effectiveness of simple model and feature perturbations to precomputed features and show that they perform at par with data augmentation, while being significantly more computationally efficient. We additionally propose the use of label augmentation in self-training for sequential outputs, wherein instead of filtering out low confidence pseudo labels, we "fix" them using a paraphraser that is trained on unpaired caption text. We show a large performance boost using label augmentation, which is additive to input data augmentation, and significantly improves performance on low-resource image captioning over current state-of-the-art results by over 15%.

Acknowledgements

We thank Yash Shah for his insightful suggestions and help with the experimental setup.