Incorporating Unlikely Negative Cues for Distinctive Image Captioning

1 Introduction

Over the past few years, data-driven image captioning has made remarkable progress [Chen et al., 2015; Xu et al., 2015; Vinyals et al., 2015; Anderson et al., 2018; Huang et al., 2019; Cornia et al., 2020; Fei, 2021a; Fei, 2021b; Yan et al., 2021; Zhang et al., 2021; Fang et al., 2022], attracting growing interest from both academic and industrial communities. Traditionally, given a specific input image, neural image captioning models use maximum likelihood estimation (MLE) as a optimization principle to maximize the probability of generating accurate captions that match the ground-truth reference. Unfortunately, the many-to-one phenomenon [Stefanini et al., 2021; Vijayakumar et al., 2016; Fei et al., 2022a], where it is common for an image context to have multiple accurate captions with distinct features, poses a challenge for neural image captioning models. That is, models tend to generate safe but generic captions, which presents an obstacle to the widespread deployment of image captioning systems. To alleviate this issue, some researchers integrate sampling operation in the latent space to meet the requirement of diversity, under the conventional variational autoencoder framework [Wang et al., 2017; Aneja et al., 2019; Mahajan et al., 2019; Mahajan and Roth, 2020; Shen, 2022]. Besides, [Vijayakumar et al., 2016; Holtzman et al., 2019] proposed advanced decoding strategies to alleviate the problem of generic sentences. Indeed, all of the aforementioned methods enhance the diversity of image captions by providing the model with positive guidance on what to generate.

∗The corresponding author.

Taking inspiration from negative training techniques [Kim et al., 2019; Ma et al., 2021] that aim to update parameters with unlikely objectives while identifying high-frequency sentences [He and Glass, 2019; Li et al., 2019; Welleck et al., 2019], we argue that it is equally imperative to instruct image captioning models on what not to generate. Whilst negative training-based methods have proven effective at promoting sentence diversity, directly applying these techniques to image captioning comes with its own set of drawbacks. Firstly, these methods only consider high-frequency tokens or sentences as negative candidates. However, the high-frequency situation is only a sub-situation. Due to the diversity of visual scenes, descriptions that are low-frequency, yet generic and meaningless may escape punishment, resulting in a failure to address in multi-modal applications. Moreover, we have previously observed that certain generic sub-sentences which evade identification as negative candidates, ultimately compromise the fluency of captions. Secondly, these negative training methods overlook the implicit knowledge within neural networks that identifies negative candidates at multiple levels [Cornia et al., 2020]. We contend that it is more feasible to conduct negative training with richer information, e.g., hierarchical semantic representation.

To address the aforementioned issues and enhance description diversity, we introduce a new Unlikely Negative Knowledge Training training paradigm, referred to as UNKT, for distinctive image captioning. Inspired by that conventional knowledge distillation [Hinton et al., 2015], where the teacher is typically regarded as a positive role that the student seeks to emulate, instead, we train the teacher model to serve as a negative role model and instruct the student to avoid exhibiting these undesirable behaviors. Specifically, to collect a negative training set, we first employed an entropy filtering strategy that prioritizes retrieving as many many-to-one cases from the raw dataset as possible. It should be noted that usually "one" is a generic sentence and "many" are different images. Next, we trained a standard image captioning model using the aforementioned subset as the negative teacher. The negative teacher generates a set of negative candidates in response to a given image. These negative candidates are captions that seem plausible, yet dull and generic. The purpose of incorporation image-level negative candidates is to encourage the student model to generate more creative and diverse captions by avoiding behaviors that the negative teacher provides, as a result of addressing the previously mentioned drawback. Moreover, to achieve a more all-encompassing training update, we devise two negative objectives, including soft unlikelihood loss on the prediction layer and reverse KL divergence on the intermediate layer. By leveraging multilevel negative knowledge, UNKT effectively encourages the production of more descriptive and contextually vivid captions for student model.

We experimented with our proposed method of unlikely negative knowledge training, evaluating its effectiveness by implementing it into the widely-used Transformer [Vaswani et al., 2017; Cornia et al., 2020] on the MS COCO benchmark. We observe that the model tends to yield clear language patterns in the generated captions, demonstrating that UNKT has successfully learned negative knowledge with explicit supervision. More encouragingly, image captioning models equipped with UNKT perform surprisingly well under diversity evaluation and oracle performance evaluation, achieving new state-of-the-art results. This shows that our learned knowledge is not only distinct but also effectively covers rich semantic understanding. The contributions are summarized as follows:

• We propose a novel unlikely negative knowledge training paradigm for distinctive image captioning. It constructs image-wise generic sentences as the negative candidates to remind the model not to generate. As far as we are concerned, it is the first work to consider unlikely knowledge in diversified image captions;
• With the negative teacher trained with retrieval entropy data, we devise two negative training objectives to progressively calibrate multi-level semantic information to boost the performance of unlikely negative learning;
• We perform extensive experiments and analysis on the MS COCO dataset to verify the effectiveness of the UNKT framework as well as the superiority compared with previous diverse image captioning methods.

2 Background

Image captioning. The objective of the image captioning model is to infer a conditional probability distribution p(Y |X) given a matched image X and its corresponding textual description Y = {y1, . . . , yL} from dataset, where the length of the text is denoted by L. Optimizing the parameters of an image captioning model is typically achieved through maximum likelihood estimation (MLE) technique, which can also be expressed as minimizing the factorized negative loglikelihood as:

where Y<i = {y1, . . . , yi-1} and θ is the model parameters. It is common for multiple different descriptions to be generated for a single visual scene in image captioning task. This "many-to-one" phenomenon occurs frequently in datasets, such as the MS COCO dataset, where each image has five distinct sentences associated with it. However, when using MLE-based training, this phenomenon can lead to the production of overly simplistic and generic sentences by the optimized model.

Unlikelihood training. Unlikelihood training [Welleck et al., 2019] is a proposed solution for neural generation models that struggle with repetitive or overly common tokens in their output. This approach can help to mitigate the problem of undesirable behaviors in generation tasks. By minimizing the likelihood of generating negative candidates during training, the model is encouraged to generate more diverse and interesting content, as:

where Vn consists of negative candidates, e.g., overused negative frequent words, that are a subset of the vocabulary.

3 Methodology

Improving the overall quality of image captioning requires guiding the model to identify generic sentences. However, traditional negative training methods that rely on frequencybased analysis have certain limitations. Here, we propose a new negative knowledge transfer that discourages generic sentence generation while promoting distinctive image captioning. Specifically, the approach involves a negative teacher, which helps to equip the student model with enough negative knowledge to prevent undesirable behaviors and ensure smooth information flow.

3.1 Unlikely Negative Teacher Construction

To facilitate negative knowledge transfer, an expert negative teacher must be able to generate precise yet versatile descriptions for a given image. Our approach employs the Transformer [Vaswani et al., 2017] as the foundation for both the negative teacher and student networks. We emphasize the generality of this approach, which can be applied to other advanced architecture [Cornia et al., 2020], and have demonstrated its efficacy through experiments.

To construct the negative training dataset, we introduce a retrieval entropy filtering strategy that enables us to identify and gather instances of many-to-one cases. The retrieval entropy can be defined as:

where pr(xi|y) is the conditional probability calculated based on the retrieval similarity of image-text pairs from CLIP model [Radford et al., 2021], xi is the searched image for the query text y, and D represents the raw training set. A higher retrieval entropy indicates that the query text y corresponds to more related images, i.e., the many-to-one problem is more serious. Then, we select the top 50% image-text pairs with a high retrieval entropy and designate them as the negative training set D-, which contains a considerably larger proportion of generic descriptions than the raw training set. After that, we can train negative teacher pn on the negative training set D- with MLE loss in Equation 1, which will naturally produce generic sentences for any input image.

3.2 Unlikely Negative Knowledge Transferring

This section presents the Unlikely Negative Knowledge Transferring (UNKT) framework, which involves transferring knowledge between negative teacher and student models. This transfer of knowledge is achieved through the use of a multi-level approach, as illustrated in Figure 1.

UNKT for prediction layer. It is believed that the softened logits in the prediction layer contain more information than the hard ground-truth labels, such as the similarity between labels [Hinton et al., 2015]. Therefore, conventional KD transfers knowledge by narrowing the gap between the output probability distributions of the teacher model pt and the student model ps as:

With regards to Unlikely Negative Distillation, the additional knowledge contained in the softened logits generated by the negative teacher pn reflects how to generate generic sentences based on the input image. Therefore, we instead introduce a soft unlikelihood loss to maximize the distance between the predictions of the negative teacher pn and the student ps as:

where model distribution pn and ps can be computed as pi = softmax( hLi t ), where hLi is the i-th hidden state from the last L layer and t is a temperature coefficient. It is important to note that previous negative training methods have only used high-frequency words with one-hot representation as targets, which disregards the rich semantic information contained in the softened logits. For instance, generic words often have similar probabilities.

UNKT for intermediate layer. Apart from the knowledge produced by the prediction layer, there is also implicit knowledge contained within intermediate layers, such as hidden states and attention matrices within the network architecture. To enhance the efficacy of deterring unwanted behaviors, such as the generation of generic sentences, in the student model, we further consider the above knowledge into unlikely negative knowledge transferring. Specifically, the distance between features of the negative teacher and student models should also be increased. In this work, we propose a new measurement function, called mean Reverse KL divergence (RKL), to calculate the information distance as:

Algorithm 1 Unlikely Negative Distillation algorithm

Input: Training dataset D, text-image retrieval model pr, negative teacher model pn and student model ps

Output: Student model ps

• 1: ▷ Collection of the negative training set
• 2: Compute retrieval entropy [Hent(yi,D)];
• 3: Select top-50% [Hent(yi,D)] as D-;
• 4: ▷ Training of negative teacher
• 5: while convergence do
• 6: Optimize teacher pn with Lmle on D-;
• 7: end while
• 8: ▷ Negative distillation training
• 9: while convergence do
• 10: Optimize student ps by minimizing L on D ;
• 11: end while
• 12: return trained diversified student ps;

where P and Q are the corresponding feature matrices of the negative teacher and the student models, respectively, and M is the number of elements. KL(·) is a standard KL divergence computation function. Considering that semantic modeling mainly focuses on language decoder as well as operation simplicity, we only conduct unlikely negative knowledge transferring on the intermediate layers of the language decoder. The negative transferring objective of hidden states in each decoder can be defined as:

where hln and hls are the output hidden states of the l-th language decode layer from negative teacher model pn and student model ps, respectively.

On the other hand, as the attention weights can learn substantial semantic knowledge [Vig and Belinkov, 2019; Kobayashi et al., 2020], it is beneficial for the student model to further conduct UNKT on the attention matrices. Formally, we set Q, K, and V as the matrices of queries, keys, and values, respectively, and dk is a model dimension for scaling. We choose a = soft( QKT √dk ) to calculate the distance. Similar to Equation 7, the unlikely negative transferring objective of matrices is formulated as:

where an and als denote the attention matrices of the l-th language decoder layer of negative teacher and student models.

3.3 Progressive Hyper-Parameter Adjustment

The overall training loss, which combines the above multilevel unlikely negative knowledge objectives and the MLE objective, is used to train the student model end-to-end as:

where λ is a hyper-parameter that balances the importance of supervised learning as well as unlikely negative knowledge transferring. Note that MLE loss is optimized with corresponding positive image-text pairs.

When it comes to unlikely negative knowledge parts, it is preferable for the student model to have the ability to generate captions before being reminded of what not to say. Thus, we adopt a progressive optimization approach that initially increases the negative knowledge ratio linearly and then gradually decreases it. The latter stage closely resembles to the cosine learning rate decay and can be determined by:

where step corresponds to the training step, and T is total training steps without warmup.

4 Experiments

4.1 Experimental Settings

Dataset. We train and evaluate the UNKT method on the MS COCO dataset [Chen et al., 2015] that contains 123,287 images and each image is equipped with at least 5 captions. For a fair comparison, we follow the previous works [Mahajan and Roth, 2020] in the area of diverse and controllable image captioning to use the m-RNN split [Mao et al., 2014] of the COCO dataset, which divides the data into 118,287 for training, 4K and 1K for validation and testing, respectively.

Qualitymetrics. To assess the quality of the generated captions, we use five widely used evaluation metrics, i.e., BLEUn [Papineni et al., 2002], METEOR [Lavie and Agarwal, 2007], ROUGE-L [Lin, 2004], CIDE [Vedantam et al., 2015], and SPICE [Anderson et al., 2016]. In between, CIDEr focuses on semantic analysis and has a higher correlation with human judgment, and other metrics favor frequent n-grams and measure the overall fluency.

Diversity metrics. To investigate the diversity of the generated captions, we use SelfCIDEr [Wang and Chan, 2019], mBLEU, and n-gram diversity denoted as Div-n [Li et al., 2015]. All of these metrics evaluate the diversity by comparing the n-gram differences among the generated captions that belong to the same image.

Implementation details. The proposed UNKT is a general training paradigm and we expect it can be easily applied to many existing image captioning models and improve their diversity. In this paper, we choose widely-used and representative Transformer architectures [Vaswani et al., 2017] as our base models, to show the generalization ability of UNKT. Since most current SoTA image captioning models, like M2Transformer [Cornia et al., 2020], COSNet [Li et al., 2022a], ViTCAP [Fang et al., 2022], and many visionlanguage pre-training models, are based on the Transformer architecture. Both the negative teacher and the student networks hold the same setting in terms of network architecture and hyper-parameters. Specifically, following the settings of Transformer in [Vaswani et al., 2017], both visual encoder and language decoder contain 6 layers, in which the self-attention module has 8 attention heads and the number of feed-forward units is 2046. The size of hidden states is set to 512 and the dimension is 64 for query, key, and value, with a dropout ratio of 0.1. We use AdamW [Kingma and Ba, 2014] optimizer and employ 3000 steps warm-up trick to adjust the learning rate of 3e-4 during training. We linearly increase λ to 1 in first 5 epochs and then gradually decay after. For the temperature coefficient t, we simply set it to 1.

4.2 Evaluation on Quality

To evaluate the effectiveness of unlikely negative knowledge in generating accurate descriptions, we first calculate the quality evaluation metrics in the oracle setting consistent with prior works [Mahajan and Roth, 2020; Mahajan et al., 2020], i.e., taking the maximum score for each automatic metric over all the candidate captions for each image. Specifically, we infer from the Transformer-based UNKT model with sampling sizes of 20 and 100, and evaluate the oracle results of the generated captions. The results are listed in Table 1. As we can see that the standard Transformer equipped with UNKT obtain a significant improvement compared with the results of the best competitors on all the quality metrics w.r.t. both 20 and 100 samples. More encouraging, when employed with the advanced meshed memory structure in [Cornia et al., 2020], the captioning performance obtains further gains. It demonstrates that UNKT method is generalized and can be easy to transfer to other advanced models.

Moreover, we compare our UNKT paradigm with the previous SoTA baselines Transformer-DML [Chen et al., 2022] and COS-CVAE [Mahajan and Roth, 2020] by calculating the oracle scores of CIDEr with different numbers of samples. The evaluated results are shown in Figure 2, and we can observe that the proposed UNKT model consistently outperforms baselines in all settings. The high gain in quality metrics demonstrates that the proposed UNKT framework successfully avoids the generic and meaningless negative productions in the training corpus.

4.3 Evaluation on Diversity

Quantitative results. In this part, we perform a diversity analysis for the proposed UNKT paradigm based on Transformer and advanced memory strategy. We compare our model with previous SoTA methods as well as Beam Search (BS) and show the results in Table 2. Overall, the proposed UNKT model achieves higher performance on most of the diversity evaluation metrics, which demonstrates its effectiveness in expressing distinct semantic content. Besides, we also find that the incorporation of memory will degrade the diverse performance to some degree.

Qualitative results. We further provide several examples of the generated captions of the UNKT model and baseline of standard Transformer with beam search from the MS COCO dataset in Figure 3. Obviously, UNKT model generates more diverse and vivid captions, which further demonstrate the superiority of the improved negative training framework for multi-level information transfer modeling.

4.4 Model Analysis

Ablation study. We conducted an analysis to examine the impact of multi-level unlikely negative knowledge transfer objectives by removing various components: the prediction layer transfer ( w/oLpred), the attention transfer (w/oLatt), the hidden state transfer ( w/oLhid), and the entire negative training (w/oLneg), which corresponds to the standard MLE loss, in the combined loss as shown in Equation 9. The results presented in Table 3 demonstrate that all three knowledge transfer objectives, from distinct perspectives, contribute to enhancing both the diversity and quality of captioning. Notably, the substantial performance decrease observed in the "without Lhid" scenario highlights the importance and abundance of negative information in intermediate layers for effective negative information transmission.

Effect of retrieve entropy in negative dataset construction.

To evaluate the effectiveness of the retrieval entropy filtering method in gathering generic text from image-text datasets, we divide the sorted training set into two equal parts: the top 50% (D-) and the bottom 50% (D+). We then train standard Transformer models, p- and p+, on their respective subsets. The results in Table 4 reveal that the p+ model significantly surpasses the p- model in all diversity-related metrics, thereby confirming the efficacy of retrieval entropy filtering in data selection.

Furthermore, we utilize p-n and p+n as negative teachers for the student models, p-s and p+s , and perform negative knowledge transfer on both models. The outcomes presented in Table 5 reveal that p-s exhibits greater improvements in diversity than p+s , suggesting that p+s discards more valuable negative knowledge. This observation is consistent with the previous analysis, which demonstrated that p-t possesses more negative knowledge than p+t .

Effect of soft targets in knowledge transferring. To assess the effectiveness of soft targets for negative knowledge transfer, we compare them with hard targets obtained by sampling sentences using greedy search on the predictions of negative teachers, given the image. The results presented in Table 6 indicate that UNKT with soft targets can significantly enhance caption diversity, highlighting the benefits of incorporating abundant unlikely negative information, such as label similarity, in soft targets. Additionally, we randomly select sentences from the negative training set, D-, as negative targets. The substantial drop in image captioning performance confirms that the negative teacher model can generate highquality, yet hard, conditional generic captions.

Effect of progressive hyper-parameter adjustment. To evaluate the effectiveness of the progressive hyper-parameter adjustment method presented in Section 3.3, we perform unlikely negative training with a fixed λ value, which is obtained by set λ in Equation 9 to 0.5 across the convergence steps. The outcomes presented in Table 8 indicate that the progressive optimization policy enables the student model to leverage negative knowledge more effectively and comprehensively.

4.5 Human Evaluation

Apart from automatic evaluations, we conducted human evaluations to further validate the effectiveness of the UNKT method compared to previous diverse image captioning techniques. We randomly selected 300 samples from the MS COCO test set and invited three well-educated annotators to judge which of the sentences generated by UNKT and the baselines were better in terms of three aspects: informativeness, relevance, and fluency. Informativeness reflects the amount of information related to the image contained in the generated caption, relevance reflects the coherence of the generated caption with its image, and fluency reflects the likelihood of the generated sentence being produced by humans. The results of the human evaluation are summarized in Table 7. We can see that UNKT approach is overall better than all

Transformer:

• a dog and a person are watching television together
• a dog and a person are watching TV together
• a dog is watching television with a person
• a dog is watching television together with person
• a dog and a person are watching television

Transformer:

there are two sinks in the bathroom there are two sinks in the clean bathroom there are two sinks with mirrors in the room there are two sinks with mirrors in the bathroom two sinks and two mirrors in the bathroom

UNKT:

a person is watching television with a dog a dog between two feet watching television a dog and a person are looking at TV together a man is watching TV at home with his dog

• a dog sitting with a person are watching television

UNKT:

a bathroom with two sinks and two mirrors two sinks with mirrors above sinks a double sink with mirrors in the bath room there are two sinks with mirrors in the bathroom two mirrors and lights over the sink in the room

Table 8: Effect of progressive hyper-parameter adjustment.

baselines. Specifically, UNKT achieved better performance than the standard Transformer in terms of informativeness and remained competitive in fluency and relevance. Compared to both Transformer-DML and CVAE, our approach demonstrated significant advantages, particularly in fluency. These results suggest that incorporating negative knowledge can enhance the distinctive image captioning capacity.

5 RelatedWorks

Diverse Image Captioning. Recent studies have shown that MLE-optimized models tend to exhibit a bias towards safe and average versions that only contain common words and phrases from the training corpus [Mao et al., 2022; Fei, 2019; Fei et al., 2022b]. In contrast, diverse image captioning aims to train a model that can generate a variety of captions for the same image. CVAE-based models [Wang et al., 2017; Aneja et al., 2019; Mahajan et al., 2019; Mahajan and Roth, 2020; Shen, 2022; Chen et al., 2022] learn a latent space during training and then generate diverse captions by sampling different priors from the latent space. GAN-based models [Dai et al., 2017; Zhang et al., 2022] predict diverse captions by using different random noises as inputs accompanied by the given images. Diverse decoding methods [Vijayakumar et al., 2016; Holtzman et al., 2019] incorporate token-level constraints during inference. Although these diverse image captioning models are able to produce different captions during inference, all of them handle the generic response problem only from the angle of negative, thus can not capture all the features of generic sentences.

UnlikelyNegative Training. Generally, unlikelihood training moderates MLE by imposing an explicit penalty on the decoding of next target words that are predefined in a candidate set of tokens. This set typically consists of undesirable words and n-grams that contradict or have already appeared in the previous context. Unlikely negative training has been widely explored and shown promising results in text generation [Welleck et al., 2019], language modeling [Son et al., 2022], dialogue [Nugmanova et al., 2019; He and Glass, 2020; Li et al., 2020; Li et al., 2022b], machine translation [Ott et al., 2018], motion forecasting [Zhu et al., 2022], and interpretable embedding [Wu et al., 2022]. However, its influence in improving the diversity of image captioning is under-explored. To this end, our work proposes a unlikely negative training paradigm to cover the visual element pluralistic, avoiding the problem of previous work.

6 Conclusion

In this paper, we introduce an unlikely negative knowledge training paradigm to improve the diversity of image captioning. It formulates negative training as a knowledge transferring process in a multi-level perspective from a trained negative teacher learned with retrieval-entropy filtering data. By generating generic and uninteresting sentences for any given image, the negative teacher method circumvents obstacles that have previously impeded frequency-based approaches. Extensive experiments on the MS COCO dataset validate the superiority of proposed UNKT method compared with previous diverse image captioning works.