OSIC: A New One-Stage Image Captioner Coined

1 Introduction

Image captioning is an important cross-modal task of automatically generating the descriptions of the main contents for given images. Inspired by the procedure for neural machine translation, encoder-decoder architecture is most widely used for image captioning [Cornia et al., 2020], which encodes the given image into the intermediate representation via a vision encoder, followed by a NLP decoder to generate the captions. As a result, the performance of the captioner largely relies on the representations containing information for the NLP decoder. Motivated by this end, the early work aim to compress the image into the fixed-length vector [Vinyals et al., 2015] as visual features. To enrich the compact expressions, grid features [Zhang et al., 2021] are further generated by the CNNs to embed more visual information. More recently, compared with the grid features, two-stage captioners using region features [Anderson et al., 2018] have made great progress by capturing the salient region-level features.

* Corresponding author.

It is noteworthy that the region features obtained by a fixed detector (e.g., Faster R-CNN [Ren et al., 2015]) focus on the detection task, which means it can hardly provide all necessary descriptive information for the subsequent image captioning task [Kuo and Kira, 2022], due to the large information gap between the two tasks. In other words, the visual encoder in the first stage of those captioners is optimized using the detection tags instead of sentences, which causes the information mismatch in feature embedding. We call this mismatch between detection-centric features and captioningguided features task-based information gap. This gap limits the model to obtain a global optimization, and results in two major issues. Firstly, region features can hardly present all the necessary descriptive information for the target captions. For example, misdetection, insufficient detection (e.g., the scene "field" and the object relation "catching" are not detected), or redundancies (e.g., "soccer goal" and "player") are produced, as shown in Figure 1. And more to the point, these region features are represented independently [Hu et al., 2018], without visual semantic connection with each other. But as a text sequence, the captions have clear semantic order assigned for each word [Liu et al., 2017]. Secondly, region features are usually presented by the deepest pooling features [Ren et al., 2015], which may lose the local details. For example, the man's detail "white shirt" is missing, and the inadequate descriptions will also decrease the captioning performance.

In this paper, we integrate the intermediate representation and captions generation into a novel one-stage trainable model to obtain a globally optimal solution. The representation misalignment between region features and descriptive semantic features comes from the annotation gap between the detection and captioning, which is essentially the task-based information gap. So, we believe that only the text-annotated optimization based on the one-stage captioner or the collaborative optimization on multi-tasks can close the task-based information gap. Hence, we solve the problem with a onestage captioner due to its lower annotation cost. In addition, these methods use the embedded features of fixed sight, so they cannot flexibly capture effective visual information of different sizes as well as discover the visual relationships of different distances. To address the above problems, we embed the visual features using the multi-level output of the Swin Transformer (SwinT). We then calculate the dynamic correlation between the features of different sights, so as to grasp the interconnected visual representation. Finally, we consider refining the features on both spatial and channel dimensions to improve the global representation capacity of the encoder for generating richer and more accurate captions. Overall, the main contributions of the paper are summarized as follows:

• We first clearly define the task-based information gap in captioners as the representation mismatch between detection-centric and captioning-guided features. We then propose a novel one-stage image captioner (OSIC) with a dynamic multi-sight learner, which is dedicated to optimizing the captioning framework and visual representations to eliminate the gap for image captioning.
• We propose a dynamic multi-sight embedding to adaptively capture and fuse the global structure in large sight and local texture in small sight. Specifically, it computes the salience coefficient of the embedded features in different sights to embed the multi-sight information dynamically, based on the long and short distance dependences of the Swin Transformer.
• In order to improve the global-interactive ability of the

SwinT, we propose a dual-dimensional refining to nonlocally enable the features interaction in spatial and channel dimensions, so that the global representation ability of the encoder can be fully enhanced.

2 RelatedWork

2.1 Pixel Level-based Representation

The early work encode the image into an vector of fixed length as the representation for image captioning [Vinyals et al., 2015]. The major issue caused by using this representation is its heavy compresses and mixture. Inspired by the CNNs in visual extraction for image classification, grid features generated by ResNets are used in captioners. For example, the pre-trained ResNet101 generates grid features and feeds them into Transformer [Vaswani et al., 2017] to infer target words [Gao et al., 2022]. Recently, ViT [Dosovitskiy et al., 2020] and SwinT are used to extract the grid features to build one-stage image captioners. For example, to consider the semantic concepts, the VitCAP [Fang et al., 2022] introduces the visual token to predict the semantic classification. However, ViTCAP still introduces other prior knowledge (i.e., multi-label classification as the concept information) to optimize the models. These mean that ViT and SwinT are promising to reduce the learning gap between vision and text. However, existing one-stage captioners embed the visual features based on the outputs with fixed sight, without adaption to the feature embedding of different distances.

2.2 Regional Level-based Representation

The detection-based methods extract the region-level features as the visual representation by the fixed detector. The salient objects of the image can be captured as a set of feature vectors, which greatly reduces the difficulty of visual semantic embedding and improves the performance of image captioning [Wang et al., 2022a]. For example, up-down model [Anderson et al., 2018] encodes the input image with a set of objects (i.e., RoI-pooled features) detected by a frozen FasterRCNN [Ren et al., 2015] pre-trained on Visual Genome [Krishna et al., 2016]. To further compute the spatial geometry relationships [Hu et al., 2018] for image captioning, Object Relation Transformer [Herdade et al., 2019] explicitly incorporates the relative geometric position and size with the semantic relationships to enrich the embedded features. Conditioned on the region features, M 2 [Cornia et al., 2020] infers the captions through learning a multi-level representation of the relationship between regions to exploit low- and highlevel features. Note that those regional level-based methods learn prior knowledge based on the detection tags. So, the task-based information gap between the detection-centric features and the captioning-guided features makes these twostage captioners suboptimal, which may decrease the captioning performance. Therefore, BPTOD [Kuo and Kira, 2022] mines attributes and relationships from the Visual Genome dataset as an auxiliary input to represent missing information to improve performance. However, BPTOD calculates both contextual descriptions and region features by using the multi-modal pre-trained model and frozen detector, respectively, which still is not an end-to-end trainable model.

3 ProposedMethod

As shown in Figure 2, our OSIC includes a standard Transformer decoder and a dynamic multi-sight learning encoder, i.e., SwinT equipped with a newly proposed dynamic multisight embedding and a cascaded dual-dimensional refining.

3.1 Captioning Procedure

Conditioned an input image I, our OSIC infers and generates a descriptive sentence S. Firstly, multi-level grid features G = {gi},(i = 1, 2, 3, 4) are learnt by SwinT from I. Then, a linear embedding layer followed by patch-merging further extracts the multi-sights features M from the G. Specifically, M is the concatenation of grid features {gi}. After that, salience coefficients E of the grid features of different sights are calculated by DMSE through the average pooling and linear projections to squeeze and dynamically excite the salient features in relevant sights. The output of the DMSE is tiled into a feature sequence as the input of the DDR, which consists of N operation layers, performing non-local interaction in spatial and channel dimensions. Each DDR layer is followed by a feed-forward network [Vaswani et al., 2017] separately. Finally, the refined features are decoded by a Transformer decoder to generate the descriptive sentence S.

3.2 Dynamic Multi-Sight Embedding (DMSE)

Due to missing the visual details, it is inadequate to use only the representation of global compressed (i.e., the pooling features in the last layer of CNNs) for the input image, while the multi-sights of grid features can have more local details benefitting the captioning. Simultaneously, simply merging the grid features of multi-sights may confuse the visual embedding. Thus, the proposed DMSE first calculates the salience coefficients of the multi-sights features by linear projecting the global pooling of all sights. By considering the importance of global optimization for image captioning, we obtain a group of learnable coefficients based on the global linear projections of multi-sights. Specifically, we squeeze the grid features in each channel by average pooling to obtain a representative value sequence Vs, whose length is equal to the number of feature channels. Then, we connect the sequence Vs with the salience sequence by two layers of linear connections, so the salience coefficients E of multi-sights for subsequent captioning can be generated formally as:

where M denotes the concatenation of grid features {gi}, (·)T denotes the transpose operation, and Lji (·) denotes the linear projection to map a tensor with embedding size i into that of j, c is the number of channel dimension. dc is the number of the channel dimension of the squeezed features.

Then, the output of the DMSE module is obtained by multiplying the salience coefficients E by M as follows:

where Norml(·) denotes the layer normalization, which is followed by a shortcut operation. After that, the output of DMSE are fed into the following DRR for further processing.

3.3 Dual-Dimensional Refining (DDR)

Given the embedded features Me from the DMSE, we further feed them into the DRR. To improve the globalrepresentational ability, we further refine the Me via building non-local information interaction in both spatial and channel dimensions. Each non-local interaction of the two dimensions is modeled by computing the scaled dot product of them.

The layer normalization is operated at the corresponding dimension in which the dependence of pixels is computed. The processing output of the spatial position dimension or channel dimension in the DDR layer is obtained as follows:

where i denotes the interaction in either spatial or channel dimension, Normil (·) is the layer normalization operated in i-th dimension. Then, the parallel refining can be formulated as follows:

where M s r denotes the output from the refining layer of single-spatial dimension, and M c r denotes the output of the refining layer of single-channel dimension. Furthermore, the cascade refining calculates features as follows:

where Normcl (·) denotes the layer normalization operated in channel dimension, M sQ r , M sK r , andM sV r are the linear projection representations of the outputs from the non-local refining layers of the multi-head self-attention on a spatial dimension, respectively, and dc is the length of the bottom row vector M sK r (Hdv, :) of M sK r ∈ RHdv×dm , i.e., dm.

After that, the refined grid features M i r are fed into the feed-forward network and sequentially processed by repeating N times the above operation layer (where N is the number of layers). The refined features are finally decoded by a standard Transformer decoder to generate sentences.

3.4 Objective Function

We use two objective functions for optimization in the training process, following the widely used benchmarks. It consist of the cross-entropy loss (XE) for the maximum loglikelihood training and the reinforcement learning loss using the CIDEr score as a reward for self-critical training (SC) [Rennie et al., 2017]. For XE training, with respect to the parameters θ and ground truth sentence y∗ (1:T) , the XE loss is calculated for the optimization as follows:

For the SC training, the model is fine-tuned continually by optimizing the non-differentiable CIDEr score as the reward of reinforcement learning processing formally as:

where n denotes the beam size, r(·) denotes the CIDEr-D score function, and b = (∑i r(yi1:T ))/n is the greedily decoded score value generated by the current model.

4 Experiments

4.1 Experimental Settings

Datasets. The experiments are mainly conducted on the MSCOCO [Lin et al., 2014] dataset, and further generalized on Flickr8K [Hodosh et al., 2013] and Flickr30K [Young et al., 2014] datasets. MSCOCO is the most widely-used and competitive benchmark, which has 164,062 images annotated with 5 ground truth captions for each image. For the fair comparison, we follow the Karpathy's splits [Andrej Karpathy, 2017], in which MSCOCO dataset consists of 113,287 training images, 5,000 validation images, 5,000 offline testing images and 40,775 online testing images with 5 and 40 human-annotations. For training, validation and testing, the Flickr8K dataset containing 8,091 images is split into 6,091 images, 1000 images and 1000 images respectively, and the Flickr30K dataset (31,014 images) is split into 29,000 images, 1014 images and 1000 images respectively.

Evaluation Metrics. The generated sentences are fairly evaluated by using the widely-used metrics, i.e., BLEU1/4 [Papineni et al., 2002], CIDEr [Vedantam et al., 2015], METEOR [Banerjee and Lavie, 2005] and ROUGE-L [Lin, 2004]. They are denoted as B-1/4, C, M, and R for short.

Implementation Details. For training, we first train our model under XE loss for 15 epochs with a mini-batch size of 8, and an Adam optimizer whose learning rate is initialized at 4e-4 with the warmup-step of 20,000. The learning rate is decayed 0.1 times from the 9-th epoch on. We increase the scheduled sampling probability by 0.05 for every 3 epochs. After the XE training, we train our model by optimizing the CIDEr score with the SC training strategy for another 15 epochs with an initial learning rate of 4e-5, which is decayed 0.1 times every 4 epochs. For testing, we use the beam search for our model with a beam size of 2. The default random seed is set to 42. All experiments are conducted in a single NVIDIA RTX2080Ti GPU with Pytorch 1.7 platform.

4.2 Main Results

Offline Evaluation. On the offline MSCOCO Karpathy's test, we show the evaluation of each method in Table 1. The comparied methods can be roughly divided into two groups:

i) Two-stage methods, which adopt offline features directly to infer descriptions, including RFNet [Jiang et al., 2018], UpDown [Anderson et al., 2018], ORT [Herdade et al., 2019], AoANet [Huang et al., 2019], M 2T [Cornia et al., 2020], XTransformer [Pan et al., 2020], DRT [Song et al., 2021], RSTNet [Zhang et al., 2021], S2Transformer [Zeng et al., 2022a], FutureCap [Fei et al., 2022], LightCap [Wang et al., 2023b], ConCap [Wang et al., 2023a] and SCD-Net [Luo et al., 2023]; ii) One-stage methods, which optimize features extracting and captioning simultaneously, including PTSN [Zeng et al., 2022b], BPTOD [Kuo and Kira, 2022] and ViTCAP [Fang et al., 2022]. Our OSIC belongs to the one-stage method.

We first compare our method with the two-stage methods in Table 1. By closing the task-based information gap, our OSIC gains with 41.0 (+1.2%) in B-4 and 137.2 (+0.44%) in CIDEr respectively. Our proposed OSIC compares favorably with all previous methods across almost all metrics. This proves the effectiveness of our proposed OSIC. Then, we compare our method with the one-stage models, including PTSN, BPTOD and ViTCAP. In spite of additional information used in these methods (e.g., retrieved text and image conditioning from pre-trained CLIP [Radford et al., 2021] for BPTOD, and multi-label classification for ViTCAP), our OSIC achieves better performance by using the proposed DMSE and DDR. Note that our model is only trained on the image-text pairs, without other additional information, which has a lower annotation cost of the dataset than them. Moreover, ConCap and LightCap have introduced large vision and language models (LVLM), such as pretrained Clip and Bert [Devlin et al., 2018], respectively. Compared with these LVLM-based captioners, our OSIC is still competitive.

Online Evaluation. We further evaluate our OSIC on the official COCO test by submitting our generated captions to the online test server * in Table 2. It is noteworthy that the performances of our OSIC are reported with a single model, without using any ensemble models. From the observations, our OSIC again surpasses all the single models across all metrics. Moreover, the single model of our OSIC even attains competitive performances with the ensemble version of some captioners (e.g., GET and UAIC), which proves the advantage of our proposed one-stage captioner.

* https://competitions.codalab.org/competitions/3221

Qualitative Results. We show some examples predicted by the baseline and our OSIC in Figure 3. In this paper, the baseline is set as extracting visual features only from the last layer of the SwinT, followed by the standard Transformer decoder. Clearly, our OSIC catches additional details and generates more descriptions with accurate semantic relationships, since our OSIC model can directly and dynamically embed more visual sources of multi-sights. For example, our model accurately captures the key scene in the top picture (i.e., "in front of a red building"), while the baseline misses it.

Generalization on other datasets. Recently, the captioners tend to be rarely evaluated on the Flickr8K and Flickr30K, since that: 1) The number of images in them is much less than the MSCOCO; 2) Their images focus on human activities, which makes the captioner difficult to describe images on large scale scenes. However, it still remains an effective benchmark for evaluating the generalization of models. As shown in Table 3, our OSIC achieves the best results on all standard metrics, and outperforms the current models by a large margin, which proves its well-generalization.

4.3 Ablation Study

Ablation on single DMSE and DDR. From the first and second rows of Table 4, as adding the DMSE into the baseline, the performance is greatly improved across all metrics over the baseline, which proves the effectiveness of the proposed DMSE. As shown in the 3rd to 6-th rows in Table 4, we study the effectiveness of the DDR, which contains the nonlocal interactions in spatial/channel dimension, or combines them in a parallel/cascade mode, respectively. Clearly, our OSIC largely benefits from refining the features on spatial or channel dimension, with improvements of at least +11.8% in Bleu-1 and +29.3% in CIDEr over baseline. All the above demonstrates that the performance gain indeed comes from the DMSE and DDR, which largely raised the metrics.

Ablation on joint DMSE and DDR. We incorporate the DMSE and the DDR with four kinds of non-local modes into the baseline, as shown in the 7-th to 10-th rows in Table 4. Clearly, OSIC with both DMSE and DDR can further deliver better results. Especially, cascaded non-local interactions in spatial and channel dimensions generally perform the best.

Since the slight outperformance, our model in a cascade mode is illustrated in Figure 2. We also visualize the heatmaps of correlations between features to analyze the effectiveness of DMSE and DDR in Figure 4. Specifically, DMSE builds relatively explicit independence of features. DDR further refines the relationship among features globally. The sparsification in heatmaps denotes few correlations among features on the global scale. As additional modules are added, the value in dense regions denotes higher connections with other features, while that in sparse regions denotes fewer connections. For example, for the local features in the top left corner in Figure 4, DMSE supplements and enriches the originally missed connections, and also weakens the unnecessary connections. Based on DMSE, the DDR further refines the embedded features. That is, important connections are strengthened and unimportant connections are weakened, without changing the overall feature distributions. From the above ablation studies, we conclude that both the DMSE and DDR are important to ensure the effectiveness of our OSIC for image captioning.

Impact of the DDR settings. We ablate our OSICwith different settings on the modes and the number of the DDR layer, as shown in Figure 5. We vary the number of refining layers from 0 to 6. From the observation, in parallel and cascade modes, the model with only 1 layer of refining can perform better. We see that parallel refining generally performs the worst among different modes. It can also be seen that our OSIC with only 1 layer cascade refining outperforms other cases. That is, cascading the spatial and channel dimensions is the best setting to obtain the best performance, which is used in all the experiments of this paper. It demonstrates that OSIC works better without needing lots of parameters, which benefits the fast inferring and text generating.

5 Conclusion

We first define the task-based information gap that exists in current two-stage captioners, and address it by presenting a novel one-stage image captioner called OSIC. OSIC directly captures the different sight of representations of the image by a new dynamic multi-sight learning encoder refined by a dual-dimensional refining, then decodes the features into captions. The visual representation is improved by building nonlocally dual-dimensional interaction. Extensive simulations demonstrated the effectiveness of our OSIC attributed to the dynamic multi-sight embedding and dual-dimensional refining, in comparison to other related methods. We also conduct extensive ablation studies to explore the contribution of modules and settings. In the future, we will explore more efficient and robust image captioners under complex real-world conditions, such as describing the rainy images [Wei et al., 2021b; Wei et al., 2022] or blur scenes [Zhao et al., 2022].

Acknowledgments

This work is supported by the National Natural Science Foundation of China (62072151), Anhui Provincial Natural Science Fund for the Distinguished Young Scholars (2008085J30), Open Foundation of Yunnan Key Laboratory of Software Engineering (2023SE103), CCF-Baidu Open Fund and CAAI-Huawei MindSpore Open Fund.