S2 Transformer for Image Captioning

1 Introduction

As a fundamental task of visual and language reasoning, image captioning, which automatically generates a natural language description for an image, has attracted extensive attention [Vinyals et al., 2016; Cornia et al., 2019; Wang et al., 2020; Chen et al., 2021]. Originally inspired by neural machine translation [Sutskever et al., 2014], its general paradigm is: firstly encoding an image to extract visual features, and then feeding those features into an encoder-decoder framework to generate descriptions [Xu et al., 2015]. Due to its specific proprieties, such as rich visual information and sophisticated semantics of descriptions, it remain a challenging problem.

∗ Pengpeng Zeng and Haonan Zhang contribute equally to this paper.

†Corresponding author: Jingkuan Song.

For visual feature extracting, two types of features are widely adopted: region and grid features, as shown in Fig. 1a (i) and (ii), respectively. The region features are designed to explore object instances, which strongly correlate with nouns in textual descriptions (e.g., "giraffe", "grass" and "tree"). To detect explicit object boxes and output region features, existing off-the-shelf methods such as Faster-RCNN [Ren et al., 2015] are pre-trained on VG dataset [Krishna et al., 2017], which is computationally expensive and not flexible. Beyond that, the detected regions may lack contextual information (e.g., "stands on" and "in the forest") and fine-grained details (e.g., "eat leaves"). By contrast, the grid features are designed to extract all patch information to cover the whole image. Previous studies [Jiang et al., 2020; Zhang et al., 2021] revise the advantage of grid features and find them to perform better than region features both in terms of performance and time-cost. However, directly operating at grid features in a flatting manner unavoidably disrupts the spatial association between grids. One natural solution is to combine the above two visual features as visual inputs, but it suffers from computation costs and complex fusion procedures.

Furthermore, transformer-based models are applied as the encoder-decoder for high quality image captioning [Li et al., 2019; Pan et al., 2020; Zhang et al., 2021] due to its strong modeling capabilities and excellent performance, shown as Fig. 1b (i). Most of them are focused on modifying the attention block. For example, [Huang et al., 2019] proposes an "attention on attention" module, which extends self-attention mechanisms to determine the relevance between attention results and queries. [Pan et al., 2020] proposes a X-Linear attention block that fully employs bilinear pooling to capitalize on visual information or perform multi-modal reasoning selectively. [Cornia et al., 2020] proposes a M2 transformer that designs a memory-augmented attention to encode a priori information and a mesh cross attention (MCA) to take advantage of scale-wise features to fully explore rich visual semantics, shown as Fig. 1b (ii). However, M2 transformer (w/o memory) based on grid feature has suffered a performance degradation and brought a parameters increase compared with the vanilla transformer, where the results are summarized in Fig. 1b (iv). Thus, how to effectively and efficiently incorporate grid features with transformer-based architecture remains to be explored for image captioning.

(iii) Ours

(iv) Performance Comparison

(b) Different transformer-based architectures

Figure 1: (a) Comparison of different visual features. Based on grid features, our proposed SP module aims to implicitly learn spatial information about grids in a pseudo-supervised manner instead of directly using explicit region features. (b) Comparison of different transformer-based architectures, where all models adopt grid features as visual features. Our SR module simultaneously explores both low- and high-level encoded features to produce superior results while maintaining a relatively small model size.

To address the above problem, this paper proposes a novel Spatial- and Scale-aware Transformer (S 2 Transformer). Specifically, we firstly propose a Spatial-aware Pseudosupervised (SP) module that aims to solve the spatial information loss of grid features caused by the flattening operation. In practice, we utilize a number of learnable semantic clusters to quantize grid features into semantic clusters, which implicitly represent discriminative regions. Furthermore, to maintain the model size and produce superior performance, we propose a simple weighted residual connection, named Scale-wise Reinforcement (SR), module to simultaneously explore both low- and high-level encoded features, shown as Fig. 1b (iii). From the Fig. 1b (iv), we can see that compared with vanilla transformer, only adopting our SR can achieve an improvement of 1.8 CIDEr points with a slight parameters increase (i.e., 1.05M), while M2 with a mesh operation increases parameters (i.e., 4.72M) and decreases the CIDEr by 1.1. To summarize, our contributions are threefold:

• We devise a S 2 Transformer, a simple yet effective method, which extends the vanilla transformer framework to fully exploit gird visual features in terms of spatial and scale perception.
• We propose a SP module, which generates valid pseudo-region features for grid features to capture spatial information based on their clustering information. Moreover, we propose a simple SR module that further takes advantage of both low- and high-level encoded features without excessive increasing model size.
• We comprehensively evaluate our approach (S 2 Trans-

former) on the MSCOCO benchmark. Experimental results demonstrate that our method performs best while maintaining the small model size.

2 S 2 Transformer

In this section, we present a novel Spatial- and Scale-aware Transformer (S 2 Transformer) for image captioning. The overview of the architecture is depicted in Fig. 2.

2.1 Overview

Given an image I, the task of image captioning is to automatically generate a description D about visual contents in images, following the paradigm of an encoder-decoder framework. Technically, S 2 Transformer first applies a feature extraction to obtain gird features G = {gm}Mm=1 about an image, where M indicates the number of grids. As for the spatial information loss caused by flattening operation when feeding G into an encoder-decoder model, our proposed Spatialaware Pseudo-supervised (SP) module is adopted to implicitly learn possible and discriminative regions to obtain pseudo-region features P :

where N means the number of pseudo regions. Then, we use the same encoder to exploit the visual information of original grid features G and pseudo-region features P simultaneously:

where the Encoder is consistent with the vanilla Transformer's encoder without any modifications, which consists of two main components: Multi-head Self-Attention (MSA), and Feed Forward Network (FFN). Note that for the sake of concise expression, positional encoding, residual operation and layer normalization are omitted.

Different from previous Transformer-based models, which only feed the encoded feature obtained from the top encoder layer to the decoder, our proposed Scale-wise Reinforcement (SR) module is to simultaneously explore both lowand high-level encoded features to obtain augmented encoded features V :

where ¯GL (or ¯ PL) means the output of L-th encoder layers and VG and VP represent grid and pseudo-region augmented encoded features, respectively. Finally, we fuse VG and VP to obtain the final encoded features V ∗ and feed it to the decoder:

where [; ] means the operation of concatenate, WV is a learnable parameter and the decoder is the same as the vanilla Transformer's decoder. The detail of our two main components (SP and SR) is described in the next subsection.

2.2 Spatial-aware Pseudo-supervised (SP) Module

As discussed above, directly operating at grid features leads to the loss of spatial information of regions. A plain idea is introducing region features to compensate for the deficiency. However, combining gird and region features will inevitably increase the computational complexity of the model. Intuitively, if we implicitly select and aggregate discrete grid features into several sub-spaces to obtain pseudo-region features, this operation would become more flexible. Motivated by this spirit, we propose a SP module to cluster the grid features with multiple centroids without explicit supervision. The purpose of these centroids is to integrate grids features of similar semantic information together to represent possible and discriminative regions.

Formally, in SP, we first design N learnable clusters as C = {c1, ..., cN}. Following [Arandjelovic et al., 2016], we calculate the similarity between grid features and clusters by dot-product. Given each grid feature gm, it can be mapped to the n-th cluster in the following manner:

where b{n,k} is a trainable parameter. The feature representation of each center pn is obtained by a weighted integration of all grid features:

where "Norm" means ℓ2-normalization operation and ˜ cn is a learnable parameter which has the same size as cn. Thus, we define the final features P as pseudo-region features.

2.3 Scale-aware Reinforcement (SR) Module

Recently, transformer-based captioning models have been proved helpful for image captioning. However, existing models neglect the low-level semantic information from the bottom of the encoder layer during the decoding process. Although [Cornia et al., 2020] has provided a solution with a complex meshed cross-attention, we further propose a novel and simple SR module to address the above limitations by incorporating all features from each encoding layer into the top features.

For simplicity, we take gird features as an example. Specifically, given the output features ( ¯G1, ¯G2, ..., ¯GL) of each encoder layer, we first concatenate them all together:

Then, to integrate both low- and high-level visual information, we employ a Multi-Layer Perception (MLP) which can weigh the contribution of features of each layer:

where W1 and W2 are trainable projection matrices.

Since the output of the top encoder layer contains more important visual information, to prevent the insertion of additional noise perturbations, we add features G′ to ¯GL to obtain the final grid augmented encoded features VG:

where the λ is an adjustable weighting factor. In a same way, we obtain the pseudo-region augmented encoded features VP .

2.4 Training

Generally, the training of captioning model is split into two stages [Rennie et al., 2017; Zhang et al., 2021]. In the first stage, we utilize cross-entropy loss to optimize our model:

where T is the length of word sequence and w∗ 1:t-1 is the ground truth tokens in the description D.

In the second stage, we adopt the strategy of reinforcement learning, which exploits the CIDEr score as reward r(·) with self-critical sequence training [Rennie et al., 2017]:

In addition, we employ the gradient expression in [Cornia et al., 2020], which computes the reward baseline of the reward by the mean operation of rewards, rather than greedy decoding. A sample's gradient expression is defined as:

where k is the number of sampled sequences, wi1:T denotes the i-th sampled sequence, and b represents the average reward earned by the sampled sequences.

3 Experiments

3.1 Experimental Settings

Dataset and Metric. We conduct experiments to verify the effectiveness of our proposed S 2 Transformer on commonlyused image captioning dataset, i.e., MS-COCO. It consists of 123,287 images, each associated with five different descriptions. In offline testing, we follow the setting in [Karpathy and Fei-Fei, 2015], where 113,287 images, 5,000 images, and 5,000 images are used as train, validation, and test set, respectively. The online evaluation is done on the COCO online test server, where ground-truth annotations of 40,775 images are not publicly provided. We measure the captioning performance using the standard evaluation metrics, including BLEU [Papineni et al., 2002], METEOR [Banerjee and Lavie, 2005], ROUGR [Lin, 2004], CIDEr [Vedantam et al., 2015], and SPICE [Anderson et al., 2016].

Implementation Details. Following [Zhang et al., 2021], we adopt the same pre-trained Faster-RCNN [Ren et al., 2015] provided by [Jiang et al., 2020] to extract grid features, where the gird shape is 7×7 and the dimension of each grid is 2,048. In practice, our encoder and decoder both have 3 layers, where each layer uses 8 self-attention heads and the inner dimension of FFN is 2,048. The number of cluster centers N is 5 and the hyper-parameter λ = 0.2 in Eq. 9.

We employ Adam optimizer to train all models and set batch size as 50. For cross-entropy (CE) training, we set the minimum epoch as 15. If CIDEr drops in 5 consecutive epochs, we will choose the model with the best CIDEr score for self-critical sequence training. Specifically, we use an epoch decay schedule to adjust the learning rate for CE by following [Zhang et al., 2021]:

where n denotes the number of current epoch. For selfcritical sequence training, the learning rate is fixed at 5×1e-7.

3.2 Ablation Studies

The core of our proposed S 2 Transformer is to generate the high-quality visual descriptions by introducing a spatialaware pseudo-supervised (SP) module and a scale-wise reinforcement (SR) module into a vanilla transformer model. In this section, we conduct comprehensive ablation studies to prove the effectiveness of our method.

Effect of SP and SR. Tab. 1 gives the results of four control experiments to investigate the impact of our proposed SP and SR modules: i) baseline: adapting vanilla transformer model without any modifications, ii) baseline+SP: integrating SP into baseline, iii) baseline+SR: integrating SR into baseline, and iv) baseline+SP+SR: integrating both SP and SR into baseline. Obviously, the performances are enhanced by individually adding SP and SR to the baseline, particularly improving 2.0 and 1.8 points on CIDEr, respectively. Moreover, the combination of the two components achieves further improvement. Also, we report the parameters of each model for measuring its complexity. SP and SR slightly increase parameters by 0.02M and 1.05M compared with the baseline. To sum up, our proposed components achieve huge improvements with a small computational cost, indicating our methods' effectiveness.

Effect of Pseudo-region feature. In Tab. 2, we execute several experiments to examine the effect of different visual features, including grid features (G), region features (R) and our pseudo-region features (P). All models both adopt a vanilla transformer. Using only a single feature, our P performs worst, which indicates that only using pseudo-region features may lose some important visual information. Combining two features (i.e., G+P and G+R) can bring performance gains. Meanwhile, G+P obtains more significant improvement than G+R, thus indicating the practicality of our pseudo-region features. Besides, in terms of FLOPs, G + R brings an excessive increase of 0.43G while G+P brings a slight increase of 0.04G. The results demonstrate that the highly abstract pseudo-region features are sufficient and complementary for grid features instead of directly using explicit region features.

Effect of N in SP . To determine how many pseudo regions the model needs to learn, we set the range of cluster number N from 3 to 9 as shown in Fig. 3a. Note that our SP is serving for high-level semantic information extraction. From the figure, we can observe that if N is too large, it may be difficult for the model to find discriminative pseudo regions, which harms the performance of the model. On the contrary, if N is too small, much weak semantics will be discarded in large quantities, resulting in poor results. Our approach achieves the best results with N = 5 clusters.

Effect of λ in SR. To choose the best weighting factor λ in Eq. 9, we conduct a series of experiments by setting the different values of λ. The results are shown in Fig. 3b. We find that the performance drastically drops with the increase of λ and the best results are obtained when λ = 0.2. It reveals that too larger λ introduces more redundant noise for the decoder. Thus, we set λ = 0.2 in the final model.

Fair comparison with strong transformer-based baselines.

For a fair comparison, we report experimental results utilizing the same ResNext101 grid feature as the visual input shown in Tab. 3. All models are based on improved versions of the vanilla transformer. Specifically, our method achieves state-of-the-art performance on most metrics except B@4, which demonstrates superior performance without the interference of diverse features. Moreover, compared to the SOTA method RSTNet, which increases the Transformer parameters by 122.74M, our model brings only slight growth of 1.07Mon parameters. It further demonstrates that our method can effectively and efficiently incorporate grid features with transformer-based architecture.

GT1: A cat that is sitting next to a laptop.

GT2: Cat sitting right next to keyboard on laptop.

GT3: The orange cat is sitting on the laptop.

Trans: An animal sitting on a desk.

Ours: An orange cat sitting on a desk next to a laptop.

GT1: A man takes a picture of food in a restaurant.

GT2: A man taking a picture of his meal at diner table.

GT3: A man taking a photo of food on a table.

Trans: A man is eating and taking a picture of food.

Ours: A man taking a picture of a pizza with a camera.

GT1: A close up of a stop sign in a small town

GT2: A red stop sign sitting next to country road.

GT3: A close up of a stop sign in a small town.

Trans: A stop sign sitting on road.

Ours: A red stop sign on the side of a street.

GT1: Four men are posing for picture at an event. Trans: A group of men standing next to each other. Ours: Four older men posing for picture at event. GT2: A group of man standing next to each other. GT3: A group of four older men posing for a photo.

GT1: A long line of pans that are hanging on the wall.

GT2: A kitchen with pots hanging over the stove.

GT3: Hanging frying pans in commercial kitchen.

Trans: A lot of pots and pans on the counter.

Ours: A kitchen with pots and pans hanging on the wall.

Figure 4: Visualization of the proposed S 2 Transformer. Each example consists of a raw image, a learned map of cluster indices by SP, the ground-truth descriptions, and the generated description by the transformer and ours. The size of these learned maps is 7×7.

3.3 Quantitative Analysis

Compared Methods. In this section, we compare our proposed S 2 Transformer with the state-of-art methods both on offline and online evaluation, including SCST [Rennie et al., 2017], Up-Down [Anderson et al., 2018], RFNet [Jiang et al., 2018], GCN-LSTM [Yao et al., 2018], SGAE [Yang et al., 2019], ORT [Herdade et al., 2019], AoANet [Huang et al., 2019], M2Transformer [Cornia et al., 2020], X-Transformer [Pan et al., 2020], TCIC [Fan et al., 2021] and RSTNet [Zhang et al., 2021].

Offline Evaluation. In Tab. 4, we show the image captioning results of our method and compare it to the aforementioned competitors on the offline test split. Overall, our method outperforms all compared methods in terms of B@1, M, R, C, and S. Specifically, compared with the best counterpart RSTNet using extra knowledge from a pre-trained language model, our method yields better gains on all metrics, demonstrating the superiority of our approach.

Online Evaluation. To further verify the benefit of our S 2 Transformer, we estimate it on the online COCO test server. Following the compared methods, we integrate the results of four models with different initialization for testing. The comparison results are summarized in Tab. 5. It is clear that our S 2 Transformer outperforms state-of-the-art models on most metrics. Particularly, with respect to the best competitor RSTNet (ResNext152), our method S 2 Transformer with ResNext152 achieves improvements of 0.7 and 1.0 CIDEr points on 5 reference captions (c5) and 40 reference captions (c40), respectively.

3.4 Visualization

Fig. 4 provides some qualitative results to show the pseudo regions learned via the proposed SP in heat maps and the high-quality descriptions generated by our proposed model. In the heat map, different colors represent different index values, which indicate different pseudo-regions. As we can see, SP focuses on specific visual regions in foregrounds but also reserves discriminative background information, confirming the usefulness of exploiting pseudo regions to retain the spatial information. Besides, our model can generate more accurate and diverse descriptions compared to basic transformer model. More visualizations are included in the supplementary material.

4 Conclusion

In the paper, we study how to effectively and efficiently incorporate grid features with transformer-based architecture for image captioning. To achieve this target, we propose a S 2 Transformer-a simple yet effective approach that implicitly learns pseudo regions through a series of learnable clusters in a SP module and simultaneously explores both lowand high-level encoded features in a SR module. Noticeably, pseudo regions can effectively capture spatial information lost by the flattening operation of gird features. Extensive experiments on the MSCOCO benchmark and visualization analysis confirm the effectiveness and interpretability of our method. Besides, our approach does not bring excessive parameters compared with the vanilla transformer.

Broader Impact. Our paper focuses on learning image captioning tasks, which has broader application in real-world scenarios such as human-machine interaction and visualimpaired assistance. Our method provides positive impacts, including 1) implicitly learning discriminative region features instead of using explicit region features, which reduces the increase in parameters and computation, and 2) providing a simple task-specific transformer-based model, which generates more high-quality descriptions. However, it is still challenging to deploy existing models into real-world scenarios because of their susceptibility to attacks, which remains our responsibility to grow awareness of these potential dangers.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant No. 62020106008, No. 62122018, No. 61772116, No. 61872064), Sichuan Science and Technology Program (Grant No.2019JDTD0005).