Enriching Phrases with Coupled Pixel and Object Contexts for Panoptic Narrative Grounding

1 Introduction

Panoptic narrative grounding (PNG) [Gonz´alez et al., 2021] is an emerging task whose goal is to predict the panoptic segmentation of things and stuff objects described by noun phrases in a narrative caption. Compared with relevant tasks like referring image segmentation [Yang et al., 2022] and phrase grounding [Mu et al., 2021], PNG not only extends the grounding granularity from sentences to phrases but also yields finer results of segmentation masks rather than bounding boxes, facilitating a more comprehensive visual-linguistic alignment. Benefiting from these properties, PNG enjoys a in Figure 1 (a), the first work [Gonz´alez et al., 2021], which proposes the PNG task, also designs a two-stage baseline method. It first leverages an off-the-shelf panoptic segmentation model [Kirillov et al., 2019a] to generate mask region proposals for the input image, then performs matching between features of phrases and proposals. Herein, the visuallinguistic interaction is realized by cross-attention [Vaswani et al., 2017] between phrases and region proposals, where proposal contexts are aggregated to the phrase features for better matching in the second stage. However, the region proposals are usually inaccurate segments, so the aggregated proposal contexts may provide fragmentary and noisy visual object information to the phrase features. Recently, PPMN [Ding et al., 2022] proposes a one-stage method that lets each phrase directly find its matched pixels to avoid the offline proposal generation process. For visual-linguistic interaction, the one-stage method in Figure 1 (b) aggregate pixel-level visual contexts directly from the grid-form image features to the phrase features. Though pixel contexts contain fine-grained visual details, the absence of object-level visual contexts inclines to incur semantic misalignment between abstract noun phrases and concrete image pixels, yielding suboptimal object delineation.

∗Corresponding author

Given the above discussion, the visual-linguistic interaction schemes of previous methods still limit the discriminative ability of phrase features due to insufficient visual contexts, albeit their strong performances. Therefore, in this paper, we propose to enrich phrases with coupled pixel and object contexts for more comprehensive visual-linguistic interaction as illustrated in Figure 1 (c). On the one hand, pixel contexts aggregated directly from image features contain more fine-grained part details, which can assist phrases in predicting finer segmentation masks. On the other hand, object contexts, which are also derived from image features, focus more on coarse-grained entity perception and can facilitate the distinguishment of corresponding referred objects for different phrases. By coupling the pixel and object contexts, the discriminative ability of phrases could be further enhanced to excavate and match with more relevant visual objects. Concretely, our method first extracts image and phrase features by vision and language encoders. Then, phrases and a set of learnable object tokens are concatenated and fed into our proposed Phrase-Pixel-Object Transformer Decoder (PPO-TD) as queries, in which multi-scale image features from a pixel decoder serve as keys and values. Through consecutive masked cross-attention layers, phrases aggregate pixel contexts from image features to obtain fine-grained part details. Meanwhile, the representations of object tokens are progressively learned to perceive coarse-grained entity information as well. With interlaced self-attention layers in our PPO-TD, phrases are also enriched with object contexts by aggregating information from object tokens to further enhance their discriminative ability. Finally, the inner product between phrase and image features is conducted to generate a binary mask for each phrase based on the pixel-phrase matching similarities.

In order to provide more precise object contexts for phrases, we also design a Phrase-Object Contrastive Loss (POCL) to refine the learning process of object tokens representations. The Hungarian algorithm is first performed to obtain the bipartite matching between object tokens and groundtruth masks/classes. Since the ground-truth masks/classes corresponding to each phrase are also available in the PNG annotations, we can find which object tokens are matched with each phrase. Utilizing these matching relations as the supervision, we devise a contrastive loss on phrases and object tokens to increase the feature similarities between matched phrase-object pairs and decrease those between unmatched pairs. Constrained by this loss, phrase features can aggregate object contexts from more relevant object tokens to enhance their discriminative ability.

Our contributions are summarized as follows:

• We propose to enrich phrases with coupled pixel and object contexts by a Phrase-Pixel-Object Transformer Decoder (PPO-TD), where both fine-grained part details and coarse-grained entity clues are aggregated to phrase features for comprehensive visual-linguistic interaction.
• We also design a Phrase-Object Contrastive Loss (POCL) to increase feature similarities between matched phrase-object pairs and decrease those between unmatched pairs, thus providing more precise object contexts for phrases.
• Extensive experiments on the PNG benchmark demonstrate our method achieves new state-of-the-art performance with large margins.

2 RelatedWork

2.1 Referring Image Segmentation

The goal of referring image segmentation is to segment the object referred by the subject of an input sentence. Previous mainstream methods [Ye et al., 2019; Huang et al., 2020; Hui et al., 2020; Liu et al., 2021; Zhou et al., 2021; Tan et al., 2020; Feng et al., 2021; Jing et al., 2021] are based on FCN [Long et al., 2015] architecture and design various cross-modal attention mechanisms to fuse image and sentence features. With the recent success of Transformer [Vaswani et al., 2017], LAVT [Yang et al., 2022] and ReSTR [Kim et al., 2022] employ Vision Transformer [Wang et al., 2022] to directly encode and fuse image and sentence features with additional channel attention or class seed embedding. CRIS [Wang et al., 2022] transfers large-scale vision-language pretrained knowledge of CLIP for text-topixel alignment. PNG extends the grounding granularity of referring image segmentation from sentences to phrases, which could achieve fine-grained vision-language alignment. In this paper, we propose to enrich phrases with coupled pixel and object contexts for more comprehensive visual-linguistic interaction. The discussion of existing PNG methods has been included in Sec. 1, so we omit it here.

2.2 Panoptic Segmentation

Panoptic segmentation task is first proposed by [Kirillov et al., 2019b], whose aim is to unify semantic segmentation and instance segmentation tasks for a coherent scene segmentation of things and stuff classes. Previous methods [Kirillov et al., 2019a; Cheng et al., 2020; Wang et al., 2020;

Mohan and Valada, 2021] mainly focus on designing specialized branches to tackle semantic and instance segmentation in a single network. Recently, some Transformer-based methods [Cheng et al., 2021; Zhang et al., 2021; Cheng et al., 2022] propose the mask classification paradigm to replace traditional per-pixel classification for more effective panoptic segmentation. For example, MaskFormer [Cheng et al., 2021] exploits object queries to generate binary masks and their classifications simultaneously. Different from panoptic segmentation, PNG segments objects according to narrative captions for a comprehensive cross-modal understanding.

2.3 Phrase Grounding

Phrase grounding requires the model to localize objects referred by noun phrases with bounding boxes. Some early works [Wang et al., 2018; Plummer et al., 2018; Akbari et al., 2019] first extract features of region proposals and phrases separately, then perform phrase-region matching in common feature space with specially designed loss functions. The following methods [Bajaj et al., 2019; Mu et al., 2021; Yu et al., 2020; Dogan et al., 2019] focus on designing effective interaction approaches between visual and linguistic features to excavate multimodal context for more accurate phrase-region matching. The recent GLIP model [Li et al., 2022a] unifies object detection and phrase grounding into vision-language pretraining, which further boosts the grounding performance. The PNG task also extends phrase grounding from region-level localization to pixel-level segmentation, which facilitates accurate object perception.

3 Method

The overall framework of our method is shown in Figure 2, which follows the meta-architecture of Mask2Former [Cheng et al., 2022]. The input image and caption are first encoded by the vision and language encoders to obtain the image and phrase features. Then, the pixel decoder extracts multi-scale image features after the vision encoder. Our proposed PhrasePixel-Object Transformer Decoder (PPO-TD) takes the concatenation of phrase features and learnable object tokens as queries and multi-scale image features as keys and values. Through successive layers in PPO-TD, phrases are enriched with coupled pixel and object contexts to form more comprehensive visual-linguistic interaction. The inner product between phrase features and image features generates the final binary mask for each phrase. Moreover, we also design a contrastive loss on phrase-object pairs to aggregate more precise object contexts for phrases.

3.1 Multimodal Encoders and Pixel Decoder

For the input image I ∈ RH0×W 0×3 , we adopt ResNet50 [He et al., 2016] as the vision encoder to extract image features F i ∈ RHi×W i×Cif , i ∈ {2, 3, 4, 5}, where Hi = H0 2i ,W i = W 0 2i ,C if denote the height, width, and channel numbers of the i-th level of image features from the ResNet backbone. For the input narrative caption with N noun phrases, we adopt the BERT-base model [Devlin et al., 2018] to extract phrase feature embeddings R ∈ RN×Cr , where Cr denotes the channel number of phrase features.

Following [Cheng et al., 2022], we exploit multi-scale deformable attention Transformer (MSDeformAttn) [Zhu et al., 2020] as the pixel decoder to fuse and refine multi-scale image features. The pixel decoder takes {F i }5i=3 as inputs and first projects the channel numbers of these image features into the same value Ch with 1 × 1 convolution layers. Then, the projected image features, their generated sinusoids positional embedding [Vaswani et al., 2017] {P i }5i=3 ∈ RHi×W i×Ch , and learnable scale-level embedding S ∈ R3×Ch are fed into the pixel decoder for multi-scale feature refinement:

Within the pixel decoder, each pixel on the image features samples a small set of positions from features of all scales using learned offsets and aggregates their features using learned attention weights. { ¯ F i }5i=3 denote image features after multiple deformable attention layers, which fuse rich multi-scale visual information. The resolutions of { ¯ F i }5i=3 are 1/8, 1/16, and 1/32 of the original image resolution, respectively. In addition, a bilinear interpolation layer is applied on ¯ F 3 to upsample its resolution by 2×, and the upsampled feature is summed and fused with input backbone feature F 2 using lateral convolution layers:

The obtained feature ¯ F 2 has 1/4 scale of the original image and serves as the per-pixel embedding to generate binary masks for phrases and object tokens.

3.2 Phrase-Pixel-Object Transformer Decoder

In order to enhance the discriminative ability of phrase features for distinguishing referred objects and predicting accurate segmentation masks, we propose a Phrase-Pixel-Object Transformer Decoder (PPO-TD) to enrich phrases with coupled pixel and object contexts for more comprehensive visuallinguistic interaction.

Each layer of our PPO-TD is mainly composed of crossattention, self-attention, and feedforward network layers in a sequential manner. Given the effectiveness and popularity of cross-attention, previous PNG methods also leverage it to conduct visual-linguistic interaction. They utilize phrase features alone as the query and mask proposal features or image features as the key and value to aggregate proposal contexts only [Gonz´alez et al., 2021] or pixel contexts only [Ding et al., 2022] to the phrases. However, proposal contexts may contain fragmentary and noisy visual object information due to the inaccurate segments predicted by the off-the-shelf panoptic segmentation model. Meanwhile, only pixel contexts also tend to incur semantic inconsistency between abstract noun phrases and concrete image pixels since lacking object-level visual contexts may yield sub-optimal object delineation.

Therefore, we argue that both pixel contexts and object contexts are beneficial to enhancing the discriminative ability of phrase features. To introduce proper object-level visual contexts, we take the concatenation of phrase features and a set of learnable object tokens as the query of cross-attention instead of using phrase features alone. For key and value, we adopt multi-scale image features from the pixel decoder.

Take the l-th layer of our PPO-TD as an example. The N phrase features Rl-1 ∈ RN×Ch and M object tokens Ol-1 ∈ RM×Ch from the (l - 1)-th layer are first summed with their learnable positional embeddings, and then concatenated and projected by a linear transformation to obtain the input query feature Ql ∈ R(N+M)×Ch of cross-attention layer:

where [; ] is concatenation and Wq is linear transformation parameter. The key and value features are similarly obtained:

where ¯ Fl ∈ RHlW l×Ch is one of the flattened multi-scale image features { ¯ F i }5i=3 from the pixel decoder, Pl and Sl are learnable positional embedding and scale-level embedding. Following [Cheng et al., 2022], we exploit masked crossattention to constrain the attention scope of phrase features and object tokens as follows:

where Xl-1,Xl ∈ R(N+M)×Ch are concatenated phrase features and object tokens in adjacent layers. Al-1 is the attention mask and its value on position (x, y) is:

where Ml-1 = Resize(Xl-1( ¯ F 2 )T) ∈ R(N+M)×HlW l is the predicted binary masks of phrases and object tokens after normalization and binarization at threshold 0.5 in the l -1-th layer of PPO-TD. The attention mask constrains the crossattention scope to predicted foreground regions, which facilitates the aggregation of more phrase-relevant pixel contexts to the phrase features, and also refines the representations of object tokens by suppressing other distractors.

After the cross-attention layer, concatenated phrase features and object tokens are fed into the self-attention layer to communicate information with each other. By this means, phrase features can also aggregate object contexts from object tokens whose representations are adaptively extracted from multi-scale image features, while object tokens incorporate language priors to distinguish referred objects better. With coupled pixel and object contexts, the discriminative ability of phrase features is further enhanced to form a more comprehensive visual-linguistic interaction.

3.3 Phrase-Object Contrastive Loss

After L layers of PPO-TD, we take the output phrase features RL to generate final mask prediction ML ∈ RN×H0W 0 by inner product with the flattened per-pixel embedding ¯ F 2 followed by upsampling operation. Since each phrase is also associated with a ground-truth class to which the referred object belongs, we also use a linear layer to generate final class prediction CL ∈ RN×K to leverage class priors contained in phrases, where K is the number of classes. Then, the mask classification loss of phrases Lphr mask-cls , which consists of a cross-entropy classification loss and a binary cross-entropy mask loss, is formulated as follows:

where c∗j is the ground-truth class of the j-th phrase, mj , m∗j ∈ RH0×W 0 are the predicted and ground-truth masks of the j-th phrase respectively. For learnable object tokens, we optimize a bipartite matching [Cheng et al., 2022] to find the matched ground-truth mask/class for each object token. Similar mask classification loss Lobj mask-cls is also applied to object tokens, but the predicted masks of object tokens are not used as the final output results during inference.

Since phrases and object tokens are both matched with their own ground-truth masks/classes, we use these ground truths as links to obtain the actual matching relations G∗ ∈ RN×M between phrases and object tokens. Inner-product between features of phrases and object tokens is also performed to generate the matching predictions G ∈ RN×M . Based on G∗ and G, we design a Phrase-Object Contrastive Loss Lpoc (POCL) to increase the feature similarities between matched phrase-object pairs and decrease those between unmatched pairs, which is implemented with binary cross-entropy loss:

Supervised by Lpoc, more precise object contexts from more phrase-relevant object tokens can be aggregated to phrase features, which further enhances their discriminative ability. Our POCL is inspired by previous works [He et al., 2020; Pang et al., 2021; Wu et al., 2022; Li et al., 2022b] which leverage similar loss functions for supervised contrastive learning, but POCL contains intuitive designs tailored for the PNG task where richer contexts are naturally coupled.

Also noteworthy is that Eq. 9 can be rewritten as follows:

Lpoc is equivalent to bi-directional multi-label classification between phrases and object tokens, which means each phrase can be classified into multiple object token labels (plural noun) and each object token can also be classified into multiple phrase labels (same object referred by different phrases). By this means, positive phrase-object pairs obtain high classification probabilities, while negative ones do the opposite. The final loss of our model is the combination of three losses:

4 Experiments

4.1 Dataset and EvaluationMetrics

We conduct experiments on the Panoptic Narrative Grounding (PNG) benchmark [Gonz´alez et al., 2021]. PNG is constructed by transferring caption annotations of Localized Narratives dataset [Pont-Tuset et al., 2020] to the panoptic segmentation annotations of MSCOCO dataset [Lin et al., 2014]. For the linguistic domain, each narrative in PNG contains an average of 11.3 noun phrases, of which 5.1 noun phrases require to be grounded. The total number of noun phrases is 726, 445. For the visual domain, a total of 741, 697 segments (659, 298 are unique) in the panoptic segmentation annotations of MSCOCO are matched with narrative captions.

Following prior works [Gonz´alez et al., 2021; Ding et al., 2022], we adopt Average Recall as the metric to evaluate our method. For each phrase, different Intersection over Union (IoU) thresholds between segmentation prediction and ground truth are considered to determine whether the prediction is a true positive or not. A curve of recall at different IoU thresholds is obtained to calculate the Average Recall, which is the area under the curve. For each plural noun phrase, all ground-truth instances are aggregated into a single segmentation to calculate IoU with similarly aggregated prediction.

4.2 Implementation Details

Our model is implemented with PyTorch. The vision encoder and pixel decoder are pretrained on MSCOCO panoptic segmentation task for a fair comparison with prior works [Gonz´alez et al., 2021; Ding et al., 2022]. The projected channel number Ch in the pixel decoder and our PPOTD is 256. The input image is resized to 1024 × 1024. The language encoder is implemented with the base version of BERT [Devlin et al., 2018] whose embedding dimension Cr is 768. The input caption has a maximum length of 230 words, among which at most N = 30 noun phrases are required to be grounded. The number of learnable object tokens M = 100. Our PPO-TD has L = 9 layers, and the mask classification losses for phrases and object tokens are applied before each layer as auxiliary losses. AdamW is used as the optimizer with a learning rate of 1e-4 . We train our model with batch size 16 for 145K iterations on eight NVIDIA V100 GPUs. The parameters of the language encoder, vision encoder, and pixel decoder are fixed during training.

4.3 Comparison with State-of-the-Art Methods

We compare our method with previous state-of-the-art methods on the PNG benchmark. Since PNG [Gonz´alez et al., 2021] is a pretty new task, only several previous methods could be compared. Table 1 summarizes the quantitative comparison results. It is observed that our method outperforms previous state-of-the-art ones by significant margins. Compared with the latest one-stage method PPMN [Ding et al., 2022], our method achieves significant absolute performance gains of 6.7/6.8/8.2/9.1/4.3 on Average Recall metric for overall/things/stuff/singulars/plurals splits, which demonstrates enriching phrases with coupled pixel and object contexts is superior to aggregating pixel contexts only due to the coarse-grained entity clues complemented by object tokens. Compared with the two-stage baseline method proposed by PNG, our performance boost becomes 10.7 on Average Recall for overall split, which shows the offlinegenerated mask proposals may introduce noisy and fragmentary object-level contexts while our learned object tokens can provide more precise object contexts to phrases. MCN [Luo et al., 2020] tackles referring expression segmentation and comprehension with a collaborative network and its Average Recall is evaluated on only things split. Both our method and previous PNG methods largely outperform MCN, which is probably because sentence-mask/box alignment in referring tasks is insufficient to learn phrase-level segmentation well.

As illustrated in Figure 3, we also draw the recall curves of our methods and previous ones for detailed comparison.

Table 1: Comparison with previous state-of-the-art methods on the PNG benchmark, disaggregated into (a) things and stuff categories, and (b) singulars and plurals noun phrases.

When the IoU thresholds are around 0, the recall values of all methods approach 1. However, the curves of our method are consistently above those of previous counterparts at most IoU thresholds, showing that our method consistently grounds more noun phrases under different evaluation conditions. In particular, the areas under our curves are still larger than those of other methods when the IoU threshold is beyond 0.9, which indicates our method can produce more accurate segmentation masks upon very strict metrics.

4.4 Ablation Studies

We also conduct ablation studies on the PNG benchmark to verify the effectiveness of our network designs.

Component Analysis. In Table 2, we analyze the effects of our proposed components. The 1-st row denotes aggregating pixel contexts (PC) to phrases by standard cross-attention in our proposed PPO-TD, which has achieved considerably higher performance than previous methods. The result of PC shows that pixel contexts are necessary to PNG models for generating masks of high quality. The 2-nd row represents applying masked cross-attention (MA) between phrases and image features to aggregate pixel contexts from foreground regions. The improvement indicates that MA can suppress background noises to provide more phrase-relevant pixel contexts. In the 3-rd and 4-th row, OC denotes introducing learnable object tokens to provide object contexts for phrases, and POCL is the phrase-object contrastive loss which makes phrases obtain more precise object contexts. Combining OC and POCL can achieve a further performance boost of 1.2 Average Recall for overall split over the particularly strong model in the 2-nd row, which is very challenging. This result well demonstrates that enriching phrases with coupled pixel and object contexts can further improve the strong performance of pixel contexts only, which shows the effectiveness of introducing object contexts.

Number of PPO-TD layers. Table 3 shows the results of our PPO-TD with different numbers of layers. Since three scales of image features are grouped into a stage in PPO-TD for three layers, we evaluate the choices of 3, 6, and 9 layers. we can find that more layers in PPO-TD yield higher performance, which shows that iteratively enriching phrases with

Image

Caption

Prediction

Ground Truth

In this picture there is a table. On the table there is monitor, laptop, keyboard, mouse, printer, table lamp, sculpture and cables. To the top left corner there is window with window blinds. In the background there is wall.

There is a man flying a kite. Here we can see some persons in the beach and this is water. There are trees and these are the poles. This is grass and there is a plant. In the background there is a sky with clouds.

In the image we can see a dog and a cat. Cat is sitting on a table. There is a chair. On the table there is a plate, in the plate there are three boxes and mud in i. This is a plant. This is a window from the window we can see grass.

pixel and object contexts is beneficial.

Different Language Encoders. Since our language encoder is fixed during training, we also explore the effects of using different language encoders to extract the phrase features. As shown in Table 4, both the large-scale visionlanguage pretraining model CLIP [Radford et al., 2021] and large-scale language pretraining model T5 [Raffel et al., 2020] yield similar performance to BERT, which indicates our method is insensitive to the structure of language encoder if its parameters are fixed during training.

4.5 Qualitative Results

We present the qualitative results of our method in Figure 4. Our method can generate high-quality segmentation according to dense phrases. We use white dashed boxes to mark some regions where our predictions are inconsistent with the ground truths. For example, in the 2-nd row, our method segments two people near the water while the ground-truth annotation is missing. In the 3-rd row, the ground truth annotates part of the plant pot as "table" while our method yields the correct prediction on the actual table area.

5 Conclusion

In this paper, we focus on the panoptic narrative grounding (PNG) task whose goal is to segment objects according to noun phrases in a narrative caption. An essential factor of PNG is visual-linguistic interaction, where previous methods aggregate proposal contexts or pixel contexts to phrases for realizing it. However, fragmentary visual information or semantic misalignment may be introduced by these methods and their performances are hence limited. Therefore, we propose to enrich phrases with coupled pixel and object contexts with a Phrase-Pixel-Object Transformer Decoder (PPO-TD) for a more comprehensive visual-linguistic interaction. In addition, we also propose a Phrase-Object Contrastive Loss (POCL) to aggregate more precise object contexts from more phrase-relevant object tokens. Extensive experiments on the PNG benchmark show that our method outperforms previous state-of-the-art methods by large margins.

Acknowledgments

This research was supported in part by National Key R&D Program of China (2022ZD0115502), National Natural Science Foundation of China (Grant No. 62122010), Zhejiang Provincial Natural Science Foundation of China under Grant No. LDT23F02022F02, and Key Research and Development Program of Zhejiang Province under Grant 2022C01082.