WiCo: Win-win Cooperation of Bottom-up and Top-down Referring Image Segmentation

1 Introduction

Referring image segmentation (RIS) is a new type of segmentation task aiming to segment the object referred by a natural query expression. The current approaches for referring image segmentation can be broadly classified into two categories [Hui et al., 2020], i.e., top-down and bottomup methods. Top-down methods calculate the object-centric cross-modal alignment between each region proposal from pretrained detector and query for getting cross-modal instance embeddings and then decode cross-modal instance embeddings to alignment score for retrieving the most confident region proposal as segmentation result [Yu et al., 2018; Liu et al., 2019]. Bottom-up methods calculate the finegrained cross-modal alignment between each pixel and query for acquiring cross-modal pixel embeddings and then decode the embeddings to retrieve those pixels of referred object [Wang et al., 2022; Yang et al., 2022]. However, according to our observations in Figure 1, existing top-down and bottom-up methods are still perturbed by two types of errors: Polar Negative (PN) and Inferior Positive (IP). These two errors can be identified by the Intersection-over-Union (IoU) between predictions and ground truths. PN samples are those predictions that have nearly no overlap with the ground truth (IoU→0). IP samples are those predictions that ignore some components of the referred object (IoU ∈ [0.5, 0.8]).

† Corresponding Author

To analyze how top-down and bottom-up methods are disturbed by PN errors and IP errors, we visualize the IoU distribution of top-down and bottom-up methods in Figure 2. We split the distribution curve into two parts: positive set (samples with IoU > 0.5) and negative set (samples with IoU < 0.5). As shown in Figure 2 (a), the positive set of topdown method achieves higher location precision than bottomup method because the prior object information suppresses low-quality IP samples. But top-down method easily generates PN samples on negative samples due to the lack of finegrained cross-modal alignment. In Figure 2 (b), bottom-up method outputs a large number of IP samples on positive set owing to the lack of prior object information. Nevertheless, the negative set of bottom-up methods has smoother distribution than top-down methods and the PN samples of bottom-up methods are far fewer than top-down methods because finegrained information provides robust cross-modal alignment. According to the analysis above, we can conclude that topdown and bottom-up methods are highly complementary.

Intuitively, we can fuse top-down and bottom-up methods by direct combination, i.e., straightly averaging their results. However, as shown in Figure 2 (c), this scheme leads to harmful cooperation between top-down and bottom-up methods which can be attributed to the lack of feature interaction and the inappropriate integration of results. In this context, we build Win-win CoopEration (WiCo) to exploit the complementary nature of top-down and bottom-up branches by interacting with each other and integrating results of two branches in an adaptive manner, which follows "Interaction then Integration" paradigm for compensating the defect of direct combination. WiCo contains two modules: Complementary Feature Interaction (CFI) and Gaussian Scoring Integration (GSI). CFI is designed to perform interaction between features of two branches for compensating the lack of finegrained information in top-down branch and prior object information in bottom-up branch. GSI is designed to model the gaussian performance distributions of top-down and bottomup branches and adaptively integrate results of two branches by sampling confidence scores from the distributions. Figure 2 (d) shows that our framework largely reduces IP errors and PN errors and generates fine IoU distribution with the merits of top-down and bottom-up methods, which demonstrates our method is more effective than direct combination scheme for incorporating top-down and bottom-up methods.

In summary, the main contributions are as follows:

• We analyze the behavior of several top-down methods and bottom-up methods when facing PN and IP errors. According to the analysis, we discover that existing topdown and bottom-up methods are highly complementary in how to cope with PN errors and IP errors.
• We propose WiCo to adequately exploit the characteristics of top-down and bottom-up methods methods and let them effectively complement each other on both interaction and integration aspects, which can better process PN errors and IP errors than intuitive direct combination.
• Extensive experiments show that our WiCo can boost the performance of top-down and bottom-up methods methods by 2.25%∼6.66% under three common datasets: RefCOCO, RefCOCO+ and G-Ref with reasonable cost.

2 RelatedWork

Top-down Method. Previous efforts on top-down style referring image segmentation are about how to calculate better object-centric cross-modal alignment between region proposals of instances and referring expression query. For example, MAttNet [Yu et al., 2018] decomposes referring expressions into three components to match instances. NMTree [Liu et al., 2019] regularizes the cross-modal alignment along the dependency parsing tree of the sentence. CAC [Chen et al., 2019b] introduces cycle consistency between referring expression and its reconstructed caption into the reasoning part of network for boosting cross-modal alignment.

Bottom-up Method. Previous efforts on bottom-up style referring image segmentation mainly focus on densely aligning and fusing visual and linguistic features for better crossmodal pixel features. For example, early works [Hu et al., 2016a; Li et al., 2018] propose to use simple concatenation to align and fuse visual feature maps and linguistic feature vectors, respectively. For replacing simple concatenation, some prior works use cross-modal attention to focus on important pixel regions and informative keywords for longrange cross-modal context [Shi et al., 2018; Ye et al., 2019; Chen et al., 2019a]. Besides, some other works use complex visual reasoning to capture more explainable crossmodal context [Huang et al., 2020; Hui et al., 2020; Yang et al., 2021]. Recently, Vision transformer (ViT) [Dosovitskiy et al., 2020] has been proposed as a new visual network paradigm. Due to its compatibility with multi-modal data, some works use it to jointly encode visual and linguistic features for intensive cross-modal alignment [Ding et al., 2021; Li and Sigal, 2021; Wang et al., 2022; Yang et al., 2022].

3 Methods

3.1 Overall Pipeline

To ensure the generality of our framework, the WiCo is designed to be compatible with arbitrary top-down and bottomup methods. As shown in Figure 3, WiCo has three parts: topdown branch, bottom-up branch and "Interaction then Integration". Top-down branch is used to deploy top-down methods. Bottom-up branch is used to equip bottom-up methods. "Interaction then Integration" is the key component of WiCo which is used to build cooperation between top-down and bottom-up branches for achieving a win-win improvement.

Top-down style methods are essentially a cross-modal match network [Yu et al., 2018]. It uses the pretrained detector and cross-modal match network to obtain instance masks M = {m 1 ∈ {0, 1} H×W , m 2 , ...,m n }, cross-modal instance embeddings E = {E 1 i ∈ R C , E 2 i , ..., E n i } and crossmodal alignment scores S = {s 1 , s 2 , ..., s n }. In general, top-down branch outputs a instance triplet set {M, E, S} = {(m 1 , E 1 i , s 1 ), (m 2 , E 2 i , s 2 ), ..., (m n , E n i , s n )}. Extracting segmentation results Ptd from triplet set is fomulated as:

where Ptd is the segmentation logits results. The binary segmentation results are m argmax(S) .

Bottom-up methods are essentially a cross-modal fusion network [Hu et al., 2016b]. It uses a cross-modal fuse network to jointly encode images and texts to cross-modal pixel embeddings Ep ∈ R C×H×W and decode cross-modal pixel embeddings to segmentation results Pbu ∈ R H×W . Decoding cross-modal pixel embeddings into segmentation results is formulated as:

where Linear(·) denotes 1x1 convolution for logit regression and σ(·) is sigmoid function. Pbu is the probability map and the binary segmentation results are extracted from it by threshold τ (Pbu > τ). In general, bottom-up branch outputs cross-modal pixel embeddings and segmentation results.

"Interaction then Integration" is designed to exploit the complementary nature of top-down and bottom-up methods. To complement on interaction aspect, the outputs of bottomup branch and top-down branch are input into CFI (Section 3.2) for updating features and results. To complement on integration aspect, the updated results are input into GSI (Section 3.3) to integrate results.

3.2 Complementary Feature Interaction

The detailed calculation flow is illustrated in Figure 4. Suppose that we already acquire pixel embeddings Ep from bottom-up branch and instance triplet set {M, E, S} from topdown branch, we hope that CFI can let the fine-grained information of pixel embeddings and object-centric information of instance triplet set enhance each other.

Top-down for Bottom-up. For enhancing pixel embeddings Ep, we assign object-centric information of each enhanced instance embeddings ˆE to corresponding pixels according to the instance masks M and concatenate these instance embeddings with raw pixel embeddings to generate enhanced pixel embeddings ˆEp:

where E {x,y} p denotes enhanced pixel embeddings at (x, y) pixel location and E j i is the enhanced instance embeddings of i-th instance. 1{m j [x,y]=1} is the indicator function, where it is equal to 1 when the (x, y) pixel location of j-th mask is 1, and 0 otherwise. The enhanced pixel embeddings are then decoded to new bottom-up results:

where the Linear(·) shares same weights with Eq. 2.

Bottom-up for Top-down. For enhancing instance embeddings E, we use vision transformer decoder [Cheng et al., 2021] as "Feature Interaction" module to refine the instance embeddings by fine-grained information of pixel embeddings Ep. Before inputting, the instance embeddings are modulated by cross-modal alignment scores S for preserving crossmodal information:

Then, transformer decoder sets these modulated instance embeddings E ⊙ S as queries to generate enhanced instance embeddings ˆE and predict enhanced alignment scores ˆS. With new alignment scores, we can update the segmentation results of top-down branch:

3.3 Gaussian Scoring Integration

After obtaining top-down results ˆPtd and bottom-up results ˆPbu, we use GSI to integrate them for generating more robust and higher-performance results. GSI has three steps: Distribution Prediction, Score Sampling and Results Blend. The details of three steps are introduced below:

DistributionPrediction. Because of the uncertainty, we set the performance score as a latent variable following a specific distribution. Due to the excellent computability, we use gaussian distribution to model the performance distribution [Kingma and Welling, 2013]. For representing gaussian distribution, we predict the mean µ and standard deviation σ according to the results and features of two branches:

where MLP(·) denotes 3 fully connected layers, GAP(·) denotes global average pooling operation and split(·) denotes channel split operation. With predicted mean and standard deviation, we obtain the performance distribution of bottomup and top-down branches, i.e., N(µbu, σbu) and N(µtd, σtd).

Score Sampling. Performance distribution indicates the confidence score range of prediction. We sample a value from the performance distribution as the detailed confidence score of this prediction. For differentiable optimization, we utilize re-parameterization trick [Kingma and Welling, 2013] to modify the sampling process:

where IoUtd and IoUbu denote confidence score of top-down and bottom-up branch results. For optimizing the distribution prediction model, we calculate smooth-l1 loss between predicted confidence score and ground truth IoU value.

Results Blend. Note that argmax(·) is essentially a nondifferentiable operation during gradient backward, we adopt a differentiable implementation [van den Oord et al., 2017] of argmax(·) function during training pharse:

where λ ∈ {0, 1} n is a binary vector to indicate the index of the max value, one-hot(·) is the one-hot encoding function and sg(·) is the stop gradient operation. The λ is used to build differentiable segmentation results of top-down branch ˆP ′ td :

where n is the number of instances. For generating final segmentation results, we use confidence scores to calculate a weighted sum results of top-down and bottom-up branches:

The final results ˆP are used to calculate segmentation loss with ground truth mask during training phrase and are decoded to binary mask by threshold τ during inference phrase.

4 Experiments

4.1 Experimental Setup

Our model is evaluated on three standard referring image segmentation datasets: RefCOCO [Yu et al., 2016], RefCOCO+ [Yu et al., 2016] and RefCOCOg [Mao et al., 2016]. For top-down branch, MAttNet [Yu et al., 2018] is selected as the main equipment due to its simple structure and effectiveness. As for the bottom-up branch, several advanced and representative methods are selected, e.g., VLT [Ding et al., 2021], CRIS [Wang et al., 2022] and LAVT [Yang et al., 2022], to show the effectiveness and generality of our method. The data preprocessing operations are in line with the original implementation of those selected methods. Because MAttNet is an early method that has an obsolete instance extractor, Mask2Former [Cheng et al., 2021] (ResNet50) is adopted as an instance extractor to compensate for the top-down branch to avoid the cask effect, which improves the performance of MAttNet from 56.51 to 62.62 on RefCOCO val set. Based on previous works [Luo et al., 2020b; Ding et al., 2021], mask IoU is adopted to evaluate the performance of methods. To reduce the training cost, the selected models are initialized by pretrained weights and just finetune when inserting them into our framework. AdamW [Loshchilov and Hutter, 2017] is adopted as our optimizer, and the learning rate and weight decay are set to 1e-5 and 5e2. We train our models for 5,000 iterations on an NVIDIA V100 with a batch size of 24. To binarize the probability map and get segmentation results, the threshold τ is set to 0.35 to calibrate previous works [Ding et al., 2021].

4.2 Quantitative Analysis

Main Results. Table 1 reports the comparison results between our method and previous state-of-the-art methods in three common datasets, i.e., RefCOCO, RefCOCO+ and RefCOCOg. Some top-down and bottom-up methods that are easy to reproduce are selected for benchmark. Specifically, there are three combinations, i.e., VLT + MAttNet, CRIS + MAttNet and LAVT + MAttNet. Because bottom-up methods are mainstream methods, we mainly describe the performance improvement based on bottom-up methods in Table 1. Utilizing WiCo to incorporate these three model combinations, the fusion results improve the results of VLT, CRIS and LAVT by 6.09%, 3.94% and 6.66% on RefCOCO val split, 5.78%, 4.23%, 2.25% on RefCOCO testA split and 4.5%, 3.38% and 2.51% on RefCOCO testB split. Other datasets also consistently show the performance improvements of our method over the selected baseline models.

Different Results Integration Strategies. In Table 2, we attempt different results integration strategies and check if these integration strategies can boost the integration results of top-down and bottom-up branches. In terms of results integration strategies, GSI is compared to three straight strategies, i.e., "Intersection", "Union", and "Average". Although these strategies improve the performance of top-down and bottom-up methods, our proposed GSI still performs better than them, indicating that GSI provides a more perceptive and robust way to integrate results. Moreover, we build an abbreviated version of GSI to check the effectiveness of the gaussian distribution-based performance modeling, i.e., Scoring Integration (SI). Based on the experiment results that the GSI performs 0.68% better than SI, it is concluded that the gaussian distribution-based performance modeling makes sense.

Effect of Feature Interaction. Feature interaction boosts results by improving the respective results of top-down and bottom-up branches. For diagnosing if feature interaction is beneficial for final results, we conduct comparison experiments of WiCo with CFI and without CFI. As shown in Table 2, WiCo with CFI improves baseline by 3.19% and performs 2.11% better than WiCo without CFI. The experiment results show that CFI effectively improves top-down and bottom-up branches by 2.42% and 2.72%. Moreover, it also shows that feature interaction (CFI) boosts final performance on a different aspect than results integration (GSI).

Complementary Effect of Different Model Combinations.

Three kinds of combinations are constructed (bottom-up + bottom-up, top-down + top-down and bottom-up + topdown) to check the complementary effect of different model combinations in Table 3. The model combination with two same kinds of models is defined as "Homogeneous" combination. On the contrary, the model combination with two different kinds of models is defined as "Heterogeneous"

model α

+

β

IoUα

+IoUβ

IoU+

Speed

VLT ♣ +CRIS ♣ 65.65+69.52 70.15 ↑ 2.57 6.63 2.17+3.72+0.74

VLT ♣ +LAVT ♣ 65.65+72.73 73.05 ↑ 3.86 10.2 2.17+7.27+0.74

CRIS ♣ +LAVT ♣ 69.52+72.73 73.87 ↑ 2.75 11.7 3.72+7.27+0.74

CAC � +MAttNet � 63.43+62.62 62.72 ↓ 0.31 4.43 2.11+1.88+0.44

VLT ♣ +MAttNet � 65.65+62.62 69.63 ↑ 5.49 4.64 2.17+1.88+0.59

CRIS ♣ +MAttNet � 69.52+62.62 73.01 ↑ 6.94 6.19 3.72+1.88+0.59

LAVT ♣ +MAttNet � 72.73+62.62 74.33 ↑ 6.66 9.74 7.27+1.88+0.59

Table 3: The performance of different model combinations. For checking the complementary effect between different models, model α and model β are integrated by only GSI. ♣ and � denote bottom-up style methods and top-down style methods, respectively. Inference speed is acquired by counting the inference seconds of 100 samples. The increase value and decrease value are calculated by subtracting (IoUα + IoUβ)/2 from IoU+, i.e., IoU+ - (IoUα + IoUβ)/2.

combination. As shown in Table 3, the experimental results can be split into three parts: bottom-up homogeneous combinations (VLT+CRIS, VLT+LAVT and CRIS+LAVT), top-down homogeneous combinations (MAttNet+CAC), and heterogeneous combinations between bottom-up and top-down methods (VLT+MAttNet, CRIS+MAttNet and LAVT+MAttNet). Three bottom-up homogeneous combinations only improve original models by 2.57%, 3.86%, 2.75% and the top-down homogeneous combination even degrade origin models by 0.31%. However, three heterogeneous combinations consistently improve original models by a clear margin (5.49%, 6.94%, 6.66%). These results indicate that heterogeneous combinations have a stronger complementary effect than homogeneous combinations for boosting performance. In order to quantify the complementary effect, "Mutually Exclusive Rate" (MER) is defined as a metric for analyzing. MER denotes the rate of samples in which only one of the top-down and bottom-up branches outputs positive prediction (IoU > 0.5). In Figure 7, the MER of heterogeneous combinations is significantly higher than homogeneous combinations. These statistics results explain why the performance improvement of heterogeneous combinations is also remarkably higher than homogeneous combinations.

4.3 Qualitative Analysis

The key reason why GSI and SI achieve better performance than straight schemes is that they can roughly estimate the confidence of outputs to adaptively integrate the results. For checking the precision of confidence regression, we plot the correlation between the predicted confidence score and real evaluation score (IoU) in Figure 5. Whether the regression target is the performance score of top-down or bottom-up results, the plot results show that the confidence score predicted by GSI presents a more linear correlation with the evaluation score than SI, which demonstrates the gaussian distributionbased performance modeling of GSI is significant.

Some representative samples of three cases (polar negative cases, inferior positive cases and normal cases) are selected to justify the refinement for PN and IP errors. In Figure 6, first and second rows clearly depict the integration results of WiCo fix the obvious errors of original top-down and bottom-up results, which demonstrates the effectiveness of our method. In Figure 6, third row also shows that our method can adaptively fetch better segmentation results from two branches.

5 Conclusion

Existing top-down and bottom-up methods fail to handle PN and IP errors. Nevertheless, top-down and bottom-up methods can complement each other's flaws for better processing PN and IP errors according to our analysis. To fully exploit the complementary nature, we follow a "Interaction then Integration" paradigm to build WiCo mechanism for achieving a win-win improvement. Specifically, CFI is proposed to let the prior object information of top-down branch and finegrained information of bottom-up branch interact with each other for feature enhancement. GSI is designed to model the performance distributions of two branches for adaptively integrating results of two branches. We select some prominent top-down and bottom-up methods to equip our WiCo for experiments. The experiments consistently show that our WiCo can improve both top-down and bottom-up methods by a clear margin, which demonstrates the effectiveness of our methods.

Acknowledgements

This work was supported in part by the National Key R&D Program of China (No. 2022ZD0118201), the Natural Science Foundation of China (No. 61972217, 32071459, 62176249, 62006133, 62271465), the Natural Science Foundation of Guangdong Province in China (No. 2019B1515120049).