PatternCIR Benchmark and TisCIR: Advancing Zero-Shot Composed Image Retrieval in Remote Sensing

1 Introduction

In recent years, remote sensing(RS) for earth observation (EO) has become a popular research topic. The large remote sensing data obtained provides research material for many computer vision tasks. However, managing and extracting relevant images from remote sensing data has become an urgent challenge. The ability to efficiently and quickly retrieve remote sensing images is crucial.

∗ Corresponding author

Composed image retrieval (CIR) addresses this challenge by enabling the search and retrieval of images from RS image archives. Unlike traditional image retrieval, which relies on single-modality queries, CIR is a more challenging visionlanguage (VL) task that involves paired images and text. The goal is to retrieve the target image by adjusting the existing image through text. Since language is the most natural way to encode human interaction, CIR enhances the retrieval process, allowing users to refine their queries through language and obtain the desired images more effectively.

In natural image CIR, datasets consist of triplets ⟨xqi, xqt, xit⟩, where xqi is the query image, xqt is the query text, and xit is the target image. Many CIR methods [Huang et al., 2023; Yang et al., 2023; Song et al., 2024; Chen et al., 2024; Levy et al., 2024; Wang et al., 2024;

Qi et al., ] rely on these triplets for supervised training. However, obtaining massive triplets for training is difficult and expensive. Some datasets choose to use manual labeling, such as CIRR [Liu et al., 2021] and FashionIQ [Wu et al., 2021], but these datasets offer a limited variety of image classes. Consequently, CIR models trained on them lack generalization capability.

To address these limitations, zero-shot composed image retrieval (ZS-CIR) has emerged as a promising solution. ZSCIR eliminates the reliance on pre-collected triplets, allowing models to perform retrieval without needing manually annotated datasets. Several methods, including projection-based ZS-CIR approaches. [Saito et al., 2023a; Gu et al., 2023; Baldrati et al., 2023a] have demonstrated this approach.As shown in Fig.1, these methods leverage pre-trained models like CLIP [Radford et al., 2021] to train a lightweight projection ϕ that represents the query image xqi as words, and retrieval is then performed by concatenating the projected image representation with the query text xqt according to the template. Although these methods demonstrate better generalization ability, the projection ϕ only captures coarse information from the image, neglecting the remaining fine-grained details. Additionally, only text modality is used during retrieval, which results in suboptimal performance when applied to RS images.

Moreover, in remote sensing composed image retrieval (RSCIR), the only benchmark, Patterncom [Psomas et al., 2024], currently supports retrieval based solely on image attribute. It does not include query text sentences or paired triplets, making it incompatible with most CIR methods designed for fine-grained retrieval in natural image datasets. As a result, many advanced CIR techniques cannot be effectively applied to RSCIR.

To address the problems in existing research, firstly, we propose a model for expanding the attribute words into fulltext query sentences, called the Zero-Shot Query Text Generator (ZS-QTG). ZS-QTG utilizes the keywords-to-sentence language model CBART [He, 2021a] to generate the sentences. After several fine-tunings, ZS-QTG obtains the text query sentence that describes the target image while retaining the attribute words.

Secondly, based on Patterncom, we constructed the triplets ⟨xqi, xqt, xit⟩ following the CIRR benchmark. We named the new benchmark PatternCIR. Unlike Patterncom, since PatternCIR constructs subsets and triplets, the latest CIR methods can be evaluated on PatternCIR. Specifically, Resnet50 is used to extract image features and calculate similarity. Next, image pairs ⟨xqi, xit⟩ are selected according to the criteria of CIRR, and text query statements xqt are obtained using ZSQTG. Finally, the dataset is organized into the CIRR format, enabling CIR methods designed for natural images to be conveniently evaluated on PatternCIR.

Lastly, we propose a ZS-CIR method called Text-image sequential training of Composed Image Retrieval (TisCIR). TisCIR first trains a Multiple Self-Masking Projection (MSMP) module ˆ ϕ using only text queries. The MSMP module ˆ ϕ is designed to extract the conflicting information between query text xqt and query image xqi. Next, images and ˆ ϕ are used to further train a Fine-Grained Image Attention (FGIA) module, enabling it to filter the conflicting information and retain fine-grained information in image embedding. Finally, unlike all ZS-CIR methods, TisCIR can use both text embedding and image embedding simultaneously during the retrieval process. A weighted sum of the distances to the text embedding and the filtered image embedding is computed, serving as the final distance for ranking.

In the experiments, our TisCIR demonstrated excellent performance in RSCIR, surpassing ZS-CIR methods applied to natural images, achieving state-of-the-art performance. Additionally, we validated the effect of the FGIA module by visualizing the image embedding attention. We confirmed that the module can effectively remove the information that needs to be replaced while retaining the fine-grained information in the rest of the image. Our contributions are summarized as follows:

• We propose a model for generating text query, called the Zero-Shot Query Text Generator (ZS-QTG). Using ZSQTG, we constructed PatternCIR, the first fine-grained composed image retrieval benchmark in RSCIR. Compared to the existing benchmark, PatternCIR contains subsets of similar images and triplets, which enable the evaluation of all the latest CIR methods.
• We propose Text-image sequential training of Composed Image Retrieval (TisCIR), a new zero-shot composed image retrieval method for RSCIR. TisCIR uses sequential training of images and text to avoid the use of labeled triplets. Unlike existing ZS-CIR methods, TisCIR allows the simultaneous use of both image and text embeddings during retrieval, utilizing more fine-grained information.
• TisCIR improves upon the latest ZS-CIR methods by 22.95% to 62.03% in RSCIR, achieving state-of-the-art performance. The effectiveness of the TisCIR modules has also been demonstrated and visualized in the ablation study.

2 RelatedWork

2.1 Vision-langauge Model

Vision-language model (VLM) is pre-trained on large-scale datasets containing hundreds of millions of image-caption pairs, such as CLIP [Radford et al., 2021], ALIGN [Li et al., 2021]. These models utilize an image and language encoder pair that can project both images and text to embedding vectors in the same space, thus bridging the modality gap between text and image. Once trained, VLM can be applied to various VL tasks with minimal or no additional annotation cost. As a result, many researchers are using image datasets in different domains to train VLM to enhance their ability to understand images. For example, RemoteCLIP [Liu et al., 2024] is the latest VLM trained with remote sensing data, demonstrating excellent performance in the field of remote sensing images. In this paper, we use VLM in both the generation of query text statements and the CIR task.

2.2 Composed Image Retrieval

Composed Image Retrieval (CIR) is proposed to retrieve the target image xti using a query pair consisting of an image query xqi and a query text xqt [Vo et al., 2019]. This topic has been explored in the field of natural images, such as fashion and scene composition.

Mainstream CIR methods generally fall into two categories: supervised and zero-shot. Supervised methods mainly rely on labeled datasets to train fusion modules for the image and text embeddings extracted by VLM. These methods are heavily dependent on labeled datasets and lack generalization ability.

Zero-shot methods do not require labeled paired triplets during training. Currently, most zero-shot methods train projection ϕ with text or image separately. ϕ projects the embedding of a query image xqi into a token embedding, which is then concatenated with the embedding of query text xqt according to a template, thus ultimately retrieving only in the text modality. Pic2Word [Saito et al., 2023b] trains the projection module ϕ by minimizing the contrastive loss between the image latent embedding and the textual latent embedding of the predefined template "a photo of [$]". SEARLE [Baldrati et al., 2023b] employs a similar approach to Pic2Word. LinCIR [Gu et al., 2023] proposed using only text to train the projection ϕ and mitigates the modality gap between image and text by adding noise.

However, the information extracted by the ϕ in current zero-shot methods depends on the predefined template and mask, which can lead to the loss of fine-grained information in the image during the cross-modal process. In contrast to these methods, our approach can preserve the Fine-Grained information in the image and perform simultaneous retrieval using both the text and image modalities.

2.3 Keywords-to-sentence LM

To generate a fluent, visually-related query text xqt based on image attribute words, we utilize the pre-trained lexically constrained language model, CBART [He, 2021b]. In the use of CBART for image understanding, there has already been some research [Zeng et al., ]. Specifically, CBART is designed to generate a sentence S = (x1, ..., xn) by maximizing the conditional probability, given the ordered set of K keywords

Specifically, CBART consists of an action encoder and a language decoder. In the inference process, the encoder predicts the action (copy, replacement, or insertion) to be taken for each word at the t-th iteration.

Copy means the current word remains unchanged. Replacement suggests the current word should be replaced. Specifically, CBART uses a mask token ⟨M⟩ to replace the current word and sample a new word based on the language model's conditional probability plm. Insertion indicates the decoder should insert a word before the current word. Similar to the replacement action, CBART inserts a ⟨M⟩ token before the current word and then samples a word from plm.

Accordingly, the decoder can refine the sentence from St to St+1. Therefore, the complete encoder-decoder sentence refinement by CBART at the t-th iteration can be formulated as

where Lt is actions predicted by encoder at t-th iteration. In this paper, although CBART can generate the sentence St while preserving attribute words, there is no constraint to ensure that accurately reflects the content of the target image xti. Our ZS-QTG improves upon this and manages to generate accurate query text xqt.

3 Methodology

In this section, we first introduce our model for generating query text, the Zero-Shot Query Text Generator (ZS-QTG), and the overview of the model is shown in Fig.2. Then, we discuss the expansion and improvement of the Patterncom dataset, explaining how subsets and triplets ⟨xqi, xqt, xit⟩ are constructed. Finally, we introduce our proposed new ZS-CIR method, Text-image Sequential Training of Composed Image Retrieval (TisCIR), with the model training and inference process illustrated in Fig.3, Fig.4 and Fig.5. TisCIR incorporates two modules, Multiple Self-Masking Projection (MSMP) and Fine-Grained Image Attention (FGIA), for sequential training, enabling the use of both text and image information during retrieval inference.

3.1 Zero-Shot Query Text Generator

The overview of ZS-QTG is shown in Fig.2. To ensure that the sentence St generated by CBART is highly relevant to the given target image xti, we need visual guidance to select candidate words. We choose to use the contrastive score of the RemoteCLIP model, a VL model in remote sensing, for evaluating the visual-text similarity. By combining conditional probability plm and the contrastive score pit, we can adjust CBART's original word prediction to align with the content of xit.

Specifically, when sampling a word xi at position i, CBART first predicts a conditional probability plm and select top-Kw candidate words {xki }Kw k=1 with the corresponding fluent score, as:

where xk-i denotes words that remain unchanged. Then Kw candidate sentences {sk = (x1, ..., xki , ..., xn)}Kw k=1 are formed by combining candidate word xki with the context xk-i .

Then the the contrastive score is denoted as pit, which can be computed by taking candidate sentences {sk}Kw k=1 and the target image xti as input to calculate the image-text crossmodality similarity as

where Ei and Et denote the image and text encoder of RemoteCLIP. Then, after a weighted sum of Eq.(4) and Eq.(5) we have the final score as

As a result, when sampling i-th word for replacement or insertion in CBART for our model, our ZS-QTG selects the candidate word with the highest score p(xki ).

3.2 Construct PatternCIRDataset

In RSCIR, the existing Patterncom dataset only provides image attributes as labels and does not construct subsets or triplets ⟨xqi, xqt, xit⟩, making it unsuitable for evaluating fine-grained CIR methods. To construct our own PatternCIR dataset based on Patterncom, we first follow the approach used in the CIRR dataset, utilizing ResNet152 to extract image features. Then, we construct image subsets based on similarity, and subsequently, select image pairs from the subsets. Finally, using ZS-QTG, we generate to further construct triplets.

Image subset construction

The nature of the CIR task requires a set of negative images with high visual similarity, on which triplets are constructed. Without this, distinguishing between reference images and target images would be an easy task. Additionally, subset recall Rsubset@K is an important metric for evaluating the performance of CIR methods, so constructing image subsets is a necessary step in building a CIR dataset. Following the approach used in CIRR, for each attribute, we construct multiple subsets of six images that are visually similar, denoted as S = {I1, ...I6}.

Here, to construct a subset within a set of images for a given attribute D, we select one image from the image set randomly, I1 ∈ D. We sort the remaining images in D according to their cosine similarity to I1 using ResNet152 [He et al., 2016] image feature vectors pre-trained on ImageNet [Krizhevsky et al., 2017], which is denoted as ki for image Ii. We then pick five images to construct the subset as follows: First, we remove images with ki ≥ 0.94 to avoid nearidentical images to I1.Then for the rest images, we add each image based on similarity ranking, skipping an image if its similarity is within 0.002 of the last image added. If a subset of size 6 images cannot be created or the number of overlapping images with the existing subset exceeds 4, the entire set is discarded.

Image pairing and annotations

In the constructed subset S, we aim to select image pairs to form triplets as supervisory data. First, we choose a query image and a target image to form an image pair ⟨xqi, xit⟩. To this end, we designate the initial image I1 of each subset S as the query image xqi and sequentially pair it with the remaining 5 images to generate image pairs. Subsequently, we use ZS-QTG to generate the query text xqt for each image pair ⟨xqi, xit⟩, thereby constructing triplets ⟨xqi, xqt, xit⟩.

While generating query text xqi, we choose a fixed initial template as the input to ZS-QTG, which is "A satellite image of, [attribute value], [class name], with, in the background of.". Here, [attribute value] corresponds to the target image xit's attribute values (e.g., white, dense), and [class name] corresponds to its object-of-interest class names (e.g., airplane, tennis court). ZS-QTG expands this template into a full sentence while preserving the initial template words. Phrases separated by commas are treated as a whole, neither split nor having their order disrupted.

Finally, we have obtained a total of 17700 triplets, which are divided into test, validation, and training sets in a ratio of 1.5:1.5:7. The dataset is organized according to the CIRR format, enabling existing CIR methods to be easily evaluated on it. More details of PatternCIR are in the appendix.

3.3 Text-image Sequential Training of Composed Image Retrieval

Existing ZS-CIR methods perform poorly in Remote Sensing Cross-Image Retrieval (RSCIR) when only text embedding is used during retrieval. To address this issue, we propose the Text-image Sequential Training of Composed Image Retrieval (TisCIR). We design TisCIR to utilize both the query text xqt's and query image xqi's embeddings simultaneously during retrieval. However, challenges arise as some information in the two embeddings is conflicting. The query text is mainly used to describe the target image xit rather than the query image xqi, often being adjusted through human interaction. Therefore, we aim to preserve the information in the query text embedding while removing the conflicting information in the query image embedding.

Specifically, the training of TisCIR is divided into two stages. In the first stage, only the query text is used. Considering that text information has a structural nature, we propose a Multiple Self-Masking Projection (MSMP) module to extract its information in conjunction with a masked template. Then, in the second stage of training, only the query image and the trained MSMP are employed to further train the Fine-Grained Image Attention (FGIA) module. This enables the FGIA to filter out conflicting information from the image embedding and focus on fine-grained details. Finally, during retrieval, both the query text embedding and the filtered query image embedding are utilized for retrieval. Since the paired query text and query image ⟨xqt, xqi⟩ are not used during training, TisCIR is considered a zero-shot method.

Training multiple self-masking projection

The original approach [Saito et al., 2023b] involves training a single SMP [Gu et al., 2023] for all multiple masks at different positions, which yielded poor performance. This is because the single SMP might not be able to handle the diverse semantic information from different masks effectively. In particular, when masks are located at different positions, they may carry different semantic nuances that a single SMP fails to capture comprehensively. Therefore, we propose MSMP, where a separate SMP is trained for each mask, illustrated in Fig.3. During training, only the corresponding mask is retained, whereas the other parts of the text remain unchanged. This approach allows each SMP to focus on extracting the semantic information of a single mask, thereby avoiding interference from other parts of the text. As a result, this approach leads to better performance by providing more accurate and focused semantic extraction. Specifically, by isolating the processing of each mask, we can ensure that the semantic extraction is not diluted by irrelevant information, leading to more reliable and useful embeddings. The loss function of MSMP is as follows:

Figure 5: Stage III:Retrieval. FGIA filters the initial image embedding EstageIII I to obtain fine-grained information embedding EstageIII FG . The, the weighted distance between the embeddings of database images EI and them is calculated according to (9) as the basis for retrieval.

We use the MSE loss between EstageI T and EstageI CT to enable MSMP to extract information from the masked parts of the template. To more effectively bridge the modality gap between the image and text modalities, a vector is sampled from a Gaussian distribution, multiplied by a random scalar, and the resulting product is added to the embedding.

Training fine-grained image attention

Building on MSMP, we further train the Fine-Grained Image Attention (FGIA) module, with the training process shown in the Fig.4. The structure of FGIA is a multihead self-attention mechanism applied to embeddings. Specifically, the multihead self-attention mechanism allows FGIA to focus on different aspects of the embeddings simultaneously. In order to enable FGIA to filter out information that conflicts with the text, we employ the following fine-grained loss for constraint:

where EstageII I contains all the information from the initial query image xqi, EstageII FG contains the fine-grained information in xqi and EstageII CT contains the conflicting information between image and text extracted by MSMP. With the FGIA loss, EstageII FG is adjusted to alignwith the difference between EstageII I and EstageII CT . This means that EstageII FG represents the fine-grained information in the image, excluding the conflicting information.

Retrieval

After obtaining FGIA, we can perform retrieval by simultaneously using the query text xqt and query image xqi. The retrieval process is illustrated in the Fig.5. Specifically, during the retrieval process, a weighted sum distance D is calculated. This distance is based on the cosine distances between the embeddings of database images Ei and both the text embedding EstageIII T and the fine-grained information embedding EstageIII FG . The formula is as follows:

where λ is the retrieval distance weight.

4 Experiments

In this section, we discuss the evaluation of TisCIR on the PatternCIR dataset and compare it with the latest ZS-CIR

methods in natural image tasks. Subsequently, We conduct ablation experiments on the MSMP and FGIA modules to verify the effectiveness of our proposed improvements. Finally, to further demonstrate the effectiveness of the FGIA module, we visualize the similarity among the embeddings before FGIA filtering, the embeddings after FGIA filtering, and the embedding of the original image in Fig.6.

4.1 Implementation Details

Settings

For the MLP in MSMP, we use the same settings as in [Gu et al., 2023]: LN [Ba et al., 2016]-Linear-GeLU [Hendrycks and Gimpel, 2016]-Linear-GeLU-Linear-LN. We use the AdamW optimizer [Loshchilov and Hutter, 2017] with a fixed learning rate of 0.0001, a weight decay of 0.01, and a batch size of 256 for the first stage of training, whereas a batch size of 32 is used for the second stage. Dropout with a probability of 50% is applied for regularization. During the first stage of training, we use all the query texts from the PatternCIR training set as training texts, and during the second stage, we employ all the images from the training set as training images.

Evaluation metrics

In CIR, recall rate R@K represents the probability of retrieving the target image among the top K images. Among which we select R@1 as the reference for choosing the optimal training weights on the validation set. Additionally, the subset recall rate Rsubset@K refers to the probability of retrieving the target image within the top K images of a subset. Compared to R@K, Rsubset@K reduces the impact of false negatives, thereby making it an important reference metric in the CIR field.

Comparison methods

We compare TisCIR with the latest ZS-CIR methods, namely Pic2Word [Saito et al., 2023b], LinCIR [Gu et al., 2023], SEARLE [Baldrati et al., 2023b], WEICOM[Psomas et al., 2024], FTI4CIR[Lin et al., 2024], and LDRE[Yang et al., 2024]. All methods employ the ViT-L CLIP backbone for training, except for SEARLE, which uses the ViT-B backbone. Therefore, we also compare TisCIR with its variant SEARLE-XL, which uses the ViT-L backbone.

4.2 Main Results

The main experimental results are shown in Tab.1, from which it can be seen that TisCIR outperforms all ZS-CIR methods in both the recall rate R@K and the subset recall rate Rsubset@K. Among the methods using ViT-L backbone, TisCIR improves R@1 by 22.95% to 62.03% and Rsubset@1 by 13.56% to 28.29%, thereby achieving state-of-the-art performance on RSCIR. SEARLE, which uses the ViT-B backbone, exhibits poor performance, indicating that the choice of backbone has a significant impact on RSCIR. Additionally, LinCIR, which only uses text for training, performs the worst among all CIR methods using ViT-L backbone, suggesting that image information is more significant in RSCIR.

4.3 Analysis

In this section, we discuss the roles of our two improvement modules, MSMP and FGIA. We first conduct ablation experiments on both MSMP and FGIA. Subsequently, we visualize the patch similarity map of the embeddings before and after FGIA filtering, which intuitively demonstrates the effect of FGIA.

Ablation study

The results of the ablation experiments are presented in the Tab.2. We separately validate the effectiveness of the MSMP and FGIA modules. The method utilizing both MSMP and FGIA performs the best. This indicates the effectiveness of our improvements over the original method.

FGIA visualization

Visualizing CLIP image embeddings has always been a challenge due to the absence of an inverse mapping capable of mapping the image embedding back to the image. However, we can still visualize the information within the image embedding by employing certain methods, and the similarity map is the approach we chose. When CLIP extracts image embeddings, it first divides the image into patches based on the the model setup, thereby obtaining multiple patch embeddings. Subsequently, it uses a linear layer to obtain the global embeddings as the final image embedding. In this paper, we calculate the cosine similarity among the image embeddings before and after filtering by FGIA and the original image patch embeddings. We then construct a similarity map based on the patch positions and overlay it onto the original image, enabling us to observe the parts of the image on which each image embedding focuses.

The final visualization results are shown in the Fig.6. We can see that embeddings before being filtered by FGIA focuses more on the main object in the image, which conflicts with the query text. After filtering, the embeddings remove this information and focus more on the fine-grained details in the background. This visually demonstrates that our FGIA has achieved the intended design effect.

5 Conclusion

In this paper, we propose a novel model for generating CIR query text statements, namely Zero-Shot Query Text Generator (ZS-QTG) , and use ZS-QTG to construct the RSCIR benchmark dataset PatternCIR, which encompasses subsets and triplets of similar images. PatternCIR enables the evaluation of most existing CIR methods. Finally, we introduce a novel ZS-CIR method, Sequential Training of Composed Image Retrieval (TisCIR), which is distinct from existing ZSCIR methods. During training, TisCIR leverages both image and text sequences to train the MSMP and FGIA modules, thereby enabling the concurrent utilization of both modalities during retrieval. TisCIR outperforms all state-of-the-art ZSCIR methods in the RSCIR task, achieving state-of-the-art performance.