Enhancing Semantic Clarity: Discriminative and Fine-grained Information Mining for Remote Sensing Image-Text Retrieval

1 Introduction

Remote Sensing Image-Text Retrieval (RSITR) aims to retrieve texts or images with high semantic relevance to a given image or text from massive remote sensing databases collected by satellites or aerial drones. In recent years, there has been a growing interest in RSITR [Yuan et al., 2022b; Liu et al., 2024a] due to its important applications in disaster monitoring [Joyce et al., 2009], natural resource exploration and remote sensing image captioning [Yang et al. ]. This task is of great significance yet highly challenging, as it requires precise visual-linguistic alignment.

∗ Corresponding author.

Figure 1: Causes of semantic confusion: (a) Redundant visual representations, and (b) high inter-class similarity.

Towards this end, many RSITR models have emerged [Pan et al., 2023; Yang et al., 2024b; Ma et al., 2024]. Early works [Mao et al., 2018; Lv et al., 2021] achieve modal alignment by directly mapping different modalities to a semantic space or using cross-modal interaction. To optimize representation, [Yuan et al., 2022a] proposes a multi-scale visual self-attention module to filter visual redundant features. [Mi et al., 2022] improves text representation through a knowledge graph-based textual enhancement method. [Yuan et al., 2022b] designs a module that dynamically fuses global and local visual features to better understand the relationships between different visual objects. The above studies have contributed prominently to RSITR. However, due to the performance limitations of feature extractors, the retrieval effect is often unsatisfactory. Recently, visionlanguage pre-training (VLP) models have achieved significant success in the field of multimodal analysis. Inspired by this, several studies [Liu et al., 2024a; Zhang et al., 2024; Wang et al., 2024] have successfully applied the Contrastive Language Image Pre-training (CLIP) model [Radford et al., 2021] to the RSITR task. [Liu et al., 2024a] first proposes a vision-language model for remote sensing. Both [Zhang et al., 2024] and [Wang et al., 2024] promote the performance improvement of remote sensing vision-language models (RSVLMs) by collecting large and high-quality remote sensing image-text datasets.

Although promising, RSVLMs focus primarily on broad application scenarios and overlook the unique semantic confusion challenges that RSITR encounters. Specifically, current methods face semantic confusion for two key reasons: 1) redundant visual representations, and 2) high inter-class similarity. Redundant visual representations mean that in remote sensing images, the proportion of the foreground is often small, making its semantic representation easily disturbed by retrieval-irrelevant areas. As depicted in Fig.1 (a), retrieval-relevant areas (i.e., aircraft and tarmac) are often affected by the surrounding irrelevant objects (e.g., buildings and grass). High inter-class similarity refers to the obvious similarity of image-text pairs with different scenes, which undermines the accuracy of multimodal semantic representation. As shown in Fig.1 (b), although the remote sensing images belong to different scenes (i.e., school, playground, commercial area, and dense residential), their visual contents are similar, and the corresponding descriptions (e.g., buildings) are apparently similar as well. These insights prompt a pivotal research question that motivates this study: "How can we reduce visual semantic redundancy and increase inter-class representation distance to enhance semantic clarity?"

To answer this, we present a novel Discriminative and Fine-grained Information Mining (DFIM) model, which consists of two crucial modules: 1) Dynamic Visual Enhancement (DVE), and 2) Fine-grained Semantic Matching (FSM). Concretely, the DVE module first utilizes a multimodal interaction guidance strategy to integrate semantic information from different modalities, offering precise direction for enhancing discriminative features. It then adopts a dynamic fusion strategy to flexibly refine these features, thereby effectively suppressing visual redundancy. The FSM module cleverly models the fine-grained alignment between image regions and text words as an optimal transport problem [Liu et al., 2020]. By striving to minimize the transport cost between the distributions of regions and words, it can accurately distinguish subtle differences between similar pairs. Finally, we integrate the advantages of DVE and FSM to alleviate semantic redundancy and improve overall performance. The comprehensive experimental results show that our method has a significant advantage over the current state-of-the-art methods, while also possessing good visual interpretability. The main contributions of this study are summarized as follows:

• We propose a new remote sensing image-text retrieval model, DFIM, which can effectively solve the semantic confusion problem and significantly strengthen the connection between vision and language.
• To reduce visual redundancy, DVE adaptively enhances the visual discriminative features by combining multimodal interaction guidance with dynamic fusion strategies.
• To alleviate inter-class semantic confusion, FSM employs optimal transport learning strategy to identify fine-grained correspondences between image regions and text words.
• We justify the superiority of DFIM on two popular benchmark datasets (i.e., RSICD and RSITMD) with extensive experiments, where our design outperforms the state-of-theart models. Moreover, our method exhibits excellent visual interpretability.

2 RelatedWork

Remote Sensing Image-Text Retrieval (RSITR) aims to ensure semantic consistency between remote sensing images and text descriptions. Recently, RSITR has become a research hotspot. According to modal interaction methods [Pan et al., 2023] before entering the latent semantic space, RSITR methods can be roughly classified into two categories: intramodal interactionmethods, and inter-modal interactionmethods. The former [Yuan et al., 2022b; Zhang et al., 2022] performs information interactions only for the same modality, while the latter [Lv et al., 2021; Yuan et al., 2022a] performs information interactions for different modalities. Although existing methods advance through enhanced unimodal features or cross-modal interactions, their effectiveness remains constrained by limited feature extractor capabilities. Recent studies [Liu et al., 2024a; Pan et al., 2025; Yang et al., 2024c; Liu et al., 2024b]have shown that the contrastive language image pre-training (CLIP) model has been successfully applied to various tasks including RSITR. Instead, we focus on improving model retrieval accuracy while enhancing visual interpretability.

Token Pruning has become a hot research topic [Wei et al., 2023; Cao et al., 2024], aiming to dynamically reduce less important tokens based on input-dependent importance. Researchers have proposed various methods [Bolya et al., 2022; Wei et al., 2023] to remove redundant tokens in ViT [Alexey, 2020] to improve the model's computational efficiency. In addition, some studies [Chen et al., 2025; Cao et al., 2024] have reduced the computational cost of vision-language models through pruning techniques. Although these methods effectively reduce computational overhead, they may result in a certain degree of accuracy loss. On the contrary, we reduce the redundancy of visual information via token pruning, thereby extracting discriminative information and further enhancing the task-relevant representation.

Optimal Transport Strategy (OT) is initially developed to quantify the distance between two probability distributions. It is frequently used to establish correspondences between learnable features or to measure the distances between distributions. OT has been applied to various fields such as domain adaptation [Courty et al., 2016] and person re-identification [Wang et al., 2022]. [Cuturi, 2013] proposes a lightning-fast approach that efficiently tackles large-scale problems by using the Sinkhorn algorithm. Unbalanced Optimal Transport [Chapel et al., 2021] reformulates the corresponding optimization problem as a non-negative penalized linear regression problem. In contrast, we apply OT to minimize the transport cost between image patches and word embeddings, effectively capturing fine-grained semantic alignment.

3 Methodology

3.1 ProblemOverview

Given a remote sensing image-text dataset D, RSITR task is to learn the cross-modal similarity for retrieval. Formally, D consists of K image-text pairs, denoted as D = {vi, ti}Ki=1 . Drawing inspiration from the partial success of transferring knowledge from CLIP [Radford et al., 2021] to RSITR [Liu et al., 2024a; Zhang et al., 2024], we directly initialize our DFIM with the complete CLIP image and text encoders, thereby bolstering its inherent cross-modal alignment capabilities. In detail, given an image-text pair (vi, ti), the visual encoder performs patch embedding to the image vi to generate the visual representation Vi = {vcls i , v1 i , ..., vN i } ∈ R(N+1)×d , which contains global visual feature vcls i ∈ R1×d and visual patch features vloc i = {v1 i , ..., vN i } ∈ RN×d . The language encoder processes words in text ti using token embedding, converting them into a textual representation Ti = {tsos i , t1i , ...teos i } ∈ R(M+2)×d , which consists of global textual feature teos i ∈ R1×d and token features tloc i = {t1i , ..., tMi } ∈ RM×d . Finally, we compute cosine similarity as a similarity score between the output vectors of the two modalities. Considering the above introduction of the RSITR task, it is intuitive that the alignment of the two modalities determines the model's performance. However, as illustrated in Fig.1, learning an effective feature extraction network is challenging due to the semantic confusion problem. Therefore, our primary research questions are how to accurately identify visual discriminative features and how to distinguish similar pairs by learning fine-grained matching. The pipeline of our Discriminative and Fine-grained Information Mining (DFIM) method is shown in Fig.2.

3.2 Our ProposedMethod: DFIM

Our DFIM method consists of two key components: Dynamic Visual Enhancement and Fine-grained Semantic Matching, which will be elaborated in Sec. 3.2 and 3.2.

Dynamic Visual Enhancement

As discussed in Sec.1, redundant visual representations exacerbate the semantic confusion problem. To address this challenge, we propose a Dynamic Visual Enhancement (DVE) module, which dynamically enhances discriminative features to promote multimodal alignment from a global perspective. There are two important strategies: multimodal interaction guidance, and adaptive fusion enhancement.

Multimodal Interaction Guidance (MIG). To ensure that the retained tokens are essential for both modalities, we first obtain guidance information via the MIG strategy. The core of this strategy is to establish the correlation between visual and textual modalities by using learnable interaction prompts p = {p1, p2, ...pl} ∈ Rl×d , where l is the length of tokens. Specifically, we use an attention layer to obtain visual token attention weights Aw ∈ Rl×N guided by multimodal interaction information. The detailed calculation process is as:

where dk represents a scaling factor. Afterward, the visual token attention maps Aw are fed into the adaptive fusion enhancement strategy to guide the fusion process of the visual features, ensuring that the enhanced features are meaningful in both modalities, which is exemplified in Fig.2.

Adaptive Fusion Enhancement (AFE). The dynamic token pruning methods [Chen et al., 2025; Cao et al., 2024] have been demonstrated to be more effective than static token pruning because they can adaptively adjust the model's compression rate in accordance with the complexity of the input instances. Inspired by this, we propose the AFE strategy to dynamically strengthen discriminative visual features. As depicted in Fig.2, we first compute the discriminant feature score for each token. Then, a learnable threshold is used to dynamically filter the tokens based on the complexity of the instances. Finally, we introduce feature fusion operations to further enhance the visual representation. Detailed description as follows:

(1) Discriminant Feature Score (DFS). To effectively avoid discarding key tokens, our approach not only considers token importance based on class attention map [Liu et al., 2022; Cao et al., 2024], but also extends to the importance of tokens across different modalities. The DFS is obtained by averaging two types of scores:

Here, Scls is the class attention score as implemented by [Liu et al., 2022]. Stoken denotes the token attention score. We use the visual token attention weights Aw ∈ Rl×N from the MIG strategy to calculate the Stoken as follows:

where N is the length of tokens and max(Anw ) is the maximum value for the n-th token in the attention weights Aw. (2) Token Filtering. To implement instance-wise adaptive token filtering, we first compute learnable thresholds θ using the token attention weight Aw learned from the MIG strategy. The computation of θ is defined as follows:

where ˆ Aw represents the sparse token attention maps obtained by applying the sparsemax function [Martins and Astudillo, 2016] on Aw. Based on the above DFS and threshold θ, we perform discriminative tokens filtering. Specifically, we compare the DFS of each token with θ to obtain the filtering mask Mm, which is expressed as follows:

Here, vn i is the n-th token in the visual patch features vloc i . Retain primary tokens vimp i with scores above the threshold θ and exclude the rest according to the filter mask Mm.

(3) Feature Fusion. To prevent the loss of information caused by directly discarding tokens, we first aggregate the primary tokens vimp i to obtain a new global feature ˆ vcls i , which contains more discriminative information than the original global feature vcls i . In detail, we employ an embedding transformation [Qin et al., 2024] to the updated sequence of important tokens. Subsequently, we fuse ˆ vcls i with vcls i in a specific ratio. The process is defined as follows:

where mlp(·) refers to a multi-layer perceptron, f(·) represents a linear layer. λ is the balance coefficient. So far, we obtain a non-redundant global visual representation ˆ vi that contains both contextual information and enhanced discriminative properties.

Fine-grained Semantic Matching

While vcls i and teos i provide a comprehensive understanding of global alignment relationships, due to high inter-class similarity, it may lack some fine-grained matching information to distinguish subtle differences between similar objects or scenarios. Although existing methods [Yuan et al., 2022a; Zhou et al., 2024] have recognized the significance of extracting fine-grained information, most of them focus exclusively on visual fine-grained details, neglecting multimodal finegrained matching, which is of vital importance to RSITR. To mitigate this issue, we propose a Fine-grained Semantic Matching (FSM) module to enhance multimodal fine-grained awareness. Specifically, for the given N suppliers (image patches) and M demanders (words). The supplier supplies image patches to the demander, described as a vector x, and the demander receives image patches from the supplier, described as a vector y. We formulate the fine-grained matching task as an optimal transport problem [Liu et al., 2020], which is to find an optimal transportation plan P ∈ RN×M to minimize the transport cost C. It can be expressed as:

where ⟨P,C⟩F is the Frobenius inner product between the transportation (matching) plan P and cost matrix C. The matrix c(vn i , tmi ) is an element of C, which denotes the transport costs between vn i and tmi . It can be expressed as c(vn i , tmi ) = 1 - cos(vn i , tmi ). In addition, we add entropy regularization on P as H(P) = ∑nm Pnm logPnm to ensure that P is not over-concentrated on a few elements. Ultimately, the solution of Eq. 8 can limit a transportation polytope:

where Pnm represents the transport plan between the n-th image patch and the m-th word, and P contains all non-negative N×M elements, with row and column sums equal to x and y, respectively. We employ the inexact proximal point method for optimal transport (IPOT) [Xie et al., 2020] to approximate the transport plan, which is formulated as follows:

where µ and v are row and column normalized vectors, respectively, and can be computed via the iterative SinkhornKnopp algorithm [Cuturi, 2013]. To maximize inter-class distinctions and learn fine-grained matching, each image patch should be closely aligned with the corresponding text word. Thus, we define the patch-to-word alignment loss Lp2w as:

Overall, the FSM module reduces the inter-class similarity by explicitly establishing fine-grained alignment between patches and words, thereby effectively alleviating the semantic confusion problem.

Learning Strategies

Our training objective consists of two parts: 1) We leverage the bi-directional similarity distribution matching loss [Jiang and Ye, 2023] LSDM to supervise the learning of the global representations across different modalities:

where pi,j is the probability of matching pairs, and qi,j = yi,j/ ∑Kk=1 yi,k is the true matching probability. 2) We also adopt the proposed Lp2w to supervise the fine-grained matching as described in Sec. 3.2. The total loss is given by:

where α and β denote the balance coefficient.

4 Experiments

This section conducts experiments to answer the following questions. RQ1: How effective is DFIM in improving the model performance across different settings? RQ2: How do the Dynamic Visual Enhancement (DVE) module and Finegrained Semantic Matching (FSM) module contribute to the performance? RQ3: What are the learning patterns and insights of DFIM?

4.1 Settings

Datasets. We evaluate our method on the RSICD and RSITMD datasets. RSICD [Lu et al., 2017] consists of a total of 10,921 images, each image is associated with five text descriptions. Following [Yuan et al., 2022a], we divide the dataset into 7,862 training images, 1,966 validation images, and 1,093 test images. RSITMD [Yuan et al., 2022a] includes 4,743 images, with each image annotated by five sentences offering a finer-grained description than RSICD. Utilizing the same division strategy as [Yuan et al., 2022b], we split the dataset into 3,435 training images, 856 validation images, and 452 test images. Additionally, to investigate the model's performance on both significant and insignificant sample pairs, we divide the RSICD and RSITMD test sets according to [Ma et al., 2024]. The distinction between significant and insignificant images is determined by whether they contain prominent objects.

Evaluation Protocol. Following standard practices in imagetext retrieval, we evaluate the performance of DFIM by R@K (K = 1, 5, 10) and mR. R@K denotes the proportion of accurately matched pairs within the top K retrieval outcomes, while mR represents the mean of these R@K values.

Implementation Details. All experiments are conducted in a station equipped with RTX4090 24GB GPU. We use CLIP (ViT-B-32) [Radford et al., 2021] as the backbone, and employ ViT-B-32-RET-2 [Zhang et al., 2024] for parameter initialization. All input images are resized to 224 × 224 and then randomly cropped and rotated to enhance the training samples. The maximum length of the textual token sequence is set to 77. We set the epoch to 20 and the batchsize to 128 on both datasets. We use Adam [Kingma, 2014] as the model optimizer. A cosine learning rate decay is applied, starting with an initial learning rate of 1 × 10-4 for RSICD and 1 × 10-5

for RSITMD. The hyper-parameters λ, α and β are set to 0.5, 1.0 and 0.7 for RSICD, respectively. The hyper-parameters λ, α and β are set to 0.6, 0.8 and 1.1 for RSITMD, respectively.

4.2 Quantitative Comparison (RQ1)

In this section, we comprehensively evaluate our DFIM on two widely-used RSITR datasets (i.e., RSICD [Lu et al., 2017] and RSITMD [Yuan et al., 2022a]). According to different training paradigms, the methods for comparison with DFIM can be roughly divided into two categories: 1) Traditional cross-modal retrieval methods, including AMFMN [Yuan et al., 2022a], GaLR [Yuan et al., 2022b], SWAN [Pan et al., 2023], HVSA[Zhang et al., 2023] and DOVE [Ma et al., 2024]. 2) CLIP-based methods, such as AdaptFormer [Chen et al., 2022], PE-RSITR [Yuan et al., 2023], GeoRSCLIP [Zhang et al., 2024] and HarMA [Huang, 2024]. Results on RSICD Dataset. According to the comparison results on RSICD reported in Tab.1, we can find that our DFIM shows a significant increase compared to the SOTA approach HarMA. For example, R@1 improves 14.1% (23.42 vs. 20.52) and 11.7% (17.70 vs. 15.84) in sentence and image retrieval, respectively. In general, there is a 10.7% (43.14 vs. 38.95) improvement on the mR metric. Thus, our method outperforms traditional and CLIP-based retrieval methods on the RSICD dataset, reflecting its superior performance in resolving semantic confusion problem.

Results on RSITMD Dataset. Tab.1 presents the results on RSITMD. It is evident that DFIM has a high performance advantage, as it achieves considerable improvements across all metrics relative to state-of-the-art methods. Specifically, compared to HarMA, the R@1 of image-query-text is improved by 6.8% (34.97 vs. 32.74), and mR is improved by 5.2% (55.01 vs. 52.27). The above results demonstrate the effectiveness of DFIM in reducing the semantic confusion and the superiority of the scene perception capability.

Results on significant and insignificant test sets. To further explore the capability of DFIM in matching significance and insignificance objects, we conduct controlled experiments: 1) the complete test set; 2) the significant test set; and 3) the insignificant test set. The difference between significant and insignificant images depends on whether they contain salient objects or not. Fig.3 displays results on the complete, significant and insignificant test sets of the RSICD and RSITMD datasets. DFIM has achieved substantial improvements over traditional methods (i.e., SWAN and DOVE). Besides, we implement a baseline approach CLIP*, which uses ViT-B32-RET-2 [Zhang et al., 2024] for parameter initialization. Our method achieves the best performance on the mR metric for all three experimental setups. That is, our method significantly improves the retrieval effect by reducing visual redundancy and mining fine-grained alignment information, regardless of whether the object is salient or not.

4.3 In-depth Studies of DFIM (RQ2)

Contributions of the DFIM's components. To fully understand the DFIM, we explore the effectiveness of the DVE and FSM modules on the RSITMD dataset. The corresponding performances are reported in Tab.2. Effectiveness of DVE. Observing w/o FSM, we can find a 3.0% (53.36 vs. 55.01)

performance drop in the mR metric. This shows DVE can substantially improve retrieval ability. Effectiveness of FSM. Without using FSM (i.e., w/o FSM), it can be found that a 4.1% (52.73 vs. 55.01) performance drop in the mR metric. This indicates that FSM can significantly improve the finegrained matching performance. Overall, the results show that the combination of DVE and FSM can effectively mitigate the semantic confusion.

Method Sentence Retrieval Image Retrieval

R@1 / R@5 / R@10 R@1 / R@5 / R@10 mR

baseline 31.24 / 53.68 / 68.31 25.36 / 57.04 / 73.61 51.54

w/o DVE 33.27 / 55.19 / 70.06 27.38 / 58.09 / 76.15 53.36

w/o FSM 33.01 / 54.43 / 69.58 26.23 / 57.96 / 75.17 52.73

DFIM 34.97 / 57.36 / 71.25 28.72 / 59.83 / 77.92 55.01

Analysis of Hyper-parameters. The sensitivity analysis of

hyper-parameters λ, α and β is shown in Fig.4. λ represents the intensity of visual feature enhancement. α and β balance the contributions of global and local matching, respectively.

Over a wide range of hyper-parameters, our model shows

very small fluctuations. In general, variations in the hyper-

parameters do not lead to significant degradation in perfor-

mance, which demonstrates the stability of our DFIM.

4.4 Qualitative Analysis (RQ3)

To capture the learning insights of DFIM, we perform a series of detailed visual analyses. (1) Visual Token Selection Visualization. In Fig.6, we display the visualization results of token filtering on the RSICD dataset. The figure consists of textual descriptions, the original images, and the corresponding processed outputs, where the white mask indicates redundant patches. It can be observed that based on the textual descriptions, our DFIM retains key tokens and prunes irrelevant ones. For instance, DFIM preserves the "baseball field" in the second row of the example shown in Fig.6. Besides, we observe that for different scenes (i.e., airport, baseball field, and church), DFIM can adaptively distinguish between primary and redundant tokens based on the size of different targets. These visualization results highlight DFIM's capacity to accredit diverse scenes and also validate its interpretability in dealing with visual redundant representations. (2) Region-Word Alignment Visualization. In order to further evaluate the DFIM's capacity to capture fine-grained relationships, we leverage gradient-weighted attention [Chefer et al., 2021] to generate heat maps, where colors closer to red indicate more semantic relevance. As shown in Fig.5 (a), "trees", "buildings", and "pool" are accurately mapped to their corresponding image regions. It is worth noting that, although the "pool" area is very small, DFIM is still able to align the region and word accurately. In addition, there are multiple dispersed small targets (i.e., vehicles) in Fig.5 (b), we observe that DFIM can accurately match the word "vehicles" with the dispersed targets in the image. Meanwhile, DFIM can accurately identify the entire region associated with color-related words (e.g., "red" and "green"), rather than merely the target object that exhibits that color. As shown in Fig.5 (c), the region corresponding to "green" encompasses not only areas associated with green plants but also all green regions in the image. This indicates that DFIM can achieve fine-grained matching and resist overfitting.

5 Conclusion

In this paper, we propose a novel Discriminative and Finegrained Information Mining (DFIM) approach to alleviate semantic confusion in the RSITR task. We first analyze two causes of semantic confusion: visual representation redundancy and excessive inter-class similarity. To address these issues, we design a Dynamic Visual Enhancement (DVE) module and a Fine-grained Semantic Matching (FSM) module, respectively. Specifically, the DVE module can adaptively strengthen the visual discriminative features that are critical to both modalities. Meanwhile, the FSM module cleverly formulates the fine-grained matching relationship between image regions and text words as an optimal transport problem, amplifying inter-class distinctions. Comprehensive experiments validate DFIM's effectiveness in both retrieval accuracy and visual interpretability. In future work, we plan to extend DFIM to more tasks [Yang et al., 2023; Wu et al., 2025; Xu et al., 2025; Yang et al., 2021] and assess its effectiveness in diverse scenarios.

Acknowledgments

This research is supported by the National Natural Science Foundation of China (62276112, 62272435, and U22A2094).