Retrieval-Guided Completion Hashing with Token–Patch Alignment for Incomplete Cross-Modal Retrieval

1 Introduction

With the explosive growth of intelligent devices and social networks, efficient information retrieval [Jiang and Li, 2017] has become a critical topic in both academia and industry. As a promising similarity retrieval technique, crossmodal retrieval (CMR) [Zhen et al., 2019; Shen et al., 2020; Hu et al., 2023; Wang et al., 2015; He et al., 2022] aims to retrieve semantically relevant instances from one modality when given a query from another modality [Zhang et al.,

∗Corresponding author.

2022b]. Among them, hashing (CMH) [Tu et al., 2022; Zhang et al., 2022a; Bai et al., 2023; Li et al., 2024] is considered an efficient mechanism for CMR, which learns a set of hash functions to map raw data into compact binary codes, thereby enabling efficient retrieval. The recent advances in deep CMH [Jiang and Li, 2017; Tu et al., 2022; Li et al., 2018; Shu et al., 2024] have attracted widespread attention. Compared with shallow CMH [Wang et al., 2020], deep CMH achieves superior retrieval performance, owing to its powerful representation capability derived from end-toend training and nonlinear hash functions.

However, existing CMH methods generally assume that the multimodal data are complete without missing modalities. However, in practical applications, multimodal data often suffer from missing modalities due to device limitations, privacy concerns, and high annotation costs [Lang et al., 2025]. The information gap caused by missing modalities disrupts the correlations between cross-modal features, thereby affecting the discriminability of hash codes. Most existing incomplete CMH methods rely on the available modality to recover features of the missing modality. [Peng et al., 2025]. These methods rely on static data distribution assumptions and often ignore the structure of the retrieval sample space during completion. They neglect prior knowledge from globally similar samples, resulting in significant distribution deviations and inconsistencies between the completed features and retrieval objectives. In addition, images contain rich and continuous information, whereas texts are inherently sparse and abstract in expression [Zareapoor et al., 2025]. This asymmetry in information density between modalities makes it difficult for strategies relying solely on global feature alignment to establish sufficient and stable cross-modal semantic correlations.

To address these challenges, this paper proposes a novel completion hashing method, called retrieval-guided completion hashing with token-patch alignment (RGCH-TPA), for cross-modal retrieval. In the proposed model, we employ cross-instance retrieval as dynamic structural guidance for modality completion. It effectively exploits retrieval results as reliable expert signals to guide the recovery of missing modality information. By dynamically maintaining a feature memory bank and jointly optimizing completion, the proposed framework establishes a closed-loop mechanism in which retrieval feedback continuously refines the completion process. Meanwhile, to mitigate cross-modal misalignment caused by asymmetric information density, we introduce a fine-grained token-to-patch alignment strategy that enables bidirectional local semantic interaction between text and image modalities. By jointly modeling retrieval-guided completion and local cross-modal alignment, the two components form a mutually reinforcing process: the completion module supplies reliable feature foundations, while the alignment module provides precise semantic correspondences for robust hash learning. This collaborative design significantly enhances retrieval performance under incomplete-modality scenarios.

The main contributions of this work are summarized as follows:

• We propose a novel retrieval-guided modality completion strategy that leverages cross-instance retrieval results as expert priors to recover missing modalities. Instead of treating completion as an isolated feature generation, it formulates modality recovery as a collaborative optimization process driven by retrieval objectives.
• We design a token-to-patch alignment and enhancement module that explicitly models the semantic correspondence between text tokens and image patches. Therefore, it effectively alleviates insufficient cross-modal alignment caused by information asymmetry.
• Extensive experiments on three benchmark datasets show that our RGCH-TPA method consistently outperforms other state-of-the-art approaches, demonstrating its superiority in incomplete CMH tasks.

2 RelatedWork

2.1 Complete Cross-modal Retrieval

Cross-modal hashing (CMH) seeks to learn a set of hash functions that project heterogeneous multimodal data into a common Hamming space to generate fixed-length hash codes. Existing CMH methods are typically grouped into unsupervised and supervised paradigms. Unsupervised CMH methods preserve semantic proximity by exploiting intermodal correlations without manual annotations. For instance, DJSRH [Su et al., 2019] and DGCPN [Yu et al., 2021] model inter-instance semantic neighborhoods, thereby providing structural priors for discriminative binary code learning. In contrast, UCMFH [Xia et al., 2023] uses frozen CLIP encoders and contrastive learning to jointly supervise multimodal fusion and binary code generation. Unlike methods that rely solely on contrastive learning, DDSS [Huang et al., 2025] further incorporates dual-branch representation learning and dual-path instance hashing to explicitly capture intramodal topology and inter-modal semantic consistency. To enhance retrieval performance, several supervised CMH methods leverage manually annotated labels as explicit semantic priors to guide hash function learning. A representative CNN-based method, DCMH [Jiang and Li, 2017], utilizes the strong nonlinear representation power of CNNs along with label supervision to learn highly separable binary hash codes. With the emergence of large-scale vision-language models (i.e., CLIP), Transformer-based hashing methods have shown superior multimodal contextual modeling capabilities. For example, DCMHT [Tu et al., 2022] adopts ViTs to obtain more expressive visual embeddings and enables end-to-end differentiable hashing, thereby improving fine-grained crossmodal representation learning. In addition, DNPH [Huo et al., 2024] incorporates learnable semantic class proxies and alignment constraints to improve class-wise compactness and cross-modal semantic separation in the binary space. However, hard label-induced decision margins may conflict with cross-modal semantic similarity distributions due to modality heterogeneity. Therefore, DScPH [Qin et al., 2025] models asymmetric intra- and inter-class correlations and introduces a semantic-consistent penalization loss, improving its retrieval results.

2.2 Incomplete Cross-modal Retrieval

Cross-modal retrieval remains highly challenging when multimodal data is incomplete. In early studies, DAVAE [Jing et al., 2020] learns aligned latent spaces via modalityspecific VAEs and generates missing-modality latent embeddings through VAE-based synthesis, thereby bypassing explicit retrieval-space modeling. Additionally, MCCN [Zeng et al., 2021a] and PAN [Zeng et al., 2021b] facilitate missing-modality learning by constructing category prototypes. In hashing-based retrieval, early methods, such as CYC-DGH [Wu et al., 2018], employ adversarial training to generate missing-modality features in a cycle-consistent manner. Subsequently, SICH [Peng et al., 2025] introduces an autoencoder as a data recovery network to recover missing-modality features using information entropy theory. This is an important attempt for unsupervised incomplete hashing, but the lack of label supervision limits its retrieval performance. CICH [Luo et al., 2024] reconstructs global similarity through label supervision and contrastive learning, and further aligns contextual correspondences to handle missing modalities and mismatched pairs. Beyond cross-modal hashing, several works also explore robust multimodal hashing in incomplete data settings. For example, GCIMH [Shen et al., 2023] employs a teacher-student framework, in which a GCN-based autoencoder encodes incomplete multimodal structural semantics as reconstruction priors. These priors are then distilled into a student hashing network to learn semantically separable binary hash codes.

3 Methodology

3.1 Notations and ProblemDefinition

We aim to project high-dimensional image and text features into a shared K-bit discrete Hamming space, and learn unified hash codes in supervised incomplete-modality scenarios, where K denotes the hash code length. We assume that the incomplete multimodal dataset O = {Ovt , Ov , Ot } containing n instances. Ovt = {(V, T,L)}nf denotes the complete image-text pairs with semantic labels L, where nf is the number of complete image-text pairs. Ov = {(V, L)}nv denotes image-only instances, where nv is the number of image-only samples. Similarly, Ot = {(T, L)}nt denotes text-only samples, where nt is the number of text-only samples. Note that the total instance number satisfies n = nf +nv+nt. Figure 1

illustrates the overall workflow of our proposed RGCH-TPA framework.

3.2 Multimodal Feature Extraction

To fully utilize the Transformer's global contextual modeling capacity, the proposed RGCH-TPA method employs pretrained ViTs [Dosovitskiy, 2020] and GPT-2 [Radford et al., 2019] as the image and text encoders, respectively. Even when only a single modality is available, self-attention models long-range dependencies to extract robust unimodal features Fv = {gv, zv} and Ft = {gt, zt}. Here, Fv ∈ RB×(Lv+1)×d and Ft ∈ RB×(Lt+1)×d , where g represents the global class embedding, z refers to local patch/token embeddings, B denotes the batch size, and d = 512 is the shared embedding dimension. To ensure compatibility with pretrained Transformers, the proposed RGCH-TPA method applies explicit zero-padding to all missing-modality inputs before encoding, thus preserving input alignment without altering encoder interfaces.

3.3 Retrieval-Guided Completion

In incomplete cross-modal hashing retrieval, existing methods complete missing modalities without modeling the intrinsic structure of the retrieval space. To address this issue, we design a retrieval-guided completion (RGC) module in our proposed method.

The RGC module maintains a dynamic memory bank that stores the image and text sequence features in a paired manner. During model training, only complete image-text pairs are stored in the memory bank to update features from the previous epoch. When a modality is missing, our RGC module uses the remaining available modality features as queries to retrieve semantically similar cross-modal expert features from the memory bank. Specifically, for query feature Fq ∈ RSq×d and memory bank features Fm ∈ RN×Sm×d , we first average along the sequence dimension to obtain embeddings fq and fm, and then compute cosine similarity as follows:

We then select the Top-K memory samples with the highest similarity, and preserve their full sequence features as expert candidates of Fq: E = {F (1) m , . . . , F (K) m } ∈ RK×Sm×d .

To adaptively allocate expert importance, we introduce a modality-aware context router that estimates routing scores by semantic relevance between available modality features and expert features. To enhance representations, we first linearly project the remaining available modality feature Fq and expert features E ∈ RK×S×d as follows:

where Wq and Wm are learnable projection matrices. We flatten the projected features along the sequence dimension to obtain global representations ¯ Fq and ¯ E. Next, we calculate the correlations between the remaining modality and each expert via matrix multiplication, and obtain routing scores using softmax normalization:

where αk denotes the routing weight of the k-th expert for the query feature Fq, satisfying ∑Kk=1 αk = 1. This design ensures routing weights are conditioned on available modality semantics, enabling retrieval-consistent expert selection.

After obtaining routing weights, the proposed RGC module generates missing-modality features using a conditional mixture of experts (CMoE) generator. The CMoE generator comprises K parallel expert networks, each implemented as a feed-forward network (FFN). After forwarding all experts, we compute a routing-weighted aggregation of expert outputs to generate the missing-modality features as follows:

Overall, the RGC module is symmetrically designed to handle both image-missing and text-missing cases. It synthesizes missing features conditioned on available modalities and reinserts them into missing positions to form completed features ˜ Fv = {˜ gv, ˜ zv} and ˜ Ft = {˜ gt, ˜ zt}. Meanwhile, to ensure semantic fidelity between generated and original features, the proposed RGC method applies a completionconsistency regularization on complete image-text pairs during training:

where M denotes the number of complete sample pairs within a minibatch.

3.4 Token-to-Patch Alignment and Enhancement

After obtaining the completed image and text features, we introduce a token-to-patch alignment and enhancement (TPAE) module to achieve fine-grained semantic alignment between text tokens and image patches, and then construct enhanced global representations for hash learning.

Token to Patch Interaction. To measure semantic alignment between text tokens and image patches, we explicitly model their token-patch similarity matrix as follows: S = ˜ zt · ˜ z⊤ v . Then we apply softmax along the patch dimension to derive text-to-image soft alignment weights as follows:

where Ab,t,v denotes the attention allocation from the t-th text token to the v-th image patch for the b-th instance, satisfying ∑Lv v=1 Ab,t,v = 1. We aggregate image patch embeddings to produce token-aligned visual representations as follows:

Symmetrically, we perform image-guided token aggregation by normalizing the similarity matrix along the textual dimension to obtain zalign t . To ensure text tokens preserve reconstructive semantics for aligned visual patches, we design a two-layer MLP reconstructor as follows: zrecon v = Reconstructor(˜ zt). The patch reconstruction is supervised using aligned patch embeddings with a feature-level L1 loss:

This constraint forces text tokens to preserve reconstructive semantics for corresponding visual patches, improving local cross-modal feature consistency. Furthermore, to reinforce discriminative token-patch correspondence, we design an alignment loss with hard positive and soft negative patch sampling. For each text token, we select the highestscoring image patch as the hard positive: s+j = maxi Sj,i. We also sample the lowest-scoring patch among the remaining patches as the soft negative: s-j = mini Sj,i. The tokento-patch alignment loss is given as follows:

where σ(·) denotes the Sigmoid function. It aims to promote high similarity between each text token and its most semantically relevant image patch, while suppressing interference from irrelevant patches.

Then we enhance local features through bidirectional cross-modal fusion as follows:

where [·; ·] denotes concatenation, and Linear ∈ R2d→d is a learnable linear fusion projection layer.

Global Representation Enhancement. To improve the discriminability of global features, a residual MLP is introduced to nonlinearly enhance the global embeddings: ˆ gv = ResMLP(˜ gv), ˆ gt = ResMLP(˜ gt). Then, we use a symmetric contrastive learning loss to align the enhanced highorder features ˆ gv and ˆ gt. Specifically, in a minibatch M, we learn discriminative cross-modal representations by contrasting the similarity relationships between positive and negative pairs:

where τ is the temperature hyperparameter.

Finally, we apply average pooling on the fused local features and integrate them with the enhanced global class embeddings to form the final global representations:

This design preserves global semantic consistency while incorporating fine-grained token-patch correspondence, thus yielding more discriminative global representations for supervised hash code learning.

3.5 Hash Learning

The hash learning stage aims to encode the aligned crossmodal global representations into binary hash codes, while preserving retrieval discriminability and cross-modal consistency. We design a hashing layer that projects global representations into K-dimensional real-valued features to obtain Hv and Ht, and then apply the Sign function to obtain compact hash codes Bv and Bt, respectively. To improve the discriminability of hash codes and preserve the semantic structure of multimodalities, the hash loss consists of the following loss functions:

1) Intra-modal similarity preserving loss:

where Ω(∗) ij = 1 2 (h(∗) i )⊤h(∗) j measures the intra-modal similarity via inner product, Sij is the label-derived semantic similarity between the i-th and j-th instances, and ∗ ∈ (v, t).

2) Cross-modal similarity preserving loss:

where Θij = 12 (hti )⊤hvj and Φij = 12 (hvi )⊤htj denote the inter-modal similarity via inner product.

3) Quantization loss: To learn unified binary hash codes, we fuse the real-valued outputs from both modalities into a shared binary code bi = sign(hvi + hti ), and minimize the quantization errors between bi and the real-valued features of each modality:

Finally, the overall loss function of the proposed RGCHTPA method is given as follows:

where α, β, γ1, γ2, λ1, λ2 are weight hyperparameters.

4 Experiments

4.1 Experimental Settings

Datasets. To validate the effectiveness of the proposed method, we conduct experiments on three widely-used datasets: MIRFLICKR-25K [Huiskes and Lew, 2008], NUSWIDE [Chua et al., 2009], and MS-COCO [Lin et al., 2014]. Among them, the MIRFLICKR-25K dataset consists of 24,581 image-text pairs, covering 24 categories. The NUSWIDE dataset comprises 195,834 image-text pairs from the 21 most frequent semantic classes. The MS-COCO dataset contains 122,218 image-text pairs spanning 80 categories.

Implementation. To evaluate the performance of our approach in incomplete cross-modal hashing retrieval tasks, we follow the experimental settings of CICH [Luo et al., 2024]. We divide the training set and retrieval set into three predefined modality incompleteness levels. Case 1 includes 10% complete pairs, 45% image-only, and 45% text-only instances, simulating severe modality incompleteness. We further adopt two additional incomplete proportions: Case 2 (0.3, 0.35, 0.35) and Case 3 (0.5, 0.25, 0.25). We train with a batch size of 128 for 30 epochs, using K = 4 experts in the CMoE generator. We optimize the network using AdamW [Loshchilov and Hutter, 2017] with a weight decay of 0.01 for hash learning. We set the learning rate of both encoders to 1e-6, while the other modules use dataset-specific learning rates: 1e-3 for MIRFLICKR-25K, 2e-3 for NUSWIDE and MS COCO.

Metric. Following standard cross-modal hashing evaluation, we conduct two retrieval tasks: Image-to-Text (I2T) and Text-to-Image (T2I). We adopt mean Average Precision (mAP) as the primary metric for retrieval performance. In addition, Precision-Recall (PR) curves and TopK-Precision curves are employed for comprehensive evaluation.

Baselines. We compare the proposed method against nine state-of-the-art baselines, including seven complete CMH methods: UCMFH [Xia et al., 2023], DDSS [Huang et al., 2025], DSPH [Huo et al., 2023], DNPH [Huo et al., 2024], CMCL [Wu et al., 2024], DNcH [Zhu et al., 2025], and DScPH [Qin et al., 2025]. Due to the limited availability of incomplete CMH methods and the absence of public implementations, we consider only two incomplete methods as baselines: CICH [Luo et al., 2024] and GCIMH [Shen et al., 2023]. Among them, GCIMH has been modified from a multimodal hashing retrieval method to a cross-modal one.

4.2 Comparisons with State-of-The-Art Methods

To comprehensively evaluate the effectiveness of our approach under incomplete multimodal settings, we conduct comparisons with several state-of-the-art cross-modal hashing methods across three levels of incompleteness on three datasets. Due to limited space, only the 32-bit retrieval results are reported in Table 1 and Figure 2.

The results indicate that the proposed RGCH-TPA method consistently achieves the best performance across all datasets and missing-modality ratios, demonstrating its robustness under incomplete multimodal scenarios. In the most challenging Case 1, the proposed method still attains mean mAPs of 0.8776, 0.7620, and 0.7547 on three datasets, respectively, clearly outperforming other state-of-the-art hashing methods. It can be observed that the performance of all methods improves as the proportion of complete samples increases (Cases 2 and 3). Nevertheless, our method shows a more pronounced improvement and consistently outperforms others, demonstrating its effectiveness in exploiting complete data via retrieval-guided completion. Figure 2 further reports the PR and TopK-Precision curves under Case 2. The proposed RGCH-TPA method consistently yields higher precision over a wide recall range, especially in the high-recall region, showing improved ranking stability. Moreover, it achieves superior TopK-Precision for large K, confirming that the learned hash codes preserve reliable semantic neighborhoods in the Hamming space. These results verify that retrieval-guided expert cues and token-patch level alignment allow the proposed method to learn more discriminative and robust hash representations under incomplete multimodal settings.

variants as follows: a) BASE: It acts as the baseline; b) BASE+RGC: This variant adds the retrieval-guided completion (RGC) module to the baseline; c) BASE+RGC+TPI: It further adds token-patch interaction (TPI) to the model in row

4.3 Ablation Study

In this part, we conduct a systematic ablation study across three incompleteness ratio settings on the MIRFLICKR25K dataset. Specifically, we take Case 1 to construct the

2; d) RGC+TPI+GRE: It replaces the original modal similarity preservation loss in BASE; e) BASE+RGC+TPI+GRE: This is the full model, obtained by adding the global representation enhancement (GRE) component to the model in row 3.

As observed in table 2, the BASE model achieves the lowest performance. By adding the RGC module, this variant shows consistent improvements across all incomplete configurations in both I2T and T2I tasks, indicating that retrievalguided completion effectively mitigates semantic discontinuity caused by missing modalities. We further incorporate the TPI block, thereby leading to additional performance gains. It confirms the critical role of fine-grained cross-modal interaction in boosting cross-modal semantic consistency. Finally, by strengthening global representations through the GRE module, the full model achieves optimal performance across all incomplete scenarios. This demonstrates that collaborative modeling of global and local features is essential for learning discriminative hash representations, thereby validating the contribution of each component.

4.4 Parameter Sensitivity

This subsection analyzes the sensitivity of the proposed RGCH-TPA method to key hyperparameters on the MIRFLICKR-25K dataset under Case 2 with 32-bit hash codes. Specifically, we evaluate the influence of α, β, γ1, γ2, λ1, and λ2, with the corresponding results illustrated in Figure 3. It can be observed that when these hyperparameters vary over a wide range, the mAP curves for both I2T and T2I retrieval remain stable with only minor fluctuations. This demonstrates that the proposed RGCH-TPAmethod is insensitive to hyperparameter settings and consistently maintains reliable retrieval performance, confirming its robustness and practical applicability in incomplete-modality scenarios.

5 Conclusion

In this paper, we propose a novel retrieval-guided completion hashing with token-patch alignment (RGCH-TPA) framework for incomplete cross-modal retrieval. Specifically, the RGC module reconstructs high-quality missing modality features by leveraging retrieved complete-modal samples as expert guidance. The TPAE module enhances local cross-modal consistency through fine-grained token-patch semantic alignment and interaction, and performs non-linear enhancement on global features. Finally, we integrate hierarchical semantic information to generate highly discriminative hash representations. Extensive experiments on three benchmark datasets validate the superiority of the proposed framework in incomplete-modality multimodal retrieval scenarios.

Acknowledgements

This work was supported by the National Natural Science Foundation of China [Grant 62162033, 62462038], Yunnan Xingdian Talent Support Plan Project.