DSSL-Hash: Dynamic Semantic Structure Learning for Unsupervised Cross-Modal Hashing

1 Introduction

Driven by the rapid proliferation and widespread adoption of the Internet, multimodal data has experienced an unprecedented explosion. However, efficiently processing and storing such vast volumes of heterogeneous information remains a formidable challenge. To address this, cross-modal retrieval has long been regarded as the touchstone for representation learning. By learning a unified feature space, it effectively unlocks the immense potential of multimodal data while bridging the semantic gap and heterogeneity gap inherent in disparate modalities, such as text and video [Zhang et al., 2025].

Cross-modal hashing (CMH) facilitates efficient multimodal retrieval by mapping heterogeneous data into a unified Hamming space [Ding et al., 2025; Tu et al., 2025; Xu et al., 2025; Yang et al., 2025a; Zhu et al., 2025].

∗Corresponding author.

Among its variants, unsupervised CMH (UCMH) is particularly valuable for real-world applications due to its independence from semantic labels [Zhao et al., 2026]. However, UCMH still faces three critical challenges: 1) Traditional methods often fail to capture fine-grained semantics and higher-order correlations when aligning diverse modalities [Huo et al., 2024; Qin et al., 2025a]. 2) Most approaches rely on predefined static similarity matrices [Zou et al., 2025], which cannot adapt to evolving semantic relationships during training, leading to suboptimal hash code representation. 3) Existing frameworks typically depend on manual, static hyper-parameter settings [Lin et al., 2025]. Such inflexibility ignores training dynamics and limits the model's potential, as hash code learning is highly sensitive to hyper-parameter configurations [Yang et al., 2024].

To address the above challenges, this paper proposes a dynamic optimization cross-modal hashing method that integrates knowledge distillation and reinforcement learning-Dynamic Semantic Structure Learning for Unsupervised Cross-Modal Hashing (DSSL-Hash). Within this framework, we first leverage the CLIP vision-language pretrained model as the teacher network and innovatively introduce a dynamic perception matrix. Then, through a knowledge distillation mechanism, the student network is guided to capture deep semantic correlations across heterogeneous modalities. Furthermore, we introduce a reinforcement learning-based optimization strategy by employing the Deep Deterministic Policy Gradient (DDPG) algorithm to dynamically adjust both the modality feature weights and the loss function ratios based on real-time model performance. This enables the network to maintain strong adaptability under dynamically changing multimodal data distributions. The main contributions are as follows:

• To mitigate cross-modal semantic loss, we propose Dynamic Graph-Based Cognitive Optimization (DGO). DGO leverages CLIP-guided distillation for fine-grained learning and GCNs to capture multi-scale structural relations in the perception graph. Furthermore, a Hamming distance optimization strategy is employed to minimize semantic distortion during hash mapping.
• To surpass static configuration constraints, we propose RAAS. RAAS employs the DDPG algorithm to adaptively adjust modal weights and loss ratios based on realtime feedback. By bypassing manual empirical tuning, RAAS autonomously explores hyper-parameter dependencies and optimal strategies, significantly enhancing parameter interaction and optimization efficiency.
• Sufficient experiments on three benchmark datasets show that the proposed method DSSL-Hash outperforms other state-of-the-art unsupervised cross-modal hashing methods.

2 RelatedWork

Deep cross-modal hashing (DCMH) maps multimodal data into a unified Hamming space and is categorized into supervised [Zou et al., 2025; Cao et al., 2024; Huo et al., 2024; Liang et al., 2024] and unsupervised [Dufumier et al., 2025; Wen et al., 2025; Wang et al., 2024; Cui et al., 2024; Cai et al., 2024; Liu et al., 2024] methods.

Supervised Deep Cross-Modal Hashing. utilizes semantic labels to maximize discriminative power. Several studies enhance uncertainty perception: FUME [Duan et al., 2025] employs fuzzy set theory for category credibility, TCL [Qin et al., 2025b] introduces Evidential Deep Learning (EDL) for self-assessment, and PDF [Cao et al., 2024] proposes a Predictive Dynamic Fusion framework. Other methods prioritize semantic discrimination: PromptHash [Zou et al., 2025] introduces prompt learning, DNPH [Huo et al., 2024] utilizes learnable category proxies, and DScPH [Qin et al., 2025a] explores imbalanced correlations for semantic consistency. Additionally, SCH [Hu et al., 2024] and MRDH [Liang et al., 2024] refine similarity constraints via differential constraints and global similarity metrics.

Unsupervised Deep Cross-Modal Hashing. extracts similarity relationships without labels. To bridge semantic gaps, dual-coding mechanisms like UDC [Cai et al., 2024] and UDDH [Zhang et al., 2024] jointly optimize semantic and content hashes. For similarity matrix robustness, VTMUCH [Fan and Cao, 2025] uses vision-guided text mining, SMSH [Kuai et al., 2025] introduces Jaccard similarity, SACH [Cui et al., 2024] uses structure-aware contrastive learning, and FUCH [Liu et al., 2024] employs collective matrix factorization with Cauchy Loss. To capture finegrained information, HuggingHash [Wang et al., 2024] and USFTH [Yang et al., 2025b] leverage Transformer architectures, while CoMM [Dufumier et al., 2025] and InfMasking [Wen et al., 2025] utilize information-theoretic frameworks to maximize collaborative interaction.

To address hyper-parameter complexity and semantic modeling gaps, we propose DSSL-Hash. It leverages DDPG for auto-tuning, while combining Knowledge Distillation (KD) and Graph Convolutional Networks (GCNs) to capture global and local semantic structures.

3 Methodology

3.1 ProblemDefinition

The proposed DSSL-Hash method targets unsupervised cross-modal retrieval. Let O = {vi, ti}ni=1 be a dataset of n image-text pairs, where vi ∈ R1×dv and ti ∈ R1×dt denote image and text features. Guided by a teacher model, we map these to c-bit binary codes Hv,Ht ∈ {-1,+1}n×c by optimizing Hamming distance.

We adopt a mini-batch training strategy with m pairs per batch. Throughout this paper, uppercase and lowercase letters denote matrices and vectors, respectively. For an image query qv, the cross-modal retrieval task aims to return the top M most relevant instances fromO, defined as:

where Rt denotes the result set of the retrieval task, hq denotes the hash code of the query sample, and Hj∗ t denotes the binary code of the j-th candidate text sample in Ht. The operator Top-M(·) returns M samples with the largest values, which is equivalent to selecting samples with the minimum Hamming distances DH. Figure 2 illustrates the overall framework of DSSL-Hash.

3.2 Dynamic Graph-Based Cognitive Optimization

In order to solve the problem of static similarity matrix which is difficult to adapt to the dynamic change of semantic structure during the training process and the semantic loss caused by the mapping of real-valued space to Hamming space, we developed the DGO component.

Dynamic PerceptionMatrix. The dynamic perceptionmatrix in DGO is composed of two hierarchical units: intramodal feature perception and cross-modal interaction. By leveraging the RAAS component to regulate similarities across these dimensions, we construct a dynamic perception matrix SDS. This matrix serves as the structural foundation for effective feature propagation within GCNs. The formal expressions are defined as follows:

where ϕ(X, Y ) = X⊤Y ∥X∥2∥Y ∥2 represents the cosine similarity, and M ∈ Rm×dv ,N ∈ Rm×dt represent CLIP-guided teacher features, with ∥·∥2 denoting the L2 norm used to normalize vectors to unit length. Crucially, SDS is reconstructed at every mini-batch from the current teacher features (M, N) and the modality weights (α, β, γ) updated by RAAS, so the perception graph continuously evolves with the local semantic state during training.

GCN(Z (l) ∗ , SDS,W(l+1) ), where W(l+1) denotes the learnable layer weight matrix. An iterative non-linear quantization strategy is further introduced to reduce quantization loss and enhance code discriminability:

where HA∗ ∈ {-1, +1}m×c denotes the discrete teacher hash codes, and ω is a RAAS-optimized hyper-parameter controlling the quantization steepness. Larger ω pushes continuous features closer to compact bipolar hash codes.

Semantic-EnhancedKnowledge Distillation. The student model adopts a VGG-16 visual encoder for the image branch and an MLP textual encoder for the text branch, whose outputs are projected into the Hamming space as hash codes Hv and Ht. In order to improve the accuracy and discriminative power of the student model in generating hash codes, we introduce a distillation loss function Lhash, whose loss is defined as:

where hAx,i , hx,i are teacher and student hash codes for modality x ∈ {v, t}.

To further enhance the structural coherence within each modality, we introduce an intra-modal consistency loss, denoted as Lintra. The loss is defined as:

where hAx,i and hx,i denote the teacher and student hash codes for modality x. A scaling factor ρ is introduced to adjust the similarity magnitude computed by the student model for better matching. To further improve the quality of hash encoding, we construct two cross-modal loss terms: the semantic alignment term L1 and the structure-preserving term L2, to jointly optimize the cross-modal hashing representations for image and text modalities. The semantic alignment term L1 aims to align image and text representations in the hash space under the guidance of SDS. This loss can be rewritten in a unified sample-wise form:

where sij is the (i, j)-th element of the dynamic perceptual similarity matrix SDS, and σ(h⊤v,iht,j ) = (1 + e-h⊤v,i ht,j )-1 . The structure-preserving term L2 is designed to retain the intra-modal structural relationships within each modality during cross-modal alignment. It is defined as:

where 1m is a normalization factor.

To encourage structural alignment between visual and textual modalities, we minimize the loss term L2, which serves as a regularizer that promotes consistency in patterns of intramodal similarity. The final cross-modal consistency loss is defined as:

To reduce semantic distortion and quantization errors in Hamming mapping, the DGO module introduces a semantically regulated Hamming distance optimization. By redefining Hamming distance to align with sample relationships, we define the target distance based on the similarity score Sij ∈ [0, 1] as:

where ε is a tunable hyper-parameter that controls the scale. Additionally, the semantic channel is bounded by an upper limit of λuij = λij and a lower limit of λlij = max(0, λij -τ ), where τ ∈ R+ denotes the width of the semantic channel.

To model semantic relationships more precisely, we categorize sample pairs into three types and apply different constraint strategies. For partially semantic-positive pairs (i.e., 0 < Sij < 1), their Hamming distance should lie within a bounded interval:

where dH(hi, hj) denotes the Hamming distance between a pair of samples in the hamming space. The overall Hamming distance optimization loss is defined as:

where ∆lij = (max(0, λlij - dH(hi, hj)))2 and ∆uij = (max(0, dH(hi, hj) - λuij ))2 . The coefficients (ηij, ξij) are set based on semantic relations: (1, 1) for partial, (0, 1) for Sij = 1, and (1, 0) for Sij = 0. While the overall Hamming loss combines constraints for different semantic pair types, we further refine this by explicitly modeling deviations from target distance intervals. The resulting multi-level semantic constraint loss is given by:

where Λ∗ = [λ∗ij ]m×m for ∗ ∈ {l, u} are matrices of target Hamming distances. The weights W ∗ ij are: W l ij , W u ij = 1 if Sij ∈ (0, 1); W l ij = θ, W u ij =0 if Sij =1; and W l ij = 0, W u ij = η if Sij = 0. The total loss function of DGO is defined as follows:

Algorithm 1 Dynamic Semantic Structure Learning for Unsupervised Cross-Modal Hashing

• 1: Input: multimodal dataset O, hash length c.
• 2: Initialize modality weights (α, β, γ) and loss coefficients {χk}4k=1 .
• 3: while the training process has not converged do
• 4: DGOOptimization:
• 5: perform graph-based representation learning under current configuration.
• 6: compute LDGO according to Eq. (13) and update student networks.
• 7: RAAS Scheduling:
• 8: observe training state and evaluate retrieval performance.
• 9: update actor-critic networks according to Eq. (18).
• 10: reconfigure (α, β, γ) and {χk}4k=1 for subsequent optimization.
• 11: end while
• 12: Output: learned cross-modal hash codes Hv,Ht.

where {χk}4k=1 are the dynamic weight coefficients adjusted by the RAAS agent to balance each optimization constraint.

3.3 Retrieval-Aware Adaptive Scheduler

RAAS uses reinforcement learning to adaptively tune modality weights and loss coefficients. At each epoch, it performs an N-order search, filters unstable candidates through gradient-protected rollback, and retains the best configuration via elite selection.

Deep Deterministic Policy Gradient. RAAS formulates hyper-parameter tuning as a reinforcement learning task, employing a dynamic referencing mechanism to navigate the continuous search space. In this framework, the agent's actions are guided by a differential reward that quantifies the performance gap between the current exploration trial and a dynamic baseline, which is updated upon discovering superior configurations. Specifically, let mAPt denote the validation mAP after the agent applies action at at step t, and mAPbest the highest mAP observed before step t; the reward is

followed by mAPbest ← max(mAPbest,mAPt). To optimize this process, RAAS leverages the DDPG algorithm within an actor-critic architecture. Both the actor and critic networks utilize a two-hidden-layer ResNet backbone. To stabilize training, the critic adopts a target network to calculate the cumulative Q-value as follows:

where r is the immediate reward, ζ is the discount factor, and Q(s′ , a′ ; θ′ ) estimates the future value based on the next state s′ and the target actor's action a′ .

The critic network is trained by minimizing the squared error between the target y and the predicted Q-value:

where y is the target value. Subsequently, the actor network is updated by minimizing the following loss to maximize the expected Q-value:

The actor's objective is to optimize the policy µ by generating actions that yield higher rewards as estimated by the critic. The objective functions of RAAS are defined as follows:

Combining all constraints, the total objective function is:

This joint objective optimizes binary codes by preserving structural consistency and enhancing semantic discriminability, while the proposed framework remains fully unsupervised. No semantic labels are used to construct SDS, supervise the teacher network, or optimize the student hashing networks. RAAS only accesses the validation mAP at the scheduling level to update (α, β, γ) and {χk}4k=1 , without introducing label supervision into the hashing loss or network optimization. Algorithm 1 summarizes the procedure.

4 Experiments

4.1 Datasets

We evaluate DSSL-Hash on three widely used cross-modal retrieval benchmarks. MIRFLICKR-25K contains 25,000 Flickr image-text pairs annotated with 24 categories. NUSWIDE includes 269,648 web image-tag pairs with 81 semantic concepts; following common practice, we use the 10 selected categories and 186,577 corresponding pairs. MS COCO contains 123,287 image-text pairs covering 80 categories, where each image is paired with textual descriptions of real-world scenes [Zhu et al., 2023].

4.2 Baselines and EvaluationMetrics

To evaluate the cross-modal retrieval performance of DSSLHash, we conduct image-to-text (I→T) and text-to-image (T→I) tasks using mAP and Top-N precision. We compare DSSL-Hash against 11 SOTA baselines: DJSRH [Su et al., 2019], JDSH [Liu et al., 2020], DCMH [Jiang and Li, 2017], MTFH [Liu et al., 2019], AGCH [Zhang et al., 2021], CIRH [Zhu et al., 2022], SCH [Hu et al., 2024], VLKD [Sun et al., 2023], DScPH [Qin et al., 2025a], and DNPH [Huo et al., 2024] on three benchmark datasets. The results are summarized in the following tables and figures.

4.3 Implementation Details

Experiments were conducted on a Win11 server with an RTX 3090Ti GPU using 16-128 bit hash codes. We report mAP@500 and top-1500 precision. Following standard protocol, each dataset is split into database, query, and training sets. We further reserve 5% of the training set as a validation subset, which is used only by RAAS to compute rt and never for representation learning. The N-order RAAS search is set to N = 10. The initial weights are α = 0.5, β = 0.3, γ = 0.2, χ1 = χ2 = 0.1, and χ3 = χ4 = 1, and are then adapted by RAAS. Other fixed hyper-parameters are set as

τ = 0.2, ρ = 0.5, ε = 1.0, θ = 0.3, η = 1.0, and ζ = 0.99. The student and GCN networks are optimized by SGD with LR=10-3 , while the actor and critic use Adam with LRs 10-4 and 10-3 , respectively. All baselines are reproduced under identical settings on three benchmark datasets.

4.4 Comparison Results and Discussions

Results on mAP: As shown in Tables 1, DSSL-Hash consistently achieves superior performance on both the I→T and T→I retrieval tasks across different datasets and hash code lengths. Compared with recent state-of-the-art methods from 2024 and 2025, including DNPH (TOMM2024), SCH (TPAMI2024), and DScPH (TMM2025), DSSL-Hash delivers significant improvements. In addition, the PR curves in the first two columns of Figure 3 further support the mAP results. DSSL-Hash maintains higher precision across most recall ranges on the three datasets, showing stable retrieval performance under different recall levels.

Results on Top-N Precision Curves. The Top-N precision curves in the last two columns of Figure 3 evaluate retrieval stability under different numbers of returned samples.

DSSL-Hash maintains competitive precision across the three datasets, especially in the I→T task, indicating its ability to preserve reliable nearest-neighbor rankings under larger retrieval ranges. This benefits from the unsupervised modeling of cross-modal semantic relationships, which helps learn more discriminative modality-specific representations.

4.5 Ablation Experiments

In order to gain more insight into our proposed DSSL-Hash method, we performed ablation experiments as shown in Table 2. We ablated the three modules of DSSL-Hash, and in the table DSSL-Hash-D which shows that the method deletes the distillation-related loss function in the DGO component, the performance decreases by up to 13.85% due to the loss of high-quality guidance. DSSL-Hash-DGO shows that the method removes the loss function related to semantic conditioning in the DGO component, and the Hamming space is no longer reasonable, leading to a performance drop of about 7%. DSSL-Hash-RAAS shows that we delete the RAAS component part of the dynamic adaptive parameter tuning of the reinforcement learning, which leads to the model not being able to dynamically and adaptively learn the required modal feature information, resulting in a performance drop.

4.6 Adaptive Parameter Analysis

Figure 4 shows performance heat maps under fixed weight combinations of α, β, and γ, with α+β+γ = 1. Subfigure (a) shows that the best performance occurs around α ≈ 0.5 and β ≈ 0.3, indicating the importance of balanced intra-modal similarities. Subfigures (b) and (c) further show that moderate cross-modal interaction (γ ≈ 0.2) is beneficial, while excessive reliance on it without intra-modal guidance degrades performance. Additionally, Figure 5 visualizes the dynamic parameter trajectories guided by the RAAS module. The trajectories clearly show how RAAS adaptively updates the parameters throughout training. As the policy is iteratively refined, the model's mAP improves steadily, confirming the effectiveness of the reinforcement learning mechanism in discovering superior hyper-parameter configurations.

The performance gains verify the effectiveness of RAAS, which adopts a DDPG-based reinforcement learning strategy to search for suitable hyper-parameter configurations. Compared with static or grid-based tuning, RAAS explores the continuous high-dimensional search space more efficiently and avoids the burden of discrete enumeration. During training, the agent adjusts modality weights and loss coefficients using iterative feedback such as mAP, and refines its policy through a closed-loop actor-critic mechanism. The heat map evolution further shows that RAAS shifts from broad exploration to high-performance exploitation, improving accuracy and robustness by balancing modality-specific and crossmodal signals. In terms of overhead, RAAS uses an actorcritic network with two hidden layers and is invoked only N times per epoch rather than per mini-batch, introducing moderate additional training cost. At inference, RAAS is disabled, and retrieval remains standard Hamming lookup over c-bit codes.

5 Conclusion

We propose DSSL-Hash, a robust cross-modal retrieval framework that synergizes CLIP-guided distillation (DGO) for complex dependency modeling with reinforcement learning based on DDPG for autonomous hyper-parameter tuning. This collaborative design enables the model to maintain consistent performance by employing the DGO module to refine semantic alignment and the DDPG-driven RAAS mechanism to search for optimal training configurations. Extensive experiments show that DSSL-Hash significantly outperforms state-of-the-art baselines in accuracy across three benchmarks, demonstrating its robust generalization and efficiency in various retrieval scenarios. Future work will extend the RAAS mechanism to broader foundational models and more heterogeneous data modalities.