Dual Semantic Fusion Hashing for Multi-Label Cross-Modal Retrieval

1 Introduction

In the era of informatization, retrieving relevant instances from vast amounts of multi-modal data [Xu et al., 2024], such as videos, images, and text, has become a great challenge. Cross-modal retrieval (CMR) [Wang et al., 2023; Liu et al., 2024] aims to query semantically related data of other modalities by providing data in one modality, which becomes considerable attention. Early CMR methods [Ranjan et al., 2015; Wang et al., 2016] tend to learn a unified real-valued representation for each instance to eliminate heterogeneity across modalities. Nevertheless, the explosive growth in data volumes brings these methods to the bottleneck of query efficiency. Due to the advantages of retrieval efficiency and storage cost, cross-modal hashing (CMH) [Sun et al., 2023a; Sun et al., 2024a] exhibits significant potential in processing massive-scale data. CMH aims to project highdimensional features into compact hash codes in the Hamming space, thereby measuring the similarities between all instances through the Hamming distances.

∗Corresponding author.

In recent years, numerous supervised CMH methods [Zhu et al., 2024; Sun et al., 2024b] have been proposed to improve retrieval performance. They can be broadly categorized into two types: label matrix supervision [Xu et al., 2017; Shen et al., 2021] and pairwise semantic supervision [Luo et al., 2018; Liu et al., 2019]. Pairwise semantic supervision methods adopt pairwise similarity matrices to estimate instance-level semantic relationships, thereby providing more semantic guidance. Therefore, they obtain more promising performance. However, they still face some limitations. The existing CMH methods always adopt relatively coarse-grained supervision information (i.e., instance-level semantic relationship) to guide hashing learning. In other words, they ignore the intrinsic multi-label semantic correspondences, which makes it difficult to excavate more semantics. For example, as shown in Fig.1, we have three labels A [1,0,0,1], B [0,1,0,1], and C [1,0,1,0], and the number of their corresponding instances are 5, 50, and 5, respectively. According to the instance-level semantic relationship, the similarity between A and B and between A and C is equal. Since B contains a larger number of instances, the more relevant instances could be retrieved if the instances from A are semantically closer to ones from B. However, it is a great challenge how to construct a fine-grained semantic relationship to enhance the expression of semantic information.

To address the above issues, we propose a dual semantic fusion hashing (DSFH) framework for multi-label crossmodal retrieval. Our DSFH explores the multi-level semantic information from multi-label data, and further fuses both the coarse-grained and fine-grained semantic relationships, thereby making the learned hash codes more discriminative. Specifically, as shown in Fig.2, we first learn the modalityspecific representations to preserve the consistency and specificity of all modalities to some extent. Then, we adopt KMeans label clustering to capture finer-grained cluster-level relationships between instances. Finally, we fuse the coarsegrained and fine-grained supervision information to guide consensus hashing learning, thereby generating discriminative hash codes. Overall, the main contributions of this paper are summarized as follows:

• We propose a new dual semantic fusion hashing (DFSH) that fully exploits the multi-level semantic information in multi-label multi-modal data. To the best of our knowledge, this is the first work to simultaneously consider the coarse-grained and fine-grained semantic relationships for multi-label cross-modal hashing.
• We establish the cluster-level semantic relationship to capture finer-grained similarities between different instances, thereby promoting that all instances with similar semantics could be clustered together.
• Extensive experiments demonstrate that our DFSH outperforms all state-of-the-art comparison methods on three benchmarks.

2 RelatedWork

2.1 Label Matrix SupervisionMethods

Label matrix supervision Methods [Xu et al., 2017; Wang et al., 2019; Sun et al., 2023b] typically adopt the class label as semantic information to guide hashing learning, thereby improving the discriminative of the learned hash codes. For example, scalable discrete matrix factorization hashing (SCRATCH) [Chen et al., 2020] learns a common latent representation by incorporating original features and labels, thereby generating unified hash codes. To more accurately explore the correlation of heterogeneous data, discrete semantic matrix factorization hashing (DSMFH) [Qin et al., 2021] performs matrix factorization on labels to obtain latent specific representations, then learns hash codes based on these latent representations. To alleviate the interference of noisy labels, robust and discrete matrix factorization hashing (RDMH) [Zhang and Wu, 2022] directly associates hash codes with the label matrix and introduces the l2,1-norm to enhance its robustness.

2.2 Pairwise Semantic SupervisionMethods

To fully reflect the semantic information between instances, pairwise semantic supervision methods [Zhang and Li, 2014; Lin et al., 2015; Teng et al., 2023] adopt the n×n similarity matrix to capture semantic similarities between all instances. For example, fast cross-modal hashing (FCMH) [Wang et al., 2022] constructs both global and local similarity matrices to guide hash codes learning. However, the n × n similarity matrix could lead to high computational cost. To this end, scalable asymmetric discrete cross-modal hashing (BATCH) [Wang et al., 2021] adopts labels to construct pairwise similarity, thereby avoiding the use of the large n × n similarity matrix. To explore potential semantic correlations of multilabel data, adaptive label correlation based asymmetric discrete hashing (ALECH) [Li et al., 2023] proposes adaptive label correlation to fully utilize the latent label information.

Although the existing CMH methods have obtained promising performance, most of them ignore cluster-level semantic relationships in multi-labels. Traditional coarsegrained semantic similarity makes it difficult to fully mine the correlations of multi-label data.

3 ProposedMethod

3.1 Notations

Given a multi-modal dataset {V t }Mt=1 with M modalities, we denote V t = [vt1 , vt2 , · · · , vtn ] ∈ Rdt×n as the training data of the t-th modality, where n is the number of instances, and dt is the corresponding feature dimensionality. To capture the nonlinear features from original multi-modal data, we employ the radial basis function (RBF) kernel [Liu et al., 2012] on V t . Therefore, the kernelized features of t modality can be denoted as Xt = [exp( ∥xti -at1 ∥22 -2σ2 ), · · · , exp( ∥xti -ath ∥22 -2σ2 )]. Where {at1 , at2 , · · · , ath } are h anchors randomly selected from V t , and σ is the kernel width. In addition, Y = [y1, y2, · · · , yn] ∈ Rc×n denotes the ground truth labels, where yi ∈ {0, 1}c×1 is the label vector of every instance, c is the number of categories. We define yij = 1 if the j-th instance belongs to i-th category, otherwise yij = 0.

3.2 Model Formulation

Most existing methods emphasize extracting the shared information from different modalities to directly learn consensus hash codes. However, they unconsciously neglect private semantic information from each modality. To this end, we first learn modality-specific representationsHt to maximally preserve the intrinsic semantic correlations of each modality. Then, we adopt the cosine similarity between the corresponding labels of the instances to capture instance-level semantic relationships, i.e., SI = Y ⊤Y . Further, to bridge the heterogeneity gaps across different modalities, we propose a consistency learning strategy to learn consensus hash codes B under the guidance of the supervision information. In addition, we add two strong constraints, i.e., (Ht )⊤Ht = nI and (Ht )⊤1 = 0, for the bit decorrelation and bit balance. Mathematically, we can obtain the following objective function:

where α is the trade-off parameter, r is the bit length.

Most of the existing methods adopt the pairwise similarity matrix to estimate instance-level semantic relationships.

Since multi-label multi-modal data contains more complex semantic relationships, it is difficult to mine deeper semantic information by solely relying on coarse-grained instancelevel semantic supervision SI. As shown in Fig.1, when the instance-level semantic similarity between label A and other labels is identical, we hope A to be closer to the label containing more instances. This proximity increases the likelihood of retrieving more relevant instances, leading to more accurate results. To this end, we introduce the finegrained semantics SC to capture the cluster-level semantic relationships, thereby improving the retrieval performance. Hence, we adopt label distribution to explore which labels should be in close proximity or clustered together. Specifically, we regard the labels Y as the new features and conduct the K-Means algorithm to obtain cluster distribution L = [l1, l2, · · · , ln] ∈ Rk×n . Note k is the number of clusters and li ∈ {0, 1}k×1 indicates the cluster assignment of the i-th instance. We further construct the cluster-level semantic similarity relationships by SC = L⊤L. To be specific, we have

where C(i, j) = 1 represents that the i-th label and j-th label are from the same cluster.

After constructing the instance-level and cluster-level semantic relationships, we propose a dual semantic fusion learning strategy to ensure the consensus hash codes can encode multi-level semantics. Specifically, we fuse both coarsegrained semantics and fine-grained semantics to simultaneously supervise the learning of hash codes. The joint semantics can be denoted as

We believe the cluster-level and instance-level semantics should be equally important in the joint semantic relationship. Hence, we set the adjusted factor γ to 1. Afterwards, we can obtain the overall objective function as follows:

In general, we introduce the concept of fine-grained semantics to distinguish which instance semantic relationships need to be emphasized. Fine-grained semantics can provide superior supervision information, especially when the instancelevel semantic relationships between labels are the same. As shown in Fig.1, such fine-grained supervision information ensures that more instances can be retrieved. Therefore, our DSFH exhibits the following advantages: 1) It preserves both intra-modal specificity and inter-modal consistency. 2) It reveals the complex semantic relationships in multi-label data, thereby enhancing the discriminative ability of consensus hash codes.

3.3 Optimization

Since the proposed objective function is non-convex and involves three variables, we design an alternating optimization strategy to solve this model. In other words, we update one variable while keeping the others fixed.

▶W t -Step: Fixing the other variables butW t , theW t step can be simplified as follows:

Afterwards, Eq.5 can be expressed as the trace form, i.e.,

Since the orthogonal constrain, we perform the singular value decomposition (SVD) onHt (Xt )⊤, i.e.,

where P t and Qt are the left and right singular values, respectively. Therefore, we can obtainW t as follows:

▶Ht -Step: To optimize Ht , we fix the other two variables and obtain the following sub-problem,

where Gt = W tXt + αrH(Y ⊤Y + L⊤L). To avoid the large n × n matrices (i.e., Y ⊤Y ,L⊤L) to reduce both the time and space costs, we can obtain

Then, we define U = In - 1n 1n1⊤n and compute eigendecomposition on GtU(Gt )⊤ as follows:

where Θt ∈ Rl×l , Zt ∈ Rr×l , and l denotes the rank of GtU(Gt )⊤. By adopting the Gram-Schmidt process on ˆZt , we can obtain ˜Zt ∈ Rr×(r-l) .To solve Ht , we further define J t = U(Gt )⊤Zt (Θt )-1/2 ∈ Rn×l and a random orthogonal matrix ˜ J t ∈ Rn×(r-l) . Therefore, the final optimization equation of Ht can be written as:

Notably, ˆZt , ˜Zt and ˜ J t are empty when r = l.

▶ B-Step: Removing irrelevant variables, we can optimize B as follows:

Then, Eq.14 can be represented as the trace form

Finally, we can obtain the optimal solution B, i.e.,

3.4 Out-of-Sample Extension

To achieve out-of-sample extension, we further learn modality-specific hash function F t by the learned hash codes B. Afterwards, we adopt the following linear regression, i.e.,

where λ is a hyper-parameter to avoid the trivial solution. Then, the hash function F t can be solved as follows:

Given a new query Xt q , we can directly obtain the corresponding hash codes Btq of each modality, i.e.,

where ˆXt q is the RBF kernelized features of Xt q .

3.5 Time Complexity Analysis

The time complexity of our proposed DSFH mainly depends on three steps, i.e., costs of updating W t , Ht , and B. In every iteration step, the time complexity for calculating the three variables is about O(rhn + r2h), O(rhn + rcn + r3 ), andO(rn), respectively. Therefore, the total time complexity is roughly O(rhn+r2h+rcn+r3 +rn). Since r, h, c ≪ n, the time complexity is approximately O(n), which is linear to the size of training data.

4 Experiments

4.1 Datasets

To evaluate the effectiveness of the proposed DSFH, we conduct numerous experiments on three benchmarks. MIRFlickr-25K [Huiskes and Lew, 2008] has 25,000 imagetext pairs sourced from the Flickr website. Each image and text are characterized by a 512-dimensional GIST vector and a 1386-dimensional BOWvector, respectively. In our experiments, we select the instances associated with a minimum of 20 textual labels, resulting in 20,015 instances. Further, we randomly choose 2,000 instances as the query set, while the remaining image-text pairs constitute the training set. IAPRTC12 [Escalante et al., 2010] consists of 20,000 geographical images belonging to 255 categories. The features for the images and textual descriptions are 512-dimensional GIST vectors and 2912-dimensional BOWvectors, respectively. In our experiments, we randomly select 2,000 instances from the entire dataset to form the query set, and the remaining instances are used for the training set. NUS-WIDE [Chua et al., 2009] contains 269,648 image-text pairs, where each image is semantically associated with one or more of the 81 textual labels. The image instances are represented as 500-dimensional SIFT vectors, while the textual descriptions are represented as 1000-dimensional binary tagging vectors. In our experiments, we deliberately choose 184,710 data pairs with the top 10 most frequent class labels. We randomly select 1,867 data pairs as the query set and the remains as the training set.

4.2 Baselines and EvaluationMetrics

To comprehensively evaluate our proposed DSFH, we compare DSFH with 16 competitive baselines. They are 11 shallow CMH methods (i.e., RFDH [Wang et al., 2017], LCMFH [Wang et al., 2018], MTFH [Liu et al., 2019], FCMH [Wang et al., 2022], FDDH [Liu et al., 2022], BATCH [Wang et al., 2021], EDMH [Chen et al., 2022], DAH [Zhang et al., 2023b], ALECH [Li et al., 2023], WASH [Zhang et al., 2023a] and AMSH [Luo et al., 2023]) and 5 deep CMH methods (DADH [Bai et al., 2020], MDCH [Lin et al., 2021], DMFH [Nie et al., 2021], DCMHT [Tu et al., 2022], and CMGCAH [Ou et al., 2023]). Following the common evaluation metrics in the CMR field, we adopt the Mean Average Precision (mAP), the Precision-Recall (PR) curve, and the Precision at Top-N (Precision@TopN) curve to show the performance of the proposed DSFH.

4.3 Experimental Setup

In our experiments, we adopt two common CMR tasks (i.e., Img2Txt and Txt2Img) to validate the performance of all competition methods. Img2Txt and Txt2Img represent using images to retrieve relevant texts and using texts to retrieve relevant images, respectively. Experimentally, we set hash lengths from 8 to 64 bits. The number of anchors for RBF is set to 1500, and the maximum iteration step is set to 10. The hyper-parameters α and λ are set to {10-3 , 10-3 }, {10-4 , 10-3 }, and {10-4 , 10-4 } for MIRFlickr-25K, IAPRTC12, and NUS-WIDE, respectively. In addition, the number of clusters k is 400, 300, and 400 for three datasets, respectively. For fair comparisons, the experimental settings of all baselines are consistent with those reported in the original papers. To comprehensively evaluate the retrieval performance, we conduct extensive experiments on a Windows server equipped with 64GB of RAM.

4.4 Comparison with Shallow Baselines

To show the effectiveness of the proposed DSFH, we compare our method with 11 shallow baselines on three benchmark datasets. The mAP scores of all methods with different hash lengths are reported in Table 1. Additionally, Fig.3 and Fig.4 depict the corresponding PR curves and the Precision@TopN curves on the NUS-WIDE dataset, respectively. Note here that we cannot perform MTFH on the NUS-WIDE dataset due to being out of memory. According to the experimental results, we can obtain the following observations: 1) Clearly, our DSFH outperforms nearly all baselines under various experimental conditions. Specifically, For Img2Txt tasks with different hash lengths, the improvement of the retrieval performance ranges from a minimum of 0.34% to a maximum of 3.76%. For Txt2Img tasks with different hash lengths, the gains of the retrieval performance are between 0.77% and 2.65%. Such improvements could be attributed to the dual semantic fusion scheme in DSFH, which can enhance the discriminative ability of hash codes.

2) DSFH boasts the largest area under the PR curves, which indicates that DSFH achieves optimal performance in all cases. Moreover, the Precision@TopN curves show that DSFH maintains high precision with different N values. Note here that DSFH appears to be inferior to AMSH in Fig.4(b). This could be because AMSH introduces the adaptive margin matrix factors on semantic labels, thereby alleviating the rigid zero-one linear regression.

3) In general, the performance of all methods on Txt2Img tasks surpasses that of Img2Txt tasks. This discrepancy could be attributed to the text features having more discriminative semantics, whereas image features are relatively more abstract. In addition, the mAP scores for all methods increase with the length of hash codes because longer hash codes can encode more information.

4.5 Comparison with Deep Baselines

To further show the performance of our DSFH, we compare it with several shallow methods with deep features and deep methods on the MIRFlickr-25K dataset. The mAP results are shown in Table 2. Specifically, the first five rows represent the latest deep methods, and the latter four rows correspond to shallow methods fed with deep features. The deep image features are 4096-dimensional vectors extracted by the pretrained CNN-F model [Chatfield et al., 2014]. In general, although DSFHcnn is not an end-to-end deep method, it still outperforms these comparison methods. This could be because DSFHcnn can encode the more semantic information of multi-label data into the discriminative hash codes.

4.6 Parameter Analysis

In this section, we perform experiments to explore the parameter sensitivity of our DSFH, i.e., α, λ, and k. α is the trade-off coefficient that balances cross-modal consistency and specificity. λ is the regularization parameter, and k is the number of clusters for K-Means. Experimentally, we set the parameters α and λ to vary from 10-4 to 102 and from 10-4 to 10-1 , respectively. In addition, we set the k value as {100, 200, 300, 400, 500}. To study the effects of these parameters, we adopt the grid search method to obtain the optimal parameters. Firstly, we employ the grid search method to vary the α and λ values. The corresponding mAP results on NUS-WIDE are shown in Fig.5. Notably, DSFH shows strong retrieval performance under a large range of α and λ values. In addition, The mAP results with various k values on NUS-WIDE are shown in Fig.6. It can be observed that the mAP curves exhibit a uni-modal trend. If the value k is too small, the clustering result could not capture the structure of the label distribution. Conversely, if the value k is too large, the clustering results may be too fine-grained, which leads to meaningless clustering redundancy. Therefore, we can choose the optimum k to obtain the best performance.

4.7 Time Cost Analysis

To further demonstrate the training efficiency of our DSFH, we show the training time of all methods with 8 and 64 bits on the three datasets in Table 3. Compared to most baselines, DSFH offers some computation advantages, mainly for two reasons: 1) Only using simple matrix operations to optimize the overall hash model. 2) Avoiding the usage of the large n × n pairwise similarity matrix. However, the training time of DSFH is slower than that of DAH and BATCH. This could be because DAH and BATCH directly learn the common latent representations of all modalities and ignore finegrained semantic information. In summary, DSFH remains highly competitive in terms of training time cost.

4.8 Convergence Analysis

We conduct the experiments to analyze the convergence properties of the proposed DSFH. Specifically, we draw the curves of the objective value and mAP scores with respect to iteration steps on NUS-WIDE in Fig.7. We can conclude the following observations: 1) The objective value decreases rapidly within the five iterations, and then converges to approximately zero and remains stable after the 10-th iteration. 2) The mAP curves of Txt2Img and Img2Txt tasks both steadily ascend with the iteration steps. Then, DSFH reaches the highest mAP score within the 10-th iteration. In general, these results demonstrate that DSFH enjoys both rapid initial convergence and subsequent stable characteristics.

4.9 Ablation Experiments

Our proposed DSFH has two primary modules, i.e., instancelevel semantic preservation (ISP) and cluster-level semantic preservation (CSP). Therefore, we meticulously design two variants, namely DSFH-w/o ISP and DSFH-w/o CSP. Compared to the original objective function in Eq.4, the variant DSFH-w/o ISP removes the L⊤L term, and the DSFH-w/o CSP removes Y ⊤Y term, respectively. As shown in Table 4, we can observe that DSFH-w/o ISP exhibits a notable performance degradation of over 10%, because instance-level semantics offer the primary supervision in hash codes learning. In addition, DSFH-w/o CSP exhibits about 1% of performance degradation, which indicates the cluster-level semantic relationship can capture finer-grained semantics in multilabels, thereby enhancing the retrieval performance.

5 Conclusion

In this paper, we propose a novel dual semantic fusion hashing (DSFH) for multi-label cross-modal retrieval. We first propose modality-specific learning and consensus hashing learning to preserve the intrinsic consistency and specificity across different modalities. Then, we utilize label distribution to construct cluster-level semantic relationships, thereby exploring finer-grained semantic similarities between all instances. Finally, we propose a dual semantic fusion learning strategy that embeds both the coarse-grained and fine-grained semantics into consensus hash codes, thereby simultaneously preserving instance-level and cluster-level semantic relationships. Extensive experiments on three benchmarks demonstrate the outstanding performance of DSFH compared with state-of-the-art CMH baselines.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant no. 62372315), the Sichuan Science and Technology Program (Grant no. 24ZDZX0007, 2023ZYD0143, 24QYCX0375, 24QYCX0354, and 24ZHSF0545), the Mianyang Science and Technology Program (Grant no. 2022ZYDF089), and the Base Strengthening Program of National Defense Science and Technology (Grant no. 2022-JCJQ-JJ-0292).