Improved Deep Unsupervised Hashing with Fine-grained Semantic Similarity Mining for Multi-Label Image Retrieval

1 Introduction

Learning to hash has gained significant attention for image retrieval because of its outstanding retrieval efficiency and low storage cost [Luo et al., 2022; Tu et al., 2019; Wang et al., 2017]. The basic principle of hashing is to compress high-dimensional data into compact binary codes while retaining their semantic similarity.

Previous hashing methods are mostly studied in the cases of supervised end-to-end training [Tu et al., 2021a; Xie et al., 2020; Tu et al., 2021b; Wang et al., 2021]. However, supervised hashing approaches are difficult to be implemented in reality due to the prohibitive cost of large-scale data annotations. Numerous deep unsupervised approaches are presented to overcome this issue and offer a cost-effective solution to practical applications, which can be mainly summarized into two categories, i.e., similarity reconstruction-based methods [Yang et al., 2018; Yang et al., 2019; Tu et al., 2020; Shen et al., 2020; Luo et al., 2021a] and self-supervised learning methods [Lin et al., 2016; Jang and Cho, 2021; Li et al., 2021]. The first type reconstructs the binary pairwise similarity of the original data based on the pretrained neural network, and then optimizes a hashing network for generating compact and similarity-preserving hash codes with the guidance of the reconstructed similarity structure. The second type usually enforces the hash code invariant to random augmentations. Typically, recent contrastive learning-based methods [Jang and Cho, 2021; Li et al., 2021; Luo et al., 2021b] propose to maximize the mutual information between the input sample and its hash code by contrasting positive pairs augmented from the same sample with negative-sampled counterparts.

∗Corresponding authors.

However, existing methods suffer from two limitations that can harm the quality of hash codes when it comes to more challenging multi-label image retrieval [Rodrigues et al., 2020; Xie et al., 2020]. On the one hand, similarity reconstruction-based methods usually define the similarity in a coarse manner, i.e., the similarities of pairwise images are usually binary. Clearly, such structure cannot reconstruct the complicated similarity relationships in multi-label datasets. In particular, when two images share more labels, their similarity should be larger. Notably, such coarse similarity structure is incapable of reflecting this ranking information, far from depicting the complicated similarity structure. Owing to unreliable guidance, these methods usually accumulate a lot of errors during hash code learning. On the other hand, the underlying assumption under contrastive learning is that different augmentations of images share the same semantics, which is usually violated in multi-label scenarios. For example, as shown in Figure 1, random cropping could result in two augmented images with different semantics, which implies a false positive pair in contrastive learning, leading to a decline of performance for multi-label datasets.

To tackle the above issues, we propose a new unsupervised hashing method termed Hashing with fine-grAined seMantic similArity miNing (HAMAN) tailed for multi-label image retrieval. The core of our method is to explore semantics from a fine-grained view for improving similarity preserving learning and contrastive learning of hash codes. To explore complex similarity relationships in datasets, we split images into patches and generate patch features by the pre-trained network. Then we reconstruct the pairwise similarity structure by matching patch features of each image pair, serving as a fine-grained guidance for learning similarity-preserving hash codes. For better contrastive learning, we measure the fine-grained pairwise similarity of deep features from the augmented pair as the pseudo-label. The pseudo-label indicates whether two augmentations have the same semantics or not, serving as a condition to guide contrastive learning for discriminative hash codes. Extensive experiments on three datasets demonstrate significant and consistent improvements of HAMAN over rival baselines for multi-label image retrieval. Our main contributions are summarized as follows:

• We propose a novel deep unsupervised hashing method termed HAMAN, which mines fine-grained semantic similarity for effective multi-label image retrieval.
• We not only explore patch features for accurate similarity reconstruction, but also eliminate false positive pairs for conditional contrastive learning, producing similarity-preserving and discriminative hash codes.
• Experiments on three multi-label datasets verify that HAMAN significantly outperforms the state-of-the-art unsupervised hashing methods.

2 RelatedWork

Deep Unsupervised Hashing. Deep unsupervised hashing methods can be mainly classified into similarity reconstruction-based methods and self-supervised learning methods. The first category constructs the semantic structure by generating the similarity graph based on the extracted deep features. SSDH [Yang et al., 2018] utilizes the Gaussian estimation to construct the semantic structure as the guide of hash code learning. DistillHash [Yang et al., 2019] enhances the semantic structure by distilling image pairs and further improves the performance. MLS3RDUH

[Tu et al., 2020] reconstructs the local semantic similarity structure based on the intrinsic manifold structure in the feature space. GLC [Luo et al., 2021a] involves both global and local semantic consistency learning by clustering and similarity mining of deep features, respectively. The second type typically enforces the hash code consistent to random augmentation [Lin et al., 2016; Jang and Cho, 2021; Li et al., 2021]. The representative method SPQ [Jang and Cho, 2021] employs the cross quantized contrastive learning based on two different augmented views of original images. To explore the local fine-grained semantics in complicated images, our HAMAN proposes both fine-grained similarity preserving and conditional contrastive learning, producing high-quality hash codes in real-world multi-label scenarios.

Contrastive Learning. Many recent works [Wu et al., 2018; Chen et al., 2020; He et al., 2020] indicate that unsupervised image representation learning has gained great improvement benefiting from the development of contrastive learning, which significantly reduces the gap with supervised pretraining. [Hadsell et al., 2006] first attempt the representation learning by contrasting positive pairs and negative pairs. SimCLR [Chen et al., 2020] adopts a simple self-supervised learning network by replacing the memory bank with elements from the same batch and achieves considerable performance on ImageNet. MoCo [He et al., 2020] constructs a dynamic and consistent dictionary that preserves the candidate keys to perform contrastive learning. Considering that hash codes is a form of representation, recent researches [Jang and Cho, 2021; Li et al., 2021] have brought contrastive learning into deep unsupervised hashing. However, random cropping could result in augmented images with different semantics for multi-label images, deteriorating the performance of contrastive learning. Compared with these works, we propose a novel conditional contrastive learning module, which leverages the prior to guide hash code learning.

3 The ProposedMethod

To begin, the formal definition of the deep unsupervised hashing task can be explained as: X = {xi}Ni=1 signifies the training set with N samples without label annotations, which is used to learn a hash function:

where xi denotes the i-th input image and bi represents the learned l-bit binary hash code. Images with similar semantic information are expected to be encoded into binary hash codes with small Hamming distances.

3.1 Framework Overview

The architecture of our hashing network G(·) is modified from VGG-F following previous work [Yang et al., 2019; Tu et al., 2020; Luo et al., 2021a]. Specifically, it is constructed by substituting a fully-connected layer with l hidden units for the last fully-connected layer in VGG-F. Our hash code learning framework consists of Fine-grained Similarity Preserving and Conditional Contrastive Learning. In the first module, a feature extractor F (·) modified from a pretrained VGG-F by removing the last fully-connected layer is adopted to construct a fine-grained similarity structure, which guides the hashing network to produce similarity-preserving hash codes. In the second module, we also use feature extractor F(·) to calculate the similarity between two views of each image, serving as the pseudo-label to facilitate conditional contrastive learning and thus improving the discriminability of hash codes. More details are illustrated in Figure 2.

3.2 Fine-grained Similarity Preserving

In this module, we first generate the fine-grained similarity structure, and train the hashing network under its guidance for generating similarity-preserving hash codes.

Fine-grained Similarity Generation

Previous methods [Tu et al., 2020; Luo et al., 2021a] typically use a pre-trained neural network to extract deep features for images and then calculate their pairwise cosine similarity. A binary similarity matrix can be produced by thresholding the similarity [Yang et al., 2018; Yang et al., 2019; Tu et al., 2020]. However, For multi-label image retrieval, there exists complex ranking information. Hence, similarities of different samples should be considered in a fine-grained view.

Inspired by recent progress in Vision Transformer [Dosovitskiy et al., 2021], we use local patch features to characterize the fine-granted semantics in multi-label images. Then, we match the local semantics for each image pair and summarize the comparisons for obtaining fine-granted similarity. Specifically, we split each image xi ∈ RH×W×C into image patches with patch size (P, P ), resulting in a sequence of patches Ii = [I 1 i , · · · , I M i ] ∈ RM×P 2×C . M = HW/P 2 is the number of patches and C is the number of channels. The feature extractor F(·) is used to generate the patch feature f m i = F(I m i ) of each I n i . To obtain the similarity of image pair (xi, xj), we first match patch features {f m i }Mm=1

of xi with patch features {f n j }Mn=1 of xj. Then we select the largest similarity as local matching scores for each patch. Finally, all the matching scores are added as a summarization. The fine-grained pairwise similarity is formulated as follows,

where sim(u, v) represents the truncated cosine similarity of two vectors, i.e., sim(u, v) =max( uT v ||u||||v|| , 0). For symmetry, we take the average of S and ST to generate the final similarity matrix. In this way, all Sij are continuous values in the interval of [0, 1] and if two images share more local semantics, they will obtain a larger similarity. Note that for the simplest case when M = 1, the fine-grained similarity is degenerated into the cosine similarity of global deep features of the whole images. When M > 1, our module can explore and match fine-granted semantics in images from a local view. As indicated in [Chen et al., 2021], we set M = 4 for capturing complete local semantics as default.

Similarity Preserving Learning

For effective image retrieval, the semantic similarities between image pairs should be well preserved. In this part, we use the fine-grained pairwise similarity structure {Sij}Ni,j=1 to guide the training process of hashing network for producing similarity-preserving hash codes. To begin, we feed the input images into hashing network G(·) to produce hash codes {bi}Ni=1 and then calculate the continuous similarity matrix of hash codes as follows:

where Θ represents the set of hashing network parameters and the generated similarities are continuous values in [0, 1]. To preserve the semantic structure in a fine-granted view, we develop a mean square error loss to preserve the similarity of hash codes from the continuous pairwise similarity structure:

3.3 Conditional Contrastive Learning

In this module, we first generate pseudo-labels to decide the positive pairs and then introduce conditional contrastive learning for discriminative hash codes.

Pseudo-label Generation

Recently, self-supervised learning has shown promising results in various visual tasks and has been applied in deep unsupervised hashing [Chen et al., 2020; Jang and Cho, 2021]. Their basic idea is to consider two augmented views generated from each image as positive pairs and enforce them to have similar hash codes compared with negative pairs. However, random cropping could generate views with different semantics for multi-label images. Consequently, contrastive learning of hash codes cannot achieve satisfactory performance for multi-label image retrieval. To tackle this issue, we seek to generate pseudo-labels, each of which indicates whether two augmented views constitute a positive pair with the same semantics by fine-grained similarity comparison.

Specifically, we first generate a minibatch of B sampled images and produce 2B randomly transformed images {x(1) i , x(2) i }Bi=1 . Then we generate the pairwise pseudolabels {Pi}Bi=1 by calculating the similarities of the extracted features, which is formulated as follows:

where λ is a pre-defined fixed similarity threshold. Only when the similarity of the image pair is above the threshold, they can be considered as a positive pair.

Conditional Contrastive Learning

Based on the pseudo-labels, we reorganize the positive pairs in the mini-batch. For the positive pair x(1) i and x(2) i , the remaining 2(B-1) augmented views in a minibatch are considered as negative samples. Let bi ⋆ bj denote the cosine similarity of bi and bj, and the conditional contrastive learning loss is formulated as

where Z (a) i = ∑j̸=i (eb(a) i ⋆b(1) j /τ +eb(a) i ⋆b(2) j /τ ), a = 1 or 2, and τ is a temperature parameter set to 0.5 as indicated in [Chen et al., 2020]. Compared with the original contrastive learning in [Jang and Cho, 2021], we introduce additional

Algorithm 1 Training Algorithm of HAMAN

Require: Training images X = {xi}Ni=1 ; Code length l; Ensure: Parameters Θ for the hashing network G(·); Hash codes B = {bi}Ni=1 for training images.

• 1: Split each image into four patches ;
• 2: Extract patch features of all images through F(·);
• 3: Construct the fine-grained pairwise similarity matrix S by Equation 1;
• 4: repeat
• 5: Sample B images fromX and generate 2B augmented images to make up a mini-batch;
• 6: Calculate the loss by Equation 6;
• 7: Update parameters Θ of G(·) by back propagation;
• 8: until convergence
• 9: Generate hash codes B

pseudo-labels as the conditions to select positive pairs for fitting the multi-label scenarios. In this way, we remove the false positive pairs in multi-label scenarios, facilitating the generation of discriminative hash codes.

3.4 Optimization

In summary, the loss of composite hashing network learning is formulated in the mini-batch form as

where η is a balance coefficient. Notably, the parameters of the hashing network could not be updated by the standard back-propagation algorithm for the reason that the derivation of sign(·) is zero for any non-zero inputs and it is indifferentiable at zero. Accordingly, the tanh(·) is adopted to approximate the results of sign(·), and the approximate hash codes can be generated by using the tanh(G(xi)) to replace bi in Equation 2 and Equation 5. In this manner, the loss functions can be optimized by the mini-batch standard stochastic gradient descent (SGD) method. For better understanding, the entire training algorithm is described in Algorithm 1.

4 Experiments

4.1 Datasets and Setup

FLICKR25K [Huiskes and Lew, 2008] contains 25,000 images with some of the 24 labels. We randomly select 2,000 images as the query set and the remaining images are used as the retrieval set, where 5,000 images are randomly selected for training. NUS-WIDE [Chua et al., 2009] contains 269,648 images of 81 unique labels, where each image is annotated with one or more labels. Following [Jang and Cho, 2021], we use the subset with images from 21 most frequent categories. We randomly sample 100 images for each class as the query set and the rest images are used as the retrieval set, where We randomly sample 500 images for each class as the training set. MSCOCO [Lin et al., 2014] consists of 82,783 training images and 40,504 validation images. Following [Shen et al., 2020], the subset of 122,218 images from 80 categories is adopted. We randomly sample 5,000 images as the query set, and the rest images are used as the retrieval set, where 10,000 images are randomly selected for training.

HAMAN is compared with a wide variety of state-of-theart unsupervised hashing methods including two traditional shallow methods LSH [Gionis et al., 1999] and SH [Weiss et al., 2009], and nine deep learning methods SGH [Dai et al., 2017], DeepBits [Lin et al., 2016], SSDH [Yang et al., 2018], DistillHash [Yang et al., 2019], CUDH [Gu et al., 2019], MLS3RUDH [Tu et al., 2020], TBH [Shen et al., 2020], GLC [Luo et al., 2021a] and SPQ [Jang and Cho, 2021]. For fair comparison, we use raw pixels as the input for deep learningbased methods, while 4096-dimensional feature vectors extracted by the VGG-F model, which is pre-trained on dataset ImageNet, are used for two traditional shallowmethods.

The ground-truth similarity information is generated based on the ground-truth image labels. Specifically, two images are regarded as similar if they share at least one common label. We employ the Mean Average Precision (MAP), Precisionrecall curve and TopN-precision curve for evaluation. For all three datasets, we adopt MAP@5000.

We implement HAMAN using PyTorch with an NVIDIA A100 80GB Tensor Core GPU. We adopt mini-batch SGD with momentum for our model training. The mini-batch size is set to 96. The learning rates for the backbone and the added fully connected layer are fixed at 0.00001 and 0.001 respectively. We resize all training images to 224 × 224 as inputs. We adopt five widely used techniques in the following order for data augmentation: random cropping and resizing, color jitter, random grayscale, Gaussian blur and random horizontal flip [He et al., 2020]. The balance coefficient η and the similarity threshold λ are set to 1 and 0.7 as default.

4.2 Experimental Results

Table 1 shows the MAP results of HAMAN and other baselines on datasets FLICKR25K, NUS-WIDE and MSCOCO with hash code lengths of 16, 32 and 64. In addition, Figure 3 shows the Precision-recall curves and the TopN-precision curves of HAMAN and four other competitive baselines on three datasets with hash code length of 64. Accordingly, we can make the following three observations.

First, our method substantially outperforms all the competing methods with different lengths of hash codes on all three datasets. For instance, in contrast to the representative self-supervised method SPQ, HAMAN achieves an improvement of 4.4% and 4.7% for the average MAP on the dataset FLICKR25K and NUS-WIDE respectively, indicating the effectiveness of our conditional contrastive learning in positive pair selection. Second, compared with the best similarity reconstruction-based method GLC, HAMAN has 5.9%, 5.0% and 3.8% higher average MAP results on FLICKR25K, NUS-WIDE and MSCOCO, respectively. Benefiting from our fine-grained similarity exploration, HAMAN generates more accurate similarity structure of datasets and thus can improve the performance for multi-label image retrieval. Third, it can be clearly found that HAMAN achieves the best performance regarding the Precision-recall curves by comparing with several competing baselines. Fourth, as indicated by the TopN-precision curves, HAMAN outperforms other compared methods by a large margin, which demonstrates that our method can realize more effective image retrieval.

Ablation Study. To investigate the influence of different components of the proposed method, we configure several variants of HAMAN and conduct experiments on the dataset MSCOCO for comparison: (1) V1 only contains the finegrained similarity preserving module with M set to 4 by default. (2) V2 only uses the standard contrastive learning loss [Chen et al., 2020] on hash codes generated from two random augmented views of the same images. (3) V3 employs our conditional contrastive learning module with the pseudo-labels. (4) V4 involves both the fine-grained similarity preserving module and the basic contrastive learning on hash code. (5) The difference between V5 and V6 (our full model) is that V5 makes use of the fine-grained similarity preserving module with M = 9 and V6 uses the finegrained similarity preserving module with M = 4. The results are shown in Table 2. We can have similar observations on the other datasets. To begin, we find that V3 outperforms V2 by a large margin and the performance of V4 decreases greatly in contrast to V1, which indicates that basic contrastive learning could bring in a negative effect on multilabel image retrieval and meanwhile demonstrates the superiority of our conditional contrastive learning strategy. In addition, V6 achieves significant improvements over V1 and V3, revealing that both fine-grained similarity preserving module and conditional contrastive learning module can contribute to the improvement of our model and are appropriate for multilabel hash lookup retrieval. Lastly, V5 differs from V6 with respect to the number of split patches of training images. It can be inferred from the results that M = 4 is more beneficial to the fine-grained similarity generation. The potential reason is that too small patches make it hard to capture complete semantics. Hence, M is set to 4 in our model as default.

Parameter Sensitivity. We study the effect of the balance coefficient η and the similarity threshold λ through experiments on datasets FLICKR25K and MSCOCO with 64-bit hash codes. Referring to the left column of Figure 4, the performance of HAMAN is not sensitive to the value of η in the range of [1, 2.5]. The similarity threshold λ plays an important role in selecting positive pairs for conditional contrastive learning, as shown in the right column of Figure 4, our method can achieve the considerable performances with λ ranging from 0.55 to 0.75. Hense, η and λ are set as 1 and 0.7 for other experiments as default, respectively.

Visualization. We show the top 10 returned images of our method and GLC on FLICKR25K based on 64-bit hash codes in Figure 5. Benefiting from the fine-grained similarity mining and conditional contrastive learning in our method, HAMAN can successufully retrieval relevant images from the aspect of multiple semantics for multi-label image retrieval.

5 Conclusion

In this paper, we propose a novel deep unsupervised hashing method termed HAMAN for multi-label image retrieval. Our HAMAN consists of a fine-grained similarity preserving module and a conditional contrastive learning module, which explore the semantics of images from a fine-granted view. Experiments on three well-known benchmarks validate the efficacy of our approach. In future work, we expect to further extend our HAMAN to a broader range of applications such as cross-modal hashing and semi-supervised hashing.

Acknowledgements

This work was supported in part by the NSFC fund 62176077, in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 2019Bl515120055, in part by the Shenzhen Key Technical Project under Grant 2020N046, in part by the Shenzhen Fundamental Research Fund under Grant JCYJ20210324132210025, in part by Shenzhen Science and Technology Program RCBS20200714114910193 and the NSFC fund 61906162, and in part by the Medical Biometrics Perception and Analysis Engineering Laboratory, Shenzhen, China.