Attention-guided Contrastive Hashing for Long-tailed Image Retrieval

1 Introduction

Hashing, which can use low-dimensional binary code to represent high-dimensional data, is a crucial embedding technology to improve the efficiency of retrieval and reduce the cost of storage in large-scale image retrieval. Deep hashing methods, such as [Kang et al., 2016b; Yuan et al., 2020], have achieved satisfactory performance in several benchmarks. However, the existing methods are trained under ideal data which is not common in real world, many recent studies [Zhou et al., 2020] demonstrate that we need to deal with unbalanced data.

In a long-tailed dataset, a few categories in the head occupy the vast majority of all samples, while the tail categories have only a very small amount of data. When a network is trained directly on such a dataset, the head categories will dominate, causing the model to ignore the tail categories. In the fields of image classification and object detection, many methods [Ren et al., 2020; Cai et al., 2021] have been proposed to tackle the long-tailed problem. Among them, the performance of reweighting [Lin et al., 2017], re-sampling [Kang et al., 2016a], and model-based modification methods [Zhou et al., 2020] have obtained promising results. However, these methods are not suitable for long-tailed hashing tasks.

∗Contact Author

A key problem of long-tailed hashing is the information loss caused by dimension reduction operation, witch will make hash codes less discriminative. In addition, the bias caused by long-tailed data will further affect the retrieval performance. [Chen et al., 2021a] proposed the Dynamic MetaEmbedding (DME) module to solve long-tailed hashing task by transferring knowledge from head to tail categories. However, in DME, only utilizing CE loss will cause follow disadvantages: firstly, the resulting network bias has not been effectively resolved, which may still degrade the ability of the feature extractor. secondly, only completing the classification task does not fully meet the requirement of the retrieval task. More importantly, although the cross-entropy loss based on re-weighting and re-sampling has achieved good results in tasks such as classification [Cai et al., 2021] and object detection [Lin et al., 2017], experiments show that these crossentropy methods can not improve the discrimination of hash codes [Chen et al., 2021a], this means that we should constrain the hash code to enhance the discrimination. However, the method of adopting contrastive learning to enhance the discrimination of hash codes also faces difficulties on longtailed data. In a long-tailed distribution, the data in a batch is unbalanced. As shown in Figure 1(b), the lack of tail data leads to the issue of insufficient positive and negative sample pairs, which leads to the deterioration of traditional contrastive learning and is more apparent in the small batch size, so it is not suitable for the long-tailed hashing task.

In this paper, to tackle the challenges mentioned above, we propose an Attention-guided Contrastive Hashing Network (ACHNet), which adopts an attention mechanism and contrastive learning on the hash codes to solve the longtailed retrieval. The proposed method has two key points: firstly, a Cross Attention Feature Enhancement (CAFE) module, which is established based on the plus-minus values of features, is employed to alleviate the information loss caused by feature dimension reduction and provide a more discriminative hash code for subsequent contrastive learning. Then, a Category-centered Contrastive Hashing (CCH) module is utilized to optimize the hash codes to aggregate the hash codes of each category to their center. We conduct comparative experiments on unbalanced Cifar-100 and ImageNet-100 with several benchmarks, under conditions of 32 bits, 64 bits, and 96 bits code lengths, and the results on three different Imbalance Factor (IF) conditions verify the superiority of the proposed method. Our main contributions are as follows:

• We elaborate ACHNet, a network structure including attention module and category-centered contrastive objective, which provides a better hash code for the longtailed scenario to complete the image retrieval task.
• We propose CAFE to alleviate the information loss originated from dimension reduction and produce more discriminative hash codes. Meanwhile, we exploit CCH to effectively improve the discrimination of hash code in the tail category using the category-centered contrastive loss.
• Our network is compared with multiple deep hashing methods on six benchmarks containing balanced and unbalanced datasets. The results show the superiority of our method in long-tailed image retrieval task.

2 RelatedWork

Our work is related to long-tailed data learning and hashing. Since the long-tailed hashing is still an opening task, we will introduce the related work from the following two points: long-tailed learning and deep hashing.

Long-tailed learning. In the field of visual recognition, there has been a lot of research on long-tailed learning. We introduce the existing methods from two aspects: the rebalancing methods and the model-based modification methods. In the re-balancing method, the importance of tail category in loss is improved by the re-weighting or resampling method. [Chawla et al., 2002] increases the number of tail class samples by over-sampling, [Kang et al., 2016a;

Chen et al., 2021b] adopt under-sampling to reduce the number of head class samples. Although the re-sampling method is simple, it is easy to cause overfitting or loss of key samples. [Lin et al., 2017] weights the cross-entropy loss function according to the number of category samples, [Cui et al., 2019] proposes the effective number of samples to weight samples of different categories, and [Cao et al., 2019] reduces the bias of the model to the head by setting a reasonable classification boundary. Then, [Ren et al., 2020] seeks a new paradigm to starts with softmax and proposes meta-softmax method. In the model modification method, the long-tailed learning is carried out by modifying the network or design knowledge transfer module. [Kang et al., 2019] proposes decoupling learning of classifier and feature extractor, [Zhou et al., 2020] designs a double branch network to learn positive distribution data and inverse distribution data respectively. [Wang et al., 2020] classifies samples by learning multiple expert classifiers. More recently, [Cai et al., 2021] is proposed based on [Wang et al., 2020], so that different expert classifiers can learn different categories and complete the training through the one-stage end-to-end method. [Chen et al., 2021a] as the first paper on long-tailed hashing, only cross-entropy loss is adopted and there is no constraint on hash code, although there are many mature related methods. We believe that the long-tailed issue can be better solved by constructing hash layer constraints.

Deep hashing. [Shen et al., 2015] regards deep hashing as a classification problem and obtains hash codes by training classifiers. [Gui et al., 2017] improves it by enhancing the robustness of the network and [Li et al., 2018] attempts to improve performance by using deep networks. In addition, there is another kind of method that regards hashing as a regression problem, such as [Liu et al., 2012]. In these methods, [Kang et al., 2016b] designs a unique fast discrete optimization algorithm, [Yuan et al., 2020] brings the hash code closer to the preset center, but ignores the correlation between categories, which has a particularly significant impact on the case of long-tailed data distribution. There are other kinds of methods, [Wang et al., 2015] directly optimizes nDCG metrics to obtain higher quality hash codes and [Deng et al., 2018] learn hashing functions through triples.

3 The ProposedMethod

In this section, we introduce the framework of ACHNet. We first formulate the long-tailed hashing task (Section 3.1), then we elaborate on the cross attention feature enhancement module (Section 3.2) and category-centered contrastive hashing (Section 3.3). Finally, we describe the total loss function and the training algorithm of ACHNet (Section 3.4).

3.1 ProblemFormulation

Give a set of images X = {x1, x2, . . . , xn}Nn=1 and corresponding labels Y = {y1, y2, . . . , yn}Nn=1 from C different categories, deep hashing task wants to learn a feature extractor E parameterized by θ and a hashing function H, where N is the number of all image-label pairs. First, we can obtain the features of the images f = E (xn | θ) , f ∈ Rdf through the feature extractor. Then, we adopt hashing function H to obtain the hash code b = H(f) of the corresponding feature, where b ∈ {±1}q , q refers to the hash code length.

In the case of long-tailed learning, the sample number of each category is different. Head categories account for the vast majority of all samples, and the remaining tail categories contain only a small amount of data. The imbalance factor is used to indicate the imbalance degree of long-tailed data. We assume that S = {s1, s2, . . . , sc} represents the number of images in each category, where s1 is the largest number of samples in head category, the remaining categories decreased in turn. We follow the imbalance setting and adopt Zipf 's law to set the number of samples [Chen et al., 2021a].

3.2 Cross Attention Feature Enhancement

In long-tailed hashing, two crucial issues are information loss caused by dimension reduction and head category bias caused by unbalanced distribution. In this paper, we propose two modules to solve these problems. Firstly, CAFE is used to prevent information loss and provide more discriminative hash codes.

In deep hashing, we want to utilize a low dimensional binary code b to represent an image, however, the dimensional difference between a feature and its binary code determines that this mapping is a dimension reduction operation: b ∈ Rq ← f ∈ Rdf . When high-dimensional data becomes low-dimensional data, some key information will be lost, which will make the hash codes less discriminative. To alleviate this problem, we design a cross attention feature enhancement module to predict the importance of features and provide a more discriminative hash code for the next category-centered contrastive hashing.

As shown in Figure 1(a), our module consists of three full connection layers l1, l2, and l3, and their weight matrices are W1 ∈ Rdf×df(1) , W2 ∈ Rdf×df(2) and W3 ∈ Rdfa ×df(h) , where df(1) and df(2) represent the output dimensions of l1 and l2, (df(1) = df(2) ), dfa denotes the dimension of attention feature after catenate operation, and df(h) denotes the output dimension of l3. We first obtain the output f(1), f(2) as intermediate features, which is formulated as::

Considering that the hash code is binary, i.e., {+1,-1}, we believe that constructing the attention module based on the plus-minus sign of features can effectively help the same category to obtain more similar hash codes. Therefore, in our module, the two full connection layers l1 and l2 share the same initial parameters, which will ensure that the initial intermediate features have the same sign, then we adopt the operation shown in Figure 1(a) to perform the attention mechanism. Specifically, we use the tanh activation function to obtain the sign vectors vs1 and vs2 representing plus-minus sign of f(1) and f(2). There are two reasons why we don't adopt the sign function here: Firstly, it will affect the backpropagation of the network. In face, we want to make the intermediate features interact with each other, rather than just focusing on the key areas. Secondly, the sign function can only produce discontinuous binarization results. In contrast, the output of tanh exists continuously between ±1. The attention vector α is obtained by multiplying sign vectors as:

Then, α is multiplied by the intermediate feature to obtain new feature vectors. Finally, the new features are concatenated and as input to the full connection layer l3 to generate the feature fh after applying the attention mechanism:

where Concat(·) refers to concatenate. Due to the above operation, the outputs of the two fully connected layers are very similar during training, which will make the plus part in α represent the elements with the same sign in f(1), f(2), while the minus part is just the opposite, and the value is close to 0. Therefore, fa will pay more attention to the part that determines the plus-minus sign of hash codes, which will also provide a more discriminative center for the next stage of contrastive learning.

3.3 Category-centered Contrastive Hashing

In deep hashing, the existing contrastive learning methods are not suitable for long-tailed data. When training on unbalanced datasets, the scarcity of tail data will lead to the lack of positive and negative sample pairs, which will cause the instance-level contrastive loss, such as [Chen et al., 2020] not applicable to long-tailed hashing. Meanwhile, the categorylevel contrastive loss adopted in [Yuan et al., 2020] utilizes the preset hash codebook and ignores the correlation between categories, which is particularly important for long-tail data. Therefore, to solve the above issues, we propose categorycentered contrastive hashing.

Specifically, we adopt pseudo hash code center instead of the preset hash codebook to solve maladjustment of contrastive learning on long-tailed data. The process to obtain the pseudo hash center can be formulated as:

where i represents the category, and hi represents the hash code center of the ith category. Thus, we can obtain the hash code centers of all c categories {h1 , h2 , . . . , hc}. Compared with the preset hash codebook, correlation between categories is preserved, and the hash codes of tail categories can be aggregated better. As shown in Figure 1(c), when we exploit hash code center as the positive and negative samplepair, it is guaranteed that each sample has a positive sample and several negative samples.

Because Hamming distance is calculated by binary codes, so it is non-differentiable, the distance between pseudo hash codes h cannot be measured for training. Hamming distance is calculated as follow:

Analyzing Eq. (5) geometrically, we can find that the relationship between dot product b⊤i bj and cosine similarity of bi, bjcan be interpreted as:

cosine similarity is differentiable and inversely proportional to Hamming distance, so we adopt cosine similarity to measure the similarity between hash codes. The categorycentered contrastive loss of nth image is calculated as follows:

where d(·, ·) is the cosine similarity distance, τ is the temperature parameter. The hash code will be close to the hash code center, and enough positive and negative sample pairs can also help to distinguish the tail categories.

3.4 Model Training

At the end of ACHNet, We set a classifier layer behind the hash layer to complete classification task. This setting is based on the idea that better hash codes should also have better classification results, the classifier layer will help to obtain a better hash code. The total loss function is shown as follow:

where ˆy represents the prediction label of the classifier, and β is a hyperparameter. The training algorithm of our method is shown in Alg.1, it represents a computation flow in an epoch.

4 Experiment

We conduct extensive experiments to verify the superiority of ACHNet on both balanced and long-tailed benchmarks under 32 bits, 64 bits, and 96 bits code lengths.

4.1 Datasets

Cifar-100. Cifar-100 is a dataset widely used in the field of image classification and retrieval. It contains 60000 images at 32 × 32 from 10 categories. We use 50000 images as the database, with an average of 500 images per category, and the remaining 10000 images as the query set, with 100 images per category. Then we follow Zipf's law to randomly sample the training set from database, building three unbalanced benchmarks according to IF=1, IF=50, and IF=100.

Algorithm 1 Training algorithm of ACHNet

Input: Training set D, pre-trained feature extractor E(· | θ), epoch number T, hyperparameter β, W1,W2,W3 belong to attention module G(· | θm) and hash layer (· | θh), classifier (· | θc)

Output: The proposed ACHNet

• 1: for n = 1 to N do
• 2: fa = G(E(xn | θ) | θm)
• 3: hn = (fa | θh)
• 4: end for
• 5: Compute the center of each category {h1 , h2 , . . . , hc} via Eq. (4)
• 6: for epoch minibatch X do
• 7: f(1) =W1X, f(2) =W2X
• 8: α = tanh(f(1)) ∗ tanh(f(2))
• 9: fa =W3Concat(α ∗ f(1), α ∗ f(2))
• 10: hX = (fa | θh)
• 11: ˆy = SoftMax((hX | θc))
• 12: Compute LCE and LCCL
• 13: L = βLCE(ˆy, y) +LCCL
• 14: Update network parameters through optimizer
• 15: end for
• 16: return ACHNet

ImageNet-100. ImageNet-100 is one of the most popular datasets, it contains more than 120000 images in 1000 categories. According to the setting in [Chen et al., 2021a], we select 100 categories from all 1000 categories to build an unbalanced data set. The database has 13000 images in total and 1300 images in each category, while query set has 5000 images in total and 50 images in each category , the training set is randomly selected from the database. It is worth mentioning that when IF=1, for fair comparison, we only take 100 images of each category as the training set. The details of the datasets are shown in Table 1.

4.2 Implementation Details

We evaluate the proposed method on six benchmarks. Due to the research on long-tailed hashing still being an opening task, in addition to the existing long-tailed hashing methods, we also compare ACHNet with four deep hashing methods. They are DPSH [Li et al., 2016], HashNet [Cao et al., 2017], DSDH [Li et al., 2017], CSQ [Yuan et al., 2020] and LTHNet [Chen et al., 2021a]. Reference to the previous methods, we adopt mean average precision (mAP) as the evaluation metric for all experiments.

We utilize resnet34, which has been pre-trained on ImageNet, as the backbone of all methods. To be fair, we adopt the same hyperparametric setting as [Chen et al., 2021a]. RMSprop algorithm is used to optimize model parameters. On the Cifar-100, we set the learning rate as 1e-5. On the ImageNet, we set the learning rate of the feature extractor as 1e-6 and the rest as 1e-4, the weight decay is 5e-4. we adopt the cosine annealing strategy to adjust the learning rate within each epoch, the final learning rate will be 0.01 times the initial value. Other hyperparametric are set as follows: batch size=8, β=0.2 for Cifar-100 and β=0.8 for ImageNet-100, the total number of epochs is 100.

4.3 Results and Analysis

The experimental results on the six benchmarks are shown in Table 2 and Table 3. It can be seen that ACHNet has been greatly improved on most benchmarks, which verifies the superiority of ACHNet. On Cifar-100, the retrieval performance of our method is particularly significant on unbalanced benchmarks. Under the setting of IF=100, The method achieves 3.3% to 4.3% gains on different code lengths, while under the setting of IF=50, the improvement effect is also between 2.2% and 2.9%. But on the balanced dataset, our method is slightly worse than [Chen et al., 2021a], it shows the superiority of our method on long-tailed distribution data. On the ImageNet-100 dataset, our method surpasses [Chen et al., 2021a] on three training sets with different IF settings. By analyzing the experimental results, we can find that the results of ACHNet can always obtain the maximum improvement in the case of small code length, which verifies the effectiveness of the attention module in curbing dimension reduction operation loss information. In a conclusion, compare with LTHNet on unbalanced data, our method is more suitable for the long-tailed hashing task.

4.4 Ablation Study

As mentioned above, ACHNet has two key points, the cross attention feature enhancement, and the category-centered contrastive hashing. We verify the effectiveness of the proposed method through three experimental settings as follow:

Cross attention. The cross attention feature enhancement module relies on the cross attention mechanism we conceived, which predict the importance of feature based on plusminus sign of the intermediate features. In order to verify the effectiveness of this operation, we compare CAFE with the Two Branch Module (TBM) composed of only three linear layers, CCH is used in both networks. The experiment compares the retrieval performance of the two modules on ImageNet-100 and Cifar-100 datasets and sets IF=100. The results are shown in Table 4. By analyzing the experiment results, we can see that on the two unbalanced datasets, the cross attention mechanism we proposed effectively improves the performance level of the network without adding any parameters, and the effect is more significant on ImageNet- 100, which confirms the effectiveness of CAFE.

Hash code center. Different from the previousmethods, we believe that the operation using pseudo Hash Code Center (HCC) is more effective than the Preset Hash Code Center (PHCC) under long-tailed data distribution. Therefore, we use the pseudo hash code and the preset hash code in [Yuan et al., 2020] to experiment on our long-tailed hashing network to verify the correctness of our theory. Setting IF=100 and the retrieval performance is shown in Table 5. By analyzing the results, we can find that using our pseudo hash code center can get better results, with an average improvement of 3.5% and 0.7% on Cifar-100 and ImageNet-100, respectively.

Module verification. To verify the improvement of the model by CAFE and CCH, the following four groups of comparison experiments are set: (1) Only Cross-entropy Loss (CELoss) is used without CAFE and CCH; (2) CELoss and CAFE are used without CCH; (3) CELoss and CCH are used without CAFE; (4) Attention-guided Contrastive Hashing network we proposed. The experimental results are shown in Table 6. We can find that both CAFE and CCH improve the retrieval results of the network. The results verify the effectiveness of our proposed CAFE and CCH.

4.5 Parameter Sensitivity

We observe the effect of hyperparameter β on retrieval performance through experiments. On the Cifar-100 and ImageNet100 datasets, we compare the impact of different hyperparameter settings on ACHNet under different hash code lengths and set IF=100. The experimental results are shown in Figure 2. We can see that due to the different image quality and the minimum number of samples in the tail category of the two datasets, the impact of beta is also different. On Cifar100, when we set beta = 0.2, the result is the best, and much better than other β values setting. This proves that categorycentered contrastive loss plays a more important role than CEloss on Cifar-100. In addition, by analyzing the results of the three code lengths, we find that when the code length is the largest, the information loss caused by the dimensionality reduction operation is the smallest, which makes the network more robust and minimizes the impact of β. There are differences on ImageNet-100. When β = 0.6 or β = 0.8, the performance results are similar and the best. After weighing the pros and cons, we finally chose β = 0.2 and β = 0.8 as the hyperparameter settings on the two datasets.

5 Conclusion

This paper proposes an attention-guided contrastive hashing network to solve the hashing retrieval task under the longtailed data distribution. In our method, firstly, the cross attention feature enhancement module is adopted to prevent the information loss originated by mapping high-dimensional features to hash codes. Then, the output of the hash layer is constrained by category-centered contrast learning, which solves the problem of imbalance between positive and negative sample pairs of head and tail classes. Through a large number of comparative experiments, we verify the superiority of the proposed CAFEM and CCH, while the retrieval results on six benchmarks also show that the hashing retrieval performance of ACHNet is the state-of-the-art method on long-tailed data.

Acknowledgments

Our work was supported in part by the National Natural Science Foundation of China under Grant 62132016, Grant 62171343, and Grant 62071361, in part by Key Research and Development Program of Shaanxi under Grant 2021ZDLGY01-03, and in part by the Fundamental Research Funds for the Central Universities ZDRC2102.