Zero-Shot Sketch Based Image Retrieval via Modality Capacity Guidance

1 Introduction

Zero-shot learning (ZSL) aims to recognize and categorize objects or classes that have never been seen before in the training process. As a specific application of ZSL, zero-shot sketch-based image retrieval (ZS-SBIR) involves a combination of ZSL and SBIR, aiming to retrieve photos by inputting query sketches of categories not seen in the training set. Benefit from the rapid advancements of self-supervised models (e.g., DINO) and large language-vision models (e.g., CLIP), the performance of ZSL-SBIR has improved by large margins. These models have been all trained on large-scale datasets, equiping them with generalized vision features, which play a vital role in ZSL-SBIR. For instance, based on DINO, some studies [Tian et al., 2022; Wang et al., 2022; Tian et al., 2023] leveraged knowledge distillation to enhance the model's ability to generalize to unseen classes and improve its discriminability to distinguish instances and categories. Other studies attemp to adapt CLIP for SBIR, leveraging its vision-semantic alignment capability for effective knowledge transfer. For example, [Sain et al., 2023a] and [Dong et al., 2023] proposed to use prompt learning and adapters to adapt CLIP text encoder to ZSL-SBIR while preserving its generalizability.

∗Corresponding author: tracy.mok@polyu.edu.hk

In the early work of image retrieval, statistical knowledge, such as distribution characteristics, is often utilized to design hand-crafted features [Stricker and Swain, 1994; Brunelli and Mich, 2001]. Such overall distribution characteristics, however, have been largely overlooked in recent deep learning-based model training. We argue that a good understanding of representation statistics is essential for model learning too. In the context of ZS-SBIR, existing loss functions, such as triplet loss and InfoNCE loss, focus solely on the relationship between individual samples without considering the overall distribution characteristics. To address this issue, we introduce in this paper a new concept of modality capacity and design a new loss according to the law of large numbers (LLN) to guide the feature learning process. More specifically, the modality capacity examines the effectiveness of a representation space (see Figure 1) by calculating the averaged inter-class similarity among samples within the same modality. With the modality capacity constraint loss defined, we can drive the modality capacity towards an optimal state, resulting in more balanced decision boundaries. This assists the model to self adjusted to learning more robust and generalizes, in particularly when inter-class similarity is either very small or too large, where existing models fail to learn discriminative representations or the learning is biased towards a specific category. Therefore, our method can facilitate representation learning for different modalities and, finally, improve the performance of ZS-SBIR. The main contribution of this paper can be summarized:

• To our best knowledge, our work is the first to consider the effectiveness of population during the ZS-SBIR learning process.
• A modality capacity is introduced to measure the effectiveness of a representation space for retrieval.
• A novel modality capacity constraint loss is designed to guide the model to learn more robust and generalized representations by constraining the modality capacity.
• Without introducing extra parameters, our proposed method, when combined with triplet/InfoNCE loss, obtains an increase of 2.4%/1.8% and 24.0%/2.5% respectively in terms of mAP@200 compared to the baselines of CLIP and DINO.

2 RelatedWork

Zero shot learning. Zero-shot learning (ZSL) is a method that enables a model to classify data samples, whose classes are not present during training. Most ZSL approaches leverage semantic information (e.g., manually defined attributes [Luo et al., 2020] or word vectors [Akata et al., 2016]) to bridge the gap between both seen and unseen classes. These works usually employ an embedding or mapping function to establish a connection between the low-level visual features and their respective semantic vectors. They can be presented as 'visual → semantics' (e.g., [Chen et al., 2018]), 'semantics → visual' (e.g., [Shigeto et al., 2015]) or 'visual → latent ← semantics' (e.g., [Zhang et al., 2017]). Instead, we believe that models like CLIP and DINO, which are pre-trained on large-scale datasets, possess a comprehensive understanding of vision and can achieve ZS-SBIR when appropriate guidance is provided. In short, we focus on the relationship between 'vision (sketch)' and 'vision (photo)'.

ZS-SBIR. When combining ZSL and SIBR, research studies can be classified as vision-centric, vision-semanticalignment, or a combination of both. Vision-centric works often focus on properties within the vision domain. [Wang et al., 2021b] proposed using an image bank to bridge the domain gap. [Wang et al., 2021a] utilized vision feature norm to guide the sketch and photo alignment. Visual image generation is also a promising and widely-considered subdirection due to the success of GANs and their variants. [Ren et al., 2023] utilized CycleGAN to enhance the knowledge of feature extractor to further improve the generalization ability of ZS-SBIR. [Dutta and Akata, 2019] proposed a cycleconsistent generator leveraging semantic information for better knowledge about the unseen classes. Recently, several studies have demonstrated the superior performance of combining vision-related semantics with ViT models pre-trained on large-scale datasets. For example, [Dong et al., 2023] leveraged the CLIP text encoder and incorporated adaptors into DINO, resulting in competitive performance. [Sain et al., 2023a] proposed an approach that fine-tunes CLIP using a prompt learning mechanism. This enables the integration of vision-semantic knowledge into the ZS-SBIR problem. However, none of these methods considered high-level population distribution information, potentially resulting in a bised learning that is unsuitable for unexplored categories.

3 Method

In this section, we first describe our problem formulation and the baseline. By investigating population distribution of the learned features of the baseline model, a novel metric - modality capacity contraint loss - is then introduced and integrated with existing retrieval losses for a more effective representation learning.

3.1 ProblemFormulation

In ZS-SBIR, the dataset consists of two types of samples: sketches P and photos S, and is denoted as D = {(imgi, yi)|yi ∈ C}Ni=1 , where each sketch/photo imgi is associated with a corresponding label yi. The sketches and photos are divided into two subsets Dseen = {P seen , Sseen }

and Dunseen = {P unseen , Sunseen } for training and testing. More specifically, Dseen = {(imgi, yi)|yi ∈ C seen }N s i=1 , where N s = |P seen | + |Sseen |, imgi ∈ P seen ⋃ Sseen and C seen is the seen category set. Similarly, Dunseen = {(imgi, yi)|yi ∈ C unseen }N u i=1 , where N u = |P unseen | + |Sunseen |, imgi ∈ P unseen ⋃ Sunseen and C unseen is the unseen category set. There is no overlap between C seen and C unseen , that is C seen ∩ C unseen = ∅. Our goal is to train a model using the seen data Dseen to retrieve photos of the same category for a query sketch from the unseen data Sunseen .

3.2 Baseline ZS-SBIR Framework

We here provide an overview of our baseline framework for ZS-SBIR. Given a sketch/photo pair, a sketch/photo feature extractor is utilized to obtain the vision feature fs = Fs(s) and fp = Fp(p), where fs, fp ∈ Rc are the sketch and photo features, respectively, and c is the feature dimension. To establish our baseline, we employ either DINO or CLIP as the vision extractor, which is trained with different loss functions based on the respective characteristics of the extractor.

DINO-based. DINO is trained on large-scale image datasets in a self-supervised manner, to learn discriminative visual representations. Recent studies have shown that using the visual representations directly as a common space yields good results in ZS-SBIR. As a result, we employ the triplet loss to train the model based on DINO in the visual space, without leveraging semantic information.

During training, we sample a triplet consisting of a sketch anchor s, a positive photo p+ and a negative photo p-. The objective of the triplet loss Ltri is to minimize the distance d(s, p+) between s and p+, while simultaneously maximizing the distance d(s, p-) between s and p+. Here, d(·, ·) is measured using the cosine distance. The calculation of the triplet loss can be represented as:

where µ is a hyperparameter that defines the minimum desired margin between the positive and negative samples.

CLIP-based. CLIP learns joint representations of image and text, which in turn facilitates zero-shot learning. We adopt CLIP-AT [Sain et al., 2023b] and leverage text embedding to guide the feature learning by incorporating a classification loss as follows:

where I ∈ {s, p} represents a sketch or photo, sim(·) denotes the cosine similarity function, and τ is the temperature hyperparameter. In addition, several recent studies (e.g., [Sain et al., 2023a]) have demonstrated the effectiveness of combining Ltri (Eq. 1) with CLIP. Therefore, we also adopt this setting as one of our baselines. In sum, a total of three baselines are adopted based on DINO and CLIP with triplet or classification loss.

3.3 Population Similarity Investigation

To evaluate the effectiveness of representations in the image retrieval task, the commonly used metrics are P@k and mAP@k. P@k is the proportion of retrieved top-k images that are from the same category. The mAP@k is the mean AP@k of all queris where the AP@k of each query is computed by AP@k = ∑ki=1 P@i×γ(i) N with γ(i) = 1, if the i-th ranked image is from the same category as that of the input query sketch, otherwise, γ(i) = 0, and N is the total number of relevant images. These metrics provide a comprehensive performance assessment for retrieval system. However, they are non-differentiable and can not be directly interpreted as a loss function to guide the representation learning through training. To address this issue, we propose a modality capacity to measure the effectiveness of representations. Taking into account inter-category dissimilarities, we calculate the modality capacity as the averaged similarity value of all negative pairs belonging to different categories:

where I ∈ {s, p}, s and p represent sketch and photo modality, N is the data size and c(·) represents a category of a sketch or photo. mI is a normalization factor, counting the valid number of (s, s) (or (p, p)) pairs with different categories:

Table 1 shows the calculated modality capacities of sketch and photo in three datasets using CLIP and DINO, respectively, while Figure 2 illustrated the corresponding histograms. It can be shown that the mean of inter-category similarity on sketches is higher than that on photos. The mean of inter-category similarity based on CLIP is higher than that based on DINO while the standard deviation is larger, indicating that DINO learns more discriminative visual representations than CLIP but less stable. It shows that the distributions based on the same baseline architecture are quite similar across different datasets. Hence, we can find out a optimal estimation of population similarity to guide the model learning regardless the datasets used for a modality. Table 2 shows the values of mAP@all and P@200 in the sketch to image retrieval task. As shown, the lower the value of modality capacity indicates the better the image retrieval performance.

3.4 Modality Capacity Constraint Loss

The modality capacity constraint loss is defined, aiming to utilize the proposed modality capacity to guide the feature learning of the model in the training process. To this end, we calculate the modality capacity for each batch of data and minimize this modality capacity to a certain value γI. The modality capacity constraint loss is given as:

where B is the batch size. According to the law of large numbers (LLN), when the sampling size is sufficiently large, the feature capacity on the whole dataset can also be minimized to the value γI .

γI is a hyperparameter controlling the target modality capacity value within the population I, i.e., sketch or photo. For the representation learning of different modalities, the γI should be set differently. However, as γI involves the estimation of the general modality capacity for different modalities, once this value has been found on a large dataset for a modality, it can be directly applied to other datasets of the same modality.

Total loss. Our proposed modality capacity constraint loss can be applied to different baselines for ZS-SBIR. The total loss based on DINO and CLIP is respectively as:

where λ1,2,3,4,5,6 are the hyperparameters, Ltri is the triplet loss and Lcls is the classification (or InfoNCE) loss (Eq. 2).

4 Experiments

4.1 Datasets and Experimental Settings

Dataset. We evaluated the effectiveness of our proposed modality capacity constraint loss on there widely-used benchmarks: Sketchy-Ext [Liu et al., 2017], TUBerlin-Ext [Eitz et al., 2012] and a subset of QuickDraw-Ext [Dey et al., 2019a]. Sketch Ext. [Liu et al., 2017] contains 75,471 sketches and 60,502 photos, with a total of 125 categories. The TUBerlin dataset [Eitz et al., 2012] contains 204,489 photos and 40,000 free-hand sketches with 250 categories of 80 freehand sketches each. QuickDraw Ext. [Dey et al., 2019a] contains 50 million sketches over 345 categories. Following [Dey et al., 2019b], we utilized a subset of QuickDraw Ext. [Dey et al., 2019a] composing of 330,000 sketches, 204,000 photos and 110 categories for evaluation. For data partitioning, we also followed [Dey et al., 2019b] to divide Sketch Ext. [Liu et al., 2017] into 100/104 categories for training and 25/21 categories for testing, which are denoted as 'Sketchy Ext Split 1' and 'Sketchy Ext Split 2', respectively. We utilized 25 categories from Sketch Ext. [Liu et al., 2017] and 30 categories from TUBerlin-Ext [Eitz et al., 2012] for testing and utilized the rest 100/220 categories for training. Table 4 lists the values of our γs and γp for various proposed methods.

Implementation details. All the experiments were conducted on Pytorch with 11GB Nvidia RTX 3080-Ti GPU. We used Adam optimizer to train the models with learning rates of lr = 1e - 4 , β1 = 0.9 and β2 = 0.999. The input size of the images was 224 × 224. The models were trained for 60 epochs with a batch size of 64. During the training stage, all the parameters of the models were frozen except for the layer normalization. The loss weights were set as λ1 = λ4 = 1, λ2 = λ5 = 4 and λ3 = λ6 = 8. The margin µ was set as 0.3.

4.2 Comparison with ExistingMethods

We compared our proposed method with 17 state-of-theart methods in Table 3. Our models based on CLIP and DINO both outperform the listed SOTA methods on all three datasets. Specifically, our model based on DINO ('Ours (DINO-ViT) w/ Ltri ' in Table 3) achieves the best performance on the Sketchy-Ext dataset and outperforms other DINO-based models by a large margin, in terms of all metrics on three datasets. It should be noted that all these SOTA methods utilized extra information such as text embeddings and the guidance from teacher models for model training, whereas our model only relies on the pre-trained vision model and fine-tunes it by combining the triplet loss and the proposed Lmcc . This demonstrates that our method takes full advantage of the pre-trained DINO model knowledge and adapt it for ZSL-SBIR by our proposed Lmcc .

Moreover, our method ('Ours (CLIP-ViT) w/ Lcls ') obtains better performance than the best CLIP-based method, i.e., CLIP-AT [Sain et al., 2023a], which has the same architecture as our CLIP-based model and was trained exploiting classification loss and prompt learning. Without prompt learning, our model Lmcc surpasses CLIP-AT by 5.9% and 2.2%, in terms of mAP@200 and P@200, respectively, on the Sketchy-Ext dataset (Split 2), by 1.7% and 4.1% in mAP@all and P@100, respectievly, on TUBerlin-Ext dataset, and by 13% in mAP@all on QuickDraw-Ext dataset. This demonstrates the effectiveness of our proposed Lmcc .

Figure 3 presents a qualitative comparison among three methods: our proposed method ('Ours (CLIP-ViT) w/ Lcls+mcc '), CLIP-AT, and our baseline model ('Ours (CLIPViT) w/ Lcls '). It demonstrates that our model outperforms the other methods in identifying visually similar photos against the input sketch. Specifically, when considering the cow sketch (first 3 rows in Figure 3), our method accurately retrieves the top 10 photos, while both CLIP-AT and the baseline occasionally retrieve rhinoceros photos.

4.3 Ablation Studies

To evaluate the effectiveness of our proposed loss, we conducted ablation studies to examine individual components of (i) the proposed modality capacity constraint loss Lmcc s and Lmcc p ; (ii) the effect of the proposed modality capacity constraint loss in conjunction with different retrieval losses based on CLIP and DINO; (iii) the effect of using only modality capacity constraint loss on the Sketchy-Ext dataset (Split 2).

The effect of Lmcc s and Lmcc p . Table 5 and Table 6 show the ablation study results based on CLIP with Ltri and Lcls , respectively. Our observation is that removing either Lmcc s or Lmcc p leads to a decrease on both mAP@200 and P@200 on CLIP. Moreover, the impact of removing Lmcc s is higher than Lmcc p . This indicates that improving the sketch representation is more important than photo representation. This is consistent with the observation in Table 1 that the modality capacity of sketch modality is higher than that of photo, requiring more improvement for the sketch representation.

The effect of the proposed modality capacity constraint loss in conjunction with different retrieval losses. We conducted an ablation study to investigate the effectiveness of our proposed loss function (Lmcc as a whole) when combined with various retrieval losses (such as triplet loss and InfoNCE loss) and different model architectures, as shown in Table 7. The results demonstrate that incorporating our proposed loss function significantly improves the performance of ZS-SBIR. For instance, when we introduce our loss function to the method that uses CLIP ViT with InfoNCE loss, we observe a notable increase of 3.4% in mAP@200. An interesting finding is that when the model architecture is DINO ViT, the Lcls results in a poor performance of only 0.375 in terms of mAP@200. The reason behind this could be attributed to the fact that the pre-trained DINO ViT is a fully self-supervised vision model, lacking exposure to the semantic embeddings, thereby posing extreme challenge to align the vision and semantic spaces through the supervision of the InfoNCE loss. However, after applying our proposed modality capacity constraint loss , the mAP is increased from 0.375 to 0.615, which demonstrates that our proposed loss is complementary with the InfoNCE loss and is able to improve the representation though the overall similarity guidance rather than the semantic alignment.

The effect of only using modality capacity constraint loss. Table 8 shows the comparisons results with the original pretrained models and the model fined using only our proposed modality capacity constraint loss . As it can be seen, merely using our proposed modality capacity constraint loss has also significantly improved features for ZSL-SBIR. Hence, our proposed loss can also enable the model to learn discriminative representations to distinguish images like triplet loss and InfoNCE loss. In addition, the performance increase on DINO is higher than that on CLIP using our proposed loss, because the representation from CLIP contains semantic information, affecting the visual similarity distribution and confusing the guidance.

4.4 Hyperparameter Analysis

Figure 5 shows the sensitivity analysis on performance when hyperparameters are tuned across different values. Since the pre-trained models perform worse on sketches than photos (see Table 8), our our strategy is to find out the optimal setting of γs first and then search the optimal setting of γp. As shown, when γ is set in a range of 0 to 0.3, the model can be effectively improved, meaning that it does not need careful tuning of hyperparameter γ. Figure 5 also shows that it is advantageous to constrain the inter-category mean of sketches to approximately 0.2, while maintaining the mean of photos around 0.0. This finding aligns with the characteristics of the sketch and photo modalities. For images of different categories, variations in background, angle, and lighting, leading to higher similarity between images belonging to different categories. Therefore, it is reasonable to minimize the intersimilarity to zero. However, for sketches of different categories, even if the sketch shapes are different, they still share the same background and with black lines, exhibiting a certain level of similarity. Hence, it is unreasonable to minimize the inter-similarity to zero for sketches; the inter-similarity should be minimized to a value slightly greater than zero.

Figure 6: Super Class Based Image Retrieval. Top: Baseline (CLIPViT w/ Lcls ). Bottom: Ours (CLIP-ViT w/ Lcls + Lmcc ).

4.5 Inter-Category Similarity Analysis

Table 11 displays the mean and standard deviation of intercategory similarities after fine-tuning CLIP and Figure 4 shows the density distribution comparisons of inter-category similarities of baselines and our models. Our results reveal distinct patterns for baselines utilizing respective Lcls and Ltri . Specifically, the Lcls aligns both photos and sketches to a shared semantic space, making mean of the inter-category similarities of photos and sketches becomes extremely similar (0.11 vs 0.11). On the other hand, the Ltri involves the interaction between the sketch and photo branches during the training process, allowing for different mean values (0.18 vs 0.10) to be achieved for sketch and photo modalities. When Lcls is combined with our Lmcc , the inter-category mean value of sketches increases. This is because that when the photo population distribution is constrained with a target mean, the sketch population can better preserve original knowledge from the pre-trained model due to the stabilization of the counterpart (from 0.91 to 0.35 vs from 0.91 to 0.18). The preservation of such knowledge would be advantageous for aligning photo and sketch spaces. However, when combined with Ltri , we constrain the two modalities to have highly similar population characteristics (γs = 0.1 and γp = 0.0) to achieve better overlay of photos (0.09 ± 0.28) and sketches (0.14 ± 0.20) in a common space.

5 Other Applications

Image-to-image Retrieval. we investigate whether controlling γs or γp individually can benefit the tasks of sketchto-sketch or photo-to-photo retrieval. To do so, we employ the setting of 'Ours (CLIP) w/ Lcls ' to fine-tune CLIP for retrieval . Table 9 demonstrates that using proper value of γI can yield better single-domain retrieval performance.

Super class based Photo/Sketch retrieval. In addition to traditional explicit category/text-based image retrieval methods, there is a growing need to leverage learned features for performing 'fuzzy match', which is widely accepted and applied in modern text-based retrieval systems. When it comes to image based retrieval systems, we refer such 'fuzzy match' task as super class based photo/sketch retrieval, where a super class is a description less implicit than the category. Table 12 illustrates the superior performance using our method. Figure 6 further demonstrates that our method yields more accurate result in the super class based image retrieval task.

6 Conclusions

In this paper, we investigate the modality capacity to effectively guide the feature learning process for ZS-SBIR. We propose a modality capacity constraint loss that enables control over the inter-category similarity during the training process. Our loss function seamlessly integrates with existing retrieval loss functions, including triplet loss and InfoNCE loss. Extensive experiments have been conducted to demonstrate the effectiveness of our approach, which outperforms the state-of-the-art (SOTA) methods by a significant margin. Additionally, our proposed loss function has the potential to benefit other applications, such as sketch-to-sketch, photo-tophoto, and even challenging super class-based image retrieval tasks. We will explore applying the modality capacity to other multi-modality tasks as our future work. Hopefully, this work can inspire the research community on more effective and indepth modality capacity utilization strategies.

Acknowledgements

The work described in this paper was supported, in part, by grant from the Innovation and Technology Fund (Grant number ITP/028/21TP) and The Hong Kong Polytechnic University (Project code: CD95).