Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priors

1 Introduction

Over the years, there has been a keen interest in the representation learning capabilities of generative models, given their proficiency in generating vivid images. Consistent with the perspective that a generative model must attain the semantic understanding ability to produce high-fidelity samples. Early works [Vincent et al., 2008; Vincent et al., 2010] have demonstrated that generative models can be employed for discriminative tasks through non-trivial methods.

∗Corresponding author

1 Our model is named after vermouth. Indeed, the properties of diffusion and blending of wines and herbs in vermouth share similarities with our design.

Recently, diffusion models have emerged with stunning performance in the image generation field, creating realistic and incredibly detailed images [Dhariwal and Nichol, 2021; Ho and Salimans, 2022; Ho et al., 2020]. Specifically, largescale text-to-image diffusion models [Rombach et al., 2022; Parmar et al., 2023] can seamlessly integrate and modify semantic information in an end-to-end manner, which allows the synthesis of images featuring diverse objects, scenes, and styles [Zhang et al., 2023b; Brooks et al., 2023]. Given this phenomenon, we deem that large text-to-image diffusion models, such as SD [Rombach et al., 2022] model, have acquired high-level and low-level semantic cues through extensive exposure to large-scale image-text pairs. Nevertheless, the question of how to extract the latent knowledge embedded in the diffusion process and harness this knowledge for visual perception tasks remains an unsolved challenge.

Visual perception tasks necessitate the establishment of distinct decision boundaries pθ(y|x) among categories, an objective not initially envisioned in the design of diffusion models. This achievement is typically attained through supervised learning [Liu et al., 2021; Liu et al., 2022], unsupervised contrastive learning [Caron et al., 2021], and masked image modeling followed by supervised fine-tuning [He et al., 2022; Bao et al., 2021]. In contrast, diffusion models aspire to model the inherent probability distributions pθ(x|z) of a dataset. Consequently, an inherent incompatibility exists between diffusion models and visual perception tasks. As shown in Figure 1, various methods [Zhao et al., 2023; Karazija et al., 2023] have attempted to address this issue by integrating off-the-shelf and heavy decoders or encoders with SD models. Nevertheless, the exploration of how to effectively leverage hierarchical features within SD models in a unified framework are inadequate.

The goal of this paper is to scrutinize the release of internal priors within diffusion models and their transfer to non-generative tasks in a unified manner. In contrast to approaches involving various decoders, we propose a simple yet effective method applicable to tasks with diverse granularity requirements within a unified framework. Specifically, for the input image, we introduce suitable noise and project it into the latent space of the SD model under accurate text guidance. Subsequently, ours U-head blends latent representations from different granularities, eliminating the need for designing complex and tailored decoders. Moreover, this architecture demonstrates remarkable flexibility, enabling smooth integration with the priors of Adapted-Expert, resulting in enhanced compatibility with visual perception tasks. Leveraging the flexibility and effectiveness of this module and the rich semantic features embedded within diffusion, we can seamlessly transfer the fused features to diverse tasks.

To comprehensively analyze the most effective way for unlocking the knowledge within the SD model, we investigated three downstream tasks across a range of near-realworld scenes, including ZS-SBIR, OV semantic segmentation, and few-shot classification tasks to evaluate our method. Experiment results on over 20+ datasets reveal that, despite its inherent mismatch with visual perception tasks, the SD model can still be regarded as a promising learner.

In summary, our contributions are as follows:

• To the best of our knowledge, we are the first to propose a unified framework to apply diffusion to visual perception tasks demanding different granularity semantics.
• We design a unified head capable of effectively fusing the generative priors of SD models with discriminative priors from the Adapted-Expert.
• Comprehensive experiments and analyses unveil observed rules for hyperparameters such as the noise level of latents, which can provide constructive insights and suggestions for future researches.

2 RelatedWork

2.1 VisionModels in Perception Task

The paradigm of pre-training and transfer learning has significantly advanced the domain of computer vision. In the early years, convolutional neural networks [He et al., 2016] and Vision Transformer (ViT) [Dosovitskiy et al., 2020] were viewed as standard architectures for various vision tasks, due to their exceptional perception capabilities. Nevertheless, the conventional pre-training on fixed-label datasets constrains their applicability in some complex scenarios.

Recently, CLIP [Radford et al., 2021] has garnered significant attention in various fields as its efficient visual language alignment pre-training. Some approaches leverage CLIP for tasks such as few-shot classification [Gao et al., 2023], image retrieval [Saito et al., 2023], and image segmentation [Zhou et al., 2022], which demonstrates the remarkable and rapid learning capabilities. In this paper, our focus shifts to the SD model, aiming to investigate whether it has the same ability and how to harness this potential capabilities.

2.2 Perceptual Learning with DiffusionModels

The research of transferring generative models to discriminative tasks, exemplified by BigBiGAN [Donahue and Simonyan, 2019], has sustained substantial interest over an extended period. Diffusion models [Ho et al., 2020], such as the SD model, have not only excelled in the field of image synthesis but have also drawn considerable attention in other fields. Harnessing latent features, the SDmodel exhibits versatile applications in diverse domains, including classification [Wei et al., 2023], image segmentation [Xu et al., 2023; Li et al., 2023], and depth estimation [Ji et al., 2023; Zhao et al., 2023]. However, these methods require taskspecific design such as encoders and decoders that makes them complex. In contrast, we sidestep this tailored procedure and advocate a unified framework capable of accommodating various scenarios.

We noted that the methods most akin to ours are VPD [Zhao et al., 2023] and Grounded-Diffusion [Li et al., 2023]. VPD employs tailored decoders for distinct tasks and utilizes ambiguous prompts for text guidance. GroundedDiffusion relies on an additional pre-trained grounding model and employs iterative denoising from pure Gaussian noise to get clean images. In comparison to VPD, our architecture is more lightweight, unified, and flexible. Compared with Grounded-Diffusion, our application scenarios are broader and do not necessitate additional model-assisted.

3 Method

To directly apply the SD model in non-generative domains within a unified framework, we propose Vermouth. It is devised to transfer the prior knowledge of the SD model for different visual perception tasks, eliminating the need for taskspecific designs. We will start with the preliminaries of diffusion models in Section 3.1. Subsequently, the U-head for obtaining the final representation and the Adapted-Expert to enhance compatibility between the SD priors and discriminative tasks are detailed in Section 3.3 and Section 3.4.

3.1 Preliminaries

Diffusion models constitute a category of likelihood-based models non-equilibased on brium thermodynamics. These models can characterize the data distribution p(x) by learning to reverse the forward process, which incrementally introduces noise to the data. The forward diffusion process at t-th time-step can be modeled as Markov: zt ∼ q(zt|zt-1) = N(√1-βtzt-1, (βt)I), where βt is associated with the noise schedule [Ho et al., 2020]. By employing a reparameterization trick, we can simplify this expression into a more manageable form:

Generally, diffusion models introduce noise to inputs until zT ∼ N(0, I) and samples iteratively by denoising the latent variables:

Through the proper simplification, one can predict the noise component ϵ by neural network ϵθ(xt; t) implemented by a U-net to learn how to reconstruct the input data:

where c is an additional condition such as text prompts.

The SD model employs a VQ-VAE [Van Den Oord et al., 2017] encoder to project the input into the latent space, denoted as z0 = E(x). Subsequently, a decoder is utilized to reconstruct the input, expressed as ˆx = D( ˆ z0). For latent modeling, the SD model adopts an asymmetric U-net, which is structured into three stages: down-sample, bottleneck, and up-sample, encompassing a total of 18 blocks. See the supplementary for the architectural details of the SD model.

3.2 Latent Prior in SDModel

Given the presence of multi-scale pattern in the U-net, one can extract desired features in specific blocks.

where FSD = {fi ∈ RHi×Wi×Ci |i ∼ B} and B is indexes of the specific blocks.

Recent works [Zhao et al., 2023; Xu et al., 2023] have opted for different blocks, combining with specific decoders to extract features for discriminative tasks. However, there remains a lack of clear insight regarding which blocks' output are more semantically rich. This challenge stems from the non-trivial nature of selecting specific blocks among a total of 18. We opt for a macro level to investigate the semantics within the SD model by viewing the stages as the research granularity to explore semantic information at various levels.

where stage i represent the stages index.

As a text-to-image model, the text prompt plays a crucial role in feature extraction as it serves as guidance for semantic synthesis. An intuitive approach involves using all class names s in the dataset D to form the text context:

where T is the text encoder of CLIP. However, employing the global context may lead to potential misalignment, especially considering that different inputs will contain different objects. In contrast, we enhance alignment by utilizing BLIP [Li et al., 2022] caption model as shown in Figure 2 to derive the image-aligned text prompt s = Cap(x). The multimodally pre-trained BLIP model, for a given image, excels in generating accurate descriptions while preserving semantics, encompassing both the object and its surroundings present in the image s ∈ X ⊂ D.

In addition to the output features fi at each level, the attention maps of U-net may also capture semantics. Inspired by recent work [Zhao et al., 2023] considering crossattention map as the diffusion prior equally, we are motivated to investigate the impact of the cross-attention map A = Softmax( QKT √d ), where Q and K denote the image and text hidden states. However, we find that this approach did not consistently yield positive results, as detailed in our experimental findings.

3.3 U-Head for Perception Tasks

Our U-head is designed to receive multi-scale features extracted from the SD model, facilitating the capture of language-aware visual features. As illustrated in Figure 2, to obtain global features, this module progressively fuses high-resolution features containing detailed semantics to lowresolution features with global semantics along the downsample flow:

where h is the final representation. Contrarily, detailed pixellevel features can be acquired along the up-sample flow. This approach encourages the capture of visual features at various levels of granularity, seamlessly blending coarser to finer semantic cues. Consequently, the need for customized redundant decoders 2 is mitigated to a certain extent. Then, the final output can be obtained through attention pooling or a single convolution layer noted by v = W · h.

We follow several works [Gu et al., 2021; Gao et al., 2023] that utilize the output features of the CLIP text encoder as the final classifier weight to align v with text features t = T (S), where S means prompt constructed by categories names. This strategy is beneficial to the creation of a more favorable feature space for recognition [Radford et al., 2021]. We ensure the alignment through cosine similarity:

This method offers flexibility for expanding to unseen labels by merely adjusting the input prompt S and facilitates the learning of language-aware visual representations.

3.4 Combining the Discriminative Prior

To effectively transfer learned features to discriminative tasks while ensuring compatibility, an intuitive approach is to introduce the prior knowledge of the recognition model. Leveraging the high flexibility of our U-head, we can readily fuse the diffusion prior with the discriminative prior:

where [·; ·] and Fexp means concatenation and discriminative prior respectively.

We employ ResNet-18 [He et al., 2016] to introduce discriminative prior Fexp = ResNet(x) due to its inherent possession of multi-resolution features. Thanks to the flexibility of our method, in fact, any discriminative model such as DINO-v2 [Oquab et al., 2023] can be introduced. More results can be found in supplementary material.

Furthermore, to enhance the integration between discriminative and generative prior, we incorporated an Adapter [Houlsby et al., 2019] following the discriminative model as illustrated in Figure 2. Experiments demonstrate improved performance with the adapted features. Recognizing its role in enhancing compatibility with visual perception tasks, we refer to it as the "Adapted-Expert".

2 We use the terms decoder and head to distinguish modules of different magnitudes. In general, the decoder is larger than the head and has more parameters.

3.5 Details of Training

Initially, for the input image, we employ BLIP to obtain an accurate description. Subsequently, we employ the text encoder to derive the conditions, denoted asc = T (s). Following the method proposed in the previous sections, the final representations are obtained as Equation 10. After getting the prediction according to Equation 9, the cross-entropy loss is applied for training:

4 Experiments

We verify the effectiveness of our method compared to other traditional vision models over 20+ different datasets grouped into 3 tasks, including ZS-SBIR, OV semantic segmentation, and few-shot classification. In addition, we also show our results on a faster training schedule and validate the effectiveness of each key component of our method.

4.1 Experimental Settings

Unless specified, we use SD 1-5 [Rombach et al., 2022] and freeze all the parameters of the SD model to preserve latent knowledge. For system-level comparisons, we select a few typical methods that employ different pre-training strategies, such as DINO [Caron et al., 2021] (contrastive learning), ConvNeXt [Liu et al., 2022], Swin-Transformer [Liu et al., 2021] (supervised learning), MAE [He et al., 2022] (masked image modeling), and BeiTv3 [Wang et al., 2023] (Multi-modality learning).

ZS-SBIR. Following the general setting [Liu et al., 2019], we report the mean Average Precision (mAP) of our method on three datasets: Sketchy, TU-Berlin, and QuikDraw. Given the heterogeneity of the sketch and image domain, this task serves as a robustness test. We train our model for 1 epoch, and the learning rate is set to 1e-4.

OV Semantic Segmentation. Following the general setting [Zhou et al., 2022], we train on the COCO-Stuff dataset and evaluate on the validation set of five datasets: ADE20K150 (ADE-150), ADE20K-847 (ADE-847), Pascal VOC (VOC), Pascal Context-59 (PC-59), and Pascal Context-459 (PC-459). Training our model for 120k iterations under weak data augmentation default and 8k iterations for the fast schedule, we report the mean Intersection over Union (m-IoU) at a single scale.

Few-shot Classification. Following the setting of CLIPAdapter [Gao et al., 2023], we report the 16-shot classification accuracy on 11 datasets: ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, EuroSAT, UCF101, DTD, and SUN397. We train our model for 100 epochs by default and 10 epochs for a fast training schedule.

Detailed information on training procedures and datasets can be seen in the supplementary material.

4.2 Main Results

In this section, we present the main results and compare with the counterparts under the default settings. The configuration Table 1: Main configuration of our method.

across three tasks is outlined in Table 1. Each row corresponds to a key configuration. Detailed explanations of each configuration provided in Section 4.3.

ZS-SBIR

ZS-SBIR is an appealing task as it resembles real-world scenarios. For a given sketch query xs, natural images xi of the same category are retrieved based on feature similarity, which is usually measured by mAP. This task requires models to balance domain heterogeneity between sketch and image and identify unseen categories. To fairly assess the performance, we augment the counterpart models with an Adapter and align visual features with the text features, as mentioned in Equation 9 to facilitate knowledge transfer and mitigate the semantic gap.

As demonstrated in Table 2, our method outperforms all the traditional models, indicating that internal features in the SD model are more adept at handling abstract representations of sketches. This phenomenon is illustrated in Figure 3, where the disparity between images and sketches diminishes as a result of light perturbation introduced at the appropriate time step (e.g., t = 200). This leads to a more regular distribution within the feature space.

OV Semantic Segmentation

OV semantic segmentation is the most challenging task, requiring the model trained on limited categories to achieve precise recognition of arbitrary or even noisy categories (e.g.,

.

ADE847 contains 847 categories, most of which are noisy).

For the counterpart models, we employ UperNet [Xiao et al., 2018] architecture and fine-tune the entire model by default. As shown in Table 3, we achieve comparable or even superior results compared to the tailored methods that combine specific decoder and discriminant backbone. It's noteworthy that we only have 5.9 million learnable parameters and therefore do not possess an advantage in terms of the learnable parameters. This is attributed to the fact that we only fine-tuned the "Normalization" layer (only 0.2 million parameters) in U-net. However, leveraging the U-head with efficient fusion capabilities and the excellent representation of the SD model, we still achieved a better performance against some tailored methods.

Few-Shot Classification

Few-shot classification is a challenging task that demands learning from only a few samples and generalizing efficiently across multiple scenarios. For the competitor model, we fit a linear layer (i.e., linear prob) to assess its generalization ability. Due to space constraints, we only report the top-1 accuracy in the 16-shot setting.

As shown in Table 4, our method outperforms MAE on all datasets except for UCF101, OxfordPets, and SUN397. Notably, in comparison to the MAE pre-trained on IN-21K, we achieve 16.15% improvement on IN-1K. While the overall performance does not exceed the discriminative models, comparable results are attained on specific datasets, such as FGVCAircraft and EuroSAT. This indicates that the inherent advantage of discriminative models in recognition tasks, due to paradigm differences with generative models, is diminishing without using any additional tricks.

An interesting observation reveals that SD models generally do not perform well on fine-grained datasets since the model may not fully understand the distinction of each class at a fine-grained level. Some failure cases are illustrated in the supplementary material.

4.3 Sensitive Analysis

In this section, we explore the potential factors affecting the performance of our method through ablation studies conducted on a faster schedule in ImageNet, Sketchy, and ADE20K (semantic segmentation). Table 5 reveals several intriguing properties.

ddpm schedule 42.74 54.97

(d) Inversion method. The results of different

inversion methods have slight differences.

Table 5: Sensitive analysis of potential factors affecting model performance in three tasks. The default setting is marked in bold

(b) Time steps

in classification and image retrieval tasks, indicating that latent representations at intermediate time steps t ∈ [100, 200] yield better performance. However, this phenomenon is not presented in the segmentation task. This characterization is consistent with the visualization of different time steps in Figure 4a. Specifically, as the images exhibit slight corruption, leading to the removal of fine-grained details, intermediate time steps focus more on capturing subject-level information, whereas forward time steps exhibit a perception of the entire image. Therefore, for tasks like semantic segmentation, which demand for local semantics, superior results are achieved within the time steps t ∈ [10, 100] as these steps preserve the detailed features.

Stage in U-net. As outlined in Section 3.1, the SD model comprises distinct stages, each encompassing different features. We selected individual stages or their associated combinations at stage granularity and examined their performance. As shown in Table 5a, exclusively extracting features from a single stage yields sub-optimal results due to

• (b) Prompt. An image-aligned prompt provides better guidance.

the incompleteness of semantics. However, there is a marked improvement when combining features from the mid-stage with either the up-sample or down-sample stage. This phenomenon is consistent with Figure 4b, illustrating that different stages encapsulate semantics at various granularities. Combining the mid stage with either the up-sample or downsample stages results in more comprehensive semantic fusion and, consequently, superior performance.

Text prompt. The key component of our method is the image-aligned prompt. In comparison to an empty prompt (FSD = U-net(zt, t, c = ∅)), image-aligned prompts can accurately reflect the content of the image and provide precise guidance. As shown in Table 5b, the prompt obtained by BLIP brought 12.7%, 4.52%, and 0.73% boosts on the three tasks compared to the random prompt. We attribute this to the fact that the image-aligned prompt is consistent with the usage of pairwise data pre-training for SD models, which allows us to retain the semantic bootstrapping abilities, resulting in better performance reasonably. Contrarily, an empty or random text prompt will lead to potentially harmful guidance.

CLIP token. Some methods [Zhang et al., 2023a] have demonstrated that using the output of the second-last layer in ViT performs better. In the case of the CLIP, the final layer is the projection layer. We are interested in exploring whether a similar phenomenon occurs in our model, given the significant role of the CLIP text encoder in our method. As illustrated in Table 5c, utilizing the token from the second-to-last layer (wo. proj) leads to enhancements of 5.12% and 1.21% in classification and segmentation respectively. This implies that unprojected text features exhibit clearer decision boundaries and are more suitable as final classifier weights. However, performance on Sketchy has a slight decrease, which we hypothesize to be potentially associated with the diverse perceptual characteristics of different layers in the CLIP text encoder [Gandelsman et al., 2023].

Inversion method. Recently, certain approaches [Mokady et al., 2023] have revealed that distinct inversion strategies for acquiring noisy inputs result in varied outcomes in image generation tasks. Therefore, we juxtapose two inversion methods, specifically, DDIM Inversion [Song et al., 2020], and the DDPM schedule [Ho et al., 2020]. DDIM inversion inverts latent variable z0 into its corresponding noised version zt by continuousizing ordinary differential equations (ODEs). DDPM schedule adds randomness to the latent variables under the control of βt.

As illustrated in Table 5d, the disparities between the DDPM schedule and DDIM inversion are minimal. This observation suggests that the discriminative cues within the SD model remain largely unaffected by the method of noise incorporation. However, employing the SD model to extract features from a clean image is suboptimal. This inefficiency is attributed to the fact that clean latent variables do not belong to the latent space of the SD model as it is trained to predict clean images from noised version.

Cross-attention map. Given the mechanism of injecting semantics guidance through the cross-attention in the SD model, we were prompted to investigate whether such a mechanism could also be advantageous in recognition scenarios. We select the average cross-attention map from different stages in the U-net and concatenate it with our image feature, denoted as F = [FSD; Fexp; A].

As outlined in Table 5e, only a subtle differences in performance have been observed. The cross-attention map located in the up-sample stage yields an improvement of 1.42% and 0.25% in IN-1K and ADE20k but registers a decrease of 0.23% in Sketchy. This may be caused by inaccurate attention due to the abstract nature of the sketch.

Discussion. We analyzed to identify the potential factors influencing the behavior of our model. The experimental results indicate the existence of shared optimal settings applicable to both dense prediction and global recognition tasks, such as text prompts. However, certain factors, including the crossattention and clip projection, result in inconsistent responses.

4.4 Ablation Study

We conducted ablation studies on our two pivotal designs, U-head and Adapted-expert. To assess the effectiveness of Uhead, we established a baseline using a straightforward fusion technique, involving a convolution operation on the obtained diffusion features followed by a global summation. A consistent average improvement of 0.7%, 0.58%, and 2.54% is observed when compared to the baseline, which indicates that U-head ensures both structural unity and effectiveness. Furthermore, when combined with the Adapted-Expert, an additional improvement of 0.45%, 0.4%, and 6.88% is observed across all three tasks. This indicates that our method, following knowledge fusion with the discriminative model, acquires more discriminative cues, thereby enhancing discriminative accuracy. In summary, experiments across all tasks verify the effectiveness of the crucial design in our method.

5 Conclusion

We introduce Vermouth, a simple yet effective unified framework that is designed to transfer the generative priors of diffusion models to discriminative tasks. Leveraging the BLIP model, we capture description as context conditions, preserving the inherent advantage of semantic guidance of SD models. To accommodate various downstream tasks, we introduce a lightweight head capable of seamlessly integrating discriminative and diffuse representations within a unified framework. Experiments involving multi-tasks conducted on the unified architecture illustrate the generality and efficiency. Through the careful selection of time steps and other key components, Vermouth effectively migrates the rich visual semantics of the SD mode in downstream classification, retrieval, and segmentation tasks and demonstrates a promising performance. This exploration will not only offer valuable guidance on harnessing and optimizing the potential of SD models but also inspire further research into developing more efficient frameworks.

Acknowledgements

This work was supported in part by the National Key R&D Program of China under Grant 2023YFA1008600, in part by the National Natural Science Foundation of China under Grant 62106184, Grant U22A2096, and Grant 62036007; in part by the Young Talent Fund of Association for Science and Technology, Shaanxi, China, under Grant 20230121; in part by Xi'an Science and Technology Plan Project under Grant 23GJSY0004; in part by the Technology Innovation Leading Program of Shaanxi under Grant 2022QFY01-15; and in part by the Fundamental Research Funds for the Central Universities under Grant QTZX23042 and Grant KYFZ24012.