Semantic-Space-Intervened Diffusive Alignment for Visual Classification

1 Introduction

Cross-modal alignment aims to integrate information from different modalities to capture semantic relationships within complex data [Baltruˇsaitis et al., 2019; Dang et al., 2024b]. It utilizes more discriminative textual representations to enhance visual classification, effectively mitigating biases caused by the diversity of visual data, lighting conditions, and background noise [Dang et al., 2025; Dang et al., 2023; Dang et al., 2024c]. Although cross-modal alignment generally outperforms single-modal learning, its effectiveness can decline due to semantic ambiguities between modalities [Meng et al., 2019; Dang et al., 2024a]. This is primarily due to substantial differences in semantics, structures, or representational forms across modalities, which pose significant challenges for cross-modal alignment in handling high heterogeneity.

∗Corresponding author

To alleviate modality heterogeneity, existing methods can be broadly divided into two groups: distance metric-based alignment methods and knowledge distillation-based alignment methods. The former approach uses techniques like MaximumMean Discrepancy (MMD) [Gretton et al., 2012], Correlation Alignment (CORAL) [Sun et al., 2016], and the Wasserstein Distance [Lee et al., 2019; Deng et al., 2025] to explicitly minimize the differences in feature distributions or representations between modalities [Fu et al., 2025]. The latter approach leverages [Huo et al., 2024; Aslam et al., 2023; Xu et al., 2024] to transfer knowledge from the teacher modality to the student modality, enabling different modalities to generate similar feature representations. However, as shown in Figure 1(a), due to the significant differences in class-wise sample distributions and feature value ranges between modalities, directly applying a one-step mapping often fails to find an effective projection, failing to capture the underlying distribution of textual features fully. Furthermore, the complex relationships between modalities make it challenging for one-step alignment methods to adequately address modality-specific distributions that are highly correlated with each modality's intrinsic discriminability. Consequently, semantic ambiguity remains, leading to significant inter-class confusion in the aligned features.

To address the above challenges, we propose a SemanticSpace-Intervened Diffusive Alignment (SeDA) method, as illustrated in Figure 1(b). SeDA leverages the Markov reverse process of diffusion models to smoothly learn distributions over multiple steps, focusing on the semantic consistency of multimodal features. This approach alleviates the semantic ambiguity caused by the heterogeneity of textual and visual features, which arises from direct one-step mapping. SeDA consists of three key modules: the Diffusioncontrolled Semantic Learner (DSL), the Diffusion-controlled Semantic Translator (DST), and the Progressive Feature Interaction Network (PFIN). Specifically, we propose a Bistage optimization framework that models a modality-shared semantic space as an intermediary to enable a three-stage projection: from visual space to semantic space and then to textual space. In the early stage of the diffusion process, the DSL module progressively removes irrelevant low-level visual information by regularizing the distance between the features learned from the PFIN module and the category center of the original visual features, thereby modeling semantic space from visual representations. In the later stage, the DST module guides the transformation of semantic space to the textual representation space by measuring the distributional differences between semantic and textual features. Additionally, the PFIN module designs a diffusion network based on a cross-attention mechanism for multimodal feature fusion, ensuring the gradual introduction of textual information into mapped features through interaction at each diffusion step. SeDA not only ensures similarity between the mapped feature distribution and the textual distribution but also considers modality-independent semantic information in both visual and textual representations.

Extensive experiments are conducted on the general dataset NUS-WIDE, the domain-specific dataset VIREO Food-172, and the video dataset MSRVTT, including performance comparisons, ablation studies, in-depth analysis, and case studies. The experimental results show that SeDA models a semantic space as a bridge for the visual-to-textual projection, alleviating modality heterogeneity and achieving the transformation from visual features to textual features. The contributions of this paper are as follows:

• A novel framework SeDA is proposed to enable the alignment between visual and textual features by modeling a diffusion process. To the best of our knowledge, SeDA is the first work to use diffusion models for crossmodal alignment in classification.
• The developed diffusion process improves the visual-totextual feature projection by modeling a semantic space,
• which may capture higher-level semantic relationships between visual and textual representations, serving as an "intermediary layer" that effectively reduces the heterogeneity between different modalities.
• SeDA is a model-agnostic framework that can integrate into various visual backbones. It effectively learns the underlying distribution of textual features, providing a feasible approach for future research.

2 RelatedWork

2.1 Cross-Modal Alignment

Distance Metrics-based Alignment Methods focus on minimizing or maximizing metrics to bring data from different modalities into a common feature or decision space. Common metrics include Euclidean distance, cosine similarity, and covariance differences. For instance, Coral [Sun et al., 2016] aligns source and target features by minimizing the Frobenius norm of their feature differences. Deep Coral [Sun and Saenko, 2016] extends this by incorporating decisionlevel information and utilizing covariance matrices as a new alignment metric. Similarly, CLIP [Radford et al., 2021], ECRL [Wang et al., 2024], and TEAM [Xie et al., 2022] enhance cross-modal alignment by computing cosine similarity between visual and textual representations, effectively bridging the gap between modalities in the shared space.

Knowledge Distillation-based Alignment Methods transfer knowledge from one modality to another by employing a pre-trained teacher model to guide a student model. This ensures that both modalities generate similar representations in a shared alignment space. For example, C2KD [Huo et al., 2024] utilizes bidirectional distillation and dynamically filters samples with misaligned soft labels to improve alignment. MM-PKD [Aslam et al., 2023] employs a multimodal teacher network to guide an unimodal student network through joint cross-attention fusion. Furthermore, PKDOT [Aslam et al., 2024] leverages entropy-regularized optimal transport to distill structural knowledge, enhancing stability and robustness in the multimodal distillation process.

2.2 DiffusionModels

Diffusion models are inspired by non-equilibrium thermodynamics. DDPM [Ho et al., 2020] gradually adds noise to the data distribution and trains a neural network to learn the reverse diffusion process, thereby denoising images corrupted by Gaussian noise. Most diffusion model studies focus on generative tasks, such as image generation [Wang et al., 2023], 3D content generation [Li et al., 2024], and video generation [Ho et al., 2022]. Recently, some work has applied diffusion models to discriminative tasks [Yang et al., 2025]. For example, DiffusionDet [Chen et al., 2023] formulates object detection as a diffusion denoising process, progressively refining noisy bounding boxes into object boxes. DiffusionRet [Jin et al., 2023] models the correlation between text and video as their joint probability and approaches retrieval as a gradual generation of this joint distribution from noise. DiffuMask [Wu et al., 2023] uses text-guided cross-attention information to locate class- or word-specific regions, resulting in semantic masks for synthesized images.

3 ProblemFormulation

The goal of this research is to improve the classification performance of cross-modal alignment networks by leveraging learning using privileged information (LUPI), where textual data is available only during the training phase. The dataset is composed of paired visual data V and textual data T, with the rare but informative textual data T serving as privileged information to enhance the representation of visual data V . Specifically, the training set consists of N visualtextual pairs DN = (v1, t1), (v2, t2), ..., (vn, tn), while the test set contains only M visual samples DM = v1, v2, ..., vm. A visual encoder Ev is employed to extract visual features xv = Ev(V ), while a textual encoder Et processes the textual data to extract features xt = Et(T ). The extracted visual and textual representations are aligned in subsequent modules. Finally, the aligned features are fed into a classifier to produce the prediction ˆ C, with model performance evaluated using specific metrics.

4 Method

This study proposes a Semantic-Space-Intervened Diffusive Alignment method (SeDA), which utilizes a modality-shared semantic space as an intermediary to enable the mapping fromvisual representations to textual representations, thereby effectively mitigating the heterogeneity between different modalities, as shown in Figure 2. SeDA consists of three primary modules: the Progressive Feature Interaction Network (PFIN), the Diffusion-controlled Semantic Learner (DSL), and the Diffusion-controlled Semantic Translator (DST). The DSL module operates in the early stage of the diffusion model, learning modality-independent semantic space from the original visual distribution. The DST module works in the later stage, projecting the semantic space to the textual features space. The PFIN module progressively introduces textual information by redesigning the diffusion model's network structure. Details of these modules are described below.

4.1 Progressive Feature Interaction Network (PFIN)

In the Progressive Feature Interaction Network module, we use cross-attention mechanisms and multimodal feature fusion to enable deep interaction between visual and textual information. This ensures effective integration and alignment of visual and textual features, enhancing cross-modal representation.

Unlike vanilla diffusion models [Ho et al., 2020], which predict ϵ = ϵθ (xi, i) using a UNet, We design the Feature Interaction and Reconstruction Network to predict ˜xr = Xθ(xis , i, xv) during the training phase of SeDA.

Specifically, given the visual features xv ∈ RB×dk and the textual features xs ∈ RB×dk , where B represents the batch size and dk denotes the feature dimension. In the forward process, Gaussian noise is added step-by-step to the data xs, following a Markov process with a predefined variance schedule {βi}Ti=1 :

where x0s ∼ q(x), N(.) means a Gaussian distribution. αi = 1-βi and ¯αi = ∏ij=0 αj,

According to Equation 1, the noisy textual feature at step i in the forward process is defined as xis = √ ¯αixs+ √1 - ¯αiϵ, where ϵ ∼ N (0, I) is the added Gaussian noise. As i → +∞, xis undergoes a gradual convergence towards the standard Gaussian distribution.

The fused feature representation Ft is generated through the cross-attention mechanism, and the process can be represented as:

where xis are used as the Query(Q), and the visual features xv are used as the Key(K) and Value(V).

SinusoidalPosEmb(t) (sinusoidal position embedding function) is used to embed the time step t, obtaining the time feature vector et ∈ RB×dk :

whereW1 andW2 are weight matrices, and ϕ is the activation function.

Subsequently, we concatenate the output Ft from the attention module with the encoded diffusion time feature et as the input to the denoising decoder:

The denoising decoder is a multi-layer perceptron (MLP), which contains an intermediate layer with linear transformations and activation functions to encode the features, as well as a linear layer to compute the output distribution.

4.2 Diffusion-Controlled Semantic Learner (DSL)

Due to the heterogeneity between the visual and textual modalities, directly mapping from the visual modality to the textual modality poses challenges. To address this, we propose a bi-stage alignment strategy. In this module, we first map the visual features space to the semantic space by constraining the category centers of the visual features.

During this process, we construct the representation of ˜xr by calculating the structural consistency loss between the interactive feature ˜xr and the original input visual feature xv.

The center of the visual features for category c is denoted as µc, and similarly, the center of the interactive features µc can be computed as:

Next, the structural consistency loss is defined as:

where the first term ensures the constraint of the feature centers, and the second term measures the offset between features.

Building upon cross-modal matching alignment, to enhance the model's performance in downstream visual classification tasks, we introduce a constraint cross entropy LCE that combines visual prediction results ˆ y = softmax(ˆx0r ) with real labels y:

The overall structural consistency constraint loss can be defined as:

where α1 and β are hyperparameters used to control the contributions of the classification loss and others, respectively.

4.3 Diffusion-Controlled Semantic Translator (DST)

In the later stage of the diffusion model, the DST module focuses on learning the underlying distribution of textual features while preserving semantic information. It facilitates the transformation from the semantic space to the textual features space.

To optimize the underlying data generation distribution, which is typically achieved by minimizing the variational lower bound (VLB) of the negative log-likelihood, We follow the DDPM [Ho et al., 2020] setup and minimize the KL divergence between the two distributions q(xir |xi-1 r , x0r ) and pθ (xi-1 r |xir ):

To simplify the optimization process, we reformulate it into a Mean-Squared Error (MSE) loss function as follows:

Similar to the DSCC module, we introduce the crossentropy constraint to assist in the training of the DCT module:

where α2 and γ are hyperparameters used to control the contributions of the MSE loss and the classification loss, respectively. The LCE has been given in Equation7.

Finally, the diffusion model training process for this bistage can be expressed as follows:

where T represents the diffusion model time step and t represents the staged step.

4.4 Inference Phase of SeDA

In the inference process, the goal is to iteratively reconstruct the original data by optimizing the likelihood pθ(x0r ). The reverse process is defined by:

where µθ is the predicted mean, and ˜ βi = 1-¯αi-1 1-¯αi βi is the variance term.

According to [Song et al., 2020], µθ can be calculated using the predicted feature ˜xr from the FIRN module:

By performing a step-by-step reverse denoising operation:

the aligned feature ˆx0r is finally obtained.

Subsequently, the aligned feature ˆx0r is fed into a fully connected classifier to compute the predicted logits for the final classification.

5 Experiment

5.1 Experiment Setting

Datasets

To assess the effectiveness and generality of SeDA, we conducted experiments on image and video classification tasks across three datasets. Details are provided below.

• VIREO Food-172[Chen and Ngo, 2016]: A singlelabel dataset with 110,241 food images in 172 categories and an average of three text descriptions per image. It includes 66,071 training and 33,154 test images.
• NUS-WIDE[Chua et al., 2009]: A multi-label dataset of 203,598 images (after filtering) in 81 categories, with textual tags from a 1000-word vocabulary. It has 121,962 training and 81,636 test images.
• MSRVTT[Xu et al., 2016]: A video dataset with 10,000 YouTube clips and 200,000 captions. We used 7,010 videos for training and 2,990 for testing.

Evaluation Metric

For the VIREO Food-172 and MSRVTT datasets of the single-class prediction task, we use the accuracy of Top-1 and -5 following [Meng et al., 2019]. For the multi-label dataset NUS-WIDE, we calculate Top-1 and -5 overall precision and recall following [Wang et al., 2017; Gong et al., 2013].

Implementation Details

In this experiment, we chose Adam as the optimizer for the model, with a weight decay of 1e-4. The learning rate for all neural networks was set between 1e-4 and 5e-5. The learning rate decayed to half of its original value every four training epochs. For the loss weights mentioned in the training strategy, we selected α1 and α2 between 0.1 and 2.0, the time step T between 900 and 1500, the staged step t between 0

and 500, while β and γ were chosen from [0.5, 1.0, 1.5, 2.0]. Our experiments were conducted on a single NVIDIA Tesla V100 GPU, using PyTorch 1.10.2, and the batch size is 64.

5.2 Performance Comparison

We conducted a comprehensive comparison involving 5 visual modal backbones and 9 alignment frameworks, with results summarized in Table 1. The visual modal backbones were implemented based on the methods outlined in their respective papers, including ResNet-50 [He et al., 2016], RepVGG [Ding et al., 2021], RepMLPNet [Ding et al., 2022], ViT-B/16 [Dosovitskiy et al., 2020], and VanillaNet [Chen et al., 2024]. For the alignment framework, ViT-B/16 was used to encode the visual channel, while BERT [Devlin, 2018] was employed for the textual channel. In addition, we validated the effectiveness of our method on ResNet-50. The cross-modal alignment methods were adapted from their original designs to fit the LUPI task, including ATNet [Meng et al., 2019], CLIP [Radford et al., 2021], TEAM [Xie et al., 2022], ITA [Wang et al., 2022], SDM [Jiang and Ye, 2023], MM-PKD [Aslam et al., 2023], C2KD [Huo et al., 2024], MGCC [Wu et al., 2024], and MoMKE [Xu et al., 2024]. A fully connected single-layer network served as the classifier.

• The proposed method, SeDA, outperforms other algorithms across three datasets. This is attributed to our method's ability to learn the underlying distribution of textual features, helping alleviate semantic confusion between categories.
• SeDA is a general framework that can combine various visual backbones, such as ViT-B/16 and ResNet50, to bring them performance gains, which showcases its model-agnostic capability.
• Models in the Alignment Framework generally outperform the Visual Modal Backbone. This can be at-

tributed to the incorporation of more discriminative textual information, which effectively optimizes the visual representation.

• SeDA shows a more significant improvement in Top1 accuracy than in Top-5. This is mainly because it focuses on learning the distribution of textual features, aiming to accurately provide the most discriminative first label rather than focusing on improving the precision in retrieval ranking.

5.3 Ablation Study

This section examines the performance of different modules based on ViT-B/16 in SeDA, with the results presented in Table 2.

• On the VIREO Food-172 dataset, Diffusioncontrolled Semantic Translator (+T) module played a key role in improving Acc-1, achieving a 3.7% increase compared to the baseline method. However, there was a slight decrease in Acc-5.
• Adding the Progressive Feature Interaction Network (+I) module led to improvements across both datasets. This indicates that it (+I) further enhances feature interaction capabilities.
• Diffusion-controlled Semantic Learner (+L) module ensured a certain improvement in the Top-1 metric while achieving a more significant enhancement in the Top-5 metric. Notably, on the NUS-WIDE dataset, Rec-5 improved by 2.6% compared to the baseline method.

5.4 In-depth Analysis

Robustness of SeDA on Hyperparameters

This section evaluates the robustness of SeDA in different hyperparameters. We select the weight parameter γ, time step T and stages step t from {0.5,1.0,1.5,2.0}, {900,1200,1500,1800} and {0,300,500}. SeDA is largely insensitive to changes in hyperparameters, demonstrating strong robustness in hyperparameter selection. For γ, the model performs best when γ = 1.5 This is because lower γ values tend to rely overly on specific features at the expense of class-level information, whereas higher γ values place excessive emphasis on class information, disrupting the learning of the feature distribution space. Furthermore, the model achieves its best performance when the diffusion model time step T is set to 1500, as smaller T fails to fully learn useful information for transferring the visual modality to the text modality, while larger T may introduce additional noise. Notably, two-stage learning is beneficial, but excessively long staged step t in the second stage may damage the structural information of the original features, leading to a slight drop in performance.

The Effect of Different Methods in Each Module

To further investigate the impact of different methods in the DSL, DST, and PFIN modules on model performance, we used ViT-B/16 as the baseline and conducted experiments on the VIREO Food-172 dataset, as shown in Tabel 3. Four aspects were analyzed: training strategies, diffusion model network structures, modal interaction methods, and stage training methods. Fixing text features limits flexibility, while joint optimization improves cross-modal understanding. UNet is better suited for pixel-level learning, while MLP, which outperforms UNet, is more effective for feature-level optimization. Cross Attention proved superior for modal interaction, effectively capturing correlations. Bi-stage framework effectively integrates semantic information from visual and text features, enhancing alignment and classification capabilities.

5.5 Case Study

Performance of Visual to Textual Feature Distribution Transformation

We randomly selected a category to compare the textual and visual features of the ViT-B/16, features aligned using traditional L2-norm alignment (as shown in Figure 4(a)), and features reconstructed by our method during the diffusion model sampling process across different time steps (Figure 4 (b)). The feature distribution produced by traditional alignment methods demonstrates minimal changes, with significant discrepancies still evident between the aligned visual and textual features. This suggests that traditional methods fail to bridge the gap between the two modalities fully. In contrast, the features reconstructed using our method initially capture the semantic characteristics of visual features, gradually moving toward textual features as the sampling process progresses. This dynamic migration process ensures that visual features are effectively transformed into a semantic space consistent with textual features. Furthermore, our method not only achieves accurate mapping between the two modalities but also enhances the expressiveness of the textual features by preserving and utilizing semantic information.

Table 3: Results from the combined experiment using different strategies and constraint functions on the VIREO Food-172 dataset.

Effectiveness of Semantic Disambiguation We selected five categories with severe confusion in the Baseline model, as shown in Figure 5(a). The corresponding confusion matrix for SeDA is shown in Figure 5(b). The results clearly demonstrate that confusion between semantically similar categories is significantly reduced, leading to an effective improvement in classification performance. Additionally, Figure 5(c) presents a detailed analysis of the prediction results for specific samples. For example, in the case of the "deep fried chicken wings" sample, the model accurately predicted the correct category with high confidence after incorporating textual information. This indicates that our method effectively leverages semantic-related information to enhance the distinction between easily confused categories. On the other hand, for samples with substantial background noise (e.g., "Braised beef with brown sauce"), the Baseline model tends to misclassify them into irrelevant categories such as "noodles" or "garlic" due to noise factors like "chopsticks" and "onion" in the original image. However, SeDA successfully extracts critical features, accurately narrowing the predicted category to the semantic domain related to "beef,"

thereby significantly mitigating classification errors caused by semantic ambiguity.

6 Conclusion

To address the heterogeneity between modalities, this paper transfers the multi-step denoising process of diffusion models to the cross-modal alignment of visual representations, proposing a semantic-space-intervened diffusive alignment method (SeDA). Using semantic space as an intermediary, a bi-stage diffusion model alignment is designed: the DSL module first captures the semantic information of visual features, and the DST module gradually translates semantic features to textual features. This method effectively projects from visual representations to textual representations. Despite SeDA alleviating the heterogeneity between different modalities, there remains room for improvement in fine-grained alignment. Future work will explore fine-grained modeling with diffusion to further improve alignment accuracy, especially in complex scenarios and datasets.

Acknowledgments

This work is supported in part by the Shandong Province Excellent Young Scientists Fund Program (Overseas) (Grant no. 2022HWYQ-048).