Three Minds, One Student: Online Multi-Teacher Knowledge Distillation for Multimodal Recommenders

1 Introduction

Multimodal recommendation seeks to improve collaborative filtering by utilizing rich item-level features, such as images and text. To achieve this, current approaches often rely on complex fusion methods, such as attention-based aggregation or multi-stage fusion processes, including early or late fusion, to integrate diverse modalities into a unified representation and improve recommendation performance.

∗ Corresponding author.

https://github.com/MICLab-Rec/MTS4MM

These methods implicitly assume that different modalities can be automatically balanced through learned fusion strategies, thereby leveraging their complementary strengths. However, recent empirical evidence suggests that this assumption is often violated in practice. In particular, multimodal recommendation models often suffer from modality imbalance, in which dominant collaborative signals-most notably item IDs-overshadow auxiliary modalities during joint learning. To gain a clearer understanding of this issue, we conducted a pilot study utilizing modality-specific variants of a representative multimodal graph model, MGCN, as shown in Figure 1. The results reveal two key findings: ⟨1⟩ the jointly fused model is predominantly influenced by the ID modality, with the contributions from image and text signals being minimal in the final representation; ⟨2⟩ current fusion strategies do not effectively leverage the complementary nature of different modalities; as a result, the fused model's performance is often worse than that achieved by independently modeling and inferring from each modality.

These findings expose fundamental limitations in fusionbased multimodal recommendation systems: ⟨1⟩ heterogeneous modalities carry preference knowledge with varying strengths and reliability, which makes them vulnerable to dominance effects during joint optimization; ⟨2⟩ fusion mechanisms implicitly assume that modality representations are well-aligned and equally informative, leaving little control over how modality-specific knowledge should influence recommendation outcomes. Consequently, fusion becomes a bottleneck that suppresses important modality signals rather than facilitating effective collaboration among them.

These insights suggest that the core challenge of multimodal recommendation lies not in designing increasingly sophisticated fusion modules, but in regulating how preference knowledge from different modalities should be utilized and combined. Instead of forcing heterogeneous modalities to compete within a shared fusion space, it is more principled to treat each modality as an independent, potentially imbalanced source of preference knowledge, and to transfer what is genuinely beneficial for recommendation selectively.

We propose Multi-Teacher Single-Student Online Distillation for Multimodal Recommendation (MTS2 4MM), a new framework that rethinks multimodal recommendation by shifting from direct feature fusion to a controllable knowledge transfer approach. In this framework, we create multiple teacher models that specialize in complementary preference views, incorporating ID-based collaborative signals along with semantic information from images and texts. A unified student model selectively distills transferable knowledge from these teachers, focusing on both ranking and representation objectives. By separating modality learning from direct fusion and explicitly managing the contributions of each modality during the knowledge transfer process, MTS2 4MM effectively reduces modality dominance and enhances robustness against modality noise and misalignment. Additionally, we introduce a Modality-Specific Preference Extractor that differentiates between shared and modality-specific preference components, allowing for fair utilization of various modalities without increasing inference costs. Extensive experiments conducted on five real-world datasets show that MTS2 4MM consistently surpasses state-of-the-art baselines, achieving improvements of up to 7.22%. The contributions of this work can be summarized as follows:

• To the best of our knowledge, this work represents the first attempt to systematically reformulate multimodal recommendation as a controllable knowledge-transfer problem, rather than relying on conventional fusion-based paradigms.
• We propose MTS2 4MM, a multi-teacher single-student online distillation framework that reformulates multimodal recommendation as a selective knowledge transfer problem, explicitly regulating modality contributions without additional inference cost.
• Extensive experiments on five real-world datasets demonstrate that MTS2 4MM consistently outperforms stateof-the-art baselines and effectively mitigates the dominantmodality issue.

2 ProblemDefinition

We clarify the conventional multimodal recommendation problem and the knowledge-transfer-based multimodal recommendation problem proposed in this paper.

2.1 Conventional Multimodal Recommendation

Given a user set U, an item set I, and an interaction matrix Y, each item i is associated with multiple observed modalities X(m) i . Conventional multimodal recommendation systems aim to encode each modality and combine the representations into a single item latent space using a fusion function, such as attention or gating, using the merged item representation along with the user embedding to predict user preference:

The main goal of the conventional approach is to develop an effective fusion strategy F(·) to combine diverse modality features to enhance recommendation performance.

2.2 Knowledge Transfer for Multimodal Recommendation

Given a user set U, an item set I, and an interaction matrix Y, each item i is associated with multiple observed modalities X(m) i . We consider a set of modality-specific teacher models {T (k) }|M| k=1 , where each teacher captures a distinct source of preference knowledge (e.g., ID-based collaborative signals or semantic information from images and texts). Given a useritem pair ⟨u, i⟩, the k-th teacher produces a modality-specific preference signal:

π(k) u,i encodes the teacher's preference view over item i for user u. Our goal is to learn a unified student model S to predict user preference by selectively transferring knowledge frommultiple teachers:

where Lrec is the recommendation loss (e.g., BPR), D(·) denotes a knowledge transfer (distillation) loss that enforces consistency between the student and the k-th teacher.

Compared to conventional work that focuses on how to fuse modality features, we explicitly address what preference-relevant knowledge should be transferred, enabling an effective balance of modality contributions and mitigating the dominance-signal problem.

3 Methodology

3.1 Modality Graph Construction

Following previous studies [Mao et al., 2021; He et al., 2023; Zhou et al., 2023a], we construct modality-specific user-item graphs to encode high-order collaborative signals associated with each modality.

User-item collaborative graph. Given the user-item interaction matrix Y ∈ {0, 1}|U|×|I| , where Yu,i = 1 signifies an observed user behavior on item i, we construct the user-item behavioral graph as follows:

We explicitly decompose the degree matrix of the bipartite graph into user-side (Du = ∑i Yu,i ) and item-side (Di = ∑u Yu,i) components when describing item-to-user aggregation. Furthermore, to alleviate the scale discrepancy caused by node degrees, we apply symmetric normalization:

Modality-specific item-item graph. For each content modality m ∈ M = {v, t}, we construct a modality-specific graph based on raw modality features X(m) ∈ R|I|×dm . Specifically, we compute the cosine similarity between item features and use it to define edge weights:

where x(m) i denotes the feature vector of item i under modality m. For each item i, we retain its top-k nearest neighbors Nk(i) and construct a sparse KNN adjacency matrix A(m) , where A(m) i,j = 1 if j ∈ Nk(i), else 0. Similar to the useritem graph, we apply symmetric normalization to obtain:

where D(m) is the diagonal degree matrix of A(m) .

3.2 Modality Preference Extractor

ID preference encoder. Given the user embeddings Eu ∈ R|U|×d and item embeddings Eid i ∈ R|I|×d , we encode collaborative preference signals by propagating representations on the normalized user-item graph ˜ A. Specifically, starting from the initial embeddings E(0) ui = [Eu;Eid i ], we perform ℓ layers of graph convolution:

To capture high-order collaborative dependencies while avoiding over-smoothing, we aggregate representations from all layers via layer-wise mean pooling. The final ID-based collaborative preference representation is given by:

ID-conditioned modality encoding (image/text views). For each content modality m ∈ M = {t, v}, we construct modality-specific preference representations under the collaborative semantic space defined by item IDs. Following prior works, we first transform raw modality features into the latent space using modality-specific encoders. Specifically, we employ a Sentence Transformer Ftext(·) to encode textual content and a convolutional neural network Fimg(·) to extract high-level visual features from item images:

Rather than directly treating modality features as independent item representations, we condition them on item ID embeddings to inject collaborative preference bias. Concretely, given the item ID embedding E(m) i , we apply a feature-wise gating mechanism:

where σ(·) denotes the sigmoid function and ⊙ is the element-wise product. This operation constrains modality information to modulate ID-based collaborative semantics, instead of forming standalone modality embeddings.

To capture high-order modality-induced relations, we propagate the conditioned item representations on the modality-specific graph ˜ A(m) and aggregate representations from all propagation layers via layer-wise mean pooling:

where E(m,0) i = E(m) i . Finally, item-side modality signals are propagated to the user side through the normalized useritem incidence matrix, and we obtain the ID-conditioned modality-specific preference representation over all nodes:

3.3 Modality-Specific Teacher Construction

Based on the extracted preference representations from different views, we further construct a set of modality-specific teachers to provide complementary supervision signals. Formally, we define a teacher set {T (m) }m∈M, where each teacher corresponds to a distinct preference view. In this paper, M = {ID, V, T }, including the collaborative (ID) view and the available content views (V and T).

Teacher parameterization. Each teacher T (m) is parameterized by a view-specific preference encoder and maps a user-item pair ⟨u, i⟩ to preference-aware representations. Importantly, teachers share the same encoder architectures as the student for their corresponding views, ensuring that teacher signals are structurally aligned with student representations.

Specifically, for the collaborative teacher T ID , user and item ID embeddings are propagated on the normalized useritem graph ˜ A using the ID preference encoder described in Eq. (9), yielding collaborative preference representations:

For each content modality m ∈ {v, t}, the modality teacher T (m) adopts the ID-conditioned modality encoding described in Eqs. (11)- (13). Specifically, rawmodality features are first projected and gated by item ID embeddings, propagated on the modality-specific graph ˜ A(m) , and then aggregated from the item side to the user side via the normalized incidence matrix R, yields modality-specific preference representations:

Teacher outputs. Different modalities encode preference knowledge with modality-specific inductive biases and uneven effectiveness at the representation and ranking levels; therefore, we design each teacher to expose complementary embedding-level and ranking-level outputs.

Given a user-item pair ⟨u, i⟩, each teacher provides two types of outputs to guide student learning:

1. Embedding-level output. Each teacher exposes its preference representations eT (m) u and eT (m) i , which correspond to the user-side and item-side embeddings extracted from E(T (m) ) . These embeddings convey view-specific semantic information for representation-level distillation.
2. Rank-level output. Each teacher also induces a ranking preference over a candidate item set I. Specifically, teacher logits are computed as:

softmax(·) maps the logits over a candidate set into a normalized probability distribution, explicitly modeling relative preference ordering rather than independent scores, and τ smooths the output distribution, helping prevent it from collapsing to a one-hot vector dominated by the top-ranked item.

By modeling teachers as preference extractors specific to each view, we allow the student to selectively absorb transferable knowledge from multiple perspectives on equal footing.

3.4 Student Model Construction

Student parameterization. The student model adopts the same view-specific encoding architectures as the teachers, including the ID preference encoder and the ID-conditioned modality encoders. Given the normalized graphs constructed in Sec. 3.1, the student extracts collaborative and modalityspecific preference representations following the same propagation and aggregation procedures:

Modality Fusionwith Shared Semantic Component. After extracting collaborative and modality-specific preference representations, we first compute a shared semantic component (C) between content modalities. Specifically, we compute scalar compatibility scores using a shared query network q(·) and obtain normalized weights αV and αT via softmax:

We calculate the shared semantic component as a weighted sum of modality-specific representations. Additionally, we decompose the modality-specific representations into residual components concerning the common content representation:

These residuals are modulated by collaborative preference representations via sigmoid-based preference gates:

We further aggregate modality-aware preference representations using mean pooling and combine them with ID preferences to obtain a unified student representation:

where Avg(·) averages over existing terms and preference scores are computed as ˆ yu,i = (eSu )⊤eSi .

Although attention and gating are applied, they do not induce modality dominance. Attention separates modalitycontent into both shared content and modality-specific components, while gating conditions modality-specific signals based on collaborative context.

3.5 Training and Inference Process

Training Process. While the teachers serve as knowledge providers, the student is optimized end-to-end under recommendation and distillation objectives.

1. Student Training Objective. Following previous works [Ong and Khong, 2025; Xu et al., 2025], we use BPR [Rendle et al., 2012] as recommendation loss for the student model, for each training triplet (u, i+, i-):

where λ is the L2 regularization weight. Furthermore, we apply InfoNCE [Oord et al., 2018] to encourage consistency between different modalities for both users and items:

where InfoNCE(·) is implemented with cosine similarity.

1. Multi-Teacher Distillation. We distill knowledge from all available teachers into a unified student via two complementary objectives: rank-level and embedding-level distillation.
2. (i) Rank-level distillation. For a mini-batch of users, we construct an in-batch candidate set C as the union of positive and sampled negative items. Given a user u, the student computes logits over C and induces a soft distribution:

Each teacher k ∈ {1, . . . , |M|} produces its own distribution p(k) T (j | u) using embeddings from the same view. We uniformly ensemble all teachers to form the target distribution:

The rank-level distillation objective is defined as

We use τ 2 as a scale to counteract the gradient shrinkage caused by temperature scaling, thereby preventing the distillation signal from diminishing at high temperatures.

(ii) Embedding-level distillation. We further align student and teacher representations on candidate items. For each view k, let ES,(k) C and ET ,(k) C denote the matrices of student and teacher item embeddings restricted to C, respectively. We minimize a normalized mean squared error:

where norm(·) denotes ℓ2-normalization to mitigate scale mismatch across teachers.

1. Teacher Online Update Process. Each teacher is designed to preserve a modality-specialized preference function. To this end, we adopt a two-stage online update scheme that ensures each teacher remains a strong specialist for its own view, while being continuously aligned with the student's evolving cross-modal representations.
2. (i) Single-modality preference learning. For the k-th teacher, we optimize its view-specific encoder using the Bayesian Personalized Ranking (BPR) objective on observed training triplets (u, i+, i-):

where ˆ y(k) u,i denotes the prediction produced by the k-th teacher under its corresponding modality. This objective enables each teacher to explicitly capture modality-specific preference cues, avoiding premature dominance by ID-based collaborative signals during multi-view learning.

• (ii) Cross-modal synchronization. After updating the teacher with LTrec , we further synchronize it with the student via an exponential moving average (EMA) [Tarvainen and

Valpola, 2017]. Let Θ(k) T denote the parameters of the k-th teacher and Θ(k) S the corresponding parameter subset of the student. The update rule is given by:

where µ ∈ (0, 1) is the momentum coefficient. EMA acts as a temporally smoothed information bridge that injects the student's cross-modal knowledge into the teacher and stabilizes teacher representations across mini-batches.

The overall training loss is:

Inference Process. At inference time, we discard all teachers and only run the student forward pass to obtain eSu and eSi , and rank items by:ˆ yu,i = (eSu )⊤eSi . (31)

4 Experiments and Results

4.1 Experimental Settings

Dataset. To comprehensively and fairly evaluate the models' effectiveness, we conducted experiments using five publicly available datasets encompassing a variety of recommendation scenarios (such as movies) and different densities from the Amazon review [Lakkaraju et al., 2013] and a video platform [Ni et al., 2025]. We select five datasets of varying sizes: Baby, Sports, Clothing, Electronics, and MicroLens.

Metric. To ensure a fair comparison, we align our approach with the settings used in prior studies [Zhou and Shen, 2023; Ong and Khong, 2025]. We utilize two widely accepted evaluation metrics for top-K recommendation: Recall@K and NDCG@K. We present the average scores for all users in the test dataset for both K=10 and K=20.

Baselines. We compare MTS2 4MM against several stateof-the-art (SOTA) recommender models. (1) General Model: The following SOTA models include the matrix factorization (MF, BPR) [Rendle et al., 2012] and a graphbased model LightGCN [He et al., 2020], which are chosen for comparison. (2) Multimodal model: To ensure robust evaluation of the proposed model, several state-of-theart multimodal recommendation systems have been selected for comparison, including the matrix factorization model (VBPR [He and McAuley, 2016]) and various graph-based models (MMGCN [Wei et al., 2019], GRCN [Wei et al., 2020], DualGNN [Wang et al., 2023], SLMRec [Tao et al., 2022], BM3 [Zhou et al., 2023b], MGCN [Yu et al., 2023], FREEDOM [Zhou and Shen, 2023], LGMRec [Guo et al., 2024], DAMRS [Xv et al., 2024], SMORE [Ong and Khong, 2025]), COHESION [Xu et al., 2025].

Implementation Details. To ensure a fair comparison, we utilized the unified open-source MMRec framework [Zhou, 2023] to develop our proposed model and replicate existing recommendation systems. For each chosen baseline, the hyperparameters were adjusted according to the optimal configurations reported in the relevant published papers. Further details can be found in the supplementary material file.

4.2 Overall Performance

The comparison between MTS2 4MM and baselines is shown in Table 2. In summary, we have the following observations: (1) Overall Effectiveness. The results show that MTS2 4MM consistently outperforms all baselines in terms of Recall and NDCG across five real-world datasets. The stable improvements indicate its superior ability to capture user preference signals and recommend relevant items. (2) Comparison with Fusion-Based Multimodal Models. State-of-the-art multimodal baselines (e.g., SMORE) achieve competitive performance by integrating multiple modalities. However, their gains are often limited by modality imbalance and ineffective fusion, where dominant collaborative signals overshadow auxiliary modalities. In contrast, MTS2 4MM consistently achieves higher performance, with relative improvements of approximately 4%-7%, highlighting the limitations of direct fusion strategies.

4.3 Hyper-parameter Analysis

Figure 3 analyzes the sensitivity of MTS2 4MM to the distillation weights λrank and λemb. We observe that moderate values of both hyper-parameters consistently yield the best performance across datasets, while overly small or large weights lead to performance degradation. This trend indicates that effective multimodal learning requires a balanced integration of ranking-level preference transfer and embedding-level representation alignment. Overall, MTS2 4MM shows stable performance within a broad range of hyper-parameter settings, demonstrating its robustness to hyper-parameter choices.

4.4 Ablation Study

The ablation results of MTS2 4MM are reported in Table 3. We summarize the key observations as follows: (1) Effectiveness of multi-teacher distillation. Removing either rank-level distillation (KDrank) or embedding-level distillation (KDemb) consistently degrades performance, indicating that preserving both preference ordering and representation alignment is crucial for effective transfer of modality-specific knowledge. (2) Importance of modality-aware student modeling. Eliminating the modality-preference gating module or all components leads to substantial performance drops, demonstrating that explicitly regulating modality contributions in the student is essential for mitigating modality imbalance and achieving optimal recommendation performance.

4.5 Case Study

Figure 4 compares MTS2 4MM with its modality-specific variants on the Baby and Sports datasets. Models relying on a single modality, either during inference or throughout both training and inference, consistently underperform the full model, indicating that no individual modality is sufficient for accurate preference modeling. In contrast, MTS2 4MM achieves substantial improvements over all single-modality variants, with relative gains of up to 27.7% on Baby and 31.1% on Sports. These results highlight both the presence of modality imbalance and the effectiveness of the proposed multi-teacher distillation framework in selectively integrating complementary modality signals.

4.6 Time Complexity Analysis

We analyze the time complexity of MTS2 4MM by its main components. For the ID-based user-item graph, one GCN layer costs O(|Eui|·d/B), where |Eui| is the number of edges, d is the embedding dimension, and B is the batch size. With Lui layers, the total cost is O(Lui|Eui|d/B). For each content modality m, propagation on the item-item graph costs O(Lm|E (m) ii |d). The BPR objective incurs a cost of O(2dB) per mini-batch. Rank- and embedding-level distillation introduce a linear overhead O(dB|C|), where |C| is the in-batch candidate size. Overall, as shown in Figure 5, MTS2 4MM maintains lower computation and storage requirements than competitive baselines such as COHESION, and SMORE.

4.7 t-SNE Visualization Analysis

Figure 6 presents t-SNE visualizations of the embeddings for both the teacher and student models across different modalities. The left panel shows individual modality embeddings for the Teacher and Student models, while the right panel displays the fused embeddings (Sall) of the Student model after modality integration, we have the following key observations: ⟨1⟩ The student model successfully integrates modality-specific representations into a shared space without any single modality dominating, indicating balanced fusion; ⟨2⟩ Both teacher and student models show well-aligned embeddings, demonstrating effective cross-modal knowledge transfer through EMA; ⟨3⟩ Randomly selecting a user's two positive samples and one negative sample, the t-SNE visualization confirms that the model can effectively distinguish between positive and negative samples, highlighting its ability to learn meaningful representations.

5 RelatedWork

Early approaches treat modality features as auxiliary signals, while recent methods emphasize modality-specific modeling and graph-based integration to capture high-order relations. Self-supervised objectives are widely adopted to improve cross-view consistency and robustness to noisy or incomplete modalities, and advanced designs further mitigate fusion bias through multi-stage fusion and graph or spectral regularization, as exemplified by COHESION [Xu et al., 2025], SMORE [Ong and Khong, 2025], and PGL [Yu et al., 2025]. However, these fusion-centric paradigms tightly couple modalities within unified architectures, implicitly assuming balanced and reliable modality contributions, which often leads to dominant-modality bias and limited controllability in practice. In contrast, multimodal distillation reframes multimodal learning as a knowledge transfer problem, enabling flexible and efficient utilization of heterogeneous expertise. Although this direction remains under-explored in recommender systems, recent studies such as MMKD (PromptMM) [Wei et al., 2024] and multi-teacher routing or consistency-based distillation methods [Sun et al., 2024; Feng et al., 2025] demonstrate its potential. Our work addresses the critical gap by systematically modeling modalityspecific preference knowledge and enabling selective, controllable transfer via multi-teacher online distillation.

6 Conclusion

In this paper, we proposed MTS2 4MM, a multi-teacher single-student online distillation framework that reframes multimodal recommendation as a selective knowledge transfer problem. By treating each modality as an independent preference teacher and distilling both ranking and representation knowledge into a unified student, MTS2 4MM effectively mitigates modality imbalance and avoids dominant-modality bias. These results highlight the promise of knowledge-driven multimodal learning as a robust alternative to conventional fusion-based recommendation paradigms.

Acknowledgements

This work is supported by the Natural Science Foundation of China under Grant No. 92567204 and No. 62472196.