Aggregation Mechanism Based Graph Heterogeneous Networks Distillation

1 Introduction

Graph is a universal language for modeling complex systems and is widely used to represent entities and their relations in a variety of domains [Dwivedi et al., 2023; Hong et al., 2024a], such as social networks [Xia et al., 2022; Sharma et al., 2024], protein-protein interaction networks [Liu et al., 2020; Jha et al., 2022], citation networks [Kipf and Welling, 2016; Yang et al., 2021], etc. The success of GNNs lies in their aggregation mechanisms, which facilitate information propagation and capture complex relationships. Effective aggregation depends on the scope (how far the model looks) and the pattern(how information is combined). Early GNNs[Kipf and Welling, 2016; Jha et al., 2022] used fixed, layer-wise schemes to aggregate local neighbor information. This was enhanced by Graph Attention Networks (GATs)[Velickovic et al., 2017], which applied attention to assign dynamic weights to neighbors. More recently, Graph Transformers (GTs)[Yun et al., 2019; Chen et al., 2023] have introduced multi-head self-attention to capture global dependencies and richer interactions, overcoming limitations like over-smoothing and limited receptive fields. These developments greatly extend GNNs' capacity for complex graph mining tasks.

∗Corresponding authors.

The complex aggregation mechanism and computational heft of GNNs can complicate their integration into latencysensitive, large-scale applications. To address this, leveraging a Multi-layer Perceptron (MLP) for swift, streamlined deployment becomes appealing. A promising approach is to transfer insights from the GNN to an MLP, thus balancing potency with efficiency. GNN-to-MLP methods in graphs have recently received widespread attention and investigation. The GLNN [Zhang et al., 2022] is a pioneering work advocating the distillation of a proficiently trained GNN into an efficient MLP, adhering to the traditional logit distillation approach and prediction mimicking. NOSMOG [Tian et al., 2023] enhances the student MLP's capacity to grasp graph topology by appending structural encodings to the initial node attributes in its input layer. It also innovates by incorporating noise adversarial training as an additional module to bolster the MLP's robustness. VQGraph [Yang et al., 2024] introduces the VQVAE technique in graph processing, condensing the teacher GNN's node embeddings into a compact codebook, and leveraging the ordering of query nodes relative to this codebook as a distillation signal. Currently, Graph Transformers are increasingly taking the place of GNNs in graph mining due to their superior global attention and scalability. Distilling the one-to-all attention aggregation pattern of the GT model into an efficient MLP has not been studied yet.

However, existing GNN-to-MLP distillation methods, mostly derived from classical knowledge distillation, fail to consider the unique role of aggregation mechanisms in graph learning. A toy experiment (Fig. 1) demonstrates this gap by comparing last-layer node embedding correlations between original GCN, GAT, GT models and their distilled MLP counterparts. The results show that differences in aggregation scope (e.g., GAT vs. GT) and pattern (e.g., GCN vs. GAT) significantly affect distillation outcomes. As aggregation becomes broader and more complex, the correlation between teacher and student embeddings declines, revealing a growing representational gap that existing methods fail to bridge. These findings highlight the critical influence of aggregation mechanisms on representational alignment. Without explicitly addressing these differences, current approaches struggle to transfer structural knowledge essential for accurate student approximation. This leads to three key challenges: (1) Aggregation scope mismatch: GNNs operate over varying receptive fields, while MLPs lack such structural context, making knowledge transfer difficult. (2) Pattern misalignment: Diverse aggregation strategies yield different structural representations, complicating the learning of consistent node semantics. (3) Rigid alignment strategies: Many methods rely on direct feature or logit matching, ignoring model heterogeneity and limiting student model effectiveness.

As shown in Fig. 2, the persistent challenges highlight the necessity of our holistic framework AMEND (Aggregation Mechanism Enhanced GNN Distillation), aiming to eliminate the impact of aggregation mechanism during GNN distillation through three key components. First, we propose Multi-scope Aggregation Context Preservation for preserving local and global dependencies, ensuring the student model captures both neighborhood information and broader structural contexts. Second, we design Aggregation-enhanced Centered Kernel Alignment, which aligns the aggregation patterns between the teacher and student models using kernelized similarities, incorporating graph structural information to transfer the teacher's aggregation behavior. Third, Manifold Mixup Soft Matching, which generates mixed embeddings through shuffling and interpolation, ensuring the student model mimics the teacher's structural knowledge while standardizing logits for efficient knowledge transfer. Together, these components form a unified approach that overcomes the challenges posed by varying aggregation mechanisms, guaranteeing effective distillation. To fully evaluate the proposed method, we conduct extensive experiments on 8 regular graph datasets and 4 large-scale graph datasets to compare with state-of-the-art methods. The experimental results demonstrate the effectiveness and superiority of AMEND. Our contributions can be summarized as:

• We are the first to systematically investigate the aggregation mechanisms in GNN distillation, introducing a new perspective on structure-aware knowledge transfer.
• We propose AMEND, a novel framework designed to enhance GNN-to-MLP distillation incorporating multiscope aggregation context preservation, aggregationenhanced kernel alignment, and manifold mixup soft matching, providing a comprehensive solution to bridge the gap between GNNs and MLPs.
• We perform extensive experiments on 8 regular and 4 large-scale graph datasets, and AMEND achieves superior performance compared to state-of-the-art methods.

2 RelatedWork

2.1 Graph Neural Networks

Graph Neural Networks (GNNs) have become crucial in graph representation learning due to their message passing paradigm. Vanilla GNNs aggregate node information from the local neighborhood [Kipf and Welling, 2016; Chen et al., 2018; Hong et al., 2021; Lin et al., 2025], which limits the receptive field of node aggregation and the discrimination of information flows. To improve the discriminative of first-order neighbors, attention-based GNNs [Velickovic et al., 2017; He et al., 2023; Fountoulakis et al., 2023] achieve the heterogeneous information filtering by adaptively adjusting the weights of node aggregation. Additionally, Graph Transformers (GTs) have emerged as a powerful GNN in graph representation learning, addressing limitations of traditional message-passing GNNs such as over-smoothing, oversquashing, and difficulty in modeling long-range dependencies and heterogeneous node types. GTs leverage the attention mechanism to capture global context and complex relational patterns for high-order node aggregation. The core innovation of GTs lies in their ability to apply attention across nodes in a graph, effectively modeling interactions without being constrained by locality. Graphormer [Ying et al., 2021] extended the Transformer architecture to graph data by incorporating spatial encoding and structural encodings. GTN [Dwivedi and Bresson, 2020] expanded the node raw attributes with Laplacian eigenvectors and sent them to a vanilla transformer encoder for graph structure capturing. Integrating GCNs and GT to exploit neighborhood messages in global modeling has also received considerable attention recently. They can be categorized as combining GTs with GCNs to enhance the structure-awareness of GTs using incomplete message propagation [Wu et al., 2021; Chen et al., 2022; Ramp´aˇsek et al., 2022] and integrating the structural bias [Hussain et al., 2022; Zhao et al., 2021; Deng et al., 2024] into the self-attention matrix to improve their expressiveness. Despite their strengths, GNNs are computationally intensive and require significant resources for both training and inference [Zhang et al., 2022; Lin et al., 2024], limiting their deployment in resource-constrained environments. This motivates us to distill the aggregation mechanism into more lightweight models such as MLP, which offers faster inference and lower resource consumption while aiming to retain the performance benefits of GNNs.

2.2 Knowledge Distillation on Graphs

Knowledge Distillation (KD) [Hao et al., 2024; Hong et al., 2024b; Wang et al., 2025; Yang et al., 2023] aims to transfer the knowledge embedded in a cumbersome teacher to a simpler student. Traditional graph-to-graph KD focuses on transferring insights from larger, deeper GNNs to more compact student GNNs. Notable methods include LSP [Yang et al., 2020] and TinyGNN [Yan et al., 2020], which emphasize the preservation of localized structural patterns, and RDD [Zhang et al., 2020] enhanced the reliability of node and edge representations to ensure the student GNN accurately mirrors the teacher GNN's essential characteristics. Distilling GNN knowledge intoMLPs seeks faster reasoning, lightweight deployment, and scalability free from graph size constraints. GLNN [Zhang et al., 2022] first introduced GNN-to-MLP following vanilla predictive mimicking with the soft label from a teacher GNN.. KRD [Wu et al., 2023] employed a reliable sampling strategy to train MLPs with highly confident knowledge, ensuring robust performance despite the simplified architecture. NOSMOG [Tian et al., 2023] integrated structural and attribute features into the MLP inputs, creating a structure-aware model enhanced by adversarial feature augmentation for noise robustness. Additionally, VQGraph [Yang et al., 2024] introduced a code-based distillation method and performed sort alignment by leveraging quantization techniques. Despite the success of GNN-to-MLP, due to their similar parameter spaces and feature transformations, challenges persist when extending to different node aggregation GNNs (e.g., GT). The more complex structure of GTs, characterized by global attention mechanisms, complicates the direct application of conventional feature and predictive alignment strategies. Such methods are inadequate for effectively transferring the rich knowledge embedded in GNNs to MLPs, necessitating more sophisticated distillation techniques to bridge the heterogeneous models and ensure the student model benefits from the teacher's capabilities.

3 Methodology

3.1 ProblemDefinition

A graph can be represented by G = {V, E}, where V and E are the node set and edge set, respectively. The graph size can be dented as N, and the nodes attribute is formed with a feature matrix X ∈ RN×d , where d is the feature dimensions. The adjacency matrix A ∈ RN×N indicates the graph topology, where Ai,j = 1 denotes node vi is connected with node vj, otherwise Ai,j = 0. D is a diagonal matrix representing the degrees of the nodes, Dii = ∑j Aij. For node classification, the most important graph mining task, the prediction targets are Y ∈ RN×c , where c is the number of node classes. Given the labels YL, the goal is to predict the labels YU for unlabeled nodes. The GNN-to-MLP task aims to distill the aggregated node embedding from large GNNs to a lightweight MLP.

3.2 AggregationMechanismEnhanced Distillation

The overview framework of our proposed AMEND is shown in Figure 2. First, AMEND preserves both local and global dependencies by constructing a multi-scope context for node representations. This ensures that the student MLP can emulate the diverse aggregation ranges of the teacher model. To further enhance the transfer of aggregation patterns, AMEND explicitly aligns the aggregation patterns between the teacher and student models using a kernel-based similarity metric ACKA, ensuring that the MLP can replicate the nuanced aggregation patterns of the GNN, even without a native graph-aware mechanism. Lastly, AMEND facilitates knowledge transfer by manifold mixed embeddings, allowing the teacher's expressive capacity to be distilled into the student. This step smooths the differences in representational capabilities by combining soft matching strategies and embedding mixup to propagate knowledge effectively.

Multi-scope Aggregation Context Preservation. To tackle the challenge of aggregation scope in knowledge distillation, we propose strategies to preserve multi-scope neighborhood aggregation contexts in the GNN teacher model. This ensures effective knowledge transfer regardless of variations in the teacher model's aggregation scope. Thus, we extend the teacher model's outputs with additional embeddings that explicitly encode aggregation scope information, complementing the general embeddings generated by the teacher model.

Specifically, to effectively capture neighborhood information across multiple aggregation levels, we first construct a node propagation sequence through multi-scope message propagation. This sequence encodes aggregation information at different hops and is defined as:

where ˆA = ˜D-1/2 ˜ A ˜D-1/2 is the Laplace normalized adjacency matrix, and k is the number of hops. The resulting node embeddings are then processed by a global aggregation module to capture higher-level dependencies:

where Global(;Θ) denotes the global aggregation function effectively capturing both local and global structural patterns. It is parametrized with Θ and can be implemented by a selfattention approach.

In this way, the general GNN teacher model's output for distillation can be combined with the scope information by a self-weighted readout function, allowing adaptive aggregation by assigning different importance to various scopes:

where αi are learnable weights and Zi=H(L) ik:(i+1)k represents the embeddings corresponding to each scope in final H(L) .

Furthermore, inspired by graph transformers [Chen et al., 2023; Chen et al., 2022; Zhou et al., 2024a], we also incorporate position encoding (X ← X+Xpe) to capture the relative positions. Enhanced by random walk-based position encoding, this allows the model to consider a broader scope when performing aggregation. In the practical distillation process, position encoding is shared across teacher and student models, ensuring consistent knowledge transfer.

Aggregation-enhanced Centered Kernel Alignment. To effectively enable the student MLP to mimic the teacher GNN model's patterns to perform node aggregation, we propose Aggregation-enhanced Centered Kernel Alignment (ACKA)

as an intermediate supervision method. ACKA enhances the structure capture ability of student MLP by explicitly aligning aggregation patterns between the teacher and student models.

ACKA performs as a metric for representation similarity, which aligns aggregated representations by leveraging kernelized similarities between node embeddings. Its formulation is:

where the Hilbert-Schmidt Independence Criterion (HSIC) is empirically estimated by:

and C is the centering matrix C = I - 1b 11⊤, K, L ∈ Rb×b are kernel matrices derived from the teacher and student embeddings, representing their aggregated node dependencies.

To further enhance the alignment of aggregation patterns, ACKA chooses to integrate graph structural information into the kernel function through the Structure-Refined Gaussian Kernel (SRBF). SRBF activites the paired-wise node aggregation and informs the structure-aware similarity function determining how much influence neighboring nodes should have for knowledge distillation. SRBF kernel is defined as:

where Zi is the embedding of node i. The SRBF kernel ensures that node similarity computations are structure-aware, allowing the student to capture the adjacency-activated aggregation dynamics modeled by the teacher. To transfer the aggregation behavior of the teacher model to the student model, we introduce the ACKA loss function, which aligns the structure-enhanced kernelized representations between the teacher and student. The loss is formulated as:

where ZT ∈ Rb×d1 and ZS ∈ Rb×d2 denote the node embeddings from GNN and MLP models, respectively, and SRBF(·) is the kernel function defined in Eq. 6.

From a theoretical perspective, ACKA can be interpreted as the upper bound of Maximum Mean Discrepancy (MMD) with an additional constant term [Zhou et al., 2024b]. This implies that maximizing ACKA is equivalent to minimizing the upper bound of MMD between the teacher's aggregated embeddings and the student's transformed features. By transferring aggregation behavior rather than raw feature representations, ACKA provides an intuitive and effective distillation mechanism. Additionally, another advantage of ACKA is its dimension-independent design, which accommodates the different representation spaces of the teacher and student models. This is particularly important when the teacher (e.g., a Graph Transformer) and the student (e.g., an MLP) have significant architectural differences. By aligning kernelized aggregation patterns, ACKA ensures that the student model captures the teacher's structural aggregation patterns, even when their embeddings operate in different dimensions or scales.

Manifold Mixup Soft Matching Distillation. Considering the representative capacity differences between GNNs and MLP, we propose a novel manifold mixup soft matching distillation method in order to propagate the teacher GNN's insights of the augmented mixing node distributions into the MLP during the distillation process, improving the efficiency of knowledge transfer. The key components of this method include the generation of mixed representations through shared random shuffling and linear interpolation, and a temperature-shared KL divergence loss.

Suppose, the node embedding generated by the teacher GNN model is denoted as ZT ∈ Rb×d1 , where b and d1 are the batch size and embedding dimensions of GNN, respectively, and MLP's node embedding is ZS ∈ Rb×d2 . To create mixed representations, we apply a random shuffle to the node embedding. Let Z′T represent the shuffled version of ZT obtained by randomly permuting the node indices. The mixed embedding for the GT model is then computed as:

where λ ∈ [0, 1] is a mixing coefficient drawn from a Beta distribution, λ ∼ Beta(α, α). The same shuffle indices are used to generate Z′S from ZS, and the mixed embedding for the MLP model is:

The mixed embeddings are fed into the classification heads of both models, yielding predicted logits ˆ Y mix T and ˆ Y mix S for the GNN and MLP models, respectively:

In the prediction mimicking stage, we use the Z-score logit standard distillation technique [Sun et al., 2024] to alleviate the challenges of a lightweight student in predicting logits with a comparable range and variance as a cumbersome teacher, given the capacity gap between them. The objective function of our prediction soft alignment can presented by:

where Z(X) = X-µ σ is the Z-score standardization, ϕ denotes the SoftMax function, τ is the shared-temperature (0.5 for all experiments), and ˆ YT = gT (ZT ), ˆ YS = gS(ZS) are the clean predict logits based their node embeddings, respectively. The mixup performs as graph rewiring augmentation which can improve the robustness of distillation. Furthermore, teaching the GNN's predictions of mixed data distributions to the MLP can somewhat mitigate logit collapses due to differences in model ability to make rigid predictive alignments (a problem that has been identified in the previous literature [Hao et al., 2024; Lao et al., 2023]).

The overall loss function for student MLP training is a weighted combination of classification task loss, ACKA loss, and logit distillation loss:

where Ltask is the commonly used classification crossentropy loss, β and γ are the tread-off hyper-parameters.

3.3 AlgorithmAnalysis

The pseudo-code of our proposed AMEND framework is illustrated in Algo. 1. The time complexity mainly depends on the self-attention module and for one layer is O(b(K + 1)2d), where b,K, d denote the batch size, pre-aggregation hops, and hidden dimensions, respectively. In student training, the computational complexity primarily derives from ACKA and MLP, which can be formulated as: O(b2d + dL), where L is the student model depth. Manifold mixup is implemented through an efficient parameterization that requires only one additional forward process for the classification head, which has a negligible impact on the overall complexity. In addition, the teacher model and position encoding can be pre-trained and pre-computed offline, which improves the efficiency of our model training. In model inference, AMEND shares the same complexity with vanilla MLPs (O(dL)), and its space complexity is O(d2L), enabling fast reasoning and lightweight deployment.

Algorithm 1 AMEND Algorithm

Input: graph G = {V, E}, node feature matrix X, and precomputed position encoding Xpe

Output: optimized parameters of the student MLP S, predict node labels ˆ Y.

• 1: Model initialization and Dataset Partitioning.
• 2: Pretrain the teacher model T with cross-entropy loss.
• 3: #Student MLP Training
• 4: for Epochs do
• 5: #Aggragation Context Preservation
• 6: ZT = T (X, E,Xpe),
• 7: ZS = S(X,Xpe);
• 8: #Aggregation-enhanced CKA
• 9: LACKA ←ACKA(ZT ,ZS) in Eq. 7;
• 10: #SharedManifold mixup
• 11: Zmix T = λZT + (1 -λ)Z′T ;
• 12: Zmix S = λZS + (1 -λ)Z′S ;
• 13: ˆ YT , ˆ YS ← gT (ZT ), gS(ZS);
• 14: ˆ Y mix T , ˆ Y mix S ← gT (Zmix T ), gS(Zmix S );
• 15: #Logit distillation
• 16: Llogit = Lmix +Lpred in Eq. 11;
• 17: #Overall loss compute
• 18: LS = Ltask +βLACKA +γLlogit in Eq. 12;
• 19: Gradient backward and model optimization.
• 20: end for
• 21: return S, ˆ Y

4 Experiments

4.1 Experiments Setting

Datasets. To fully evaluate our proposed method, we use 8 public regular graph benchmarks [Yang et al., 2021], i.e. Cora, Citeseer, Pubmed, Computer, Photo, Corafull, Coauthor-CS, Coauthor-Physics, and 4 large-scale graphs [Hu et al., 2020], i.e., Ogbn-Arxiv, Aminer, Reddit, and Ogbn-Products. The details of these 12 datasets are in Appendix A. For each dataset, we follow the dataset protocol in [Chen et al., 2023], where 6/2/2 of the nodes are used as training/validation/test sets, respectively.

We select GT as the teacher model for distillation because it presents a more challenging and representative case. GT leverages a global aggregation scope and an attention-based aggregation pattern, which demand that the student MLP replicate both its extensive receptive field and intricate aggregation mechanisms. These characteristics make GT an ideal testbed for evaluating the effectiveness of GNN distillation methods. In our experiments, we report the mean and standard deviation of ten separate runs. We employ accuracy to measure model performance, use validation data to select the optimal model, and report results for test data.

Baselines. Consistent with the comparative experimental setup of traditional knowledge distillation frameworks, we compare a variety of train-from-scratch baseline models, i.e., MLP, GNN-teacher (GCN [Kipf and Welling, 2016], GAT [Velickovic et al., 2017], SAGE [Hamilton et al., 2017], NAGphormer [Chen et al., 2023]), and 3 state-of-the-art GNN-to-MLP methods, i.e., GLNN [Zhang et al., 2022], NO- SOMG [Tian et al., 2023], and VQGraph [Yang et al., 2024]. The details of baseline methods are in Appendix B.

4.2 Performance on Eight Regular Graphs

We compare the proposed AMEND to vanilla supervised MLP, GNN, GT, and state-of-the-art distill-to-MLP methods in the same experimental setting across eight datasets, with the results reported in Table 1. The results show that the vanilla MLP consistently exhibits the lowest performance, emphasizing the need for effective knowledge distillation from more complex models. GNN-based models (i.e., GCN, GAT, and SAGE) outperform the MLP thanks to the message passing for graph structure learning. The NAGphormer, representing the GT teacher, demonstrates superior performance compared to all GNNs, affirming the effectiveness of its global attention mechanisms and more sophisticated model architecture. Our proposed method consistently outperforms all other distill-toMLP methods (GLNN, NOSMOG, VQGraph) across eight datasets. The ∆ values indicate the performance gains of our method over the baseline models and other distill-to-MLP approaches, with significant improvements observed in several datasets: 1.79% higher than NOSMOG on the Citeseer, 2.41% higher than VQGraph on Pubmed, and 2.34% higher than NOSMOG on CS, etc. These results confirm the effectiveness of our proposed AMEND method for GNNto-MLP, which leverages the strengths of the graph transformer teacher and addresses the aggregation mechanism differences, resulting in substantial performance improvements and underscoring the potential for lightweight deployment and fast inference in graph data mining tasks.

4.3 Performance on Four Large-Scale Graphs

Figure 3 shows the performance comparison of our AMEND against SAGE, NAGphormer, GLNN, and NOSMOG across four large-scale graphs (i.e., Ogbn-Arxiv, Aminer, Reddit, Ogbn-Products.) in a few-shot setting with varying training label rates (For the first two datasets, we randomly selected two non-overlapping 10% nodes as the validation and test sets, respectively, and doubled 1% for the last two datasets.). The results indicate that the model performance steadily improves as the supervision increases, and our method almost surpasses all other distill-to-MLP approaches. For example, on Ogbn-Arxiv, our method achieves the highest accuracy at all label rates, with a notable improvement of approximately 5% over NOSMOG at the 0.1% label rate. On Reddit and Ogbn-Products, our method maintains superior performance, especially at higher label rates (e.g., ∼2% higher than NOSMOG at 6‰ label rate on Ogbn-Products). These findings highlight the robustness and effectiveness of our approach, particularly in utilizing limited labeled data, underscoring its potential for scalable and efficient deployment in large-scale graph scenarios.

4.4 Ablation Study

In the ablation experiments, we investigate the impact of ACKA and manifold mixup modules within AMEND , and the results are reported in Figure 4, where ACKA→RAD represents we replace ACKA with the Representational Similarity Distillation (RAD) in NOSMOG [Tian et al., 2023]. The results clearly demonstrate that removing either ACKA or manifold mixup significantly drops accuracy across all datasets. For example, on the Cora dataset, removing ACKA decreases accuracy from over 91% to approximately 89%. The impact of manifold mixup is also notable, as its removal causes a drop in performance across all datasets, such as from 96% to 93% on the Photo dataset. These results confirm that both ACKA and manifold mixup are critical components of our method, significantly contributing to its superior performance. Results underscored the importance of these modules in effectively transferring knowledge and enhancing model performance in graph data mining tasks.

4.5 Parameter Sensitive Analysis

In Figure 5, we explore the sensitivity of hyper parameters β and γ in overall objective function Eq. 12 on three citation graphs. β and γ represent the contributions of the ACKA and manifold mixup logit distillation, respectively. The results indicate that the optimal performance is achieved with β = 10 and γ = 0.1. According to the definition of LACKA, its value range is [0, 1]. We monitored the values of each component of the loss function during training and found that, with β = 10, γ = 0.1, the scales of LACKA and Llogit were comparable to the task loss component Ltask, leading to optimal model convergence.

4.6 Visualization

To visually compare the node embeddings learned by different methods, we used the TSNE algorithm [Van der Maaten and Hinton, 2008] to create scatterplots of the test node representations from the Pubmed dataset, as shown in Figure 6. The figure demonstrates that the node embeddings extracted from our student model exhibit better class separability and intra-class compactness compared to those from the current state-of-the-art GNN-to-MLP approach. This indicates that AMEND effectively transfers the complex graph modeling knowledge from the GT teacher into an efficient MLP. Additionally, we visualize the node correlations between the local aggregation (GCN→GLNN and SAGE→NOSMOG) and global aggregation (GT→AMEND) in Fig. 7. Results demonstrate that our approach has a more regular and structured correlation with the teacher model than NOSMOG.

5 Conclusion

In this paper, we proposed the AMEND framework for effective and efficient knowledge transfer from GNNs to MLPs. The framework introduces a multi-scope aggregation context preservation strategy to enable the student MLP to preserve the teacher's broad and varying aggregation scopes effectively. Additionally, a pattern-guided alignment mechanism addresses aggregation pattern discrepancies so that the student MLP can accurately replicate the structural aggregation behaviors of the teacher GNN. We further incorporate a manifold mixup distillation approach to improve the efficiency and robustness of the student model by capturing the teacher's insights into mixed data distributions. Extensive experiments on 8 regular and 4 large-scale graph datasets, combined with ablation studies and visualization analyses, validate the superiority of the proposed method over existing baselines.

Acknowledgments

This work was partially supported by the Natural Science Foundation of Jiangsu Province (Grant No. BK20222003), the Collaborative Innovation Center of Novel Software Technology and Industrialization, and the Sino-German Institutes of Social Computing. The corresponding authors are Mingkai Lin (mingkai@nju.edu.cn) and Wenzhong Li (lwz@nju.edu.cn).