SALE-MLP: Structure Aware Latent Embeddings for GNN to Graph-free MLP Distillation

1 Introduction

Graphs have emerged as essential data structures for modeling non-Euclidean relations and their interactions. From applications such as protein interactions [Jha et al., 2022], entity alignment [Chaurasiya et al., 2022; Surisetty et al., 2022], and social media [Zhang et al., 2022b], these interconnected networks effectively represent the intricacies of such systems. To learn rich node and graph-level representations from these graph structures, GNNs employ message-passing mechanisms by aggregating information from neighboring nodes [Kipf and Welling, 2022]. As a result, GNNs achieve state-ofthe-art performance in tasks such as node classification, link prediction, and graph classification, demonstrating their potential in handling non-Euclidean data [Hamilton et al., 2017; Veliˇckovi´c et al., 2018; Wu et al., 2020].

Despite their success, GNNs face critical challenges such as high latency and computational overhead. Due to the neighborhood aggregation process, the applicability of GNNs in real-world scenarios remains limited [Hamilton et al., 2017; Kipf and Welling, 2022]. To address the latency bottleneck [Liu et al., 2022], techniques such as pruning [Huang et al., 2024], partitioning [Modak et al., 2024], and quantization [Ding et al., 2021; Tailor et al., 2021; Zhao et al., 2020] aim to reduce computational complexity. However, these methods fail to eliminate the message-passing overhead in GNNs. Alternatively, knowledge distillation (KD) [Zhang et al., 2022a; Tian et al., 2022; Lu et al., 2024; Yan et al., 2020; Malik et al., 2024] transfers the knowledge of computationally intensive teacher models to lightweight student models. KD methods replicate the teacher's output through soft labels or latent embedding spaces. MLPs are often selected as student models in such distillation approaches due to their strong memory capabilities [Szegedy, 2013]. By leveraging only node features, MLPs can effectively capture the knowledge transferred from GNNs while maintaining lower inference latency.

Though lightweight, student MLPs struggle to fully capture the crucial graph-specific structural knowledge from GNNs. Traditional MLP-based methods, such as GLNN [Zhang et al., 2022a], rely solely on node features. They lack explicit structural information and often lead to suboptimal performance. Additionally, DeepWalk-based methods like NOSMOG[Tian et al., 2022] are unsuitable for online training or inductive scenarios, where models must generalize to unseen nodes without accessing the entire graph. Maintaining the whole graph in such cases is resource-intensive and undermines the purpose of large-scale graph distillation. While structure-aware methods [Chen et al., 2022; Wang et al., 2024] improve performance, they struggle with heterophilic graphs. Furthermore, methods [Wu et al., 2023c] attempt to make MLPs more structure-aware through knowledge transfer from GNN embeddings. However, it remains unclear whether distilling the GNN latent embedding space is the best approach for student MLPs, particularly for node classification tasks and their effectiveness is not well established. This raises the question: can we make MLPs structure-aware while maintaining graph-free inference?

In this work, we propose SALE-MLP, a novel, flexible, and adaptable structure-aware (G2M) KD method. SALE-MLP learns structure-aware latent embeddings from node features by distilling GNNs into MLPs, without relying on GNN embeddings or explicit graphs as input. SALE-MLP enhances the student model's generalization by mimicking GNN-like message passing and improving the embedding space to better represent graph structures. Our approach can integrate with any unsupervised graph-structure loss and adopt diverse GNN teachers, making the framework fundamentally loss, teacher, and task-agnostic (suitable for both node classification and link prediction). It excels in unseen scenarios where structural information is vital. Extensive experiments on node classification and link prediction tasks show that SALE-MLP outperforms existing G2M distillation methods in both transductive and inductive settings. The performance gain is particularly notable (3-4%) in practically relevant inductive settings. Furthermore, ablation studies validate the effectiveness and efficiency of SALE-MLP.

2 Preliminaries

Notation: Consider a graph G = (A,X) with the node set V = {v1, v2, . . . , vN} and edge set E, where N is total number of nodes. Let the node features be X ∈ RN×d with d as the size of node features. A is the adjacency matrix, where Aij = 1 if node i and j are connected, else, Aij = 0. Furthermore, Y ∈ RN×C represents the label matrix where C is the number of classes. Finally, labeled nodes are denoted as (V L, XL, Y L), and unlabeled nodes as (V U , XU , Y U ).

Experimentation Setting: We conduct experiments in two setups: (i) Transductive setting: The entire graph G is accessible while training using Y L and the model is evaluated on the unlabeled nodes XU to predict Y U (ii) Inductive setting: The unlabeled nodes V U are partitioned into two disjoint sets of nodes V U = V U obs ∪ V U in , representing observed nodes and inductive nodes, respectively. This creates two disjoint graphs: Gobs based on nodes V L ∪ V U obs and Gind based on nodes V U in . While training Y = Y L and only G = Gobs is accessible to the model while evaluation is performed using the entire graph G. The inductive setting closely resembles real-world scenarios as the entirety of large graphs is either unavailable or too compute-intensive to train on.

GNN Distillation: The message-passing mechanism is essential for learning node representations in GNNs by aggregating information from its neighboring nodes. It typically involves two key steps: message aggregation and representation update due to which the computational complexity of GNNs increases drastically. So, GNN distillation methods aim to train a student model that replicates the performance of a GNN while reducing computation. The student model is trained using a combination of adjacency matrix A, node features X, GNN output logits ˆ y, and GNN embeddings. The student is considered message-passing-free if the neighborhood aggregation step is unnecessary, or graph-free if no structural information is extracted from the graph during inference. The distillation loss, such as KL divergence [Hinton, 2015], minimizes the difference between the student's output and the GNN's output.

3 RelatedWork

GNNs require substantial time and memory for training and inference over relational data. Graph knowledge distillation addresses this challenge and can be categorized into two types: graph-to-graph distillation and G2M distillation. While graph-to-graph distillation methods [Chen et al., 2021; Joshi et al., 2022; Lassance et al., 2020; Ren et al., 2021; Wu et al., 2022a] successfully train compact student GNNs, methods like LSP [Yang et al., 2020] and TinyGNN [Yan et al., 2020] leverage structural insights from teachers. Additionally, RDD [Zhang et al., 2020] uses the reliability of nodes and edges to improve the performance of student GNNs. Although effective in compression, these student GNNs still inherit the computational overhead of message passing, limiting their practical applicability. In contrast, G2Mmethods, based on structural awareness, are subdivided into structure-aware (SA-G2M) and non-structureaware (NSA-G2M) approaches.

3.1 Non-Structure (NSA-G2M) Distillation

To tackle the problem of complex message-passing, MLPbased student models distill knowledge from teacher GNNs and replicate the teacher's performance while reducing memory and computational complexity. GLNN [Zhang et al., 2022a] pioneered G2M distillation by mimicking GNNs using teacher prediction probabilities as soft labels, along with ground-truth labels as hard labels. By simply trying to replicate the GNN output logits, MLPs achieve a significant performance improvement over MLPs trained solely using ground-truth labels, demonstrating the drastic usefulness of distillation methods. Furthermore, GSDN [Wu et al., 2022b] enhanced this approach by incorporating mixup augmentation [Han et al., 2022] on input features and enforcing locality-based consistency. However, these methods rely on logit-based distillation, ignoring neighborhood and graph structure, which leads to information loss and poor generalization on unseen nodes. FF-G2M [Wu et al., 2023a] addresses this by using spectral-graph theory to preserve highfrequency information during the distillation process, ensuring that high pairwise differences in teacher logits between neighboring nodes are maintained.

Additionally, KRD [Wu et al., 2023b] handles this information loss by introducing a novel strategy of training student MLPs with only confident knowledge points using a reliable sampling strategy. Recently, AdaGMLP [Lu et al., 2024] proposed an ensemble G2M distillation approach that improves the generalization capabilities of the student MLP to some extent. They also use feature masking and alignment of AdaBoosted MLPs, which help mitigate incomplete/corrupted node features and graph structures that are prevalent in realworld graphs. While these methods benefit from graph-free MLP inference, not enforcing structural awareness causes them to lose crucial information, leading to suboptimal performance, especially in inductive or few-shot learning. As most real-world graph systems are inductive and face time and memory constraints, it becomes increasingly important for these G2Mmethods to incorporate structural information.

3.2 Structure-Aware (SA-G2M) Distillation

Structure-aware MLPs carry more information about graphs and can better generalize on unseen nodes in inductive scenarios. Thus, researchers have proposed various approaches to use structural information into student MLPs. SA-MLP [Chen et al., 2022] explicitly passes structural information along with node features. However, it is impractical for largescale graphs as it requires access to the complete graph during inference, making it impractical for large-scale graphs. NOSMOG [Tian et al., 2022] utilizes DeepWalk-generated positional features and also aligns GNN-MLP latent spaces using representational similarity loss. However, the use of neighborhood approximations limits its effectiveness with unseen nodes in inductive settings.

[Wu et al., 2023c] used class prototype embeddings to distill structural information, but the extent to which the MLPs are made structure-aware is limited. Similarly, SSL-GM [Wang et al., 2024] proposed a self-supervised framework that aligns embedding spaces without explicit structural information. However, the effectiveness of such GNN latent space alignment remains questionable due to limited expressiveness [Xu et al., 2019] and over-smoothing issues [Li et al., 2018]. Moreover, completely GNN-free methods [Winter et al., 2024] show substantially reduced performance compared to G2M strategies, as GNN knowledge is not utilized. This shows the critical need for G2M distillation that effectively utilizes graph structural information with strong expressiveness and generalization, while maintaining graph-free inference. Our proposed SALE-MLP addresses these limitations by aligning node feature representations with graph topology, optimizing both structural awareness and efficiency for inductive scenarios, while also better approximating GNN output and ground truth.

4 Proposed Approach

This section presents SALE-MLP, which consists of two primary components: the Latent-Space Projector (LSP) and the MLP classifier, as illustrated in Figure 1. These two components together map the node features to a latent space, which is then used to make the final predictions of the student MLP. Additionally, we demonstrate our method using DeepWalk [Perozzi et al., 2014]. For more details on the implementation and additional results (including other unsupervised structural losses), read the supporting material 1 .

4.1 Structure-Aware Student MLP

Our approach enhances GNN distillation by aligning the student MLP's representation space with the graph structure. Specifically, nodes with similar structures or that are closely connected in the graph should have similar vector representations. Node representations generated by unsupervised structural losses [Perozzi et al., 2014; Grover and Leskovec, 2016; Post˘avaru et al., 2020; Tang et al., 2015] are well-established in generating representations that align with the graph topology and are supported by efficient implementations for large graphs. One such loss, DeepWalk, learns attribute-free graph node embeddings by generating random walks to explore the neighborhood structure and uses the SkipGram model [Mikolov et al., 2013] to align the representations of nearby nodes. Since the student MLP model takes the node features as input, the random walk-based SkipGram model is instead applied to the embedding space of the MLP in our method.

Given a random walk node sequence for a node i, Si = {v1, v2, . . . , v|Si|} of length |Si|, the embedding f(vi) for each node vi (where 1 ≤ i ≤ N) is generated. Representation f(vi) predicts the context of vi within a window of size t by modeling probability Pr(vj | vi) as mentioned in Eq. 1.

where vj ∈ Si and f(.) is node-embedding that maps node attributes of vi to latent-embedding space. To train f(.), the loss function LDwL, an InfoNCE approximation [Church, 2017] is typically used for efficient implementation. The equivalent loss is defined in Eq. 2. The constraints in Eq. 3 encourage neighboring nodes to move closer to vi by maximizing the probability of the random walk Si. The constraints in Eq. 4 represent a contrastive loss, pushing the embeddings of the negative nodes in ¯ Si farther away. The set ¯ Si contains k randomly sampled negative nodes for each node j in Si.

1 https://github.com/ganzagun/SALE-MLP

4.2 TrainingMLPs by GNNs Distillation

Given a cumbersome pre-trained GNN, the goal of distillation is to train a lightweight MLP using both ground truth labels and soft labels (from the teacher). For any labeled node v ∈ V L, the ground truth label is yv while zv is the corresponding soft label, as predicted by the teacher GNN. The node classification objective is captured by cross-entropy CE(.) loss between the student prediction ˆ yv and the ground truth label yv:

GNN distillation is performed using the KL-Divergence DKL(.) between the MLP prediction ˆ yv and the soft labels zv, as generated by GNN teacher:

4.3 Overall Loss

The final objective function L is defined as the weighted combination of ground truth cross-entropy loss LGT , soft label distillation loss LSL, and the unsupervised structural loss (DeepWalk loss LDwL in this case):

where α is a trade-off that balances the impact of LDwL, and λ balances LGT and LSL. By incorporating LDwL, the distillation process is instead based on structure-aware MLPs. Additionally, the joint end-to-end training of both losses aligns the representation space, ensuring that neighbor embeddings are close even when labels differ (heterophily), thus preserving distinguishably. Furthermore, MLP overfitting, particularly with limited training labels, is mitigated by regularizing the unsupervised structural loss. Hence, it replicates teacher GNN's generalization in the student MLP with structureaware embeddings via soft labels. Additionally, we perform pretraining of the latent space encoder, aiding faster convergence, especially for larger graphs. Efficient DeepWalk implementations can also be utilized with separate training. Algorithm 1 summarizes the proposed SALE-MLP.

5 Experimental Details

Datasets: The experimentation utilizes six widely-adopted public benchmark datasets: (Cora, Citeseer, Pubmed) [Sen et al., 2008], (Amazon-Photo, Amazon-Computer) [Feng et al., 2022], and the large-scale graph ogbn-arxiv [Hu et al., 2020]. These datasets are selected based on their prevalent usage across state-of-the-art G2M methods (GLNN, NOSMOG, KRD, AdaGMLP). These span different scales (small: Cora/Citeseer, medium: Pubmed/Amazon, large: ogbn-arxiv) and network types (citation networks and co-purchase networks). Furthermore, for Cora, Citeseer, and Pubmed, we

Algorithm 1 Proposed SALE-MLP

• 1: Input: An attributed network G = (V,E,X);
• 2: window-size = w;
• 3: embedding-size = d;
• 4: walk-per-node = γ;
• 5: walk-length = t;
• 6: Graph node features: Z m;
• 7: Num of pretraining steps: q;
• 8: Output: Node embeddings f(v) for each v ∈ V and Classifier E(.) for node classification;
• 9: f(.), E(.) ← initialize parameters;
• 10: *Pretraining Latent Space Encoder* 11:
• 12: for t = 1 to q do
• 13: Sample K negative samples of vi ∈ V for i ∈ |V|
• 14: Calculate unsupervised loss LDwL using Eq. 2
• 15: Perform backpropagation on f(.) 16:
• 17: end for
• 18: *MLP Distillation* 19:
• 20: for t = 1 to T do
• 21: S ←generate a set of randomwalks on G;
• 22: Sample | ¯ Si| negative samples of vi ∈ V for i ∈ |V|
• 23: Calculate unsupervised loss LDwL using Eq. 2
• 24: Calculate final loss L using Eq. 7
• 25: Perform backpropagation on f(.) and E(.) 26:
• 27: end for
• 28: return f(.) anb E(.);

follow splits from [Kipf and Welling, 2022] and for the Amazon dataset, we use splits from [Zhang et al., 2022a] i.e., using 20-shot for training, 30-shot for validation, and remaining for testing. While ogbn-arxiv uses standard splits [Hu et al., 2020]. Detailed dataset statistics are provided in the supporting material.

Implementation details: We take GCN[Kipf and Welling, 2022], SAGE[Hamilton et al., 2017] and GAT[Veliˇckovi´c et al., 2018] as teacher GNN model. The model has dimension 128 and 2 hidden layers for all datasets except ogbn-arxiv for which 256 dimensions and 3 hidden layers are used. The choices are consistent with other G2M methods. For SALEMLP, we perform a grid search for the hyper-parameters below on the validation data:

# Hidden layers = {2,3} # Walks = {1,2,5} Walk len = {3,5,10} Pre-train Epochs = {1,2,5,10} λ = {0.0, 0.1, ... , 1.0} α = {1, 1.5, 2, 2.5, 3, 3.5, 4} Hidden layer Dimensionality = {64,128,256}

Baselines: We chose GLNN [Zhang et al., 2022a], NOSMOG [Tian et al., 2022], KRD [Zhang et al., 2020] and AdaGMLP [Lu et al., 2024] for comparison. Their publicly available codebases are replicated along with reported best hyperparameters for evaluation. Additionally, as NOSMOG is not a graph-free method, we also report results for NOSMOG by removing the Deepwalk embeddings component for a fair evaluation of methods with graph-free students. For AdaGMLP and KRD, we note that the teacher GNN used in the original work is larger than our teacher GNN.

6 Results and Analysis

In this section, we evaluate SALE-MLP across several key aspects: its performance, inference time, impact of various loss component, robustness and hyperparameter sensitivity

6.1 Classification Performance

SALE-MLP is evaluated in both transductive and inductive settings, with results for SAGE shown in Table 1. In both the reported settings, SALE-MLP consistently outperforms existing methods and even surpasses the teacher GNN across all datasets, demonstrating several key advantages:

Transductive Setting: While NSA-G2M methods (KRD, AdaGMLP) benefit from accessing prediction node logits during training, SALE-MLP still achieves superior performance. Notably, SALE-MLP outperforms NOSMOG even with its DeepWalk embeddings, highlighting the effectiveness of our structure-aware latent space alignment.

Inductive Setting: SALE-MLP demonstrates substantial improvements over existing methods, particularly on the largescale ogbn-arxiv dataset. NSA-G2M methods show significant performance degradation compared with transductive settings due to their lack of structural awareness which impacts most in inductive settings. While NOSMOG's performance heavily relies on DeepWalk embeddings as the performance without DeepWalk drops across the dataset. In contrast, SALE-MLP maintains consistent performance across both settings due to its inherent structure-aware embedding space, achieved without requiring explicit graph access. Furthermore, statistical significance of the experiments is reported in the supporting material.

6.2 Inference Time

The Figure 2 illustrates the accuracy-inference time trade-off on PubMed (transductive setting). SALE-MLP achieves the highest accuracy with only 0.2ms inference time, approaching GLNN's efficiency (0.096ms) while significantly outperforming it in accuracy. The variation in inference times across methods is attributed to the number of layers, hidden dimensions and ensemble of models (for AdaGMLP) associated with the best-performing model. While the higher inference time of NOSMOG can be attributed to the process of finding the neighboring nodes and calculating the mean embedding. Notably, SALE-MLP achieves 150× speedup over SAGE GNN (33.2ms) while delivering superior performance, demonstrating its practical utility for real-world applications.

6.3 Loss-Component Ablation

An ablation study by systematically removing individual loss components is also conducted, with results reported in Table 2. The analysis reveals several key insights:

Impact of Structure-Aware Learning (LDwL): Removing LDwL causes a significant performance degradation, showing that structure-aware latent space is crucial for both effective knowledge transfer and maintaining better classification.

Role of Soft Labels (LSL): Without GNN distillation via LSL, the model struggles to capture the teacher GNN's inductive bias and generalization capabilities, highlighting the importance of teacher guidance.

Ground Truth Supervision (LGT ): The removal of LGT has minimal impact, as LSL provides sufficient supervision.

Notably, SALE-MLP without LSL achieves performance comparable to the teacher GNN, and with distillation providing complementary information, explains why SALE-MLP ultimately outperforms the teacher GNN (Table 1).

6.4 Consistency with Graph Topology

To quantify SALE-MLP's effectiveness in capturing graph topology, we evaluate the cut-value (CV) between model predictions and graph structure [Zhang et al., 2022a]. The mincut problem partitions N nodes into K disjoint subsets while minimizing edge cuts, formulated as [Dhillon et al., 2004]:

Following [Bianchi et al., 2020], replacing C with model predictions ˆ Y yields:

Here, CV is proportional to the consistency between the topology of the graph and model predictions. And a Higher CV indicates better capture of graph-structural information.

CV values across datasets are reported in Table 3. Inductive bias in GNN is best at understanding graph topology leading to the best GNN performance. While GLNN benefits from GNN distillation compared to MLP, the structure awareness introduced in NOSMOG further improves the CV. Finally, SALE-MLP outperforms all G2M methods further proving the advantage of having structure-aware latent space derived directly from the graph topology. Additionally, Table 4 shows the impact of removing the loss components. And as observed removing LSL minimally impacts CV, indicating SALE-MLP effectively captures structural information through embedding alignment alone.

6.5 Performance on Heterophilic Dataset

Next, we report the performance of SALE-MLP on highheterophily datasets Actor [Tang et al., 2009] and Wisconsin (https://tinyurl.com/Wiscosn) in inductive settings. High heterophily poses unique challenges for G2M methods, requiring effective integration of both node features and graph structure.

As shown in Table 5, SALE-MLP outperforms other distillation methods in both the datasets, with significant gains in Wisconsin. MLP performs better in Actor, given sufficient training data, due to over-smoothing in GNNs. However, leveraging the neighborhood information, GNNs regain their edge in a few-shot scenario. Further, SALE-MLP outperforms MLP by providing a sweet spot with information from both structure and MLP, if trained using heterophilic specific GNNs. The larger improvement in Wisconsin can be attributed to the fact that the number of nodes inWisconsin is smaller and it is easier for SALE-MLP to capture the relation between structure and node features. Moreover, while SALEMLP does not explicitly address heterophily, its structureaware latent embeddings naturally combine node features and structural information, leading to improved performance in heterophilic conditions. Additional experiments with varying heterophily ratios are presented in the supporting material.

6.6 Generalization Ability of SALE-MLP

Evaluating generalizability is another aspect explored for different approaches in Cora and Citeseer datasets. We trained and plotted the performance of SALE-MLP and other G2M methods with limited labeled samples per class (k-shots). This experimental setup is crucial for understanding how well the model can generalize when only a small amount of labeled data is available, simulating real-world scenarios where labeled data is often scarce.

Figure 3 shows SALE-MLP's superior generalization capabilities across different k-shot scenarios. The model exhibits strong performance even with few labeled examples (lower k values). As more labeled examples are introduced, SALEMLP shows sharp initial accuracy improvements, indicating effective learning from minimal supervision. The performance gradually saturates at higher k values, suggesting efficient knowledge extraction from the available labeled data. While KRD showing competitive performance to SALEMLP due to its reliable node selection strategy that helps capture important data patterns. However, throughout the range of k values, SALE-MLP consistently outperforms all G2M methods including KRD.

6.7 Different Training Strategies

An ablation study is conducted to understand the most optimal strategy by examining how the model performs under different strategies. We investigate three distinct training strategies for SALE-MLP: (i) end-to-end training without pretraining f(.), (ii) unsupervised pre-training of f(.) (warmstart) followed by freezing f(.) while training the classifier E(.), and (iii) pre-training f(.) for a few epochs, followed by subsequent end-to-end fine-tuning of both f(.) and E(.).

The performance of these approaches on the Cora and Citeseer datasets is reported in Table 6. It can be observed that warm starting f(.) and fine-tuning it end-to-end yields the best results. This superior performance can be attributed to two reasons: this strategy better handles the relatively slower convergence of the unsupervised structural loss (Deepwalk loss). And training in a structure-aware space effectively prevents MLP from overfitting on node features.

6.8 Input Feature-Space Analysis

To understand the effectiveness of the generated latent space, we analyze different feature spaces using t-SNE plots. Figures 4(a) and 5(a) show the node-feature vs. node label plots, representing the input space used by most G2M methods (GLNN, KRD, AdaGMLP). The poorly distinguishable class clusters in this space explain the relatively lower performance of methods like GLNN that directly utilize node features. Figures 4(b) and 5(b) illustrate NOSMOG's enhanced input space, incorporating DeepWalk embeddings. The improved cluster separation explains NOSMOG's superior performance over GLNN. Finally, Figures 4(c) and 5(c) depict SALEMLP's latent space, which demonstrates the clearest cluster separation, validating its superior performance through better structural representation learning.

6.9 Hyperparameter Sensitivity Analysis

We examine the impact of λ and α on the performance of SALE-MLP. Figure 6 shows model sensitivity to λ ([0,1] with step of 0.1). λ < 0.6 is favorable with little impact when varied between [0,0.6]. This translates to the supervision of the teacher being pivotal, but relying more on direct supervision limits the learning of the student (λ > 0.6). Figure

6 illustrates sensitivity to α ([1,4] with step of 0.5). Performance peaks around 2.5, signifying a balance between LDwL and other losses, as overall performance drops with lower or higher values. Here, lower values degrade latent embedding quality, while higher values reduce influence of teacher.

7 Conclusion

This paper addresses the issue of graph-free structural awareness in existing G2M distillation approaches. We propose SALE-MLP, which aims to align the node features to graph topology in a latent space using the unsupervised structural loss. The latent space learns both graph structure and node features, achieving the best performance compared with various SOTA methods. With extensive experiments on six datasets, we demonstrate that SALE outperforms GNN by 3.11%, other SOTA methods by 6.47%, and MLP by 16.75% on average in the inductive setting. In addition, the time was reduced by 150× compared to GNN. Also, we analyze consistency and generalization. Ablation studies on loss components, training strategies, hyperparameter sensitivity, latent representation, and heterophily settings establish the effectiveness of SALE-MLP. Finally, experiments with different teachers and varied structural loss further demonstrate the broad applicability and adaptability of SALE-MLP.

Contribution

Harsh Pal, Sarthak Malik and Rajat Patel contributed equally.