Efficient Tuning and Inference for Large Language Models on Textual Graphs

1 Introduction

Textual graphs are pervasive in the real world, like academic networks [Wang et al., 2020], webpages [Mernyei and Cangea, 2020] and e-commerce datasets [Chiang et al., 2019]. In the early stages, shallow embedding methods [Mikolov et al., 2013; Zhang et al., 2010] and graph neural networks [Kipf and Welling, 2017; Hamilton et al., 2017; Veliˇckovi´c et al., 2018] are combined to analyze and process this data like Figure 2a. However, static shallow embedding methods struggle to capture context-aware information and complex semantic relationships, limiting their ability to exploit the richness of text attributes, particularly in graph tasks.

† Corresponding Author

Large language models (LLMs) such as ChatGPT [OpenAI, 2023], LLaMA2 [Touvron et al., 2023], and Falcon [Almazrouei et al., 2023] have recently demonstrated significant potential in understanding language, effectively capturing the semantic richness of text attributes. This success has prompted researchers to utilize LLMs on textual graphs for enhanced performance. Some works [Chien et al., 2021; He et al., 2023b; Duan et al., 2023] explore the application of Language Models (LMs) on textual graphs to improve node features. These features are subsequently aggregated by GNNs to generate the final representation like Figure 2b. However, these approaches, as pointed out by many preceeding researchers [Zhao et al., 2022; Xue et al., 2023;

Pan et al., 2024b], may be sub-optimal: the nodes' textual and structural features are encoded separately by the LLMs and GNNs at two different stages. An apparent solution to this issue is the joint training of LMs and GNNs, while this option introduces the efficiency challenges, as memory and time complexity during training and inference may become prohibitively expensive for both institutions and customers.

This work aims to provide an efficient and effective solution to enable the joint modeling of the textual and topological information on textual graphs. We propose an efficient tuning algorithm for large language models on textual graphs, named ENGINE, to address the challenges mentioned above. As illustrated in Figure 2d, during training, we freeze the parameters of LLMs and add a tiny tunable ladder (G-Ladder) alongside each layer of LLMs. Within each G-Ladder, we adopt message passing to integrate structural information, thereby enhancing the quality of node representations. The key idea behind the G-Ladder is that only an extremely small portion of the parameters are updated, resulting in a significant reduction in memory consumption. Furthermore, since the parameter update of ENGINE does not depend on the gradient computation of the LLMs, which allows us to precompute node embeddings, storing them in the cache for subsequent reuse, thereby significantly reducing the time complexity during training, as shown in Figure 1. In the inference phase, where LLM-induced latency is typically high, we introduce an early exit operator post each ladder to expedite the inference process. This variant, termed ENGINE (Early), facilitates dynamic early exit of node embedding encoding, accelerating inference and mitigating the risk of overthinking [Zhou et al., 2020]. Through extensive experiments, our method demonstrates SoTA performance compared to baselines on textual graphs across various domains. Compared with existing LLM-based approaches, our proposed framework significantly improves training efficiency and reduces inference latency. This suggests a promising direction for combining LLMs with GNNs in textual graph analysis.

Our contributions can be concluded as:

• We propose a memory- and time-efficient tuning method for LLMs on textual graphs named ENGINE. The LLMs and GNNs are combined through a tunable side structure which significantly reduces the training complexity without impairing the joint model's capacity.
• Two additional variants of ENGINE are proposed to further improve the training and inference: (i) ENGINE (caching) precomputes node embeddings for training samples, places them into cache, and reuses them during training; and (ii) dynamic early exit dynamically determines whether to exit the feed forward of the LLM layers and returns the prediction.
• Extensive experiments demonstrate the effectiveness of our method with the lowest training cost compared to previous methods. Moreover, ENGINE with caching speeds up training time by 12x, and ENGINE (Early) achieves up to 5x faster inference with a slight maximum performance drop of 1.17% across 7 datasets.

2 RelatedWorks

2.1 LM-basedMethods for Textual Graphs

Representation learning for textual graphs has gained significant attention in the field of graph machine learning. Rather than relying solely on shallow or hand-crafted features (e.g., bag-of-words, skip-gram) as seen in previous works [Kipf and Welling, 2017; Veliˇckovi´c et al., 2018; Hamilton et al., 2017; Zhu et al., 2023a; Zhu et al., 2023b; Zhu et al., 2023c] , LM-based methods harness the potential power of Language Models (LMs) to handle textual graphs. These methods combine LMs and GNNs in a cascading (Figure 2b) or iterative structure (Figure 2c). Specifically, in cascading structure methods, the initial step involves finetuning pre-trained language models on downstream tasks. Subsequently, these fine-tuned models are employed to generate text embeddings, serving as the initial node features for GNNs. For example, GIANT [Chien et al., 2021] enhances the meaningfulness of node embeddings by fine-tuning LMs using neighbor information. SimTeG [Duan et al., 2023] adopts a two-stage training paradigm: initially fine-tuning LMs through graph-related tasks (e.g., node classification, link prediction), and subsequently training GNNs based on the embeddings generated by LMs. TAPE [He et al., 2023b] utilizes LLMs to augment text descriptions and fine-tunes PLMs based on these augmented texts. The subsequent step involves using the fine-tuned PLMs to generate text embeddings, serving as the initial node attributes for GNNs. These methods only model partial information, limiting their ability to learn comprehensive features.

Instead of adopting the cascading structure, GraphFormers [Yang et al., 2021], GLEM [Zhao et al., 2022], and LEADING [Xue et al., 2023] combine LMs and GNNs in an iterative or co-training structure, training LLMs and GNNs in a co-training way. However, these co-training paradigms face significant scalability issues. Encoding all neighbors by LMs introduces high-cost fine-tuning and inference overhead due to the substantial number of parameters in language models.

To address these non-trivial challenges, we propose to combine LMs and GNNs in a side structure as depicted in Figure 2d. We introduce a novel method designed to implement a parameter- and memory-efficient tuning method for LLMs on textual graphs named ENGINE.

2.2 Parameter-Efficient Fine-Tuning (PEFT)

Recently, large language models (LLMs) have achieved remarkable success in NLP domain [OpenAI, 2023; Touvron et al., 2023]. PEFT methods aim to adapt large pretrained models to various downstream applications without fine-tuning all parameters due to prohibitive costs. For instance, adapter-based methods [Houlsby et al., 2019] inject lightweight neural networks (adapters) into Transformer layers and exclusively fine-tune these adapters for model adaptation. LoRA [Hu et al., 2021] assumes a low intrinsic rank in weight changes during model tuning. It proposes optimizing the decomposition of original weight matrices in self-attention modules, multiplying optimized matrices during deployment to obtain the delta of self-attention weights, thus significantly reducing the parameters requiring fine-tuning. Similarly, IA3 [Liu et al., 2022] modifies attention weights for both key and value, as well as the feedforward activation, through element-wise multiplication. Prompt-based tuning [Lester et al., 2021; Li and Liang, 2021; Pan et al., 2023] prepends trainable soft prompts to the input text while keeping the entire pre-trained models unchanged. While most methods mainly focus on achieving competitive performance through tuning few parameters (parameterefficient), they often fall short in terms of memory efficiency. Ladder Side-Tuning (LST) [Sung et al., 2022] addresses this issue by training a ladder-side network that separates trainable parameters from the backbone model. Because it does not require backpropagation through the backbone network (LMs), LST significantly reduces memory requirements.

Although PEFT has achieved great success in the Euclidean domain including natural language and computer vision [Jia et al., 2022; He et al., 2023a; Pan et al., 2024a], how to effectively and efficiently apply such approaches to nonEuclidean domains like graphs is still under exploration. The main challenge lies in the lack of experience of how to fully utilize structural information to enhance node-level representations. In this work, we propose an efficient tuning method for LLMs on textual graphs, named ENGINE, which explicitly utilizes structural information and enhances node-level representations through our designed lightweight G-Ladder.

3 Method

In this section, we will introduce the notations used in this paper. Subsequently, we will present the proposed efficient tuning method for LLMs tailored to textual graphs in Section 3.2. Finally, efficient inference will be discussed in Section 3.3.

3.1 Notations

Given a textual graph G = {V, {tn}n∈V, A, Y }, where V is the node set consisting of N instances, tn ∈ T Qn represents a sequential text for its node n ∈ V, T is the tokens dictionary, and Qn is the sequence length, A ∈ RN×N denotes the adjacency matrix, and Y denotes labels for each node. To enhance scalability, a sampling function Γ(·) is applied to a large graph to obtain a set of small subgraphs {Gn}n∈V, where Gn represents the subgraph for the target node n ∈ V. In this study, the focus is on the node classification task of textual graphs. Specifically, given a set of training nodes Vtr, a classification model is trained on these nodes and evaluated on the remaining test nodes Vte. Formally, given a set of training nodes and their induced subgraphs G tr = {Gn}n∈Vtr , the optimal predictor fθ∗ is formulated as

where yn denotes the true label of target node n ∈ Vtr and ˆ yn is the predicted label. It is noteworthy that our method can be applied to other tasks as well, including graph classification and link prediction. For instance, in graph classification, each instance aligns with a graph, analogous to the subgraph of the target node in the node classification task. The exploration of these applications is left for future work.

3.2 Efficient Tuning for LLMon Textual Graphs

The efficient tuning of large language models for graph data is under exploration [Duan et al., 2023; Xue et al., 2023]. However, existing methods are parameter-efficient but not memory-efficient, primarily focusing on fine-tuning smallscale language models like BERT [Kenton and Toutanova, 2019] and DeBERTa [He et al., 2020] for textual graphs. Moreover, these methods encounter challenges in effectively utilizing structural information explicitly during fine-tuning. In this work, we present a parameter-efficient and memoryefficient tuning method for LLMs while also leveraging structural knowledge explicitly, as depicted in Figure 2 (Right).

Given an input graph G, a sampling function Γ(·), such as random walk with restart [Qiu et al., 2020; Zhu et al., 2022], will be applied to obtain subgraphs G tr = {Gi}i∈Vtr for target nodes. Each subgraph consists of several nodes, and each node contains a textual description ti. The textual descriptions of nodes within a subgraph will be packed into a batch B = {ti}Bi=1 and fed into LLMs. In each layer of LLM, token-level representations of each node will be calculated:

where LLM Layerl (·) denotes the l-th layer of LLM, and Hl ∈ RB×Q×D means the token-level representations in ith layer. In the first layer, the input H0 is equivalent to B.

Lightweight G-Ladders

In this work, we focus on node-level tasks, intricately related to the quality of node-level representations. Our goal is to enhance these representations during model tuning. To achieve this, a readout function R, such as mean pooling [Mesquita et al., 2020], is applied to token-level representations:

where zl i ∈ R1×D denotes the representation of node i ∈ Vtr. Then these node-level representations are fed into GNN Ladders (G-Ladders) to improve the quality of node embeddings through structural information, ensuring accurate comprehension of the node's semantics from a global perspective. The improved node-level representations ˆ Z l are derived as

where P l (·) : RB×D → RB×K is a projector that maps the node-level representations into low dimensions (K ≪ D), thereby reducing the subsequent computation in G-Ladders. GNN is one type of graph neural networks that includes message passing. In this paper, for simplicity, we benchmark three different architectures, including GCN [Kipf and Welling, 2017], SAGE [Hamilton et al., 2017], and GAT [Veliˇckovi´c et al., 2018] and pick the best one (SAGE) for modeling G-Ladders. Nevertheless, the choice of the GNN architectures does not bring heavy influence to the final performance (refer to Table 11 in Appendix). λl is a learnable coefficient for combining information in the current layer l and the previous layer l - 1. It is modeled by λl = sigmoid( ωl T ), where ωl is a learnable zero-initialized scalar, and T is a temperature set as 0.1.

Finally, the representation of the target node ˆ zL i from the final layer's output ˆ Z L is fed into a linear classifier C for classification. The training loss L is computed using the crossentropy loss CE(·, ·) between predictions and true labels:

What Makes ENGINE An Efficient-Tuning Method?

As depicted in Figure 2 (Right), the efficiency of ENGINE stems from three aspects. First, similar to existing PEFT methods, the parameters of LLMs are frozen and only lightweight G-Ladders are updated during training, ensuring parameter-efficient. Second, our method innovatively integrates G-Ladders with LLMs via a side structure, eliminating the need for backpropagation through the LLMs. This contrasts with prior methods that insert additional tunable parameters within LLMs, requiring costly backpropagation memory, such as cascading or iterative structures [Chien et al., 2021; Zhao et al., 2022]. Last, our side-structure design allows us to precompute and cache node embeddings for reuse, further boosting computational efficiency. In short, our method is both parameter-efficient and memory-efficient, as demonstrated by empirical evidence in Section 4.5.

3.3 Efficient Inference for LLMon Textual Graphs

The high latency linked to LLM inference can hinder model deployment in real-world settings. Various strategies are proposed to speed up model inference, including model pruning [Gordon et al., 2020], model distillation [Sanh et al., 2019], and dynamic early exit [Xin et al., 2020; Hu et al., 2023].

In this work, for simplicity, we adopt a similar mechanism as dynamic early exit, which can be seamlessly integrated with our method. Specifically, we add a lightweight singlelayer MLP as an early exit classifier C l after each G-Ladder. During the model tuning, these classifiers are directly connected to the downstream task's training objective, e.g., the cross-entropy loss between the true label y:

Later during inference, early exit will be triggered based on the classifiers' uncertainty measure, such as information entropy [Xin et al., 2020; Hu et al., 2023] and confidence [Xie et al., 2021]. However, these criteria heavily depend on datasets, and careful selection is necessary for different datasets. For example, on simpler datasets, a lower value of information entropy may be chosen as the threshold, considering the trade-off between efficiency and performance. In this work, we utilize 'patience' criteria [Zhou et al., 2020] to guide dynamic early exit, allowing us to avoid the need for meticulous design. Patience-based criteria imply that if consecutive p early exit classifiers predict the same results, where p represents the number of layers set at 2 across different datasets in our experiments, the model stops inference early and makes a prediction.

Since these early exit classifiers are also included in the side structure, the gradients continue to flow exclusively within the side structure as shown in Figure 2 (Right), making our method tuning-efficient and inference-efficient.

4 Experiments

In this section, we first introduce the datasets used in Section 4.1. Then, we will illustrate the baselines and experimental setup in Section 4.2 and 4.3 respectively, and conduct experiments on these datasets to demonstrate the effectiveness of our proposed method in Section 4.4. Then, we will study the training efficiency of ENGINE compared to other methods in Section 4.5. Lastly, the analysis of hyper-parameters and an ablation study will be provided. Furthermore, additional experiments (e.g., link prediction) are in Appendix.

4.1 Datasets

In this work, we adopt seven commonly used textual graphs to evaluate our proposed ENGINE: Cora [Sen et al., 2008], CiteSeer [Giles et al., 1998], WikiCS [Mernyei and Cangea, 2020], OGBN-ArXiv [Hu et al., 2020], ArXiv-2023 [He et al., 2023b], OGBN-Products [Hu et al., 2020] and ElePhoto [Yan et al., 2023]. We utilize collected raw text data of these datasets by previous works [Chen et al., 2023; He et al., 2023b; Yan et al., 2023]. Details of these datasets can be found in Appendix A.

4.2 Baselines

To assess the effectiveness of our proposed method, 17 baselines in 5 main categories of approaches are employed. The 5 categories are: (i) traditional GNN models, (ii) Graph Transformers, (iii) LM-Based methods, (iv) recent works designed for textual graphs, and (v) PEFT methods. Briefly, traditional GNN models include GCN [Kipf and Welling, 2017], SAGE [Hamilton et al., 2017], and GAT [Veliˇckovi´c et al., 2018]. Graph Transformers include GraphFormers [Yang et al., 2021] and NodeFormer [Wu et al., 2022]. Full fine-tuned LM-based methods include BERT [Kenton and Toutanova, 2019], SentenceBERT [Reimers and Gurevych, 2019], and DeBERTa [He et al., 2020]. Recent works for textual graphs include Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction (GIANT) [Chien et al., 2021], Learning on Large-scale Text-attributed Graphs via Variational Inference (GLEM) [Zhao et al., 2022], LLM-toLM Interpreter for Enhanced Text-Attributed Graph Representation Learning (TAPE) [He et al., 2023b], and A Frustratingly Simple Approach Improves Textual Graph Learning (SimTeG) [Duan et al., 2023]. PEFT methods include Lowrank Adaptation of Large Language Models (LoRA) [Hu et al., 2021], IA3 [Liu et al., 2022], The Power of Scale for Parameter-Efficient Prompt Tuning (Prompt Tuning) [Lester et al., 2021], and Ladder Side-Tuning (LST) [Sung et al., 2022]. Details of these methods are in Appendix B.

4.3 Experimental Setup

For traditional GNN methods, we utilize grid search to obtain optimal results. For LM-based methods, i.e., BERT, SentenceBERT, DeBERTa, we fine-tune all parameters of these models on training nodes. Methods tailored for textual graphs are implemented using their official codes and reproduced under our settings. The GNNs utilized in these methods are selected from GCN, SAGE, and GAT, choosing the most effective one. Notably, GraphFormers is originally designed for link prediction task, GraphFormers⋆ refers to our adaptation of their official codes* to the node classification tasks. The hyperparameters of the baselines can be found in Appendix C. Regarding our method, ENGINE can be applied to any LLMs. In the main content, we applied ENGINE to the widely used open LLM named LLaMA2-7B [Touvron et al., 2023] to show the effectiveness. Experimental results for other language models, like e5-large [Wang et al., 2022], can be found in Appendix D. We report the mean accuracy with a standard deviation across five different random seeds.

4.4 Performance Analysis

From Table 2, we can draw the following conclusions:

Firstly, static shallow embedding methods combined with GNNs (i.e., GCN, SAGE, GAT) perform inferiorly compared to recent methods that combine LMs with GNNs. This indicates that static shallow embedding methods may struggle to capture context-aware information and complex semantic relationships, limiting their ability to fully exploit the richness of text attributes, thereby achieving suboptimal results. For instance, on OGBN-ArXiv and OGBN-Products datasets, LM+GNN methods (i.e., SimTeG, GLEM, GIANT) significantly outperform GNNs with shallow embedding methods, exhibiting an absolute performance gap of around 5%.

Secondly, pure LM methods (i.e., BERT, SentenceBERT, DeBERTa) perform inferiorly to LM+GNN methods on textual graphs. This demonstrates that, compared to pure LM methods which neglect intrinsic graph structure, combining LMs with GNNs can generate more semantic and structureaware node embeddings. For instance, on the Ele-Photo dataset, GIANT achieves 81% accuracy, outperforming its base LM model BERT by approximately 13%.

Lastly, our method is superior to current LM+GNN methods. Specifically, ENGINE further outperforms the previous SoTA method SimTeG, achieving an absolute improvement of over 2% on the Cora dataset and an absolute 3% improvement on the WikiCS dataset. Besides, our method also shows significant improvements over all the other PEFT methods (i.e., LoRA, IA3, Prompt Tuning, LST), showcasing the superiority of ENGINE in fine-tuning LLMs on textual graphs. Additionally, ENGINE (Early), which combines dynamic early exiting, achieves comparable performance with ENGINE while improving the efficiency of model inference, as further discussed in Section 4.5.

* https://github.com/microsoft/GraphFormers

These statistics show the effectiveness of our method for generating context-aware and complex semantic nodes embeddings on textual graphs.

4.5 Efficiency Analysis

Training Efficiency

In this section, we assess the training efficiency of several baselines and ENGINE. The additional trainable parameters in our method are integrated with a frozen LLM through a side structure. This allows us to precompute node embeddings using frozen LLMs and store them in the cache for reuse, consequently enhancing training efficiency. We also present the training efficiency of our method with caching.

In Table 3, GIANT and GLEM employ full fine-tuning of language models with the largest trainable parameters. In contrast, SimTeG utilizes LoRA to fine-tune e5-large and subsequently trains GNNs. These methods consume low memory because they employ small language models. However, scaling them up to large language models proves challenging due to their high training costs. Furthermore, it is evident that previous PEFT methods demand substantial time and memory resources for tuning LLaMA2-7B. In contrast, ENGINE achieves the lowest time and memory costs, and the inclusion of caching in ENGINE significantly reduces computation expenses, achieving 12x faster training compared to without caching (21m vs 4h 23m).

Inference Efficiency

For inference, ENGINE goes through all layers in LLMs, which may be impractical in some realistic scenarios. Therefore, we incorporate dynamic early exit seamlessly into our method, named ENGINE (Early). ENGINE (Early) dynamically exit based on the complexity of samples to save inference time. Table 4 records the inference time for running all test samples using our method (with or without dynamic exit). Figure 3 depicts the early exit ratios across different layers.

In Table 4, the speed advantage of ENGINE (Early) over ENGINE is clearly evident, primarily due to reduced intensive computation within LLM layers. Notably, ENGINE (Early) attains 5x faster inference speed, mainly attributed to the fact that over 90% of samples exit early in the 5th layer as depicted in Figure 3. Additionally, ENGINE (Early) is faster than other textual graph baselines while achieving better performance.

4.6 Sensitivity Analysis

The number of G-Ladders. G-Ladder can be incorporated alongside each LLM layer. In our investigation, we examine varying numbers of G-Ladders. For simplicity, we analyze three strategies: (1) Adding a G-Ladder every 10 layers (i.e., 0, 10, 20, 32 where the 0-th layer represents the word embedding layer, and the 32-nd layer is the last layer used in all strategies). (2) Adding a G-Ladder every 5 layers (i.e., 0, 5, ..., 25, 32). (3) Adding a G-Ladder every 2 layers (i.e., 0, 2, ..., 30, 32).

Table 5 presents the results. The first strategy consistently performs worse than others, likely due to having the fewest learnable parameters. Although the last strategy involves the most learnable parameters, it falls short of the second strategy on small datasets (e.g., Cora), suggesting that a moderate number of parameters is optimal. Throughout our experiments, we adopt the second strategy due to its balanced consideration of effectiveness and efficiency.

The patience in dynamic early exit. To accelerate inference, we adopt patience-based early dynamic exiting in our work. In this section, experiments are conducted with varying patience values: 2, 3, 4, and 7. The patience of 2 means terminating in advance and returning results if two consecutive layers produce identical results. Notably, the patience of 7 signifies the absence of dynamic early exiting.

Figure 4: Sensitivity analysis of patience p in dynamic early exit.

Results from Figure 4 illustrate the trade-off between performance and efficiency. In our experiments, we set the patience to 2 for most datasets, achieving comparable performance while significantly reducing inference time, denoted as ENGINE (Early).

4.7 Ablation Study

In this section, we investigate the impact of components in our method. Specifically, 'constant λ (0.5)' indicates setting λ as a constant value (i.e., 0.5) in Equation 4 rather than a learnable coefficient. 'w/o struct. info.' denotes the removal of message passing in G-Ladders.

Table 6 highlights that structural information is crucial for enhancing the quality of node representations, showing around a 14% absolute improvement on Cora and WikiCS datasets. Additionally, a learnable coefficient λ outperforms a constant value because the coefficient can adapt to the characteristics of datasets.

5 Conclusion

In this paper, we present ENGINE, an efficient and effective framework for incorporating large language models in textual graphs. Our proposed approach introduces a lightweight, tunable GNN-based side structure (G-Ladder) alongside each layer of the LLM, to explicitly model the structural information of the textual graphs. The key insight is that the parameter update of ENGINE does not depend on the gradient computation of the LLMs, resulting in exceptionally efficient training compared to concurrent counterparts. Building upon this, we introduce two variants with caching and dynamic early exit to further enhance training and inference speed. Empirical studies demonstrate that our ENGINE outperforms the state-of-the-art approaches across multiple realistic textual graph datasets, in terms of performance, training efficiency, and inference efficiency.

Acknowledgments

This work was supported by the Key Research and Development Projects in Zhejiang Province (No. 2024C01106), NSFC (No. 62272411), the Tencent WeChat Rhino-Bird Special Research Program (Tencent WXG-FR-2023-10), the National Key Research and Development Project of China (2018AAA0101900), and Research funding from FinVolution Group.

Contribution Statement

The paper's methodology and experimental design were collaboratively conceived by all contributing authors. Both Yun Zhu and Yaoke Wang made significant and equal contributions to this work, thereby meriting the designation of co-first authors. Siliang Tang holds the role of corresponding author for this publication.