HiTuner: Hierarchical Semantic Fusion Model Fine-Tuning on Text-Attributed Graphs

1 Introduction

Many graphs in the real-world possess textual features, and can be termed as Text-Attributed Graphs (TAGs). In TAGs, nodes typically represent textual entities like documents or sentences, while edges capture the relationships between different entities. Graph Neural Networks (GNNs) have emerged as a promising paradigm for TAG-related tasks due to their ability to propagate messages between neighboring nodes [Kipf and Welling, 2017; Luo et al., 2024]. This capability allows them to effectively utilize a small number of labeled examples and generalize well to large-scale graph tasks [Hamilton et al., 2017]. Typically, the text associated with each node must first be transformed into vector compatible with GNNs. Traditional shallow text encoders, such as skip-gram or bag-of-words models, are often employed for this purpose by converting text into numerical node features. However, these non-contextualized embeddings limit the ability of TAG-related models to learn rich semantic knowledge, as they lack contextual awareness to exploit textual attributions.

∗Corresponding author

Recently, Large Language Models (LLMs), with cuttingedge developments such as ChatGPT and LLaMA2 [Touvron et al., 2023] excel in general language understanding, have introduced new possibilities for processing TAGs. LLMs leverage attention-based transformers to adeptly capture the contextual semantics of text, demonstrating superior generalization and zero-shot learning capabilities across diverse linguistic scenarios [Wu et al., 2024; Pan et al., 2024]. However, when used for specialized domain-specific tasks in TAGs, e.g., citation graph categorization of papers in some specialized areas, these models may exhibit limitations due to limited task-specific knowledge, in some cases, even produce erroneous or hallucinatory responses [Shen et al., 2024]. In order to narrow the semantic gap between pretrain and downstream tasks, "pretrain-then-finetune" has become the mainstream paradigm. Fine-tuning on task-specific labeled TAGs data theoretically enhances performance significantly, while the vast size of LLMs, which often includes billions of parameters, necessitates considerable computational resources [Tang et al., 2024; Perin et al., 2024]. In contrast, prompting offers a more resource-efficient alternative by adjusting the inputs to tailor outputs for specific tasks [Yu et al., 2024]. Nevertheless, crafting effective prompts still requires extensive validation data to refine. A pertinent question arises: How to design efficient fine-tuning methods to address the resource-consumption issues of LLMs in specific tasks within TAGs?

Conversely, smaller-scale Pre-trained Language Models (PLMs), such as BERT [Devlin et al., 2019] and RoBERTa [Liu et al., 2019], can be flexibly fine-tuned for downstream tasks to leverage the rich semantics encoded in domain-specific textual features. In this work, we use PLMs to refer to relatively small language models that can be trained and fine-tuned under limited resources in the academic research labs. Fine-tuning involves using labeled samples as additional training signals in a task-specific manner, thus finetuned PLMs can be adapted to domain-specific terminology. Despite these advancements, the scarcity of labeled data in specialized domains complicates the training and fine-tuning of PLMs within TAGs, often leading to poor generalization of unseen data. Thus, the primary challenge remains: How to develop strategies that allow small-scale PLMs learn sufficiently generalized representation from limited labeled samples within TAGs.

While initial LM-based training frameworks excel at capturing semantic information, relying solely on textual attributes may hinder the representation learning in TAGs due to issues such as noise or incompleteness in the original text. Modeling node relationships to enable message passing offers a strategic solution, with a common approach being to cascade LM-generated text representations intoGNNs [Chen et al., 2024]. To illustrate this concept further, the three training pipelines, as previously mentioned, are described in Figure 1. Non-contextualized shallow feature learning involves using shallow text encoders to process text data without semantic understanding. LLMs leverage the linguistic capabilities to learn representation without task-specific optimization. Fine-tuned PLMs for specific tasks allow better adaptation to particular requirements, improving performance. As a result, those context-aware representations are more flexible and robust than static, shallow features [Apidianaki, 2023].

Based on the above observations, the objective of this research is to address the challenges of empowering textattributed graph learning by developing resource-efficient methods for fine-tuning LLM-based representations. Toward this goal, we propose HiTuner to enhance both performance and scalability while addressing resource limitations. Initially, PLMs are fine-tuned on domain-specific datasets to acquire detailed knowledge pertinent to the field. Then, we extract different hidden states of LLM to aggregate versatile hierarchical text representations, harnessing their capacity to integrate extensive knowledge. To ensure diversity in the learned representations, hidden representations of different transformer layers in LLMs are selected for further process. Subsequently, we propose a mixed approach to tailor text representation processes to better align with the specific needs of the domain represented in the TAGs. This targeted approach can optimize computational efficiency by minimizing the computational overhead associated with LLMs. Finally, HiTuner employs a confidence network to adaptively fuse these diverse representations, enhancing the accuracy and reliability for subsequent analyses. An illustration of HiTuner is shown in Figure 2. The key contributions of this paper are as follows.

• Distilling Hierarchical Contextualized Representations for Resource Efficiency: To harness the complementary semantic information at different levels, we strategically select and integrate hidden states from LLMs without prohibitive computational demands.

• Task-oriented Knowledge Infusion for Semantic Comprehension: We develop a hybrid representation that amalgamates the extensive knowledge of LLMs with the targeted insights of fine-tuned PLMs, incorporating structure-related semantics related to TAGs.
• SOTA performance and Great Scalability: Extensive experiments across various TAG benchmarks show that HiTuner significantly boosts baseline performance, while also demonstrating adaptability and generalization capabilities.

2 Preliminaries

In this section, we establish some relevant notations and formalize the research problem of this work. Primarily, we introduce the definition of TAGs and three popular learning paradigms associated with them.

2.1 Text-Attributed Graphs

A general graph G = (V, E) consists of a set of nodes V, and a set of edges E connecting these nodes. While TAG can be formally represented as GT = (V, E, T ), where T is the document set. Each node v ∈ V is associated with a sequential text tv ∈ T and the corresponding label yv. In this work, we focus on node classification, aiming to predict labels for unlabeled nodes V U based on a given set of labeled nodes V L.

2.2 GNN-based Paradigm

GNNs are message-passing frameworks for graph-structured data that aggregate information across neighboring nodes to learn the vectorized representation. Mathematically, the update process of GNNs for each node v can be expressed as:

where u is the neighbor of node v, N (v) denotes the set of neighbors of v, and AGG (·) and UPD (·) are aggregation and update functions, respectively. The initial node feature vector x(0) v is a numerical feature extracted from the text tv.

2.3 Pre-training of PLMs

PLMs encode the vector of a word by dynamically combining it with surrounding words. A text document tv ∈ T can be tokenized into a sequence of tokens Sv = {sv,i}|Sv| i=0 , where sv,i denotes a specific token-id. A PLM takes the sequence of tokens Sv as input, and produces the hidden states of each token:

Generally, the BERT-like language model can be fine-tuned for various tasks by simply adding a classifier on top of the [CLS] token, a special classification token placed at the beginning of each sequence. This token utilizes the selfattention mechanism to capture the contextual information relevant to the specific task. Here, we denote the final hidden state of the [CLS] token sv,0 as semantic-aware representation sv:

To predict specific outputs for downstream tasks, one can simply adopt a multilayer perceptron (MLP) as a predictor and the model parameters are trained with

2.4 LLMs with Prompt

LLMs are first pre-trained on large-scale text datasets to learn common sense and then generate outputs based on a prompt P to process the input without modifying the model parameters, rather than being fine-tuned for specific downstream tasks. Specifically, prompting methods use templates containing unfilled slots, which modify the original text into a textual string prompt. For example, the task is to classify the topics to which papers belong based on their abstracts [ABSTRACT]. By inserting some prompts in the sentences, we can create a template with placeholders for specific informa-

We denote this new sentence with the prompt [TOPIC] inserted as SP|v, where P represents the inserted prompt tokens.

LLMs are built with multi-layer transformer encoders that use attention-based mechanisms to aggregate information in the sequence of tokens SP|v. Thus, these hidden states can be viewed as representations at different levels:

where ENC(l) (·) denotes the l-th layer of LLM, and {h(l) v,i }|SP|v| i=0 means hidden state sequence in l-th layer for target node v. Note that |SP|v| indicates the length of the sequence. A readout function [Reimers and Gurevych, 2019] can be applied to token-level representations:

Here, h(l) v denotes the resulting hidden state sequence from the l-layer of LLM, and this work adopts mean pooling.

3 Proposed Framework

In this section, we begin by illustrating our motivation through two experiments. Building upon these insights, we then introduce the proposed framework, referred to as HiTuner, which uses fine-tuned PLMs with domain expertise as tuners for hierarchical LLM contextualized representations.

3.1 Motivation

Hierarchical Insight fromContextual Layer

While the attention mechanism is applied uniformly across all transformer layers of LLMs, previous research has shown that different contextual layers retain lexical meaning at varying levels of abstraction [Yuan et al., 2023]. We revisit the learning capabilities at varying levels of LLMs in Figure 3. In this setup, the hidden state sequences from various layers of LLMs are then directly fed into an MLP trained for classification tasks. The analysis indicates that deeper layers capture richer semantics but may degrade beyond a certain depth, and the optimal layer varies across datasets. This suggests that the traditional method of extracting the last layer of the LLMs as a representation may not be the optimal strategy, because the information captured by different contextual layers can be complementary. This observation inspires us to exploit multiple hidden states from LLMs to better understand hierarchical semantic information in text.

Leveraging Domain Expertise in Fine-Tuning PLMs

To validate the previously discussed concepts, Figure 4 illustrates experimental evidence comparing the performance of fine-tuned PLMs with unfine-tuned LLMs across three distinct datasets from different domains. Remarkably, LLMs excel across diverse domains, even with a minimal labeling rate of 1%, while the performance gains between 1% to 10% supervision are modest. In contrast, fine-tuned PLMs fail at low labeling rate settings, but show a significant improvement in ACC from 1% to 5%, and maintain a slight edge over LLMs at 10% supervision. These results not only confirm the limitations of unfine-tuned LLMs in specific domains but also highlight that fine-tuning may fail to transfer sufficient targeted knowledge for downstream tasks under label-scarce scenarios. This phenomenon inspires us to leverage task-specific knowledge in fine-tuned PLMs to compensate for the domain semantic gaps, while utilizing the linguistic capabilities of LLMs to overcome label scarcity.

3.2 Distilling Hierarchical Contextualized Representations

From various contextual layers of the frozen LLM, hidden state sequences {h(m) ∈ RDL }Mm=1 are pre-computed in advance, where M<L. Each vector h(m) captures unique information pertinent to its respective context or level. Generally, handling such high-dimensional data often requires dimensionality reduction techniques to focus on the most informative features. We define a denoised projector P : RDL → RDP to filter out noise and redundant information. h(m) is first linearly transformed by specific weight matrix W(m) ∈ RDL×DP , which maps the high-dimensional data into a new space. The transformed data is then subjected to a threshold-based filter, where higher values of τ result in a greater proportion of the input being filtered out,

where Pτ (·) creates a differential response to distinguish between different types of feature activation, potentially adding robustness against noise and providing additional contextual information.

3.3 Semantic and Structure-aware Augmentation

Enhancing Task-oriented Semantic Understanding. The unsupervised representations learned by LLMs capture the language regularities in large-scale corpus, but may lack sensitivity for specific tasks. Integrating fine-tuning processes in deploying LLMs, such as updating parameters, requires a lot of resources and time, so we choose to incorporate the learned task-oriented knowledge by small-scale PLMs into the semantic representation at a lower cost. Denoting s as the fine-tuned PLM representation, we simply design the following mixed strategy:

where λ is a trade-off parameter to balance the influence of the fine-tuned and unsupervised representations. Overall, m(m) encodes both general knowledge and task-specific refinements, enabling efficient task adaptation with lower finetuning overhead.

Incorporating Global Structural Information. Incorporating topological information significantly boosts the capability of models to interpret structure-related semantics within TAGs. This is beneficial in specialized domains like academic research, where analyzing citation patterns can help categorize papers. To effectively leverage the information from a few labeled samples and extend their influence to more unlabeled samples, HiTuner inputs the mixture representation m(m) to GNNs for accurate comprehension of node semantics from a global perspective. Each layer progressively encodes complex structural and contextual information into the node representations. Formally, we denote the GNN-based paradigm as

where ΘG is the trainable parameters, and E provides the structural context. The final node output z(m) encoded by GNNs is better to meet downstream graph-related tasks such as node classification and link prediction.

3.4 Confidence-based Fusion

Subsequently, HiTuner integrates the enhanced representations {z(m) }Mm=1 to infer final predictions. To assess the reliability of each representation, we compute its true class probability, i.e. the likelihood assigned to the correct label. Therefore, higher true class probabilities indicate greater model reliability, thereby serving as a critical measure for assessing confidence. In situations where only a limited subset of samples is labeled, confidence network is employed to estimate the certainty of these predictions:

Here, t(m) v quantifies the predictive confidence of node v of the m-th level contextualized representation and fCONF (·) is achieved by MLP combined with a sigmoid function to normalize the output between 0 and 1.

Furthermore, a relative confidence metric ψ is introduced to dynamically adjust the influence of the m-th level by comparing to the aggregate confidence of other levels:

Then, fusion weights are derived using the softmax function to ensure that contribution of each level is proportional to both its assessed and relative confidence:

Therefore, this weight can dynamically adjust according to the confidence, preventing low-confidence representation from affecting the final results, expressed as

Finally, we define the loss function used to train the model as the cross-entropy loss between predictions and true labels:

All trainable parameters in the proposed method are denoted by Θ = {ΘG,ΘC, τ}. The entire training process only requires training Θ, and the subsequent learning does not involve the updating of LLM parameters. By leveraging the knowledge from smaller PLMs, we can efficiently enhance the semantic representation without incurring the high resource expenses caused by full-scale fine-tuning of LLMs.

4 Experiment

Our experimentation focuses on the following questions:

RQ1: Can HiTuner enhance the performance in modeling TAGs compared with baselines?

RQ2: Are all components comprising of HiTuner valid?

RQ3: Is the HiTuner sensitive to the fusion of two kind of semantic knowledge in different domains?

RQ4: Does HiTuner have excellent scalability?

4.1 Implementation Details

In this section, we perform node classification with low labeling ratio to valid the effectiveness of HiTuner. All experiments are conducted on NVIDIA A100 GPUs with 80GB memory. By default, we employ LLaMA2-7B [Touvron et al., 2023] and BERT as the backbones, and SAGE as an instance of GNNs. For a fair comparison, we run 5 times and report the mean result and the standard deviation.

Datasets

We evaluate the proposed method on various types of datasets, with the relevant statistics summarized in Table 1. These datasets encompass three citation networks, including CiteSeer, Cora and PubMed. Along with a social network dataset (Instagram), an E-commerce dataset from Amazon (Electronics-Photography, namely Photo) and a Wikipediabased dataset (WikiCS). Specifically, the ratio of nodes used for the train/valid/test stage is 10%/10%/80%.

Comparison Methods

Following [Chen et al., 2024], we compare HiTuner with three types of comparison methods:

(1) Benchmark models: For GNN-based method, we select GCN [Kipf and Welling, 2017] and SAGE [Hamilton et al., 2017], while MLP as a baseline. We test 3-layer architectures with a hidden dimension of 256, and each layer is accompanied by a batch operation. We also consider initial node features with three textual feature extraction methods:

• Non-contextualized Shallow Embeddings (FS): Text embeddings extracted using shallow methods, such as BoW, TF-IDF, and word2vec.
• LLMEmbeddings without Fine-tuning (FLLM): Text embeddings taken directly from a LLM without the application of specific task labels. Additionally, the hidden states based on prompts.
• Fine-tuned PLM Embeddings (FFLM): Text embeddings extracted from BERT-like PLM that has been finetuned on a specific dataset.
• Standard fine-tuned PLMs: We select three popular LMs: BERT (bert-base-uncased) [Devlin et al., 2019], DeBERTa (deberta-base) and SentenceBERT (bert-base-nlimean-tokens) [Reimers and Gurevych, 2019]. In this setting, we employ a simple cascading structure, where PLMs are first fine-tuned on the downstream datasets, and the generated text embeddings are input as the initial node features for the downstream classifier.

(3) LLM-enhanced methods: We further compare the following SOTA methods that combine LLMs and GNNs:

• TAPE [He et al., 2024] leverages LLMs to generate pseudo labels and explanations as enhancements, and concatenates themwith original text for PLMs.
• ENGINE [Zhu et al., 2024] combines LLMs and GNNs via a tunable side structure to adopt message passing at each layer of LLMs to integrate structural information.
• GraphAdapter [Huang et al., 2024] introduces a residual learning procedure to pre-train GNNs, which serve as efficient adapters when integrated with LLMs.

4.2 Overall Comparison (RQ1)

We present the comparative analysis in Table 2. From the experimental results, we derive the following observations:

Observation 1: Incorporating semantic information into features significantly improves performance. A consistent performance hierarchy is evident among different feature types: FS may suffice for basic tasks, but it lacks the depth to grasp semantic nuances. In contrast, LLM-generate message greatly improves the model's ability to handle ambiguity and context changes. On one hand, FLLM effectively captures language regularities from extensive corpora, but the improvement offered by LLMs remains limitations without a well-designed prompt. On the other hand, FFLM trained with supervised information demonstrates superior performance with large variations on several datasets.

Observation 2: Contextualized representation cascaded GNNs provide a strong baseline. The standard PLM pipeline, which focuses primarily on tasks like text classification, neglects the inherent correlations within global graph structures. PLMs with structure-aware cues obtain relative improvements ranging from 4% to 8%, indicating that natural language understanding is important on these datasets. On Cora and Photo datasets, the advantages of using GCN and SAGE are more pronounced, demonstrating the role of graph structures for enhanced predictive performance. However, the inclusion of neighboring information occasionally leads to performance degradation, as in PubMed, which may be caused by "heterophily" problem in graph.

Observation 3: With enhancing semantic information, HiTuner achieves state-of-the-art performance. By distilling hierarchical contextualized representations from LLMs and leveraging semantics of category labels from fine-tuned PLMs, HiTuner strikes a balanced synergy between these two approaches, surpassing the performance of each method when applied independently. HiTuner can effectively model TAGs with outstanding performance across a wide range of datasets, highlighted by the best average performance (81.47%), demonstrates its effectiveness in combining semantic understanding with graph-based learning.

4.3 Ablation Study (RQ2)

We evaluate all components by removing each item from HiTuner in turn: (a) w/o structural information: replacing GNN with MLP; (b) w/o hierarchical representations: only take the last layer of LLMs as input. (c) w/o task-specific knowledge: λ is set to 0, training the HiTuner without any task-specific knowledge. (d) w/o confidence-based fusion: simple average fusion is used to instead. Observation 4: HiTuner benefits from integrating complementary textual and structural semantic. From Table 3, we observe that removing any components leads to decrease performance. Furthermore, the importance of these components across different datasets. Overall, removing structural information in the most significant performance drop across all datasets, aligning with the nature of TAGs that structure-related information plays a more pivotal role than pure text. Excluding task-specific knowledge also impacts performance, confirming that fine-tuning complements general LLM-derived features by adapting them to dataset-specific nuances.

4.4 Parameter Sensitivity (RQ3)

The trade-off parameter λ is chosen from 0.1 to 0.9 with step size 0.1, and the number of layers M varies from 2 to 8. As demonstrated in Figure 5, we arrive at Observation 5: HiTuner effectively balances effectiveness and model complexity across diverse datasets. λ regulates the mixing ratio of general language knowledge and task-specific knowledge, where higher λ values increase the influence of task-specific knowledge but may impact the model's capacity for generalization to unseen data. Performance fluctuates noticeably as M changes, suggesting that adding more layers does not consistently improve results. Overall, moderate λ values combined with moderate M values achieve desired performance.

4.5 Scalability Study (RQ4)

We conduct a scalability study focusing on the performance across varying proportions of training data, and different LLMs (Vicuna1.5-7B [Chiang et al., 2023] and Baichuan27B [Yang et al., 2023]). As depicted in Figure 6, HiTuner consistently outperforms several baseline models as the training ratio increases, highlighting its ability to effectively integrate features for enhancing generalization. Furthermore, as shown in Figure 7, the choice of LLMs does not have a significant impact on the performance, demonstrating the favorable compatibility of the proposed framework. It can derive Observation 6: HiTuner is a generalizable and scalable framework, proficient in adapting to diverse training environments and model architectures.

5 Conclusion

To enhance the task-specific understanding and reasoning capabilities of LLMs, we proposed HiTuner, a framework designed to bridge the gap between pre-training and downstream objectives. Focusing on fully unleashing the potential of LLMs for TAGs, HiTuner efficiently distilled hierarchical contextualized representations while minimizing unnecessary computational overhead. Specifically, we fine-tuned smallscale PLMs, infusing task-oriented knowledge at a lower cost and making the framework more adaptable to diverse contexts. Furthermore, to enhance the quality of node representations in TAGs, we integrate these refined representations into GNNs, enabling them to capture and interpret structurerelated semantics within TAGs. Extensive experiments across various TAG benchmarks reveal that HiTuner significantly boosts baseline performance.

Acknowledgments

This work is in part supported by the National Natural Science Foundation of China under Grants U21A20472 and 62276065, the Fujian Provincial Natural Science Foundation of China under Grant 2024J01510026, and the Fujian Provincial Department of Education Youth Project of China under Grant JZ230011.