GPL4SRec: Graph Multi-Level Aware Prompt Learning for Streaming Recommendation

1 Introduction

Recommender systems (RS) have become a fundamental component of online platforms like Amazon, Netflix, and YouTube, driving personalized user experiences and content discovery. However, traditional recommendation models, which are trained on offline static datasets, often suffer from performance degradation when deployed in streaming scenarios [He et al., 2020]. To overcome this limitation, the concept of streaming recommendation has been introduced, focusing on the dynamic updating and deployment of recommendation models in response to continuous data streams [He et al., 2023; Zhang et al., 2024; Yang et al., 2024].

Recently, Graph Neural Networks (GNNs) have stood out as a groundbreaking approach in traditional static recommendation scenarios [He et al., 2020; Guo et al., 2023], relying on their strong capability to capture complex collaborative patterns between users and items. Nevertheless, due to the necessity for continuous model updates in streaming scenarios, these conventional GNN-based recommendation models can not be directly applied to handle streaming data. Therefore, some recent studies have concentrated on two primary areas: dynamic graphs and graph fine-tuning. First, dynamic graph methods, such as EvolveGCN [Pareja et al., 2020] and ROLAND [You et al., 2022], typically utilize recurrent neural networks on the graph to capture and track ever-evolving user preferences. However, these methods struggle to capture rapidly changing user preferences, exhibiting over-stability [Ostapenko et al., 2021] in incremental learning where model overly reliant on historical knowledge and fail to adapt to sudden and dramatic shifts. Second, graph fine-tuning approaches, such as DEGC [He et al., 2023] and G-Tuning [Sun et al., 2024], mainly focus on more efficiently updating the parameters of pre-trained graph models as new interaction data arrives, enabling faster and more precise adaptation to the evolving user preferences from new data.

∗ Corresponding author.

However, the above graph fine-tuning methods may face the problem of Catastrophic Forgetting [Chang et al., 2017a] in streaming environments. Recently, a novel method known as graph prompt learning has emerged as an effective and adaptive technique for mitigating the issue of catastrophic forgetting. These methods, such as GPF [Fang et al., 2023] and GraphPrompt [Liu et al., 2023], not only aid in preventing catastrophic forgetting by maintaining the integrity of the model fundamental knowledge but also ensures effective learning and adaptation to new and diverse data patterns through node prompts and structural prompts. Node prompts mainly focus on continual evolutions in individual users and items attributes, while structural prompts primarily emphasize the incremental changes of one-hop useritem relationships. However, the design of existing graph prompt templates [Fang et al., 2023; Yang et al., 2024; Liu et al., 2023; Zhang et al., 2024] focusing solely on the incremental changes of nodes and edges is suboptimal. In fact, these changes within the graph are not merely incremental variations in nodes and edges but also cascaded, profoundly influencing the entire graph. This cascading nature of these changes implies that incremental updates simultaneously affect multi-hop cascading relationships within the graph. The neglect of such multi-hop cascading relationships hinders the full integration of the model historical knowledge and new data, which is crucial for more accurately capturing shifts in user preferences. Therefore, as shown in Table 1, designing a prompt template that not only captures incremental changes in nodes and edges but also accounts for multi-hop cascading changes across graph layers presents a unique challenge. To address the problems mentioned above, we propose GPL4SRec, a Graph Multi-Level Aware Prompt Learning for Streaming Recommendation. Specifically, we first pre-train a graph encoder on a large volume of historical data to effectively capture users long-term preferences. Then, we design three distinct types of graph prompts: node-aware, structureaware, and layer-aware prompts, which serve to guide the pre-trained encoder in more effectively capturing users shortterm preferences. This is achieved by accounting for the incremental changes in nodes and interactions, as well as the cascading modifications in multi-hop relationships between users and items. Our theoretical analysis demonstrates that the design of these prompt templates plays a pivotal role in achieving superior performance. Finally, experimental results also demonstrate that our model significantly outperforms the state-of-the-art approaches, especially in mitigating the issue of catastrophic forgetting in SRec. In summary, the main contributions of our work can be concluded as follows:

• We propose a novel graph multi-level aware prompt learning framework for streaming recommendation that effectively mitigates the issue of catastrophic forgetting in streaming scenarios.
• We design three types of graph prompt templates: nodeaware, structure-aware, and layer-aware prompts, which effectively guide the model to capture both incremental and cascaded changes within the graph.
• We provide a theoretical analysis showing that our prompt templates are essential for achieving superior performance. Moreover, extensive evaluations on three datasets demonstrate the state-of-the-art performance of our model in streaming recommendation.

2 RelatedWork

There are multiple topics related to our GPL4SRec. In this section, we first review existing researches on streaming recommendation, followed by a discussion on current dynamic graph learning and graph prompt learning techniques.

2.1 Streaming Recommendation

Traditional RS, trained on static datasets, struggle to accurately capture the evolving user preferences when applied to streaming scenarios. Therefore, streaming recommendation [Chang et al., 2017b; Wang et al., 2018] has been introduced to address this challenge by updating and deploying recommendation model in response to continuous data streams. Early researches [Chandramouli et al., 2011; Lommatzsch and Albayrak, 2015] primarily focused on analyzing the popularity, recency, and trends of items , followed by the introduction of matrix decomposition, collaborative filtering, and other methods combined with online clustering technology [Li et al., 2019; Li et al., 2016]. Recently, GNNs have become a research hots pot in the field of recommendations due to their powerful relationship modeling capabilities [Ying et al., 2018; Chen et al., 2020; Chen and Wong, 2020]. However, the challenge lies in adapting GNNs, initially trained on static datasets, to effectively manage the continuous data stream in streaming recommendation scenarios [Wang et al., 2020; Xu et al., 2020b].

2.2 Dynamic Graph Learning

Dynamic graph learning has attracted significant attention in recent years, with researches primarily categorized into snapshot-based and event-based methods, depending on their temporal granularity. Snapshot-based approaches (such as EvolveGCN, DGCN, and ROLAND [Pareja et al., 2020; Li et al., 2020; You et al., 2022]) construct dynamic graphs from scratch and leverage recurrent neural networks to capture temporal changes. While effective in modeling smooth transitions, these methods often struggle with abrupt shifts in user preferences and are prone to noise in user and item representations [Ostapenko et al., 2021], thereby limiting their applicability in streaming environments. In contrast, event-based methods [Trivedi et al., 2019; Ma et al., 2020; Xu et al., 2020a] focus on capturing graph evolution at finer temporal granularity, but challenges remain in balancing computational efficiency and temporal precision.

2.3 Graph Prompt Learning

Graph prompt learning has emerged as a powerful technique due to its adaptability and effectiveness in graph tasks. Recently, several significant advancements in this area have been proposed. GraphPrompt [Liu et al., 2023] introduces a universal prompt template with a learnable readout function, enabling the unification of multiple downstream tasks. GPF [Fang et al., 2023] proposes a universal prompt-based tuning method for pre-trained GNNs, aligning pre-training and downstream tasks through feature-space adjustments, without the need for task-specific prompt designs. GPT4Rec [Zhang et al., 2024] first introduces graph prompt learning for streaming recommendation through designing special prompt templates. GraphPro [Yang et al., 2024] incorporates time-aware and structural prompts for dynamic recommendations, extending the capabilities of pre-trained models to accommodate evolving data. Although these methods have achieved acceptable results, the suboptimal design of existing graph prompt templates remains a critical challenge. Our research complements the shortcomings of existing prompt templates and designs three types of prompts taking into account both the incremental and cascading changes within graph, which can better guide the seamless integration of historical knowledge and new data for mitigating the problem of catastrophic forgetting in streaming recommendation.

3 Preliminaries

In this section, we first formalize the task definition for streaming recommendation. Then we briefly introduce the definition of graph incremental learning for streaming recommendation used in this paper.

Definition 1. Streaming Recommendation. Real-world RS must process a continuous stream of user-item interaction data, denoted as D. This continuous data stream is partitioned into sequential snapshots [D1,...,Dt-1,Dt,...,DT ] with an equal time span. At each time snapshot t, the model needs to optimize its performance on Dt by combining previous knowledge from historical snapshots [D1,...,Dt-1]. The recommendation performance is subsequently evaluated across the entire timeline.

Definition 2. Graph Incremental Learning. Given the data stream defined above, the graph snapshots G are represented as [G1,...,Gt-1,Gt,...,GT ] on the data snapshots D, where Gt = Gt-1 + ∆Gt. Gt = (At,Xt) is an attributed graph at time t, where At and Xt are the adjacency matrix and node features of Gt. ∆Gt = (∆At,∆Xt) represents the changes of graph structures and node information at time t. Therefore, the goal of graph incremental learning (GIL) is learn ∆Gt(Dt) sequentially while transferring previous knowledge to new graph segments effectively. Mathematically, GIL aims to learn the optimal graph structure St and paramters Wt at each segment t, formulated as:

where (St,Wt) ∈ (S,W). The S and W are corresponding search spaces, respectively. The function Lt(St,Wt,∆Gt) denotes the loss for the snapshot t, evaluated on ∆Gt.

4 Methodology

The GPL4SRec framework shown as Figure 1 consists of two main stages: graph pre-training learning and graph prompt learning. During graph pre-training, we pre-train a graph encoder on massive historical interaction data, enabling it to effectively capture user long-term preferences. During graph prompt learning stage, we primarily focus on designing node-aware, structure-aware and layer-aware prompts to capture multi-level changes within the graph, further adapting user evolving short-term preferences. Finally, theoretically analyze the critical role of prompt template optimization in achieving improved performance.

4.1 Graph Pre-training

To better capture the evolving nature of user preferences, we decompose them into long-term preferences (LTP) and shortterm preferences (STP). LTP generally remains stable over time, influencing multiple snapshots and is often shaped by factors such as gender, occupation, family and education. In contrast, STP exhibits rapid fluctuations and is more relevant within specific time windows, typically driven by contextual factors such as user emotions. In our framework, the model is designed to learn and integrate both LTP and STP within its parameters, adapting to the dynamic shifts in user preferences over time. Therefore, to effectively capture and encode LTP, we pre-train a graph encoder using extensive historical interaction data, ensuring that LTP is accurately reflected in the user embeddings. The learning objective is based on contrastive learning, optimizing the LTP representation:

where s(·, ·) represents a similarity function such as inner product in our experiment, Pu and Nu are respectively the set of positive instances and negative instances for u. Within our proposed framework, we incorporate the Bayesian Personalized Ranking (BPR) [Rendle et al., 2012] algorithm in conjunction with other graph self-supervised learning strategies [Wu et al., 2021; Huang et al., 2021].

4.2 Graph Prompt Learning

After grasping LTP through graph pre-training representation learning, we mainly focus on capturing STP via graph prompt learning. Specifically, we design node-aware, structure-aware and layer-aware prompt to capture the comprehensive essence of graph patterns in streaming recommendation.

Node Aware Prompt Template Design

Node-aware prompts focus on the attributes or properties of individual nodes within the graph, such as user and item characteristics in recommendation models. By emphasizing the node level, GPL4SRec effectively captures the nuances of node-specific data, providing a deeper understanding of user behaviors and item features. This level of analysis is crucial for tasks that require personalized recommendations or detailed attribute evaluation.

Specifically, the node aware prompts consist of a set of learnable parameters, denoted as NP = [np1, ..., npn]∈ Rn×f , where n is the number of node prompts and f is the feature dimension. These node prompts serve as targeted cues, guiding the model in interpreting and integrating new information related to users or items. We first employ an attentional mechanism based on softmax function to help automatically determine how these prompts transform each node's representation. We then introduce an additional learnable prompt weight λi to further refine the node prompting mechanism and help prevent the model from overfitting to noise and irrelevant information in the new data. The λi can be learned by using a simple multi-layer perceptron (MLP) based on the node feature as follow:

where the w represents the learnable parameters which are shared across all nodes and σ denotes the sigmoid function to enforce λi ∈ [0, 1], denoting the confidence of node xi to be prompted. Therefore, the prompted node is X = [xi], which represents the search space of node. These prompts undergo an attention mechanism that selectively adjusts the weights of data most pertinent to the current node.

Structure Aware Prompt Template Design

In addition to node aware prompts, structure aware prompts are devised to interact with the more extensive connectivity and relationship patterns within the graph. These prompts play a vital role in comprehending and adjusting to alterations in the overall graph topology, like the appearance of novel interaction patterns or the development of existing ones.

The structure aware prompt is designed as follows: a set of learnable parameters, denoted as SP = [sp1, ..., sps]∈ Rs×f for the nodes that adaptively aggregate the structure information via prompting adjacency matrix A = [aij]. To be specific, given a node xi, the node ˆxi prompted by structure aware prompts can be computed:

where eij = fmlp(xi∥xj∥tij) denotes the feature of edge (i, j) which is obtained by a MLP based on node xi, xj and interaction time tij . The prompted structure A = [aij ] represents the search space of structure. Therefore, these prompts can guide models to capture dynamic structural changes for adapting to the evolving nature of the data.

Layer Aware Prompt Template Design

The importance of node and structural information on the graph is widely recognized. Nevertheless, current research has also revealed that the relationships among layers are extremely critical, as different layers focus on distinct information. For example, in recommendation models, different layers respectively pay attention to the relationships between user-user, user-item and item-item. Therefore, designing a layer-aware prompt that can better capture the cascading relationships among layers poses a great challenge.

To address this challenge, we design the layer aware prompts as follows: a set of learnable parameters, denoted as LP = [lp1, ..., lpl]∈ Rl×f for the nodes via prompting each graph layer embedding xki . Given a node xi, the node ˜xi prompted by layer aware prompts can be computed:

where K denotes the total layers of GNNs and xki represents the kth layer representation of xi. L = [βk i ] represents the search space of layers. These prompts will motivate the model to learn the multi-hop relationship between the different layers of node xi and better extract the relevant node information in the deeper layers of the GNNs.

4.3 Theoretical Analysis

In this section, we conduct theoretical analysis to guarantee the correctness of the proposed graph prompt learning algorithm within the context of streaming recommendation.

THEOREM 1: GPL4SRec's performance is determined by our design of prompt templates, and it satisfies at least the ability of fine-tuning globally using full data.

PROOF: Given a pre-trained graph encoder f and a streaming task including graph and dataset over time D = {(G1, Y1), .., (Gt, Yt), (Gt+1, Yt+1), ..., (GT , YT )}, we update the pre-trained model ft+1 to maximize the likelihood of predicting the correct labels Yt+1 at the time t + 1:

where we use ft to initialize model parameters ft+1. The optimization has the upper bound as [Zhang et al., 2023]:

where ∆Gt denotes the graph gap and the optimization process after using our prompt template is:

where gϕ(∆Gt) is the our proposed graph prompt templates including node, structure and layer levels. Hence, the final object function for the prompt is:

where (A,X, L) is the candidate search spaces of prompted (A,X, L) = prompt(∆At,∆Xt,∆Lt). (∆At,∆Xt,∆Lt) is the incremental and cascading changes in Gt. Equation 13 shows that fixing ft and then prompting the graph gap ∆Gt is equivalent to directly optimizing Equation 11, which is the upper bound of Equation 10. Therefore, Equation 13 can show that our proposed graph prompt learning method satisfies at least Equation 10 fining-tuning with full data and its performance is determined by our prompt templates. □

5 Experiments

In this section, we conduct experiments with the aim of answering the following questions: Q1: How do our proposed GPL4SRec perform compared with other baselines? Q2: What is the influence of key components of GPL4SRec? Q3: How is the robustness of GPL4SRec? Q4: Whether is GPL4SRec sensitive to the hyper-parameters? Q5: How efficient is the training of GPL4SRec in streaming scenarios?

5.1 Experimental Settings

Datasets. We use three public datasets that cover diverse realworld scenarios in streaming recommendation. The Taobao dataset records the implicit feedback from taobao.com, which is a Chinese e-commerce platform, during a period of 10 days. The Koubei dataset, which is provided for the IJCAI'16 contest, documents 9 weeks' worth of user interactions with local stores on Koubei within Alipay. The Amazon dataset is composed of a collection of product reviews from Amazon that spans 13 weeks. Detailed information about these datasets can be found in Table 2.

Evaluation Protocols. In our evaluation, we simulate realworld dynamics using graph snapshots taken at different intervals (weekly/daily). We use a two-step sliding window to learn from current data and predict future changes. Following the Pre-train and Fine-tune paradigm, we pre-train on most of the dataset, fine-tune, and evaluate on later snapshots (see Table 2). For consistency, the same method is applied to all baselines. Dynamic GNNs start fine-tuning with pre-training weights. Results are averaged on future snapshots, and standard metrics like Recall@k and NDCG@k (k = 10 and 20) are applied, in line with prior work [Yang et al., 2024].

Baseline Methods. We include the recent dynamic graph learning methods, graph prompt learning approaches, and traditional graph fine-tuning as our baselines:

• Dynamic Graph Neural Networks. These graph neural networks are tailored to dynamic scenarios, updating embedding with time sensitivity to reflect graph changes. We benchmark our approach against notable models: EvolveGCN-O and EvolveGCN-H [Pareja et al., 2020] and ROLAND [You et al., 2022].
• Graph Prompt Learning Methods. This line focuses on leveraging prompts to get task-specific knowledge related to downstream tasks, thereby becoming a unified approach for downstream tasks. We compare our approach with notable models: GPF/GPF-Plus [Fang et al., 2023], GraphPrompt [Liu et al., 2023], GPT4Rec [Zhang et al., 2024], and GraphPro [Yang et al., 2024].

Implementation Details. We run all methods in PyTorch [Paszke et al., 2017] with Adam [Diederik, 2014] optimizer on an NVIDIA GeForce 4070Ti GPU. In our experiment, the batch size b and the demension of embeddings d are set 2048 and 64. The layers k of GNNs are set 3. We train all models 300 epochs at every snapshot. We apply grid search to find the optimal hyper-parameters for each model. The ranges of hyper-parameters are [16, 32, 64, 128] for the size N of nodeaware prompts NP, the size S of structure-aware prompts SP and the size of layer-aware prompts LP. GPL4SRec is trained with a learning rate of 0.001.

5.2 Overall Perference (Q1)

In this section, we report the experimental results of different methods in the Table 3, where R and N are abbreviations for Recall and NDCG, respectively. It can be clearly observed that our proposed GPL4SRec surpasses both graph prompt learning methods and dynamic graph learning methods, highlighting the effectiveness of our graph pre-training strategy and graph prompt learning methods.

First, this superior performance of GPL4SRec can be attributed to two key factors: 1) Our graph pre-training representation learning that adeptly captures and encodes constant LTP of users during pre-training stage, 2) Our graph multilevel aware prompt templates design, which includes three types of prompts: node-aware, structure-aware, and layeraware prompts. These well-designed prompts ensure seamless knowledge transfer from the pre-trained model across temporal snapshots. Such adaptive learning helps model to better capture incremental and cascaded changes such as STP of users in the graph, which is very important to understand user behavior in streaming recommendation.

Second, there are more findings in these comparative experiments. The different performance of baseline methods shows the complexity of streaming recommendations. Graph prompt learning methods broadly outperform dynamic graphs and fine-tuning methods, indicating the advantage of graph prompt learning in streaming recommendation, especially in mitigating catastrophic forgetting.

Finally, different graph prompt learning methods show certain performance differences, and our proposed GPL4SRec exhibits better performance than other prompt learning methods. We argue that the key point of differences is the design of graph prompt templates. Specifically, GPL4SRec is carefully designed with three types of prompts, which help to comprehensively capture the incremental and cascading changes within the graph and more effectively integrate the historical knowledge and new data, thus generating new and complete knowledge for streaming recommendation.

5.3 Ablation Study (Q2)

In this section, we focus on GPL4SRec and test the efficacy of its various designs in regard to the node-aware, structureaware, and layer-aware prompts. The results are shown in Table 4. We have the following observations:

• w/o NP. We studied the effect of node-aware prompts in adaptation to node changes. Comparing prompted and unprompted models, the former responded better to user and item changes, showing node-aware prompts aid context adaptation for improved personalization.
• w/o SP. We measured the impact of structure prompts on adapting to graph incremental changes. Our results demonstrated that these prompts accurately represent structural patterns, highlighting their significance in capturing onehop graph relationships between users and items.
• w/o LP. We focused on exploring the effect of layer-aware prompts. Our ablation studies have demonstrated that such prompts are capable of capturing cascading relationships from diverse perspectives within the graph, thus enabling a more thorough comprehension of graph data.

5.4 Robustness Analysis (Q3)

Within the framework of GPL4SRec, graph pre-training is of critical significance for capturing user LTP and for enhancing the adaptability of the proposed graph prompt learning method. To conduct a comprehensive assessment of its robustness and adaptability, a systematic exploration of a variety of pre-training strategies was implemented, with MixGCF [Huang et al., 2021] and SGL [Wu et al., 2021] incorporated. This exploration encompassed two core objectives: on the one hand, to verify the performance of GPL4SRec under diverse experimental settings; on the other hand, to guarantee the consistency of objectives between pre-training and prompt tuning, which is of utmost importance for bolstering the overall effectiveness of the model. As shown in the experimental data presented in Table 5 and Table 6, GPL4SRec exhibits remarkable robustness across a wide range of pretraining strategies. It not only attains substantial enhancements in predictive accuracy but also demonstrates outstanding adaptability in complex graph scenarios. Notably, the integration of MixGCF and SGL contributes to strengthening the generalization capabilities of model while simultaneously maintaining its stability, thereby further highlighting the remarkable superiority of GPL4SRec in efficiently handling Table 6: The average performance with SGL as our base model. The numbers in bold indicate statistical improvement (p < 0.01) by the pairwise t-test comparisons over the other baselines.

complex streaming recommendation tasks based on GNNs.

5.5 Hyper-parameter Sensitivity (Q4)

We conducted a detailed hyper-parameter study on the model with a particular focus on the prompt size. Specifically, we explored the impact of varying the node-aware , structureaware, and layer-aware prompt sizes on the Amazon dataset. As shown in Figure 2, in the initial stage, adding the prompt size was found to enhance the performance of model. More prompts could represent complex user-item interactions in a more comprehensive manner, facilitating the model capture of user preferences and item characteristics. However, when the prompt size surpassed a specific threshold, further increments led to diminishing returns. The additional size not only scarcely augmented the explanatory capacity of model but also exerted negligible influence on performance improvement and even caused unnecessary computational overhead. This underlines the significance of precisely ascertaining the optimal prompt size to strike a balance between performance and computational efficiency. It further implies that there exists an optimal range for the prompt size, and exceeding this range would prove detrimental. Future research could focus on delineating this range and its mechanisms.

5.6 Efficiency (Q5)

As shown in Figure 3, we presented the average training time of different models on three datasets per epoch. GPL4SRec rivals fine-tuning and prompt-learning methods in terms of efficiency and surpasses several advanced models in performance. The high efficiency stems from using lightweight graph prompts, which integrate new data seamlessly with low cost, cutting training time and enhancing overall efficiency. Moreover, compared with existing dynamic graph methods, GPL4SRec skips complex graph structure adjustments, averting extra computational costs from frequent reconstruction. Thus, it keeps an efficient training rhythmwith large-scale data, strengthening its edge in efficiency. This makes GPL4SRec a highly promising option for scenarios where rapid model training and deployment are crucial.

6 Conclusion

This study proposes GPL4SRec, a graph multi-level aware prompt learning framework for streaming recommendation task. First, GPL4SRec utilized a graph encoder pre-training on extensive historical data to capture the LTP of users. Then, by introducing node-aware, structure-aware, and layeraware prompts, GPL4SRec could focus on the incremental and multi-hop cascading changes between users and items for accurately capturing the STP of users, thereby mitigating the issue of catastrophic forgetting. Further, our theoretical analysis verified the importance of our prompt template design for GPL4SRec's superiority. Finally, our experiments on three public datasets and three base models confirmed its effectiveness, outperforming existing approaches.

Acknowledgments

This research was partially supported by the NSFC (62376180, 62176175), the National Key Research and Development Program of China (2023YFF0725002), Suzhou Science and Technology Development Program (SYG202328), and the Priority Academic Program Development of Jiangsu Higher Education Institutions.