Learnable Data Augmentation and Contrastive Pre-training for Temporal Link Prediction

1 Introduction

Many real-world networks exhibit dynamic behavior, where interactions or edges carry associated timestamps denoting their occurrence. These are identified as temporal (or dynamic) graphs [Kazemi et al., 2020]. Such dynamism necessitates specialized methods for temporal graph analysis, diverging from traditional graph representation learning techniques [Barros et al., 2021]. Initial endeavors in embedding temporal interaction graphs predominantly yielded static representations [Nguyen et al., 2018; Zhang et al., 2017]. A significant limitation of these approaches is their inability to accommodate newly introduced nodes or edges as graphs progress over time. To address this dynamic element and the need for time-sensitive representations, a new wave of research ushered in the concept of temporal graph neural networks (TGNNs) [Sahili and Awad, 2023], which are crafted to model both temporal nuances and structural dependencies, demonstrating adaptability to emerging nodes and edges [Kumar et al., 2019; Pareja et al., 2020; da Xu et al., 2020; Zhang et al., 2023b; Zhu et al., 2023]. A common theme among these models is the employment of neighborhood aggregation, which dynamically assimilates contextual information essential for node representation learning. Nevertheless, a potential shortcoming is the predominant focus of neighborhood aggregation on immediate contexts and recent history, which may inadvertently sideline the broader global context and invaluable long-term historical insights.

∗Corresponding author

With the advent of graph transformers [Ying et al., 2021; Kreuzer et al., 2021; Dwivedi and Bresson, 2020], a fresh perspective on representation learning in temporal networks has been established. Temporal graph transformers (TGTs) [Cong et al., 2023] have been designed in alignment with traditional practices, primarily utilizing a sequence-based framework, emphasizing the sampling of extensive sequences to encompass both the global context and the historical behaviors associated with a target node. By synergizing these sampled sequences with diverse encodings, TGTs demonstrate a keen ability to identify and interpret prolonged dependencies within the temporal graph. As a testament to their efficacy, TGTs have achieved benchmark performances in a range of temporal graph learning challenges [Wang et al., 2021a; Cong et al., 2023].

The Problem. While Transformer-based approaches have proven effective, they often rely heavily on ample labeled data for model training [Wang et al., 2021a; Cong et al., 2023], which becomes a bottleneck in scenarios with restricted supervision. A promising solution lies in pre-training on graphs, which leverages the intrinsic structures of the graph through a self-supervised approach [Hu et al., 2019; Hu et al., 2020]. Within this context, contrastive learning has emerged as a potent method for pre-training on graphs, particularly for extracting underlying information [Qiu et al., 2020; You et al., 2020; Zhu et al., 2020]. Yet, the application of contrastive learning with graph transformers on temporal graphs remains an under-explored area. This gap is especially pronounced in the realm of temporal graph representation learning under limited supervision. Motivated by this observation, our paper delves into the problem of contrastive learning of graph transformers on temporal graphs.

Challenges. Pre-training graph transformers on temporal graphs holds significant promise but remains an underexplored avenue. Within the domain of temporal graph representation learning, the methodology diverges significantly from the commonly explored contrastive learning that utilizes GNN backbones. Unlike these, graph transformers typically operate in a sequence-based fashion. Therefore, a typical practice is to devise a contrastive strategy tailored for sequence-based learning on temporal graphs. Yet, when considering contrastive learning on graphs, prevalent data augmentation techniques [You et al., 2020] like DropEdge [Rong et al., 2020] and DropNode [Feng et al., 2020] often employ random masking of raw data. However, calibrating the degree of such augmentations is complex, potentially leading to excessive (over-augmentation) or insufficient (underaugmentation) alterations. This poses a conundrum: in the context of contrastive learning on temporal graphs, how to craft an adept contrastive strategy for sequence-based learning that harmoniously integrates an optimal augmentation technique?

Present work. To address the aforementioned challenges, we introduce a novel framework named ContraTGT, for the Contrastive learning of Temporal Graph Transformers. At its core, ContraTGT leverages a dual-view graph transformer, which is tailored to extract sequences for a given target node, encompassing both spatial and temporal views. Such a design facilitates the assimilation of comprehensive contextual information and traces the historical evolution associated with the target node, thereby refining temporal node representation learning. In tandem with this, to proficiently pre-train the dual-view graph transformer, we architect a sequencebased contrastive strategy, which aligns guarantee with the requirements of contrastive learning tailored for temporal graph transformers. Moreover, we introduce an adaptive, learnable data augmentation technique. By selectively masking elements within the dual-view sequences, this method crafts optimized augmentations, thereby enhancing the efficacy of the contrastive learning process.

We leverage dual-view node sequences sampled from temporal graphs, which are derived from both spatial and temporal perspectives, to formulate our contrastive strategy. Particularly, for each node sequence, we employ node masking-specifically, by dropping nodes within the sampled sequences, to act as the data augmentation. The foundational reason behind this approach is that the sequences, even when subjected to such masking perturbations, retain an approximation of the principal context and historical trends associated with the target node. As a result, these modified sequences can effectively function as augmented counterparts in the contrastive learning process.

While random masking serves its purpose, as highlighted earlier, it lacks precision in controlling the extent of augmentation, which can lead to over-augmentation or underaugmentation, potentially compromising the quality of contrastive learning. Therefore, we introduce a learnable data augmentation mechanism, designed to selectively mask nodes within the dual-view sequences. To steer this learnable data augmentation, we define two essential metrics: consistency and diversity. The former, consistency, gauges the prediction alignment between augmented and raw data. A higher consistency suggests that the augmentation closely mirrors the key features of the raw data, leading to aligned predictions. Conversely, diversity evaluates the variance between the augmented representation and its raw counterpart. A greater diversity indicates a more desirable augmentation. This rationale stems from the idea that, beyond capturing primary behaviors, diversity encompasses supplementary or secondary data features-essentially, the minor perturbations. These perturbations help to amplify the distinctions between augmented and raw data, aiding in the label propagation from labeled to unlabeled data. Collectively, these metrics allow for meticulous control over the noise introduced, ensuring optimal augmentations tailored for contrastive learning in temporal graphs.

Contributions. In summary, our contributions can be outlined as follows. We tackle the novel research problem of contrastive learning for graph transformers applied to temporal graphs, which represents a significant and under-explored domain. To address this challenge, we introduce a novel model ContraTGT, designed specifically for the contrastive learning of temporal graph transformers. ContraTGT not only samples sequences from both spatial and temporal perspectives but also incorporates a learnable data augmentation mechanism to generate optimized augmentations for these sequences. We conduct extensive experiments on five realworld datasets, showcasing the superior performance of ContraTGT when compared to state-of-the-art baseline methods.

2 RelatedWork

Graph Representation Learning. The evolution of graph embedding techniques, ranging from early structural mapping [Grover and Leskovec, 2016; Tang et al., 2015] to message-passing paradigms [Hamilton et al., 2017; Velickovi´c et al., 2018], has laid the foundation for modern graph analysis. To capture the evolutionary nature of real-world networks, Dynamic Graph Embeddings (DGEs) extend these concepts into discrete-time snapshots [Pareja et al., 2020; Sankar et al., 2020; Zhang et al., 2023a] and continuous-time event streams [Zhu et al., 2023; Zhang et al., 2023c; Wang et al., 2021c]. While effective, traditional DGEs often struggle to balance fine-grained temporal resolution with global structural dependencies. Recently, Graph Transformers have emerged to capture long-range interactions through self-attention mechanisms [Dwivedi and Bresson, 2020; Kreuzer et al., 2021]. To mitigate the quadratic computational overhead of full attention, sequence-based architectures like TCL [Wang et al., 2021a] and DyFormer [Cong et al., 2023] have been proposed. Building on this, CoLA-Former [Zhao et al., 2025] further refines temporal modeling by integrating to enhance scalability. Parallelly, Graph-Mamba [Wang et al., 2024] introduces the State-Space Model (SSM) paradigm to the graph domain, leveraging selective scan mechanisms to achieve linear-complexity modeling of long-sequence dependencies, thereby surpassing the efficiency bottlenecks of conventional Transformers.

Self-Supervised Learning and Augmentation. Graph Contrastive Learning (GCL) has become a dominant paradigm for learning perturbation-invariant representations without manual labels [Hu et al., 2019; Qiu et al., 2020; You et al., 2020]. In temporal contexts, TCL [Wang et al., 2021a] utilizes time-depth encodings for link-level contrast, while DyFormer [Cong et al., 2023] employs dual-view sequence contrasting. However, these methods typically rely on random sampling or fixed heuristics, which may overlook underlying temporal trends or lead to suboptimal augmentation-specifically, the issues of over-augmentation or under-augmentation [Bo et al., 2022]. To address these limitations, several advanced augmentation strategies have been explored. For instance, JOAO [You et al., 2021] adopts an adversarial framework to select optimal pairs from a predefined pool, though it often neglects representation consistency. Similarly, GCA [Zhu et al., 2021] introduces adaptive augmentation based on node centrality, yet its dropping mechanism remains static and nonlearnable during training. Unlike these approaches, our proposed ContraTGT framework devises a learnable data augmentation strategy guided by both consistency and diversity, specifically optimized for capturing the intricate evolutionary patterns in temporal graphs.

3 Preliminaries

Graph and Temporal Graph. A static graph is defined as G = {V,E,X}, where V is the set of nodes, E ⊆ V ×V is the set of edges, and X ∈ R|V |×dX denotes the node feature matrix. A temporal graph incorporates temporal dynamics, represented as G = {V, E, X, T, ϕ, φ}. Here, T denotes the time domain, while ϕ : (V, T) → {0, 1} and φ : (E, T) → {0, 1} serve as indicators for the existence of nodes and edges at a specific timestamp t ∈ T , respectively.

Problem Formulation. Given a node v at timestamp t, we aim to derive a representation h(v,t) (simplified as hv) by leveraging its contextual and historical information. We focus on the dynamic link prediction task. For an interaction (v, u) at time t, the existence probability pt(v,u) is calculated as:

where ∥ denotes the concatenation operation. The model is optimized using binary cross-entropy loss over a set of training tuples Ttr:

where Uv denotes the uniform distribution for negative sampling. To address scenarios with limited supervision, we employ self-supervised contrastive learning to capture underlying structural nuances.

Graph Transformers. The Transformer architecture [Vaswani et al., 2017] utilizes multi-head self-attention (MSA) to model dependencies within a sequence H(l-1) = [h(l-1) 0 , . . . , h(l-1) n-1 ] ∈ Rn×dl-1 . In layer l, the input is projected into Query, Key, and Value matrices:

where W(l) Q,K ∈ Rdl-1×dKl and W(l) V ∈ Rdl-1×dVl . The attention output is computed via:

The hidden state is updated through a Feed-Forward Network (FFN) with Layer Normalization (LN) and residual connections:

Stacked L layers produce the final sequence representation H(L) for downstream tasks. We denote the parameter set of the Transformer as θt.

4 The ProposedModel: ContraTGT

As illustrated in Fig. 1, ContraTGT consists of three main stages: (1) dual-view sequence sampling (Sect. 4.1), (2) contrastive pre-training with learnable data augmentation (Sect. 4.2), and (3) fine-tuning for downstream tasks (Sect. 4.3).

4.1 Dual-View Sequence Sampling

To circumvent the quadratic complexity of global attention on large graphs, we adopt a sequence sampling strategy to capture local context and historical dynamics.

Spatial Sampling. For a target node v, we sample its ls nearest neighbors based on interaction timestamps. These neighbors are arranged in descending order of time, as recent interactions typically exert more influence. For node v0 in Fig. 1(b), the spatial sequence is (v0, v1, v3, v2, v4). For each node, the time interval ∆t is computed relative to the current timestamp tnow and transformed via time encoding [da Xu et al., 2020].

Temporal Sampling. Temporal sampling captures global trends preceding the target node's latest interaction. If v's latest interaction occurred at tlast, we sample lt interactions immediately preceding tlast in the graph. The involved nodes are chronologically ordered to form a sequence, e.g., (v0, v1, v4, v2, v7) in Fig. 1(b). This allows the model to capture historical trajectories beyond the immediate spatial neighborhood.

4.2 Pre-Training with Contrastive Learning

We propose a contrastive learning strategy driven by learnable data augmentation to avoid the sub-optimal views generated by random masking.

Consistency and Diversity. Guided by [Bo et al., 2022], we optimize augmentations ˆx = Ψ(x; θΨ) using two metrics: Consistency: Ensures the prediction f(hˆx) remains invariant compared to f(hx).

Diversity: Maximizes the representation distance d(hx, hˆx) to provide a challenging view.

Figure 1: The overall framework of ContraTGT. (a) Temporal graph input. (b) Dual-view sequence sampling for target node v0. (c) Pretraining with learnable masking based on consistency and diversity. (d) Fine-tuning for link prediction between v0 and v5.

Algorithm 1 Pre-Training of ContraTGT

Require: G, ls, lt, α, ρ;

Ensure: θt, θΨlm .

• 1: while not converged do
• 2: Sample batch B ∈ G;
• 3: for each interaction (v, u, t) ∈ B do
• 4: hv , hˆ v = NRL(v); hu = NRL(u);
• 5: Compute p(v,u), p(ˆ v,u) via Eq. (1);
• 6: Laug ← Laug + LΨc + αLΨd;
• ▷ Eq. (9)
• 7: Sample negative q and compute hq;
• 8: Lcon ←Lcon +Eq. (10);

• 9: Update θΨlm by minLaug (fix θt);
• 10: Update θt by minLcon (fix θΨlm );

Algorithm 2 Node Representation Learning (NRL)

Require: Node v;

Ensure: hv, hˆ v.

• 1: ssv, stv ←Spatial/Temporal Sampling for v;
• 2: Ss v, Stv ← Transformer encoding of ssv, stv;
• 3: ˆ ssv, ˆ stv ← Ψlm(Ss v ), Ψlm(Stv );
• 4: hv = TF(ssv)∥TF(stv);
• 5: hˆ v = TF(ˆ ssv )∥TF(ˆ stv );

• 6: return hv, hˆ v.

The SharPen Function. Algorithm 3 implements the gradient-preserving Top-K selection mechanism.

5 Experiments

In this section, we conduct extensive experiments to evaluate the effectiveness of ContraTGTon five benchmark datasets.

5.1 Experimental Setup

Datasets

We evaluate our approach using five publicly available benchmark temporal networks: soc-wiki-elec 1 , ia-slashdot-reply-

1 https://networkrepository.com/soc-wiki-elec.php

LearnableDataAugmentation. We implement a learnable masking module Ψlm that adaptively selects nodes to mask within a sequence. Given a representation sequence S, the augmented features ˆ sx are:

where Atten(·) calculates attention weights via a parameter matrix W, and SharPen(·) yields a gradient-preserving binary mask based on the masking rate ρ.

Using a link prediction head p(v,u), we define the augmentation loss Laug as a weighted sum of consistency (LΨc) and diversity (LΨd):

Contrastive Objective. The Transformer is pre-trained by maximizing the agreement between a node v and its augmentation ˆ v, while minimizing it with a negative instance u:

Optimization alternates between Laug (updating θΨlm ) and Lcon (updating θt).

4.3 Objective

For downstream tasks, the representations from spatial and temporal views (Hs,Ht) are concatenated: hv = Hs[v] ∥Ht[v]. The final model is fine-tuned using supervised binary cross-entropy (Eq. 2).

4.4 Algorithms and Complexity Analysis

Pre-training Paradigm. Algorithm 1 details the iterative training process. The function NRL(·) (Algorithm 2) handles the dual-view encoding and learnable masking.

Complexity Analysis. Pre-training complexity is dominated by the Transformer encoding, which is O(B(l2 s d+l2 t d)) per batch. Since sequence lengths ls, lt are fixed, the model scales linearly with the number of nodes, ensuring efficiency on large-scale temporal graphs.

Algorithm 3 SharPen Function

Require: h,K, ϵ; Ensure: f . 1: m← 0, f ← 1; 2: for i = 1 to K do 3: w← 1-maskprev; y ← SoftMax(h+ log(w+ ϵ)); 4: m←m+y ∗w; f ← f · m; 5: return f ;

dir 2 , soc-sign-bitcoinotc 3 , Social Evolve 1 month [Madan et al., 2011] and Ubuntu 4 . Table 1 provides insights into the ratio of nodes that share multiple interactions (edges), where "-" denotes that the statistics is not provided by the authors.

For every dataset, interactions are randomly divided in a 1:1:8 ratio for training, validation, and testing.

Baselines

To comprehensively evaluate our proposed ContraTGT against state-of-the-art approaches, we consider several stateof-the-art baselines categorized into three main groups 5 : (1) Static GNN models: GraphSAGE [Hamilton et al., 2017]; (2) Temporal GNN models: JOIDE [Kumar et al., 2019], CAWN [Wang et al., 2021c], TIP-GNN [Zheng et al., 2022], TAPGNN [Zheng et al., 2023]; (3) Approaches with advanced techniques: DyRep [Trivedi et al., 2019], TGAT [da Xu et al., 2020], APAN [Wang et al., 2021b], TGN [Rossi et al., 2020], TIGER [Zhang et al., 2023d], and FTM [Cao et al., 2023]; and (4) Graph Transformer approaches: TCL [Wang et al., 2021a] and DyGFormer [Yu et al., 2023], and CoLA-Former [Zhao et al., 2025]; and (5) State Space Models: GraphMamba [Wang et al., 2024].

5.2 Performance Evaluation

In this part, we evaluate the performance of ContraTGT on the task of temporal link prediction on temporal graphs.

Evaluation with limited training data

In this experiment, we limit the training data to only 10% of the interactions to simulate scenarios with limited supervision. We conduct the experiments under both transductive and inductive settings. In the transductive setting, the model is evaluated on the entire test dataset, while the inductive setting focuses on predicting interactions involving at least one new node that did not appear in the training data.

2 https://networkrepository.com/ia-slashdot-reply-dir.php

3 https://networkrepository.com/soc-sign-bitcoinotc.php

4 https://snap.stanford.edu/data/sx-askubuntu.html

5 For the temporal baselines, we only consider continuous-time models.

The performance comparison in Table 2 and Table 3 presents results for the transductive and inductive settings, respectively. Our ContraTGToutperforms all competing approaches on Wiki, Slashdot, and Bitcoin-OTC datasets, while achieving slightly inferior performance to Graph-Mamba and CoLa-Former on Ubuntu dataset. Graph-Mamba benefits from effective long-sequence state modeling, while CoLaFormer leverages semantic consistency through contrastive learning.

Evaluation with more training data

To further assess the efficacy of our proposed model with an increased volume of training data, we executed additional experiments, varying the quantity of training data while keeping the validation and test sets constant at a ratio of n:1:5, where n = {1, 2, 3, 4, 5} represents the proportion of the training data. These experiments were carried out on two datasets (iaslashdot-reply-dir and soc-sign-bitcoinotc), and we selected the most competitive baselines for comparison. The results, with AUC as metric in a transductive setting, are depicted in Fig. 2.

We have the following observations. (1) In scenarios with limited supervision, such as when the proportion of training data is 1 or 2, our proposed model, ContraTGT, surpasses the baselines, underscoring the strength of our model enhanced by pre-training with learnable data augmentation. (2) As the amount of training data increases, the baselines begin to demonstrate comparable, or in some cases, superior performance. This indicates that with ample training data at their disposal, end-to-end models like TAP-GNN, Tiger, and FTM are capable of fully leveraging label information to optimize their models. Typically, these models can surpass those that rely on pre-training when sufficient data is provided, at which point our model still maintains superior performance.

5.3 Model Analysis

We analyze ContraTGTfrom four perspectives: ablation, scalability, augmentation comparison, and parameter sensitivity.

Ablation study

To evaluate the contribution of each component in ContraTGT, we conduct an ablation study by comparing with several degenerate variants: (1) without pretrain: we use the ContraTGT architecture without the pre-training step; (2) no spatial: we remove the spatial-view sequences, by only using the temporal-view sequences for node representation learning. (3)no temporal: we remove the temporal-view sequences, by only using the spatial-view sequences for node representation learning.

We show the results of ablation study (using AUC as the metric) in Fig. 3, and make the following observations. Firstly, the performance consistently drops when the model pretraining step is omitted. This finding underscores the efficacy of the proposed contrastive learning strategy, which is significantly enhanced by the learnable data augmentation. Secondly, when the spatial sequences are omitted, ContraTGT exhibits a notable performance decline. This underscores the indispensable nature of spatial information for node representation learning within transformers applied to temporal graphs. Thirdly, the absence of temporal sequences also leads to a general performance decrease. This demonstrates the value of incorporating temporal sampling into ContraTGT. Notably, both spatial and temporal sequences seem to contribute comparably to the overall performance.

Scalability

To assess scalability via experiments, we utilized the largest temporal graph in our study, namely Ubuntu. We sampled several temporal graphs with increasing sizes, specifically 100k, 300k, 500k, 700k, and 900k interactions, to assess their training and inference times. For each temporal graph, we maintained the same data split as in our main experiment, with a 1:1:8 ratio for the training, validation, and testing sets. The training (per epoch) and inference times are reported in Fig. 4. Our observations reveal that both training and inference times for our proposed ContraTGT, scale linearly with the increase in temporal graph size. This linear scalability underscores ContraTGT's capability to be efficiently applied to large-scale temporal graphs, matching the complexity analysis in Section 4.4.

Comparison of different augmentations

To assess the effectiveness of our proposed learnable data augmentation method, integrated with the graph transformer featuring dual-view sequence sampling, we introduce two simple yet efficient random masking methods: MaskNode and MaskHidden. For MaskNode, we randomly mask a few nodes in a sampled node sequence according to the predefined masking rate ρ and then compute its representation. Meanwhile, for MaskHidden, we first compute the representation of the entire sequence and then randomly mask a few dimensions with respect to the predefined masking rate ρ, akin to the dropout mechanism. We compare the performance of these three approaches on two representative datasets, presenting the results in Table 4. Note that, for each dataset, we tune the masking rate ρ in the range of [0.1, 0.7] for each method to achieve optimal performance. We observe that our proposed model generally outperforms the other two random masking methods.

We further compare different data augmentation methods using consistency and diversity as evaluation metrics, where higher values indicate better performance. Our learnable data augmentation is compared with MaskNode and MaskHidden on the Bitcoinotc dataset, and the results are shown in Fig. 5. We observe that, as training advances, it becomes ev- TGN 0.886±0.008 0.896±0.007 0.802±0.056 0.867±0.017 0.880±0.016 0.692±0.007 0.773±0.006 0.809±0.006 0.701±0.006 0.909±0.007 0.881±0.011 0.852±0.005 0.824±0.007 0.859±0.004 0.763±0.004

TIGER 0.832±0.010 0.845±0.012 0.696±0.008 0.939±0.050 0.947±0.003 0.804±0.007 0.763±0.014 0.806±0.008 0.694±0.016 0.952±0.002 0.938±0.004 0.918±0.007 0.826±0.004 0.878±0.009 0.609±0.014

FTM 0.878±0.017 0.888±0.015 0.760±0.015 0.919±0.006 0.934±0.004 0.807±0.002 0.799±0.011 0.826±0.012 0.728±0.004 0.928±0.003 0.908±0.001 0.849±0.003 0.846±0.004 0.862±0.019 0.779±0.001

CoLA-Former 0.992±0.006 0.992±0.005 0.972±0.009 0.930±0.036 0.950±0.026 0.902±0.042 0.863±0.043 0.886±0.037 0.803±0.030 0.928±0.065 0.908±0.087 0.867±0.094 0.931±0.028 0.948±0.018 0.867±0.017

ContraTGT 0.997±0.001 0.998±0.001 0.981±0.001 0.955±0.003 0.960±0.003 0.895±0.004 0.899±0.008 0.901±0.012 0.824±0.040 0.943±0.007 0.935±0.010 0.877±0.008 0.857±0.007 0.892±0.010 0.820±0.006

Table 3: Inductive learning task results for link prediction.

ident that our model persistently surpasses the randommasking techniques, namely MaskNode and MaskHidden, securing markedly enhanced consistency and diversity. This trend highlights the efficacy of our learnable data augmentation strategy, which evidently produces more advantageous data augmentations when contrasted with random masking methods. The superiority of these data augmentations plays a pivotal role in amplifying the success of our contrastive learning approach. Moreover, the progressive improvement of our proposed model with each epoch is noteworthy. In contrast, the performance trend of the random masking approaches appears more variable, particularly in terms of consistency.

Parameters sensitivity

We evaluate the sensitivity of several important hyperparameters in ContraTGT, and show their impact in Fig. 6. For the length of spatial and temporal sequences, i.e., ls and lt, the performance seems to be stable w.r.t.the length, while longer sequences might be slightly better. For the weight of α in Eq. (9), it seems that a smaller α can benefit the performance, showing that consistency is more important in determining the performance. For the masking ratio ρ, the performance seems to be stable across different ρ's. In particular, ρ ∈ [0.25, 0.50] might be a good choice.

The smooth performance across hyperparameters can be attributed to three factors. (1) The effectiveness of our learnable data augmentation, driven by these metrics, ensures that the model is pre-trained with the most informative augmentations even with different ls and lt, contributing to the smooth performance across varying sequence lengths. (2) The smoothness in the masking ratio may be attributed to the effectiveness of the learnable data augmentation mechanism. This mechanism is designed to selectively choose the most informative elements for augmentation, even under extreme masking ratios, leading to promising augmentation generation. (3) For the wiki dataset, the graph Transformer backbone armed with dual-view sequences achieves nearly 100% performance, leaving limited room for significant improvement. As for the Ubuntu dataset, its relatively large size and the utilization of 10% interactions for training may already provide sufficient data for training the graph Transformer.

6 Conclusions

In this work, we introduced a dual-view graph transformer called ContraTGTto extract sequences tailored for specific target nodes, encapsulating both spatial and temporal perspectives. To optimize the contrastive learning of this dualview transformer, we put forth an innovative, learnable data augmentation method, which involves selective masking of elements within dual-view sequences, generates superior augmentations, thereby amplifying the potency of the contrastive learning approach. Extensive experiments on five public temporal network datasets demonstrate that our model can consistently outperform all baselines, especially in small training set conditions.

Acknowledgements

This research was funded by the Natural Science Foundation of Zhejiang Province of China under Grant (No. LMS26F020043,LHZSD24F020001,LZ25F020012).