CausalNET: Unveiling Causal Structures on Event Sequences by Topology-Informed Causal Attention

1 Introduction

Causal discovery within event sequence data stands as a critical pursuit with far-reaching implications for a multitude of real-world applications. Its fundamental objective revolves around the discernment of causal relationships between events within sequences. Through this discernment, valuable insights into the underlying generative mechanisms of the data emerge, empowering us to make well-informed decisions. One illustrative context lies in Artificial Intelligence for IT Operations (AIOps), where event sequences encapsulate a diverse array of system events, ranging from historical alarm occurrences to user interactions. Grasping the intricate causal connections within these sequences serves as a linchpin for a host of advantages. For instance, by pinpointing the root cause of alarms, optimizing the allocation of resources gains precision, and the ability to anticipate potential system breakdowns becomes feasible [Gong et al., 2023].

∗Corresponding author

Research on causal discovery within event sequences falls into three main categories. Constraint-based methods identify causal graphs via conditional independence tests, e.g., [Spirtes and Glymour, 1991; Runge et al., 2019; Bhattacharjya et al., 2022]. Score-based methods search for an optimal graph using a score tester, as demonstrated in [Bhattacharjya et al., 2018; Zhu et al., 2019]. Granger-based methods [Xu et al., 2016a; Zhang et al., 2020b; Id´e et al., 2021] identify causality by evaluating if one event type influences another's prediction. However, these methods usually operate under the assumption that event sequences across topological nodes are independent and identically distributed (i.i.d.), leading them to address events on distinct topological nodes in isolation. Contrarily, some real-world situations often present events influenced not only by events on the same node but also by those from neighboring nodes. Addressing this, Topological Hawkes Process (THP) [Cai et al., 2022] is proposed to learn causal relationships from non-i.i.d. event sequence data.

However, several challenges still exist for causal discovery within event sequences in real-world scenarios. Notably, state-of-the-art methods like THP require manual kernel specification for the Hawkes process [Hawkes, 1971], limiting its flexibility in modeling intricate real-world events. Moreover, many approaches aim to obtain an optimal causal graph by searching for a Directed Acyclic Graph (DAG) with the highest likelihood within the exponentially expanding DAG space as the number of event types increases. This expansion raises scalability concerns for search-based methods, often yielding suboptimal outcomes [Li et al., 2022]. In a word, prevailing methods are entangled in unrealizable assumptions, constrained model flexibility, and scalability issues stemming from inefficient causal graph optimization.

In our pursuit to refine causal discovery for real-world applications, we have developed the innovative CausalNET model. Positioned at the heart of CausalNET, we propose a causal-attention-based Transformer that targets event prediction. Instead of indiscriminately considering all historical events, it leverages a trainable causal graph among event types to determine events that have a direct causal relationship to the upcoming ones. Besides, we introduce a causal decay matrix to characterize the influence of the underlying topological network on causal dependencies among events, to tackle the problem of causal discovery on non-i.i.d. event sequences. Moving away from traditional search-based methods that grapple with scalability and efficiency concerns, our approach involves a gradient-based causal graph optimization via Gumbel Softmax. During training, we alternate between fine-tuning the Transformer module and the duo of the causal graph and decay matrix. Furthermore, considering causal discovery tasks usually require the final graph to be acyclic, we design a post-processing strategy to prune the causal graph learned from the above training process to be an exact DAG.

Our main contributions are listed as follows:

• We propose CausalNET, a novel model for causal discovery on event sequences with the topological network. It supports flexible event sequence modeling and addresses the constraint of the i.i.d. assumption.
• We devise a causal-based self-attention mechanism, which enables our model to gradually enhance its understanding of the causal relationships among different event types while learning to predict future events.
• We conduct experiments on both real-world datasets and synthetic datasets and demonstrate the superiority of the proposed model. Notably, our model achieves remarkable F1-score enhancements exceeding 10% over the state-of-the-art methods across both real-world datasets.

2 RelatedWork

Causal Discovery. Causal discovery, i.e., causal structure learning, has emerged as a crucial area of research due to its applications in various domains. General approaches for causal discovery can be grouped into three classes: constraint-based, score-based, and functional causal model (FCM)-based. Most constraint-based methods lie in the principle that causal relationships between variables or events can manifest as specific patterns of conditional independence, including the widely used algorithms PC [Spirtes and Glymour, 1991] and PCMCI [Runge et al., 2019]. Score-based methods usually aim to search for a DAG with the best score as the causal graph. However, the search for DAGs is an NPhard problem [Chickering, 1996] as the number of candidate DAGs increases exponentially with the number of causal graph nodes. This motivates the following works such as NOTEARS [Zheng et al., 2018], DAG-GNN [Zheng et al., 2018] to cast the searching problem as a continuous optimization problem. More recent works also try to solve this problem from the perspective of reinforcement learning [Zhu et al., 2019] and generative flow network [Deleu et al., 2022; Li et al., 2022]. FCM-based methods model the data generation process using a functional causal model and identify causal relationships in data based on it. While traditional methods usually implement the FCMs as simple linear and non-linear models with specific noise such as ICALiNGAM [Shimizu et al., 2006] and DirectLiNGAM [Shimizu et al., 2011], recent studies have begun to use more flexible neural networks, e.g., [He et al., 2021; Cheng et al., 2022].

Causal Discovery on Event Sequence Data. While most of the aforementioned methods assume well-structured data such as variables or time series sampled with a regular time interval [Gong et al., 2023], real-world events usually emerge irregularly and asynchronously. Therefore, these methods usually demonstrate inferior results on event sequence data. To address these challenges, specific causal discovery methods have been developed for event sequences. Among them, Granger causality-based methods are well-developed and can be categorized into two main classes. On the one line, some methods utilize the Hawkes process [Hawkes, 1971] to model the event generation process and infer causal relationships between events. Typical examples include ADM4 [Zhou et al., 2013], MLE-SGL [Xu et al., 2016b], NPHC [Achab et al., 2017], L0Hawkes [Id´e et al., 2021], THP [Cai et al., 2022], and SHP [Qiao et al., 2023]. The Hawkes process is a stochastic process that aims to describe the mutual excitation effects between events through a conditional intensity function [Hawkes, 1971]. Benefiting from the Hawkes process's good interpretability and compatibility with Granger causality, these methods can infer causal relationships between events from the conditional intensity function. On the other line, with the advancement of deep learning, some studies have begun to employ neural point processes to model event sequences and learn causalities [Zhang et al., 2020b].

However, most of these methods still run on the independent and identically distributed assumption of event sequences. In particular, THP attempts to solve this problem by introducing a graph convolution to handle the topological information, but it suffers from inherent flexibility issues due to the manually specified kernel function and efficiency bottleneck due to the inefficient DAGs searching approach. In parallel work, [Liu et al., 2024] proposes the TNPAR model to solve the issues of THP. TNPAR transforms continuous-time event sequences into discrete-time sequences and leverages a multi-layer perceptron (MLP) to model the event generation process. However, in practice, this simple MLP-based architecture still exhibits limitations in flexibility and scalability.

3 ProblemDefinition

Consider a topological network denoted as GN = (N,EN), where N represents a collection of topological nodes, and EN signifies the interconnections among these nodes. Additionally, let D denote the discrete space of DAGs GV = (V, EV ), where V denotes a set of event types and EV denotes the direct edges between them. Specifically, vj → vi denotes a direct edge from vj to vi, signifying that a type-vj event can cause the occurrence of a type-vi event.

Given an event sequence with L events: S = {e1 = (t1, v1, n1), ..., ei = (ti, vi, ni), ...}Li=1 , where ti ∈ T indicates the timestamp of the i-th event, vi ∈ V stands for its event type, and ni ∈ N represents the topological node where it occurred. We assume that each event instance ei = (ti, vi, ni) is generated from a latent process of the form:

where parents(ei) denotes the historical events that are of the types vj ∈ V with a direct edge vj → vi ∈ EV . The term ϵ denotes jointly independent noise variables. fi is a function generating ei from parents(ei) and ϵ. The goal of this paper is to learn the causal structure GV among event types that drive this event generation process from the observable event sequence S and the underlying topological network GN .

4 CausalNET

In this paper, we propose CausalNET to uncover causal relationships among different event types from event sequences and the underlying topological network (Figure 1). At first, we present a causal-attention-based Transformer to model the latent event generation process. To characterize how causal dependencies among events of diverse types evolve under the impact of the topological network, we introduce a causal decay matrix. Then, we implement a continuous optimization on the distribution of possible causal graphs via Gumbel Softmax. During training, we optimize two main modules alternatively in an iterative framework: the Transformer module, and the causal graph together with the decay matrix.

4.1 Causal-attention-based Transformer

Event Embedding. Given an event sequence with L events: S = {ei}Li=1 , where ei = (ti, vi, ni). Let us denote ˆ V = [ ⃗ v1, ⃗ v2, ..., ⃗ vL] ∈ R|V |×L where ⃗ vi is a one-hot vector of the event type vi, and ˆN = [ ⃗ n1, ⃗ n2, ..., ⃗ nL] ∈ R|N|×L where ⃗ ni is a one-hot vector of the topological node ni. Embedding representations for the events can be formulated as:

where Xv = Wv ˆ V ∈ Rd×L, Xn = Wn ˆN ∈ Rd×L. Specifically, Wv ∈ Rd×|V | and Wn ∈ Rd×|N| are trainable embedding matrix for event types and topological nodes respectively. Xt = [⃗ t1, ⃗ t2, ..., ⃗ tL] ∈ Rd×L is the temporal encoding matrix which records the timestamp of each event. In practice, we adopt the temporal encoding procedure proposed by [Zuo et al., 2020], and there are alternatives such as [Zhang et al., 2020a]. Finally, X = [ ⃗ x1, ⃗ x2, ..., ⃗ xL] ∈ RL×d , and ⃗ xi represents the embedding of the i-th event in the sequence.

Self-Attention with Causal Graph. Consider a sequence of historical events up to ei at timestamp ti, to infer the typevi+1 event's intensity at the next timestamp ti+1, we need to consider the cumulative influence of all types of events before ti+1. [Zuo et al., 2020; Zhang et al., 2020a] propose to compute the pairwise influence of each historical event on the next event via self-attention. However, none of them takes into account the latent causalities that really drive the generation process of the event sequence, and many unnecessary irrelevant events are introduced during prediction. Here, we present a causal-based self-attention mechanism with a causal graph (M = {mi,j}|V | i,j=1 ) sampled from the GumbelSoftmax distribution [Jang et al., 2016] of a causal probability graph (Figure 1). This generates a hidden vector that summarizes the influence of relevant historical events up to ti:

where ti is the timestamp of the i-th event, and ⃗ xi denotes the corresponding event embedding. We consider historical events starting from es instead of e1 because considering historical events that occurred too long ago will not only introduce unnecessary noise but also increase the length of the sequence to be processed. This, in turn, would hurt the efficiency of the model. In practice, we set a hyper-parameter ξ, which means that only the historical events that occurred during [ti+1 - ξ, ti+1) are considered when predicting the event at ti+1. Especially, the event es at ts and the event ei at ti are the first and last ones respectively, during the interval. g(·) = ⃗ xjWV ∈ Rdv is a linear transformation whose output is the value(v) in the attention terminology [Vaswani et al., 2017]. ˆ f(·, ·) is a similarity function between two events:

where ⃗ qi = ⃗ xiWQ ∈ Rdk , ⃗ kj = ⃗ xjWK ∈ Rdk , and · denotes dot product. f(·, ·) is the softmax function, whose output is a normalized attention score/weight between events. In particular, mvj,vi+1 is used to mask irrelevant historical events for predicting the event at ti+1. Specifically, only the subset of historical events that are of the types vj ∈ V with a direct edge vj → vi+1, i.e., mvj ,vi+1 = 1, are considered.

The softmax function defined in Equation (4) fully adheres to the philosophy of Transformer Decoder. Specifically, it allocates attention weights to historical events solely based on the pairwise similarity between the current event (ei) and each historical event (ej≤i). However, experiments show that such a softmax function may fail to help the model learn a good causal structure from the event sequence data of complicated real-world scenarios (Appendix B.3). Therefore, we further reformulate the softmax function as follows:

where ∆i = exp( ⃗ qi ·⃗ki+1 √dk ) + ˆn, and ˆn is the number of the padding events 1 . The first term of ∆i describes the similarity between the event ei and ei+1. It plays a similar role as the sampled causal graph, i.e., adjusting the attention weight to each historical event (ej≤i) based on the relationship between each historical event and the forthcoming event ei+1. The size of ˆn reflects the number of recent historical events for ei+1. When there are few historical events for it, we pad the sequence with meaningless events and n will be larger, thus reducing the attention weight for each historical event to prevent overestimating the influence of specific events.

1 The embedding representation of a padding event is set to be ⃗0. Since exp(⃗ qi ·⃗0/√dk) = 1, the sum for ˆn padding events will be ˆn.

Incorporating the Influence of Topological Network. So far, we have incorporated a trainable causal graph among event types into the attention mechanism via a causal mask after softmax. However, in many real-world scenarios, the structure of the underlying topological network will also influence the generation process of events. [Cai et al., 2022] found that ignoring the topological structure may lead to unobservable confounders and unstable results for causal discovery. Intuitively, events occurring at two topological nodes can only exert causal influence when there is at least one path between the nodes, and the influence should wane as the nodes' distance increases. Therefore, to consider the influence of the underlying topological network, we further extend such a causal attention mechanism to a topology-informed causal attention mechanism via the adjacency matrix among topological nodes and a trainable parameter matrix describing node properties. Specifically, we extend Equation (5) to:

and replace the similarity function ˆ f(·, ·) in Equation (3) with ˜ f(·, ·). ˆ Ak is a binary matrix construted from the adjacency matrix A ∈ R|N|×|N| , where ˆ Aknj,ni+1 = 1 if and only if there is at least one k-hop path between the topological node nj and ni+1. Besides, to emphasize the influence between events occurring on the same topological node, we set all the diagonal elements of ˆ Ak to be 1. ϕ ∈ RK×|N|×|N|×|V |×|V | is a trainable parameter matrix introduced to characterize the property of the underlying topological network. In particular, ϕk,nj ,ni+1,vj ,vi+1 describes the decay coefficient of the causal influence from a type-vj event (on node nj) to a type-vi+1 event (on node ni+1) after propagating along a k-hop path "nj → . . . → ni+1". Given its special positioning, we refer to this matrix as the causal decay matrix. sig(·) is a sigmoid function used to normalize the value of ϕ to (0,1). Intuitively,

ϕ plays a similar role to the graph attention score [Veliˇckovi´c et al., 2018] because both of them are used to aggregate the information from the topological neighbors of each node.

In addition to the self-attention module, we pass the hidden vector ⃗hti through a position-wise feed-forward neural network, which consists of two linear transformations with a ReLU activation function in between. This generates a new hidden vector as follows:

Auxiliary Tasks for Transformer Optimization. In order to optimize the causal-based Transformer, we employ two event prediction tasks. The first task is predicting the likelihood of the next event based on historical events. Since we have obtained the hidden representation of historical events, the remaining question is how to map it to the likelihood of the next event. To solve this, we adopt the continuous conditional intensity function proposed by [Zuo et al., 2020]:

for t ∈ (ti, t], where fv,n(x) = βv,nlog(1 + exp(x/βv,n)) is a softplus function with the softness parameter βv,n > 0 to eusure a positive intensity. αv,n is used to control the importance of the interpolation between the current timestamp ti and the future timestamp t, and bv,n represents the occurrence probability of a type-v event at topological node-n without the influence of history information. Given the intensity function as λv,n(t|Hti ), the probability density function of a type-v event at topological node-n can be expressed as:

and the log-likelihood for the whole event sequence S = {(t1, v1, n1), ..., (ti, vi, ni), ...}Li=1 can be formulated as:

The logarithm term should be increased to explain why a type-vi+1 event actually occurred on topological node-ni+1

at ti+1, and the integral term should be decreased to account for the absence of any event between (ti, ti+1). [Rasmussen, 2018; Id´e et al., 2021] provide detailed mathematical derivations of the probability density and log-likelihood functions.

The second task is to predict the attributes of the next event, including its timestamp, event type, and topological node. In practice, we pass the hidden vector of the i-th event through three independent linear layers to predict the (i+1)-th event. This involves regression (timestamp) with squared error loss and classification (type and node) with cross-entropy losses.

4.2 Continuous Optimization of Causal Graph

Due to the large search space of DAGs and complex acyclicity constraints, most search-based methods cannot handle largescale settings and are prone to getting stuck in local optima. Therefore, recent studies begin to recast the combinatoric search problem of DAGs into continuous optimization problems [Zheng et al., 2018; Ng et al., 2022]. Here, we implement a continuous optimization of the causal graph via Gumbel Softmax [Jang et al., 2016], which provides a differentiable way to draw samples from a discrete distribution. This procedure includes two steps in turn. At first, we leverage the sigmoid function to transform a trainable parameter matrix ϑ into the causal probability graph P = {pi,j}|V | i,j=1 , which characterizes a discrete categorical distribution. Specifically, pi,j represents the probability of the causal edge vi → vj to be true. Then, we draw a sample of the causal graph from the distribution using Gumbel Softmax as follows:

where τ is the temperature parameter that controls the smoothness of the Gumbel Softmax. Especially, as τ approaches 0, the Gumbel Softmax distribution will be equivalent to the original categorical distribution. In addition, {gi,j}|V | i,j=1 are i.i.d. samples from the Gumbel(0,1) distribution, where u ∼ Uniform(0,1) and g = -log(-log(u)).

4.3 Training Procedure

So far, we have introduced a causal-based Transformer and a Gumbel Softmax-based causal graph together with the causal decay matrix. Next, we will describe how these two modules interact with each other during the training process.

As shown in Figure 1, we implement an EM-style training framework like [Id´e et al., 2021] and [Cheng et al., 2022] via plugging these two modules into a two-stage iterative framework, and optimize them alternatively. In the first stage, given a causal graph sampled from the distribution of the causal probability graph (P = σ(ϑ)) together with the causal decay matrix between topological nodes (ϕ), we train the causalbased Transformer module (f (θ) ) to fit the event sequence (S) by minimizing the following loss function:

where L is the log-likelihood function defined in Equation (11), and λ1 a hyper-parameter. Lt, Lv, and Ln are loss functions for predicting the timestamp, event type, and topological node of the next event based on historical events.

In the second stage, given a causal-based Transformer model (f (θ) ), we optimize the causal graph (ϑ) together with the causal decay matrix (ϕ) by minimizing the loss function:

where || · ||1 is the L1-norm to enforce sparse connections for the learned causal graph. h(·) is a differentiable characterization of graph acyclicity called DAG-ness [Zheng et al., 2018], which is defined as follows:

where ◦ denotes Hadamard product, and h(A) ≥ 0. The matrix A ∈ R|V |×|V | denotes a DAG if and only if h(A) = 0.

The discovered causal graph ˆO is identified based on the causal probability graph P and the causal decay matrix ϕ:

Specifically, if max{pi,j × ϕk,m,n,i,j} is penalized to a value below the threshold ε, we deduce that there does not exist a causal relationship vi → vj , and set ˆ oi,j to be 0.

4.4 Post-processing with RollbackMechanism

While gradient-based causal graph optimization is more efficient than directly searching DAGs and evaluating them, the downside is that we cannot ensure the acyclicity of the causal graph. To solve this, GOLEM [Ng et al., 2020] proposes to gradually increase a threshold ω and remove all the edges with absolute weights smaller than ω until the pruned graph is acyclic. However, in real-world datasets, significant noise may lead to some negative edges having higher weights than true causal edges. Experimental results show that this pruning strategy might mistakenly remove many edges that should be predicted as positive (Appendix B.2).

Therefore, we design a more flexible pruning strategy. At first, we sort the edges in ˆO according to their weights (i.e., max{pi,j × ϕk,m,n,i,j}). Then, we remove an edge with the lowest weight and compare the DAG-ness of the pruned graph with the original graph. If removing this edge reduces the graph's DAG-ness, we will keep the deletion and update the graph. Otherwise, we will roll back the operation, i.e., recover the deleted edge. One by one, we will attempt to remove each edge of ˆO in ascending order of weight until the pruned graph's DAG-ness decreases to zero, which means that the pruned graph has become an exact DAG.

5 Experimental Setup

Real-world Datasets. We adopt two challenging realworld datasets from telecommunication networks 2 . The first is 24V 439N Microwave (Micro-24), which has 24 alarm types, 439 topological network elements, and 64,599 alarm events in total. The second is 25V 474N Microwave (Micro25), which contains 25 alarm types, 474 topological network elements, and 48,573 alarm events in total. Causal discovery on such datasets aims to uncover causal relationships among alarm types. To explore the applicability of our model in other domains (potentially those without topological network underlying event sequences), we further include the IPTV dataset [Luo et al., 2015], which records the history of TV watching behavior of each user and each TV program category denotes an event type. Since the ground-truth causal graph between TV program categories is not available, we conducted qualitative analysis on this dataset (Appendix B.1).

2 https://competition.huaweicloud.com/information/1000041487/ dataset

Synthetic Datasets. In addition, we also generate a range of synthetic datasets via gcatle's API [Zhang et al., 2021], which simulates event sequences based on the topological Hawkes process [Cai et al., 2022] engineered with a classical exponential decay kernel function. In the experiments, to generate datasets with different event interactions and temporal effects, we reformulate the original topological Hawkes process by replacing the default kernel function with some other kernel functions from the field of point process and event modeling. The distinctions among these synthetic datasets lie in the kernel function, the number of event types, the number of topological nodes, and the event sequence length (total number of events). Without further specification, each synthetic dataset utilizes an exponential decay function as the kernel, comprising 30 event types, 60 topological nodes, and an event sequence length of 30,000. Detailed statistics of the datasets are shown in Appendix B.4.

Baseline Models. We compare our model against the following 10 baselines: constraint-based methods: PC [Spirtes and Glymour, 1991] and PCMCI [Runge et al., 2019]; scorebased method: RL-BIC [Zhu et al., 2019]; FCM-based method: ICALiNGAM [Shimizu et al., 2006]; Granger-based methods: ADM4 [Zhou et al., 2013], NPHC [Achab et al., 2017], CAUSE [Zhang et al., 2020b], SHP [Qiao et al., 2023], THP [Cai et al., 2022], and TNPAR [Liu et al., 2024].

Evaluation Metrics. We adopt 4 widely used evaluation metrics of causal discovery as follows: F1 Score (F1), True Positive Rate (TPR), False Positive Rate (FPR), and Area Under the Receiver Operating Characteristic Curve (AUROC).

6 Experimental Results

Results on Telecommunication Network Alarm Datasets. Here we compare the performance of our proposed model CausalNET and its pruned version CausalNET* with all baseline models. Table 1 presents the experimental results for each model on the two real-world datasets. Notably, CausalNET exhibits significant advantages over all baseline algorithms across multiple metrics, with an improvement of over 10% in F1 score compared to the state-of-the-art methods. Moreover, CausalNET demonstrates a remarkable improvement in AUROC. This suggests that CausalNET excels in distinguishing between causally related and non-causally related event types. While some baselines excel in controlling FPR, they present low TPR. NPHC and CAUSE achieve high TPRs, but their FPRs are also extremely high, thus ineffective in discerning real causal relationships. Compared to other baselines, CausalNET, THP, and TNPAR outperform due to their inclusion of the underlying topological network structure, while other baselines run on the i.i.d. assumption of event sequences. This demonstrates the importance of considering topology in causal discovery tasks. Both neural network-based methods CausalNET and TNPAR achieve much higher TPR than the parametric method THP, indicating that neural networks are more effective in capturing intricate event dependencies in real-world scenarios. In addition, while TNPAR uses a simple MLP to model event dependencies, CausalNET employs a causal-attention-based Transformer architecture. The significantly superior performance of CausalNET on both datasets demonstrates the advantages of its causal-attention-based Transformer architecture.

Results on Synthetic Datasets. Then, based on the synthetic datasets, we further evaluate the performance of CausalNET and six baseline models across three dimensions (Figure 2). (I) Flexibility: Figure 2(a) illustrates the performance of each model as the kernel function varies. In this experiment, we introduce Weibull distribution [Rinne, 2008] and modify its shape parameter to alter the kernel function (Appendix B.4). CausalNET and THP, while achieving comparable performance, significantly outperform other baselines. (II) Scalability 3 : As shown in Figure 2(b) 4 and Figure 2(c), across all event type number and topological node number settings, CausalNET exhibits excellent performance and outperforms all baselines except THP. Notably, THP performs well when there are fewer event types, but experiences a notable decline as the number of event types increases to 100. This is because THP needs to search for a DAG with the highest likelihood from the discrete DAGs space, whose size increases exponentially with the growth of event types. In contrast, CausalNET demonstrates outstanding performance on all datasets, as it has converted the NP-hard DAGs searching problem into a much more efficient gradient-based continuous optimization problem in the realvalued space. (III) Data Efficiency: Figure 2(d) indicates that CausalNET and THP demonstrate outstanding performance right from the start. Meanwhile, some models (e.g., SHP) are sensitive to the number of events, with their performance improvement relying on an increase in the event number.

3 When the number of events reaches 100, some algorithms fail to complete training within a reasonable timeframe. Therefore, we impose a maximum runtime of 72 hours for each algorithm.

4 TNPAR encounters out-of-memory (OOM) issues on datasets with 100 and 150 event types on Tesla V100 GPUs (32 GB).

Influence of Historical Events. In this section, we study the influence of different receptive fields on historical events. Specifically, we set ξ (max time lag) to {30s, 60s, 120s, 180s}, and only allow CausalNET to use the most recent historical events within ξ to predict the next future event. As shown in Figure 3(a), on both datasets, CausalNET excels with a 120s time lag, followed by 60s and 180s, while 30s leads to the worst performance. These results emphasize the significance of dataset-specific time lag setting for optimal model performance. On the one hand, a too-short time lag limits the receptive field of self-attention, making it unable to capture long-term dependencies or causal relationships between events. On the other hand, a too-long time lag could introduce additional noise, since usually, only events occurring within a certain time interval will affect each other.

Influence of Topological Neighbors Similarly, we investigate the influence of considering different-order topological neighbors. We set the hyper-parameter k (max hop) to {0, 1, 2, 3}, where k = 0 corresponds to not considering any topological information. As shown in Figure 3(b), on the first real-world dataset, CausalNET performs best when k = 1, and closely followed by k = 2 and k = 3. In particular, k = 0 leads to an obvious decline in model performance. On the second dataset, CausalNET gets the best performance when k = 2. k = 1 and k = 3 lead to comparable worse performance. And again, k = 0 results in the worst results. This experiment confirms the significance of considering the appropriate topological information in causal discovery tasks.

Influence of Causal Graph Initialization. In this section, we evaluate the effects of different causal graph initialization strategies on the performance of CausalNET. Specifically, for the parameter matrix ϑ, we test four common initialization strategies. Random Initialization (RI) sets all parameters with random values within [-1, +1]. Zero Initialization (ZI) sets all parameters to be 0. This is a setting without prior assumption, where all edges are initialized to have an equal probability of being true or false. Negative Initialization (NI) sets all parameters to -1. This is a setting with explicit assumption, where all edges are initially predicted to have a very small probability of being true. Positive Initialization (PI) is opposite to NI. We repeat the experiment of each initialization strategy three times using different random seeds. As shown in Figure 3(c), ZI and NI achieve the best results across both datasets. RI exhibits slightly worse results. Besides, PI demonstrates notably inferior performance, a shortcoming ascribed to its imposition of an excessively strong prior. Specifically, an excessive number of edges in the causal graph are initialized as true, whereas real-world causal graphs are usually sparse.

Distribution of Causal Probability Values. We display the distribution of the learned probability value of each edge being true (i.e., max{pi,j × ϕk,m,n,i,j}) for two real-world datasets in Figure 3(d). The results from both datasets indicate that the vast majority of values are concentrated around either 0 or 1. Therefore, by simply setting an appropriate threshold as indicated in Equation (16), we can exclude a majority of edges and thus obtain a sparse causal structure.

7 Conclusion

Uncovering causal relationships within event sequences in the real world plays a ubiquitous role. In this paper, we propose CausalNET, a novel approach for learning causal structure from event sequences. The core of CausalNET is a causalattention-based Transformer that predicts future events via attention to historical events under the guidance of a causal graph. The causal graph describes causal relationships among different event types and is trained alternately with the Transformer module in an iterative framework. Extensive experiments on both real-world and synthetic datasets demonstrate that CausalNET achieves superior performance and scalability over a range of existing methods. In the future, we plan to expand this work to handle massive-scale event sequences.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 62127808).