Dynamicity-aware Social Bot Detection with Dynamic Graph Transformers

1 Introduction

As social networks become integrated into people's daily routines, there is a prevalent occurrence of program-controlled bots masquerading as legitimate users for malicious purposes [Subrahmanian et al., 2016]. Social bots engage in detrimental activities such as propagating misinformation [Varol et al., 2017; Gao et al., 2023], manipulating public opinion [Cui et al., 2020], interfering in elections [Rossi et al., 2020] and promoting extremist ideologies [Ferrara et al.,

† Corresponding authors.

2016]. It is therefore imperative to effectively detect social bots to mitigate the detrimental societal and economic impact and to preserve the integrity of social network information.

Traditional techniques for bot detection are largely based on features, requiring extraction of either numerical feature from user information [Yang et al., 2013] or semantic features from textual information [Wei and Nguyen, 2019; Duki´c et al., 2020]. However, bot operators can often bypass bot detection through advanced countermeasures, which is commonly referred to as bot evolution [Cresci, 2020]. In fact, the detectability of the feature-based methods is vulnerable to imitation and evasion, as bot operators can effortlessly steal user information from legitimate users or intersperse a few malicious messages with many neutral ones [Feng et al., 2022b]. As a result, such methods are inaccurate in spotting disguised social bots. With the advancements in graph neural networks, some researchers employed graph-based methods [Wu et al., 2023; Feng et al., 2022a; Yang et al., 2023a] to identify the disguised social bots. They typically assume that the network structure of social bots generally differs from that of legitimate users. For instance, social bots tend to have sparser connections and randomly select users to interact with, whereas human beings prefer to connect with others who share similar characteristics [Yang et al.,

2013]. These graph-based methods are among top performers by leveraging the topological structure of social networks for bot detection. However, most of the existing graph-based detection methods interpret the social network as a static graph and fail to acquire the dynamic nature of social networks. As shown in Figure 1, there still remain two intractable issues:

Deficiency in utilizing historical interaction graph context. Similar to the case of evading detection from featurebased methods by forging numerical or semantic features, the ever-evolving social bots are meticulously engineered to interact with legitimate users and mimic their network structures [Cresci, 2020] to escape graph-based detection. However, despite the structure of social network has changed, the discrepancies in the previous interaction graph between social bots and benign users could reveal the deception of social bots and uncover their true identity. Unfortunately, conventional approaches upon static graphs solely rely on the last state of the social network and overlook the valuable historical interaction graph context. Consequently, if the social bots have already completed their disguise, it is challenging for static graph based methods to distinguish benign users from the evolved social bots.

Limitation of modeling evolving behavior patterns. Social bots evolve over time, evading detection by dynamically adapting their actions, strategies, or interaction patterns to mimic legitimate users. In contrast, genuine users do not require such adaptations and exhibit different evolution of behavior patterns compared to social bots. Discovering the evolving behavior patterns may enhance the effectiveness of social network modeling [Liu et al., 2020]. Nevertheless, static graph based methods fall short of modeling the distinct evolving behavior patterns of social bots and legitimate users, leading to erroneous results when conducting bot detection tasks.

To overcome the limitations above, we propose a new framework called BotDGT (Bot detection with Dynamic Graph Transformers). The key insight is to introduce the dynamicity modeling of social network for bot detection. To this end, BotDGT depicts a social network as a dynamic graph for modeling historical interaction graph contexts and discerning the evolving behavior patterns. Specifically, we interpret users and interactions as nodes and edges, respectively, to generate a batch of snapshots at a fixed time interval for a given social network. A structural module that employs message-passing mechanism is proposed to model the topological structure of each historical snapshot. Additionally, a temporal module based on self-attention mechanism is further employed to incorporate historical contexts and exploit the distinct behavior patterns evolution exhibited by social bots and legitimate users. Overall, our contributions are summarized as follows:

• To the best of our knowledge, we are the first to characterize a social network as a dynamic graph and effectively identify the ever-evolving social bots that disguise themselves through adapting their behavior patterns.
• We introduce a novel bot detection framework to consider both topological structure and the dynamic nature of social networks to enhance the performance of bot detection.
• We conduct comprehensive experiments on two benchmarks for bot detection, which demonstrates the superior performance of BotDGT compared to the leading methods in terms of accuracy, recall and F1-score. Further experiments substantiate the effectiveness of incorporating the dynamic nature of social networks for bot detection.

2 Preliminaries

2.1 RelatedWork

Social Bot Detection

Early methods for social bot detection are predominantly feature-based. Researchers extracted numerical features from user information and fed them into machine learning models for classification [Lee et al., 2011; Mazza et al., 2019; Yang et al., 2020] or anomaly detection [Miller et al., 2014]. Some studies employed natural language processing techniques to encode textual information, capturing semantic features to enhance the feature-based methods [Hayawi et al., 2022; Duki´c et al., 2020; Yang et al., 2023b]. However, newer generations of social bots may forge numerical features or semantic features, either by stealing legitimate users' information or interspersing malicious messages among benign ones, to evade feature-based detection.

With the advancements in graph neural networks, some graph-based methods leveraged the topological structure of social networks for bot detection [Pham et al., 2022; Shi et al., 2023; Peng et al., 2024; Zeng et al., 2024]. The study [Ali Alhosseini et al., 2019] takes the first attempt to introduce graph convolutional networks to aggregate user information from neighboring nodes for bot detection. Subsequent investigations modeled the heterogeneity of social networks and yielded leading performance [Feng et al., 2021b; Feng et al., 2022a]. However, these methods interpreted the social networks as static graphs and neglected the intrinsic dynamicity of real-world social networks, thereby falling short of detecting evolving social bots that adapt strategies to mimic legitimate users' network structure [Cresci, 2020]. To this end, we build upon previous research and present a dynamicity-aware bot detection framework. It incorporates historical context and exploits the evolution of user behavioral patterns, aiming at enhancing the performance of bot detection.

Dynamic Graph Neural Network

Dynamic graphs capture temporal information through timebased dimensions [Skarding et al., 2021; Peng et al., 2021; Sun et al., 2021]. Previous research in graph representation learning has predominantly concentrated on static scenarios, presuming fixed topological structures. However, real-world graphs, including social networks [Alvarez-Rodriguez et al., 2021; Wang et al., 2021], exhibit continual evolution and dynamic characteristics over time. Dynamic graph neural networks are designed to capture this dynamic nature and are widely adopted in various tasks, including link prediction [Xie et al., 2021; Chen et al., 2022; Sankar et al., 2020; Zhang et al., 2023], anomaly detection [Cai et al., 2021; Guo et al., 2022], and node classification [Kim et al., 2022; Pareja et al., 2020; Xu et al., 2020]. Drawing upon the previous works that employed recurrent neural networks [Chen et al., 2022] and attention mechanisms [Sankar et al., 2020; Xu et al., 2020] to model dynamic graphs, we propose a novel approach that utilizes a self-attention mechanism to leverage the dynamic nature of social network, thereby enhancing the effectiveness of bot detection.

2.2 ProblemDefinition

In this paper, we depict the social network as a dynamic graph that changes over time and capture the dynamic nature of the social network to improve the performance of the bot detection model. In this part, we first define the dynamic social network and then formulate the problem.

Definition (Dynamic Social Network). A dynamic social network is depicted as a graph G = {Gt1 , Gt2 , ..., GtN } with a series of network snapshots over time. Gtk = (V tk ,Etk ) represents the snapshot of a given social network graph at the timestamp tk, where V tk , Etk are users and interactions respectively observed at timestamp tk.

To align with the previous studies, we treat bot detection as a binary classification problem, i.e., users are classified into human (y = 0) or bot (y = 1). We formulate the problem of bot detection in dynamic social networks as follows:

Problem (Bot Detection in Dynamic Social Network). Given a dynamic social network G = {Gt1 ,Gt2 , ...,GtN }, the problem is to find an encoding function f : v → ˆ y for each node v ∈ V tk at the timestamp tk ∈ {t1, t2, ..., tN}, such that ˆ y approximates the ground truth y to maximize prediction accuracy.

3 Methodology

As shown in Figure 2, BotDGT comprises two modules to primarily capture the topological structure and dynamicity of social networks. Specifically, we produce snapshots of the social network at a certain time interval. For each snapshot, the structural module aggregates the features of neighboring nodes and generates temporary node representations containing the snapshot's topology information. Additionally, the temporal module integrates historical context and exploits changes in behavior patterns over time. The resultant node representations are subsequently utilized to differentiate social bots from genuine users.

3.1 Constructing Dynamic Graph

We first construct a dynamic social network G = {Gt1 ,Gt2 , ...,GtN } at a specific time interval ∆t. We then acquire the node representation using the information encoding procedure established in the state-of-the-art techniques from recent study [Feng et al., 2022a]. A fully-connected layer is used to transform xtk i , the feature of user i at timestamp tk, into the initial user vector htk,{0} i :

where WI , bI are trainable parameters and σ(·) is a nonlinear activation function.

3.2 Modeling Topological Structure

Upon generating a dynamic social network and initial user vectors, we propose a structural module that leverages a message-passing mechanism [Gilmer et al., 2017] to effectively model the topological structure of each snapshot. This structural module takes a snapshot Gtk = (V tk ,Etk ) and a set of initial user vector {htk i , ∀vi ∈ V tk } at timestamp tk as input. The output includes a new set of temporary node representations {stk i , ∀vi ∈ V tk }, which captures the structural information of the snapshot at tk.

Graph attention networks [Veliˇckovi´c et al., 2017] have shown superior performance when tackling graph data, by specifying different weights to different nodes within a neighborhood. Inspired by the Transformer architecture [Vaswani et al., 2017], we adopt a scaled dot-product attention mechanism for provisioning each node with the ability to learn the importance of its neighbors within a particular snapshot. Specifically, for a given a node pair (vi, vj) in snapshot Gtk , the attention weight can be calculated as such:

where {l} denotes the l-th layer of structural module and N tk (i) denotes the neighborhood of node vi at the timestamp tk. The initial user vector htk,{l} i and htk,{l} j are transformed into a query vector qtk,{l} i and a key vector ktk,{l} j . The attention weight αtk,{l} ij , which indicates the contribution of node vj to node vi at the snapshot Gtk , is calculated by the exponential scale dot product function ⟨q, k⟩ = exp( qkT √d ), where d is the input embedding dimension.

After obtaining the attention weight, we transform the initial user vector into a value vector and aggregate information from the neighboring nodes of vi. A multi-head attention mechanism is employed to capture diverse patterns and dependencies in the topological structure of the social network:

where ∥ represents the concatenation operation, αtk,{l} c,ij denotes the attention weight computed by the c-th attention head, and vtk,{l} c,j denotes the corresponding value vector.

It is worth noting that we stack L layers to allow nodes to capture more distant and global dependencies in the topological structure. The output of the last layer in the structural module is denoted as s.

3.3 Acquiring Temporal Dynamicity

While BotDGT's structural module can capture topological information from static snapshots, it insufficiently leverages historical context and fails to discover evolving behavior patterns of social bots. Inspired by [Sankar et al., 2020; Ying et al., 2021], we devise a self-attention based temporal module to further make use of the temporal characteristics of social networks for bot detection. The temporal module takes as inputs a sequence of temporary representations of node vi at each timestamp, denoted as {st1 i , st2 i , ..., stN i }. The module outputs a new sequence of user representations { ˆ zt1 i , ˆ zt2 i , ..., ˆ ztN i }, where ˆ ztk i denotes the final representation that contains both topological and temporal feature of node vi at tk.

Position Embedding Layer

Since the self-attention mechanism is unaware of the nodes' ordering information, we introduce a position embedding layer to accommodate temporal information in the sequence that can effectively reflect the dynamic nature of social networks. We consider two categories of position embedding absolute temporal position embedding and evolving temporal position embedding.

First, we embed the absolute temporal position [Gehring et al., 2017] of each snapshot as a basis to capture ordering information as follows:

where ptk,AT denotes the Absolute Temporal position embedding for the timestamp tk and EAT denotes the trainable absolute temporal position embedding parameter. Note that the absolute temporal position embedding only relies on the order of the snapshot, indicating that the nodes in the same snapshot have the same absolute temporal position embedding, i.e., the absolute temporal position embedding is independent of the nodes' features.

Second, we embed two crucial temporal signals: the local clustering coefficient and bidirectional links ratio. These signals have demonstrated their utility in countering the disguised social bots [Yang et al., 2013] and could reveal the evolving behavior patterns over time.

· Local Clustering Coefficient (LCC): it measures the degree to which a node's neighbors are interconnected. Genuine users typically engage with acquaintances (e.g., friends, family members, and colleagues) who have similar connections and thus form closely-knit communities. By contrast, social bots are usually associated with randomly selected neighbors who lack close connectivity, which results in reduced clustering coefficients when compared with legitimate users. The position embedding of the local clustering coefficient is calculated as follows:

where |etk vi | is the number of edges between neighbors of node vi at the timestamp tk, ktk vi is the sum of the indegree and outdegree of node vi at tk.

• Bidirectional Links Ratio (BLR): it is a metric in social network analysis to assess the reciprocity between an account and its followings [Yang et al., 2013]. A bidirectional link appears when two accounts mutually follow each other. This metric proves particularly useful in distinguishing between genuine users, who often own higher bidirectional link counts due to reciprocal following acquaintances with mutual follow-backs, and social bots, who exhibit lower bidirectional link counts due to their indiscriminate following behavior and lack of reciprocal connections. The position embedding of the bidirectional links ratio is calculated as follows:

where Nblinks(vtk i ) and Nfing(vtk i ) denote the numbers of bidirectional links and following interactions.

In summary, the integration of these two categories of position embeddings enables the temporal module to capture essential temporal insights from ordering information and the evolving behavior patterns of social bots.

Temporal Attention Layer

The temporal attention layer starts from gathering the outputs of the structural module and the position embedding layer:

Then we pack the representations of node vi together across the timestamps, which is denoted as ˆ Si ∈ RT×F . Finally we performmulti-head temporal attention as follows:

where ∥ represents the concatenation operation, Q,K, V are the queries, keys, and values transformed by trainable parameters W∗ ∈ RF×F respectively. Mask ∈ RT×T is a sequence mask matrix that makes sure the node at timestamp tk only attends over its historical node representation. The Mask is defined as follows:

3.4 Learning and Optimization

The goal of BotDGT is to capture both the topological structure and the dynamic nature of social networks to classify the accounts into legitimate user and social bots. We pass the output of the temporal module into a linear layer and softmax layer for bot detection:

where ˆ yi is the predicted output of node vi and ˆ zi is the representation of node vi obtained by temporal module. Finally, we define the objective function that utilizes a binary crossentropy function to classify node v into legitimate users and social bots at each snapshot:

where N is the number of the snapshots, yi is the ground truth label of node vi.

4 Experiments

In this section, we conduct extensive experiments on two benchmark datasets to answer the following questions:

• RQ1: How does our framework perform in bot detection compared to baseline methods?
• RQ2: What is the impact of removing individual architectural components on the framework's performance?
• RQ3: What is the significance of capturing the dynamic nature of social networks for social bot detection?

4.1 Experimental Setup

Dataset

We conduct experiments on two comprehensive social bot detection benchmarks: TwiBot-20 [Feng et al., 2021a] and TwiBot-22 [Feng et al., 2022b]. The datasets provide a wide range of entities and relationships, spanning the period from the inception of Twitter to the time of dataset creation, which supports our bot detection framework to model the topological structure and dynamic nature of social networks.

Baselines

We compare BotDGT with comprehensive social bot detection methods categorized into two groups: feature-based methods and graph-based methods. Our code is publicly available on GitHub 1 .

Feature-based methods generally extract the numerical features from user metadata or semantic features from textual information to identify social bots, including:

• EvolveBot [Yang et al., 2013] designs robust features that are expensive for bots to evade and utilizes machine learning classifiers to combat the evasion tactics of spammers.
• Varol et al. [Varol et al., 2017] extracts groups of features from Twitter users and leverages random forest classifier to identify Twitter bot.
• BotBuster [Ng and Carley, 2023] employs a mixture-ofexperts approach to process user metadata and textual information, thereby improving cross-platform bot detection.
• DeeProBot [Hayawi et al., 2022] extracts features from the user account and leverages natural language processing techniques to learn user representations for bot detection.
• SGBot [Yang et al., 2020] is proposed to tackle the scalability and generalization issues in social bot detection by strategically selecting a subset of training data.

Graph-based methods generally interpret social networks as graphs and leverage geometric deep learning for social bot detection, including:

• GCN [Kipf and Welling, 2016] equally aggregates features from neighbors and learns user representations, which are then passed to a linear layer for classification.
• GAT [Veliˇckovi´c et al., 2017] leverages an attention mechanism to assign diverse weights to different neighboring nodes, improving the learning of node representations.
• BotRGCN [Feng et al., 2021b] is designed to construct a heterogeneous social network graph and employ relational graph convolutional networks for bot detection.
• RGT [Feng et al., 2022a] utilizes graph transformers and semantic attention to effectively model the heterogeneity of social networks for bot detection.

4.2 Framework Performance (RQ1)

We evaluate our proposed social bot detection framework along with several representative baselines on the two benchmarks and present the results in Table 1. It is demonstrated that graph-based methods, which treat the social network as graphs, generally outperform feature-based methods. This could be attributed to the feature-based methods are easily circumvented by forging numerical or semantic features. The results underscore the importance of capturing topological structure of social networks for effective bot detection.

1 https://github.com/Peien429/BotDGT

Our proposed BotDGT outperforms other static graphbased baseline models, including the state-of-the-art static graph model, in terms of accuracy, recall, and F1-score on both TwiBot-20 and Twibot-22 datasets. In comparison with the architecture of previous static graph-based methods, BotDGT not only leverages the topological structure of the social network but also incorporates a temporal module that captures the dynamic nature of the social network. The superior performance of BotDGT could be attributed to its ability to capture the historical context of social networks and model the behavior patterns of automated bots that may evolve over time to evade detection, enabling better discrimination of social bots disguised as legitimate users by interacting with other users. Further detailed analysis of the dynamicity modeling is provided in Section 4.4.

4.3 Ablation Study (RQ2)

BotDGT comprises a structural module and a temporal module, to generate node representations for social bot detection. In this section, we conduct an ablation study on BotDGT by removing or replacing one specific component at a time to assess its significance. The components validated in this section are the message-passing mechanism in the structural module, the temporal module and the temporal position embeddings within it. Experimental results of these ablation models on TwiBot-20 and TwiBot-22 are shown in Table 3.

Effect of Structural Attention. In the structural module, we propose a scaled dot-product attention mechanism that assigns diverse weights to different neighboring nodes for propagating messages and capturing topological structure in each static snapshot. We replace the structural attention with other static graph-based methods proposed to detect social bots, including GCN, GAT, BotRGCN, and RGT. The observed performance degradation indicates that the scaled dot-product structural attention better captures the underlying topological structure for dynamicity modeling of social network.

Effect of Temporal Module. The temporal module assumes a pivotal role in modeling the dynamicity of social networks and the evolving behavior patterns of social bots. We first assess the overall impact of the temporal module and then evaluate the effect of position embeddings within the temporal module. The variant w/o TM removes the temporal module of BotDGT, characterizing the social network as a static graph. As shown in Table 3, the variant's performance experiences significant degradation when compared to the original BotDGT architecture, which indicates the importance of incorporating temporal information for effective bot detection. The temporal position embeddings are proposed to capture the temporal information in the sequence. The variant w/o pAT in TM confirms the effectiveness of capturing the ordering information of the time sequence. The variants w/o pAT in TM and w/o pLCC in TM demonstrate the importance of considering evolving behavior patterns. Overall, these results confirm the crucial role of positional embeddings in effectively modeling the dynamicity of social networks.

4.4 Significance of DynamicityModeling (RQ3)

To investigate the impact of exploiting the inherent dynamicity of social networks for bot detection, we enhance several static graph-based baselines with the proposed temporal module and compare their performance before and after this enhancement. The experimental results outlined in Table 3 reveal that the majority of the graph-based baselines with the temporal module integrated exhibit noticeable performance improvement compared with their non-enhanced counterparts.

Similar to BotDGT's result, all of the enhanced graphbased methods can achieve higher recall rates, indicating the significance of capturing the dynamic nature of social networks in detecting disguised social bots. Notably, the graphbased methods with the temporal module integrated experience a slight reduction in precision. We speculate that the increased sensitivity to social bots derived from the temporal module might lead to a slight increase in false positives. This observation also gives rise to a decrease in RGT's F1score on the Twibot-20 dataset, which initially had the highest precision and the lowest recall rate. Nevertheless, there is a notable improvement in F1-score for most graph-based methods, which can reaffirm the importance of incorporating the dynamic nature of social networks in bot detection.

5 Discussion

5.1 Granularity for DynamicityModeling

Different from the static graph-based approaches that rely on the most recent snapshot, BotDGT integrates historical context from multiple snapshots. During the process of graph construction in BotDGT, we construct a series of snapshots to depict the social network at a time interval ∆t, which determines the granularity of the dynamic social network. To explore how granularity affects dynamicity modeling, we evaluate BotDGT with various time intervals on TwiBot-20. The result, illustrated in Figure 3, shows that the model performance initially improves as the granularity becomes finer due to richer temporal information. However, a decline in the model performance is observed when the granularity becomes finer than 12 months. We speculate that while setting the granularity finer than 12 months provides more historical context and detailed evolving patterns, it also introduces more noise and short-term fluctuations of the social network, making it more challenging for the temporal module to learn consistent patterns. Furthermore, the study [Cresci, 2020] has found that the evolution of social bots is not very frequent, indicating that social bots mostly don't change their actions or strategies within a short period of time. Therefore, excessively fine granularity may not provide meaningful insights into the evolving behavior patterns exhibited by social bots.

5.2 Temporal Attention for DynamicityModeling

To further explore how temporal attention affects the performance of bot detection, we visualize the distribution of temporal attention weights averaged over test nodes on TwiBot20 at the time interval of 12 months. In Figure 4, each row represents the attention weight distribution of the snapshot at tk over its historical snapshots at t1, · · · , tk-1. As shown in Figure 4, BotDGT doesn't assign uniform weight to historical snapshots, indicating the different contributions of each snapshot for social bot detection. When predicting the most recent snapshot of the dynamic social network, BotDGT assigns more weight to the historical snapshots before 2015, rather than focusing on the more recent ones. We speculate that the reason is that advanced social bots evolved to change their behavior patterns and interact with legitimate users around 2015, which is consistent with the rise of a third wave of bots from 2016 onwards as described in the study [Cresci, 2020]. Thus, BotDGT is capable of adapting attention weight distributions to effectively incorporate historical context.

6 Conclusion

The proliferation of social network bots has led to negative consequences. While state-of-the-art bot detection methods generally represent the social network as a static graph, they tend to overlook the dynamic nature of the social network. In this paper, we propose a bot detection framework, BotDGT, to exploit the inherently dynamic nature of social networks, which incorporates the historical context and models the evolution of behavior patterns. Experimental results on real-world datasets demonstrate that BotDGT outperforms other static graph-based bot detection methods. Further studies indicate the significance of exploiting the social network's dynamic nature for effective bot detection.

Acknowledgments

This work was supported by the National Key Research and Development Program of China through grants 2022YFB3104700, 2022YFB3105405, 2021YFC3300502, NSFC through grants 62322202, U2003206, Beijing Natural Science Foundation through grant 4222030, Guangdong Basic and Applied Basic Research Foundation through grant 2023B1515120020, Shijiazhuang Science and Technology Plan Project through grant 231130459A.

Contribution Statement

The contributions of Buyun He and Yingguang Yang to this paper were equal.