EFEVD: Enhanced Feature Extraction for Smart Contract Vulnerability Detection

1 Introduction

As a core component of blockchain technology, smart contracts significantly enhance the blockchain application potential through their ability to automatically execute transactions. However, this characteristic also creates substantial security risks. Once a smart contract is deployed, its code becomes immutable, meaning that any existing vulnerabilities can lead to irreversible losses. A prime example of this is the reentrancy vulnerability found in the DAO smart contract. As shown in Fig.1, in this vulnerability, the contract executed an external call (msg.sender.call) before updating its internal state (account balance), which allowed attackers to repeatedly trigger the withdrawal function and extract funds before the balance was updated. This demonstrates that contract vulnerabilities are intimately connected with code variables (semantics) and code structures (syntax). Building on this, Liu et al. [Liu et al., 2021a; Liu et al., 2021b] summarized expert knowledge patterns for reentrancy, timestamp, and infinite loop vulnerabilities, integrating expert pattern features with semantic and synthetic code analysis to enhance vulnerability detection performance.

∗Corresponding Author

Subsequent works [Sendner et al., 2023; Li et al., 2023; Xie et al., 2023] continued to explore expert patterns for additional vulnerability types, enhancing feature extractions. However, this expert pattern-based vulnerability detection approach, which relies on domain expertise, lacks generalizability. Another category of deep learning-based vulnerability detection methods focuses on the code's modality features[Qian et al., 2023; Jie et al., 2023]. These studies comprehensively consider the two primary modalities of smart contracts-the source code and EVM (ethereum virtual machine) bytecode-along with their respective features. However, methods based on code modality feature enhancement pose high demands on datasets, and are prone to introducing overlapping redundant features between different modalities, which can limit model performance.

Therefore, this paper considers two novel dimensions of feature enhancement for smart contract vulnerability detection. First, we introduce community features for smart contracts. In domains such as social networks, financial transactions, and academic citations, the related community specifies the common interests of the individuals, which are characterized by similar patterns and preferences[Hu et al., 2023]. Given the close relationship between smart contract vulnerabilities and their semantic and syntactic properties, we define a community as a group of smart contracts with similar semantics and syntax. By analyzing patterns and vulnerabilities in similar contracts, we can extract a broader context, effectively aiding in the identification of the vulnerabilities. Second, we focus on the shared features related to two prevalent tasks in vulnerability detection, vulnerability classification and vulnerability localization. Vulnerability classification refers to deciding whether a smart contract contains vulnerabilities, while vulnerability localization involves classifying specific functions or variables within a contract for fine-grained vulnerability pinpointing. These cross-task shared features encompass not only the details of specific code segments but also the overall structure and behavioral patterns of the contracts[Xie et al., 2020; Takiddin et al., 2023], enabling the model to identify and locate vulnerabilities more effectively.

Specifically, this paper introduces an enhanced featurebased vulnerability detection (EFEVD) method for smart contracts. Initially, we employ the TextCNN and Transformer approaches for feature embedding of the internal function codes of smart contracts, and for extracting semantic features and sequence dependencies. Subsequently, a smart contract graph is constructed with functions as nodes and call relationships as edges. Using graph neural network (GNN) methods, the features encompassing the semantic and syntactic information of the contract graph are obtained. Thereafter, smart contracts are categorized into different communities based on the semantic and syntactic similarity between the contract graphs, and the commodity features are extracted. Finally, two downstream tasks for vulnerability detection are designed, classification of the contract graph and classification of the nodes within the graph. The model not only integrates the original features of the contract graph with the corresponding community features but also learns the cross-task shared features through the multitask learning architecture. This approach enhances the accuracy of the graph classification task while simultaneously locating specific vulnerable code segments through node classification.

Our main contributions can be summarized as follows:

• We explore the community features of smart contracts by defining communities based on semantic and syntactic similarities. Community features provide wider contextual information, enhancing the detection of vulnerabilities.
• We leverage shared features across two related tasks in vulnerability detection. By combining both detailed code segments and overarching contract analysis, our approach enhances the model's performance in identifying vulnerabilities.
• We employ a multitask learning architecture, learning and fusing enhanced features. We conduct extensive experiments on a real-world smart contract dataset, and our method outperforms the state-of-the-art methods, recall increases by 10.12%, F1-score increases by 8.17%, precision increases by 4%, and accuracy increases by 3.1%.

The remainder of this paper is organized as follows. Section 2 provides a comprehensive review of related work in the field of smart contract vulnerability detection. Section 3 formulates the proposed problem. Section 4 describes the proposed EFEVD method in detail. Section 5 presents the experimental information, including the dataset, evaluation metrics, and experimental results. Finally, Section 6 concludes the paper.

2 RelatedWork

Early works on vulnerability detection were primarily divided into two types, static analysis and dynamic analysis. Static analysis involves analyzing code without running the program, while dynamic analysis monitors the behavior of the code during its execution to detect vulnerabilities and errors. Chen et al[Chen et al., 2021] developed a smart contract vulnerability detection tool called DefectChecker, which uses symbolic execution to detect eight types of vulnerabilities. Li et al[Li et al., 2022] proposed a new tool called ReDefender, which uses fuzzy testing to detect reentrancy vulnerabilities. Osiris[Torres et al., 2018], which used symbolic execution to verify potential integer overflow vulnerabilities in smart contract bytecode. Oyente[Luu et al., 2016], which generates symbolic paths through symbolic execution to detect timestamp dependencies and reentrancy vulnerabilities; Smartcheck[Tikhomirov et al., 2018], an Ethereum static analysis tool that uses an syntax parser and specifies Solidity syntax rules to detect various types of vulnerabilities. Contract Fuzzer[Jiang et al., 2018], is a fuzz testing-based smart contract vulnerability detection framework.

However, static analysis often results in high false-positive rates and fails to capture runtime dynamics, while dynamic analysis is limited to executed code paths and is resource intensive. These drawbacks highlight the need for more effective strategies, which has led to an increasing focus on deep learning-based detection methods as promising alternatives.

Currently, deep learning-based methods in the blockchain domain predominantly leverage three categories of features. First, there are contract code content features. Ren et al.[Ren et al., 2023] utilized complete semantic structure information (CPSs) to construct program slices, facilitating the learning of syntactic structures, keywords, and other pertinent information. Chen et al.[Chen et al., 2023] introduced a novel semantic graph incorporating syntax and semantic information to represent the semantic details of each function comprehensively, enabling a more holistic capture of contract vulnerabilities. Liu et al.[Liu et al., 2023] pioneered the construction of a Multi-Relationship Nested Contract Graph to better characterized the syntax and semantics of smart contract code. Second, expert knowledge pattern features are proposed. Liu et al.[Liu et al., 2021b] designed specific expert patterns and integrated them with contract graph features, to construct a final vulnerability detection system. Based on this, an increasing number of vulnerability expert patterns are being progressively analyzed and proposed[Sendner et al., 2023; Li et al., 2023; Xie et al., 2023]. Finally, the contract modality feature is considered. Qian et al.[Qian et al., 2023] distinguished between two modalities: control flow graphs generated from bytecode and code semantic graphs generated from source code. Wang et al.[Wang et al., 2023] proposed the DEEPVD model, which leverages three hierarchical modality features, including statement types, post-dominator trees, and exception flow graphs. These features are meticulously designed to enhance discriminative performance.

In summary, although existing works have considered some feature enhancements, there are still some limitations. Expert knowledge pattern features require predefined domain knowledge, while multi-modal features demand high-quality datasets and are prone to introducing redundant features. Therefore, we propose the extraction of two new types of readily obtainable smart contract features to enhance feature representation and improve detection performance.

3 ProblemFormulation

In the enhanced feature extraction for smart contract vulnerability detection (EFEVD), we formalize the problem as follows:

Consider a smart contract graph g derived from the smart contract space G. Each contract graph g is composed of nodes N, representing contract functions, and edges representing call relationships between functions. We define two tasks within a multitask learning framework: graph classification (GC) for overall contract vulnerabilities, and node classification (NC) for function-specific vulnerabilities.

where GC(g) and NC(N) yield a probabilistic estimate of the presence of vulnerabilities in contract g and its functions N. The objective of the EFEVD is to minimize the crossentropy between the predicted results and the true labels.

Let ˜ G represent community features. The shared features are refined through a multitask learning process (MTL):

This formulation reflects the community and shared feature enhancement of the EFEVDmethod, highlighting the integration of feature extraction techniques to enhance the vulnerability detection results of smart contracts.

4 ProposedMethod

An overview of the proposed system is illustrated in Fig.2, which consists of three components, a community feature extraction module, a graph construction module, and a multitask downstream output module.

4.1 Community Feature Extraction

As mentioned before, we define a community in smart contracts as a group of contracts with similar semantic and syntactic features. Therefore, our first step is to extract the semantic and syntactic features of smart contracts. For the source code of smart contracts, the semantic features are embedded into vector space based on the word2vec method. The syntactic features are obtained by parsing the abstract syntax tree (AST) of the code, as shown in Fig.4. We classify smart contracts into different communities based on the similarity of their semantic and syntactic features. The similarity Simi is defined as:

where SemanSimi represents the semantic similarity between contracts, and SyntSimi represents their syntactic similarity. Contracts with a similarity exceeding a certain threshold are classified into the same community. Additionally, to mine the feature embeddings corresponding to each community, we establish edge connections between smart contracts within the same community. If the similarity between two smart contracts exceeds 80%, an edge is created between their respective nodes. This process transforms each community into a graph form, and then, based on GNN methods, we extract the feature representations for each type of community, defined as ˜ G = { ˜ G1, ˜ G2, . . . ˜ Gn}

4.2 Contract Graph Construction

For each smart contract, a smart contract graph is constructed with its functions as nodes and the invocation relationships between the functions as edges. To extract the node feature, each function is embedded into the features as a sequence. First, word2vec is used to generate word embeddings for each word in the function. Then, the TextCNN and Transformer are used to generate feature embedding for the function, better capturing short and long term sequence features.

Formally, after word2vec embedding, each function is represented as x = [x1, x2, ..., xn] ∈ Rn×d , where n is the sequence length and d is the dimension of the word vector. The TextCNN feature extractor can slide different sizes of convolutional kernels over the input sequence to aggregate local features of different lengths. Assuming M filters with R convolution kernels of size h × d are used, the feature vector obtained by the m- th filter is represented as:

Here, W m ∈ Rhm×d represents the weight matrix of the m- th filter in the convolution operation, b is the bias, f(·) denotes the activation function, and max(·) represents max pooling. Finally, all feature vectors of the filters are concatenated into a complete sequence C = [c1, c2, ..., cm] as the input to the Transformer.

The Transformer encoding layer consists of a multi-head Attention mechanism and a feedforward neural network. The self-attention layer computes attention weights between input embedding vectors to represent the importance of each input vector in the sequence. The multi-head attention layers and feedforward networks further process these encoded vectors to obtain more semantically meaningful textual representations, as shown below:

Here,Q,K, V represent the query, key, and value vectors, respectively, with dk being the vector dimension. In our model, we first utilize TextCNN with various kernel sizes to capture multi-scale features, defined as C. C is then fed into Transformer to learn inter-scale relationships. In Transformer, C is first transformed into three different sets of vectors: query (Q = CW Q), key (K = CW K ), and value (V = CW V ), where W Q, W Q, and W V are learnable weight matrix. The resulting Q, K, and V vectors are then used in the computation of attention scores and the subsequent generation of the output sequence ˜ C. The new sequence representation ˜ C generated by multi-head self-attention on the input sequence C is processed through a feedforward neural network, where a non-linear transformation is applied to each token in the sequence. The function feature embedding is denoted as Ef :

where W1, b1, W2 and b2 are the learnable weight matrices and bias vectors. The overall process of function feature embedding is illustrated in Fig.3.

After obtaining the node feature in the contract graph, the node feature is propagated according to the order of edges based on GNN. At time step t, node information is transmitted through edge ek to its end node. Specifically, the message mk is computed based on the hidden state of the start node of ek, and the edge embedding ek:

where ⊕ denotes the concatenation operation, and Wk and bk are network parameters. Upon receiving the message feature, the end node of the edge updates its current hidden state hek by considering the previous hidden state and the edge-specific update mk :

where U, Z and R are matrices, b3 and b4 are bias vectors. Finally, for the smart contract graph with |V | nodes, the overall feature embedding of the smart contract obtained by aggregating the node features is:

4.3 Multi-task DownstreamOutput

In the final output layer, we defined two tasks related to vulnerability detection: vulnerability classification and vulnerability localization, corresponding to the classification of the smart contract graph and the classification of individual graph nodes, respectively. In the process of multi-task learning, information pertinent to the tasks is interchanged and enriched through a shared feature at an early stage, leading to simultaneous refinement of both node and graph classification tasks. For the contract graph classification task, the input is the fusion of contract graph feature Ec from the multi-task learning process and contract community feature ˜ G. The output of the classifier is defined as ˆ y1 , and y1 is the true label of the sample. The cross-entropy loss function is used to train the

The loss of node classification LossNC is similar to Equation(10). We achieve multi-task learning on graphs by designing a joint loss function that combines the two masked categorical cross-entropy losses for graph classification and node classification:

5 Experiments

In this section, we conduct experiments on a real-world smart contract dataset. We endeavor to investigate the following research questions:

RQ1: How does the proposed EFEVD method perform compared with traditional detection methods and state-of-theart deep learning based methods?

RQ2: How do the proposed enhanced features in EFEVD perform with each other to improve the performance?

RQ3: How do hyper-parameters impact performance?

5.1 Experimental Settings

Datasets. The dataset used in this paper comes from a realworld smart contract dataset on Ethereum. The labels in the smart contract dataset are binary, with 1 indicating the existence of vulnerabilities and 0 indicating the absence of vulnerabilities. The dataset mainly contains four types of vulnerabilities, namely reentrancy, timestamp, overflow, and secure(No Vulnerability). The labels for the functions were obtained using the Slither tool combined with manual verification. After filtering out samples with development language types that do not match the vulnerabilities, the original dataset is preprocessed and balanced, resulting in a total of 9742 smart contract data samples. The overall dataset is divided into a train set and a test set at a ratio of 7:3, where the train set is used for model training and the test set is used for model performance evaluations.

Compared Methods. To evaluate the improvement of the model's performance compared to that of existing methods, this paper adopts several highly recognized smart contract vulnerability detection methods as comparative experimental methods. The baselines can be categorized into four aspects:

• Static and dynamic analysis: This category includes tools such as SmartCheck [Tikhomirov et al., 2018], which uses static analysis to detect common vulnerabilities. Oyente [Liu et al., 2018], on the other hand, is a symbolic execution tool for detecting security flaws. Mythril [Feist et al., 2019] combines concolic analysis and taint analysis, while Securify [Tsankov et al., 2018] identifies vulnerabilities by analyzing the data flow in contracts.
• Semantics and syntax feature-based methods: These methods leverage the intrinsic features of smart contracts to detect vulnerabilities. Rechecker [Qian et al., 2020] analyzes the semantic information to identify hidden bugs. TMP [Zhuang et al., 2020] and DR-GCN [Zhuang et al., 2020] employs deep graph networks to analyze the syntactic structure of smart contracts.
• Expert knowledge enhanced method: The AME [Liu et al., 2021a] system incorporates expert knowledge into the detection process, combining the insights of smart contract security experts.
• Multi-modal feature enhancement method: SMS [Qian et al., 2023] integrates features from two different modalities: bytecode and source code.

Implementation details. All the models are trained by the Adam optimizer. The loss function is the softmax crossentropy loss, which is consistent with the underlying model architecture. The batch size is set to 64, and the learning rate is initialized to 0.01 and is dynamic during training. All the models are trained using an NVIDIA GeForce RTX 3060 GPU and implemented with TensorFlow 1.15. Our data and code are available on GitHub 1 .

Metrics. This paper uses four evaluation metrics, accuracy, recall, precision, and F1-score, to validate the experimental results. The accuracy represents the ability to correctly identify each class. Recall represents the proportion of samples with existing vulnerabilities that are correctly predicted as having vulnerabilities, which is mainly used to measure the probability of missed detection. Precision measures the proportion of samples predicted as the positive class that are actually positive. The F1-score is the harmonic mean of precision and recall, providing a balanced consideration of the model's accuracy and coverage.

5.2 Overall Performance(RQ1)

The experimental results of the comparison of smart contract vulnerability classification tasks are shown in Fig.5. Overall, the deep learning-based detection methods are generally better than the traditional detection tools, and their accuracy can be maintained at approximately 80%. The recall results of traditional methods is less than 60%, indicating that they miss a substantial number of true vulnerabilities. Within the deep learning-based approaches, the DR-GCN and TMP methods consider only the semantic and syntactic features of code, resulting in relatively poor vulnerability detection outcomes. Building upon this, the AME method integrates expert knowledge features of code, and the SMS method incorporates multimodal features of code, both demonstrating improved detection results over the DR-GCN and TMP methods, which substantiates the efficacy of feature enhancements. However, neither of these feature enhancement approaches achieves the effectiveness of the proposed EFEVD method. EFEVD outperforms the existing state-of-the-art methods, achieving a 10.12% increase in recall, an 8.17% increase in F1-score, a 4% increase in precision, and a 3.1% increase in accuracy, demonstrating the effectiveness of the two types of enhanced features considered in our proposed method.

1 https://github.com/xawmx/MTLContractdetection

5.3 Ablation Study(RQ2)

To discover the impact of our community- and shared- enhanced features, as well as the influence of several deep learning feature extraction methods used in our model on the vulnerability detection results, we conduct an ablation analysis by altering the following components:

• EFEVD(tt-): Without employing TextCNN and Transformer for semantic feature extraction, instead exclusively utilizing Word2Vec embedding along with Transformer positional encoding.
• EFEVD(sl-): Remove the syntax similarity during dividing the smart contracts communities.
• EFEVD(cf-): Remove the community feature enhancement. Only use the contract graph feature for vulnerability classification.
• EFEVD(sf-): Remove the shared feature enhancement. That is, remove the multi-task framework. Only consider the vulnerability classification task.

Ablation studies confirm the effectiveness of the proposed enhanced feature method. As shown in Table 1, replacing certain feature embedding components in the model with simpler methods, or removing either community features or shared features, leads to a decrease in performance across all four evaluation metrics. Specifically, The removal of TextCNN models in EFEVD(tt-) led to a decrease in all the metrics, emphasizing the importance of multi-scale semantic processing for vulnerability detection. Omitting syntax similarity in EFEVD(sl-) also led to a moderately reduced performance, highlighting syntax's role in accurately dividing smart contract communities. The exclusion of community feature in EFEVD(cf-) and shared feature in EFEVD(sf-) negatively affected all performance metrics, thereby confirming the effectiveness of the two types of enhanced features we proposed for detecting vulnerabilities. Overall, the complete EFEVD model outperformed all other variants, further validating the effectiveness of the enhanced features and their combination.

5.4 Hyper-Parameters Sensitivity(RQ3)

The hyperparameters, such as the learning rate and number of epochs, have a direct impact on the speed and performance of the model. The number of epochs refers to the number of times the model is trained on the data, and an appropriate number of epochs can ensure that the model stops training after having sufficient fitting ability, thus reducing the training cost. In this section, we set four initial values of the learning rate, namely 0.1, 0.01, 0.001, and 0.0001, and record the convergence performance of the model under different conditions. To comprehensively evaluate the precision and recall performance, we still use the F1-score as the evaluation metric, and the results of the parameter experiment are shown in Fig.6. Both of the vulnerability detection tasks improve when the learning rate is set between 0.01 and 0.001. However, when the learning rate is set to 0.0001, the evaluation metric tends to decrease. Considering all aspects, we choose 0.01 as the initial learning rate and adjust the learning rate during model training by 0.01 numiter , where numiter represents the number of training groups in which 10 epochs are grouped. This helps the model to quickly find the global optimum. For the training epochs used for model training, we set the number of epochs to range from 1 to 100, and gradually increase it to observe the convergence of the loss and accuracy during model training. The parameter experimental results are shown in Fig.7. Based on the loss curve, the model's learning effect is very obvious in the first 30 training epochs, and the loss value shows a significant downward trend. When the epoch is set to 30-50, the model tends to converge, and when the epoch is set to 50-100, the model converges. Therefore, the epoch hyperparameter in this paper is set to 30.

6 Conclusion

In conclusion, the proposed EFEVD method introduces novel feature enhancement for smart contract vulnerability detection. This approach diverges from the existing domainexpertise-reliant methods by incorporating community features, which reflect common patterns that are beneficial for identifying vulnerabilities. Moreover, EFEVD leverages a multi-task learning framework that utilizes shared features across two related tasks, enhancing the model's performance. The experimental results indicate that EFEVD achieves superior performances over the current state-of-the-art methods, demonstrating its potential as an effective tool for improving smart contract security.

Acknowledgements

This work was supported by the China National Science Foundation under grants 62172079 and U23A20310.