CooBa: Cross-project Bug Localization via Adversarial Transfer Learning

1 Introduction

Bug localization, which aims to locate the corresponding buggy source code files for a given bug report, has been attracting more and more attention in software quality control [Zhou et al., 2012]. Software projects routinely receive a large number of bug reports describing the errors or unexpected results during program operation [Li et al., 2018]. Once a bug report received, programmers can use it to locate the related buggy source code files (referred to 'code files' for short). Many studies have documented the tremendous success of applying supervised machine learning models for bug localization, which are capable of automatically pointing out the related code files with respect to a new bug report [Zhou et al., 2012; Ye et al., 2014; Huo and Li, 2017]. However, much of the work on bug localization only copes with the projects with sufficient historical bug-fix information (i.e., labels) [Zimmermann et al., 2009]. The performance of these techniques could be dramatically affected by the quantity and quality of bug reports and their related code files [Rahman and Roy, 2018]. Unfortunately, in practice, it might be difficult to obtain sufficient bug-fix data. This is especially true for new projects in the first release. Such newly developed projects urgently need an automated bug localization technique to ease the burden on programmers.

∗Corresponding Author

One way to deal with the shortage of historical label data is to leverage the knowledge from label-rich projects to locate buggy code files in the current project. Recently, a few works have explored how to transfer available knowledge from label-rich projects [He et al., 2012; Ma et al., 2012; Nam et al., 2017]. For example, Huo et al. [2019] proposed the first cross-project bug localization (CPBL) model TRANP-CNN. It first extracted transferable features from bug reports and source code files of source and target projects, then generated project-specific predictions for new bugs by the extracted features. However, a straight-forward application of transfer learning for CPBL is prone to bring the noise into the model, known as the negative transfer effect. For example, the multiple projects in a CPBL task may be produced by different companies (e.g., Microsoft or Mozilla Foundation), developed for different domains (e.g., web browser or mobile application), or designed by different development concept (e.g., large scale development or agile development), etc [Zimmermann et al., 2009]. Therefore, directly transferring knowledge from a label-rich project inevitably brings information that pertains to that specific project (referred to as 'private information' in this paper). Transferring such private characteristic information is likely to hurt, instead of help, the prediction of the buggy code file in another project.

Based on the above observation, we present a new model named COOBA for cross-project bug localization. To be specific, from bug reports, the proposed COOBA learns indicative information about the bug through an encoder that is shared across multiple projects. From code files, the proposed COOBA simultaneously extracts both the public and private information. COOBA leverages an individual Graph Convolutional Networks (GCN)-based feature extractor for each project to extract the private information and a shared Convolutional Neural Network (CNN)-based feature extractor between projects to extract the public information. It further introduces adversarial learning to ensure the public information shared between projects could be effectively extracted. Extensive experiments on large-scale real-world data sets reveal that our model COOBA significantly outperforms state-of-the-art on all evaluation measures.

The main contributions of our work are as follows,

• We present an end-to-end bug localization model called COOBA, to locate the code files that need to be fixed for projects with insufficient historical bug-fix data.
• Our model leverages adversarial transfer learning to transfer public information shared between multiple projects, while preventing the private information to avoid negative transfer.
• We explore the rich structural information of the code file, and extract the private information by a novel multilayer GCN.

2 RelatedWork

2.1 Within-project Bug Localization

Most of the existing bug localization models have been evaluated on within-project bug localization (WPBL) settings. As shown in Figure 1(a), each instance representing a pair (b, c), where b is a bug report and c is a code file, with a label (buggy or clean) to indicate whether the code file c is related to the bug report b. In this setting, a supervised model is trained using the fixed bug reports history in Project p. This model is then used to locate the buggy code files that caused the inappropriate bugs described in the upcoming bug reports for the same Project p. Information Retrieval (IR) and machine learning methods are widely used in WPBL. For example, Zhou et al. [2012] proposed the BugLocator based on the revised Vector Space Model (rVSM). Kim et al. [2013] treated the bug localization as a classification task and proposed a two-phase prediction model using Naive Bayes. With the advent of deep learning, many researchers presented deep bug localization techniques. For instance, Lam et al. [2017] proposed a bug localization model combining rVSM [Zhou et al., 2012] with Deep Neural Network (DNN). NP-CNN [Huo et al., 2016] and LS-CNN [Huo and Li, 2017] learned a unified feature from natural language and programming language to locate the buggy code files.

2.2 Cross-project Bug Localization

More often than not, software engineers need bug localization beyond the same project. Cross-project bug localization (CPBL) techniques predict buggy code files even for new projects lacking in fixed bug reports history by 'borrowing' information from other projects. As shown in Figure 1(b), a supervised model is trained by the help of labeled instances in Project s (source project) and predicts buggy source files for the upcoming bug reports in Project t (target project).

Recently, several works have explored how to transfer available knowledge from data-rich projects [He et al., 2012; Ma et al., 2012; Nam et al., 2013]. For example, Turhan et al. [2009] proposed Nearest Neighbor (NN) Filter by selecting similar instances from source project to construct a new training set that is homogeneous with the target project. Nam et al. [2013] proposed another cross-project defect prediction approach, called TCA+, by extending Transfer Component Analysis (TCA) [Pan et al., 2010], which utilized the latent feature space provided by TCA for data of source and target projects. Huo et al. [2019] proposed a cross-project bug localization model TRANP-CNN that combined deep learning and transfer learning. It first extracted transferable potential features from bug reports and source code files for source and target projects, then generated project-specific predictions for new bugs by extracted features.

3 The ProposedModel

In this paper, we focus on cross-project bug localization task. For source project s, we denote Bs = {bs1 , bs2 , . . . bsms} as the set of bug reports, and C s = {cs1 , cs2 , . . . , csns} as the collection of code files, where the ms , ns is the number of the bug reports and code files, respectively. For target project t, we denote Bt = {bt1 , bt2 , . . . btmt} as the set of bug reports, and C t = {ct1 , ct2 , . . . , ctnt} as the collection of code files, where the mt , nt is the number of the bug reports and code files, respectively. It should be noted that mt is far less than ms . Besides, indicator matrices W α ∈ Rmα×nα are used to indicate a code file is buggy or clean with respect to a bug report, and α ∈ {s, t}. As an example, W s i,j = 1 indicates that code file cj is a buggy file of the bug report bi in the source project, and W s i,j = 0 indicates not.

We instantiate the cross-project bug localization as a classification learning task. In the training process, the model aims to learn prediction functions f α : Bα ×C α →W α by the input pairs (bα, cα) from the source and target projects with their labels, α ∈ {s, t}. After the model is fully trained, prediction function f t is used for determine the relationship of each pair (bt , ct ) from target project t during testing. The framework of our COOBA is illustrated in Figure 2, which consists of three integral parts: shared bug report processing, cooperative code file processing, and relevance prediction. We will elaborate on each part in the following subsections.

3.1 Shared Bug Report ProcessingModule

In our task, bug reports submitted to different project groups are written in the same natural language (English) and share the same topic (incorrect or unexpected results during program operating). We take the attitude that sharing bug report processing module between source and target projects in training is beneficial for learning natural language knowledge by one project and subsequently utilized by another one. The shared bug report processing contains an embedding layer and a bug report encoder to extract high-level indicative information about the bugs described in a bug report.

Embedding Layer

Given a bug report, we can extract all sentences from the summary and description items, which are the main part of a bug report, then integrate all sentences into one sequence. We exploit the pre-trained GloVe [Pennington et al., 2014] to map each word of the sequence into a k-dimensional embedding. The input embedding layer represents the bug report b as a embedding sequence w = {w1, . . . ,wn}, n is the number of words in the sequence. It is worth mentioning that the bug report embedding layer is shared for both source and target projects.

Bug Report Encoder

After representing the bug report as an embedding sequence w, we exploit a Bidirectional Long Short-Term Memory (BiLSTMs) [Schuster and Paliwal, 1997] to encode the input sequence w by considering information from both forward and backward directions. We concatenate the last forward hidden state -→hn and the first backward hidden state ←- h1 into a new

where the wt is the input of the Bi-LSTMs at the time step t, and the ⊕ is the concatenate operation. After the embedding sequence w passes through the encoder, we obtain the b that is the indicative representation of the bug report. For bug reports from project s or project t, we also share the encoder layer and denote the generated indicative representation as bs or bt , respectively.

3.2 Cooperative Code File ProcessingModule

To appropriately transfer knowledge from a label-rich project to a target project, we propose a cooperative code file processing module for extracting shared information of code files belonging to different projects and simultaneously extracting their private information. It is worth mentioning the cooperative nature of our approach. This is crucial because transferring knowledge directly from a source project will likely bring its private information that is not applicable to another (target) project. There are mainly four components in this module: embedding layer, private feature extraction, public feature extraction, and project-specific feature fusion.

Embedding Layer

We exploit graph structure to represent code files, thereby extending the text representation from a sequential (or grid) point of view to a graphical view. For each code file, we obtain a graph G and a node embedding matrix X from the embedding layer. Specifically, the graph G = (V, E) is generated by the Abstract Syntax Tree (AST) corresponding to the code file, which is widely used in code analysis and clone detection [Neamtiu et al., 2005]. V is the node set containing the code tokens in the AST (each node is a code token) and a synthetic root node 0 (represents the start of the program). E is the edge set containing the links between tokens in the AST. The node embedding matrix X ∈ R|V|×k contains code token embeddings initialized by GloVe. Each row of X is the embedding of a node (i.e., code token), and k is the dimension of the vector. For code files from source and target project, we set up two code embedding layers with the same structure but no parameters shared between them. Therefore, we obtain the graph G α and the node embeddings Xα then feed into the individual and shared feature extractors subsequently, where α ∈ {s, t}.

Private Feature Extraction

For private feature extraction from each project, we employ the multi-layer GCN [Kipf and Welling, 2016] with the same structure but no shared parameters as the individual feature extractor. Previous work of GCN has demonstrated the ability to operate directly on a graph and induce the embedding vectors of nodes based on the properties of their neighborhoods [Linmei et al., 2019]. For a code graph G of code file c, we introduce the ˜ A = A+I, which is the adjacency matrix of the undirected graph G with added self-connections, where I is the identity matrix. And the GCN follows the layer-wise propagation rule,

where ˜Dii = ∑j ˜ Aij, σ(.) denotes an activation function. Hl+1 and W l are the node hidden representations and the trainable weight matrix in the l-th layer; H0 = X. For a code file from project α, its corresponding individual encoder Eα in generates the private feature gα pri as follows,

where G α and Xα are obtained from code file embedding layer.

Public Feature Extraction

Besides the private features of each project, public feature extraction aims to learn the common feature among different projects. Specifically, the adversarial training is utilized to ensure effective learning public features [Wang et al., 2018a]. The adversarial training strategy [Ganin et al., 2016] normally consists of a generator and a discriminator. In our task, we use a shared feature extractor as the generator, and the discriminator aims to estimate which project the code file comes from.

Shared Feature Extractor. To extract the fundamental information shared in multiple projects, we adopt CNN, which is widely used for text processing and analysis tasks [Kim, 2014], in the shared feature extractor. Many experimental results [Kalchbrenner et al., 2014] have shown that the convolution and pooling operations enable CNN to induce lexical and semantic features over the input sequence. Accordingly, we prefer to choose convolution kernels with small sizes (e.g., 3, 5), especially including convolution kernels of size 1 to extract the lexical and semantic features in code files. For simplicity, we use Esh denote the CNN structure, the input of it is the embedding matrix Xs of source project or the embedding matrix Xt of target project. The extractor generates the public feature of code files as the following equations,

Project Discriminator. We incorporate adversarial training into shared space to guarantee that private features of projects do not exist in shared space. We employ a multilayer perceptron (MLP) network as a project discriminator to estimate which project the code file comes from. Formally, the project discriminator can be expressed as follow,

There is a min-max optimization that the shared encoder generates a representation to mislead the project discriminator and the discriminator tries its best to correctly determine the type of project (source project or target project). The adversarial training process can be formalized as follow,

where θsh and θd denote the trainable parameters of shared feature extractor and project discriminator, respectively. T α is the number of training instance of project α. Xα (i) is the i-th instance of project α.

Project-specific Feature Fusion

We obtain the private and public representations for each code file by the feature extractors. However, due to the unequal amount of data in the source and target projects, different projects may differ in how their private and public information are correlated. Thus, we design the project-specific feature fusion layer to learn the project-specific correlation patterns considering the private and public features. For a code file from project s or project t, we get a fused representation appropriately merged by a two-layer MLP network as

where the ⊕ is the concatenate operation.

3.3 Relevance PredictionModule

The relevance prediction module aims to learn the correlation patterns of bug reports and their related code files by the information obtained from bug report and code file processing module. As the lack of labeled instances, the relevance prediction module cannot perform well on the target project. We take advantage of the fact that bug reports and related code files are correlated in a similar way in both source and target projects, which is all bug reports describe how the product was damaged. Therefore, we construct the shared relevance prediction module across the source and target projects. Specifically, to measure the relevance degree of a (b,c) pair, the relevance metric in the prediction module is defined as

We encourage the representation of the relevant buggy code file to be as close as possible to the representation of the given bug report. For a bug report b, we use C+ to denote its related code file set and treat unrelated code file set as C-. Considering the related pairs (b, c+) should have lower distance than the unrelated ones (b, c-), where c+ ∈ C+ and c- ∈ C-. Thus, the task loss function for each project is

where F(., .) is computed by Eq. 10; τ is a margin, forcing F(b, c+) to be greater than F(b, c-); α ∈ {s, t}.

3.4 Training

Giving the labeled bug-fix data of project s and project t, the final training objective is defined as follow,

where λ is the hyper-parameter, Ls and Lt are computed via Eq. 11, and Ladv is computed via Eq. 8. I(b, c) is an indicator function to identify which project the input pair (b, c) comes from. It is defined as follow,

where P s and P t are source project corpora and target project corpora, respectively. At each iteration in the training process, we alternately sample a batch of training instances from P s or P t to update the parameters. In this way, we adapt Adam [Kingma and Ba, 2014] to directly minimize the final loss function L in our model.

4 Experimental Results and Analysis

4.1 Experimental Setup

Data sets. We evaluate our method on the data sets provided by Ye et al. [2014]. The data sets contain the bug reports, source code links, buggy files, API documentation, and the oracle of bug-to-file mappings, which are all publicly available 1 . Four open-source projects are collected in these data sets as follows,

• AspectJ: an aspect-oriented programming extension for Java programming language.
• SWT: an open source widget toolkit for Java.
• JDT: a suite of Java development tools for Eclipse.
• Eclipse Platform UI: a user interface of a development platform for Eclipse.

1 http://dx.doi.org/10.6084/m9.figshare.951967

Comparison methods. We compare COOBA with the following methods, including the conventional models for within-project bug localization and the recent models for cross-project bug localization:

• BugLocator [Zhou et al., 2012]: a classic IR-based bug localization method that consider similar bugs information that have been fixed before.
• DNNLOC [Lam et al., 2017]: a model combining rVSM [Zhou et al., 2012] with DNN while considering the metadata of the bug-fixing history and API elements.
• NP-CNN [Huo et al., 2016]: a deep learning method based on CNN, which leverages both lexical and program structure information from natural language and programming language.
• NN Filter [Turhan et al., 2009]: a cross-project and cross-company defect prediction method by selecting similar instances from source project to construct a training set that is homogeneous with target project.
• TCA+ [Nam et al., 2013]: a cross-project defect prediction approach, which is an extension of TCA by normalization before applying it.
• TRANP-CNN [Huo et al., 2019]: a transfer learning approach for cross-project bug localization by extracting transferable semantic features from the source project.

Among these methods, BugLocator, DNNLOC, and NPCNN are designed for within-project context. Burak, TCA+, and TRANP-CNN are recent methods for cross-project software mining problems. Followed the work of Huo et al. [2019], we select the cross-project defect prediction models Burak and TCA+ as our comparison methods. In our experiments, we use the same settings suggested in their original works. For bug localization in cross-project context, we suppose the training data includes all the fixed bug reports of source project and 20% fixed bug reports of target project. The remaining 80% bug reports of target project are used for testing. We repeat this experiment for 10 times, and 10 cross-validation is used in the experiment. We evaluate the bug localization performance with two criteria, i.e., Top-10 Rank and Mean Average Precision (MAP), which are widely adopted in bug localization based on historical bug-fix information [Zhou et al., 2012; Wang et al., 2018b; Huo et al., 2019].

4.2 Effectiveness Results

The experimental results of accuracy comparison for different methods are shown in Figure 3 and the best performance of each task is marked with ·. The top half of the figure depicted performance w.r.t. Top-10 Rank, and the bottom half w.r.t. MAP. The horizontal coordinate indicates different tasks, for example, "AspectJ → JDT" represents using AspectJ as the source project and locates the bugs in JDT. From Figure 3, we can observe: 1) Our proposed COOBA can effectively resolve the cross-project bug localization problem, and outperform the state-of-the-art in all tasks on both metrics. As we can see, COOBA outperforms all the compared methods. For example, COOBA defeats its best competitor (i.e., TRANP-CNN) by 6.5% w.r.t. MAP on project Eclipse Platform UI. It also achieves 4.1%, 4.8%, and 5.2% improvement over its best competitor w.r.t. MAP on JDK, AspectJ, and SWT respectively. In summary, COOBA performs best considering cases comprising of all possible pairs of metrics and tasks. Compared with the average results, COOBA achieves the average top-10 of 0.652 and MAP of 0.351. 2) The conventional within-project bug localization models trained in source data cannot be directly used effectively in target data. The experimental results show that the performance of all transfer learning bug localization models (i.e.,COOBA and TRANP-CNN) significantly outperforms the within-project bug localization models. For example, COOBA and TRANPCNN improve NP-CNN by 11.7% and 7.6% w.r.t. MAP on project JDT. They also achieve 10.3% and 4.5% improvement over DNNLOC w.r.t. Top-10 on AspectJ. These results indicate that there is a demand to build a specialized technique for cross-project bug localization, since directly using withinproject methods does not produce satisfactory performance.

4.3 Effectiveness of Adversarial Transfer Learning

To investigate the impact of adversarial learning on transferring knowledge from the source project to the target project, we change the model and denote it as COOBA* in this group of experiments. The main differences between COOBA and COOBA* are (1) individual feature extractors and (2) a project discriminator. The COOBA* only use a shared feature extractor to directly transfer knowledge across projects. We compare the results of COOBA with COOBA* on all tasks in Table 1 and find, in terms of MAP, COOBA performs significantly better in all tasks. The results indicate that a straightforward application of transfer learning is prone to negative transfer effect. And the adversarial transfer learning with a project discriminator significantly improves the performance of cross-project bug localization. These results demonstrate that the adversarial transfer learning indeed avoid bringing in the noise that does not pertain to the target project, thereby more effectively and reasonably transferring knowledge from the source project.

5 Conclusions

Cross-project bug localization is crucial for projects with limited bug-fix information. It can significantly extend the application horizon of bug localization techniques, e.g., locating system bugs in the first release of a new project. To produce a successful cross-project bug localization, we need to transfer knowledge from source project carefully, while avoiding bringing in the noise that does not pertain to the target project. In this paper, we propose a novel model named COOBA leveraging adversarial transfer learning. Experimental results demonstrate the effectiveness of the proposed COOBA. In future work, we plan to explore cross-project bug localization in the context of projects written in different programming languages.

Acknowledgments

This work was partially supported by Natural Science Foundation of China (No.61772284), Postgraduate Research&Practice Innovation Program of Jiangsu Province (SJKY19 0763). And the Hanghang Tong author is partially supported by NSF (1947135, 1715385, and 1939725 ).