Multi-View Robust Graph Representation Learning for Graph Classification

1 Introduction

Graph classification models represented by graph neural networks (GNN), which aggregate graph nodes information using the graph topology for representation learning, have demonstrated remarkable performance in various graph classification tasks [Wang et al., 2022; Ma et al., 2022a]. However, this learning mechanism is susceptible to cascade effects, making graph classification models highly sensitive to the graph-structure input [Zhu et al., 2021]. This results in extremely poor robustness of graph classification models against noisy data, severely hindering their implementation in real-world scenarios.

∗Corresponding author

To improve the robustness of graph classification models, researchers have proposed various approaches, such as adversarial attacks based approaches [Zhang et al., 2022; Xu et al., 2022], graph representation learning based approaches [You et al., 2020; Zheng et al., 2020], graph structure learning based approaches [Luo et al., 2021; Song et al., 2022] and data augmentation based approaches [Wang et al., 2021; Han et al., 2022], etc. These approaches all rely on driving the model to converge on the prediction of original and perturbed graphs. In consequence, when confronted with noisy inputs, models trained using the above approaches do not jitter as much in predictive metrics (e.g., accuracy) as models learned using vanilla graph classification models.

Despite the outstanding performance, we empirically find that the existing graph classification approaches cannot guarantee semantic consistency across perturbed and original graphs, which we call the semantic bias issue. In detail, for a robust graph classification model, the semantic embedding of the perturbed and original graphs should be as close as possible while maintaining the stability of the metrics. This is crucial for applying graph classification models in some critical scenarios, such as bioinformatics and healthcare. As shown in Figure 1 (a), when we add 30% perturbations (adding edges) to the input graphs of the G-Mixup model [Han et al., 2022], as observed, the semantic representations of the perturbed graphs deviates significantly from that of the original graphs.

To deal with the above semantic bias issue, we propose an instance-view consistency representation learning method that performs semantic constraints through multi-granularity contrastive learning. Specifically, we rely on dropouts in the neural network to acquire multiple perturbed representations from the same original graph. Then, we employ unsupervised contrastive learning [Gao et al., 2021] to maximise the semantic consistency of the multiple perturbed representations at both the node- and graph-level. Moreover, we introduce an instance weighting method to avoid semantic damage caused by false-negative representations in contrastive learning.

Another empirical finding is that the prediction confidence (i.e. prediction probability) of existing graph classification models can vary dramatically in response to data perturbations, even leading to a collapse of the prediction results, which we call the confidence collapse issue. Existing methods usually simulate perturbation scenarios to evaluate the robustness of graph classification models. The fact is that simulations of perturbation scenarios are restricted, and the overall metrics stability of the dataset does not guarantee that the confidence in the data would remain constant. When experiencing complex or varied disturbances in practical circumstances, predicting confidence in graph classification models collapses. As shown in Figure 1 (b), when we remove 30% edges to the input graphs, for the G-Mix model [Han et al., 2022] , the prediction confidence changes by up to 60%.

To alleviate the confidence collapse issue, we propose a class-view discriminative representation learning method that adjusts intra- and inter-class distances utilizing the prototypedriven class distance optimization technique. Specifically, we employ class prototypes, i.e. class centres, as targets to drive graph representations closer to the class prototype of the same class and away from other class prototypes. In addition, we design prototype centre loss to further increase interclass spacing by adjusting the distance between prototypes. In this way, the model can obtain discriminative representations, thereby increasing its tolerance to perturbed data and mitigating changes in prediction confidence.

Based upon the above, we effectively integrate these two solutions into a unified model, resulting in our approach, Multi-View Graph Representation Learning (MGRL), where both solutions share the common goal: to learn a robust graph representation. The motivation behind this model is that a better representation can lead to more efficient and robust prediction results. We choose several relevant baselines from graph classification robustness studies for comparison on eight benchmark datasets, and the experimental results validate the effectiveness of MGRL.

Summarily, we make the following contributions: (1) We empirically discover that existing graph classification models suffer from semantic bias and confidence collapse issues when facing data perturbations. (2) We propose a novel model, MGRL, which alleviates the semantic bias issue via instance-view consistency representation learning, and mitigates the confidence collapse issue via class-view discriminative representation learning, respectively. (3) We demonstrate the effectiveness of our model through comparison, ablation, and visualization experiments on eight benchmark graph classification datasets.

2 RelatedWork

The recent advances in deep learning have enabled graph classification models to achieve superior performance on various graph classification tasks [You et al., 2020; Sun et al., 2020; Hassani and Ahmadi, 2020; Xu et al., 2021; Chu et al., 2021; Li et al., 2022b; Ma et al., 2022a; Wang et al., 2022; Ma et al., 2022b]. These approaches typically rely on a well-designed model architecture to understand the semantics of graph data. However, the message aggregation mechanism of graph classification models makes them highly sensitive to the structure of graph data [Zhang and Zitnik, 2020; Zhu et al., 2021], resulting in severe challenges to the robustness of graph classification models

Based on the above considerations, researchers have proposed various methods to address the robustness of these graph models [Geisler et al., 2021; Luo et al., 2021; Wang et al., 2021; Li et al., 2022a; Xu et al., 2022; Han et al., 2022]. For example, [Zheng et al., 2020] and [Luo et al., 2021] enhances the robustness of graph classification models by removing task-irrelevant edges from graph data. However, these previous approaches simply sought to maintain the overall data metric when facing data perturbations. Different from previous work, we experimentally verify that existing graph robust classification models are suffering from semantic bias and confidence collapse problems. Then, we explore the robustness of graph classification models for the first time from these two perspectives .

3 Methodology

3.1 ProblemFormulation

Given a graph G = {V, E}, we denote the node set and edge set as V = {v1, · · · , vN} and E ⊆ V × V respectively. The associated node feature matrix is represented as X = {xv|v ∈ V} ∈ R|V|×d , where xv denotes feature for node v and d is the dimension of input feature. Also, we leverage A ∈ {0, 1}|V|×|V| to denote the adjacency matrix, where Ai,j = 1 if (vi, vj) ∈ E. In this paper, we aim to learn a graph neural encoder F(·) to obtain more robust representations for downstream graph-level classification task, which can be formalized as : ˆ y = F(G(X, A)).

3.2 Overview

This paper proposes a multi-view graph representation learning model (named MGRL), which utilizes the instance-view consistency representation learning method and class-view discriminative representation learning method to alleviate the semantic bias and confidence collapse problems, thereby enhancing the effectiveness and robustness of graph classification models. As illustrated in Figure 2, our model primarily contains a perturbed graph encoder and two representation learning modules with different views.

3.3 Perturbed Graph Encoder

In this part, we employ the perturbed graph encoder to obtain different perturbed representations. For graph data perturbations, most existing works rely on human priors or expert knowledge [You et al., 2020] to design some specific perturbation strategies. Due to the diversity of data distribution and downstream tasks, these methods are not always effective and may even destroy the graph's semantics [Xia et al., 2022]. To circumvent this difficulty, we introduce the standard dropout as perturbations to obtain different graph embeddings. We refer to these embeddings as perturbed representations, which share the same gold label. This operation is the effective data perturbation method capable of preserving the samples' semantics [Gao et al., 2021].

Specifically, given a graph G with N nodes, we pass it to the graph neural encoder and forward propagate K times. Since the dropout in the neural network can randomly discard different neurons, the K perturbed representations we obtain have some differences. The process can be formalized as:

where hk = {hnk ∈ Rd1 }Nn=0 denotes the node representations set of the kth perturbation, hnk denotes one node representation from hk, d1 denotes the dimension of node representation and K denotes the number of perturbation.

3.4 Instance-ViewConsistency Representation Learning

In this section, we elaborate on our instance-view consistency representation learning approach. It employs a multigranularity contrastive learning technique to perform semantic consistency learning at both node and graph levels, alleviating the semantic bias issue in graph classification models.

Node-Level Semantic Consistency Learning. We argue that a node representation obtained by a robust graph classification model possesses the following qualities: On the one hand, each node ought to maintain its semantic stability. That is, its semantic representation should not change significantly in response to data perturbations. On the other hand, each node representation should keep its semantic identity, meaning that different node representations are required to retain a specific semantic distance in the embedding space.

To achieve the above purpose, inspired by deep graph contrastive representation learning [Zhu et al., 2020; Hassani and Ahmadi, 2020], we introduce the prevalent unsupervised contrastive learning [Gao et al., 2021] to constrain node representation semantics. In detail, we select a node representation hnk from the node representation set hk as an anchor. Accordingly, the corresponding node representations from other K - 1 perturbed representation set are employed as positive representations, denoted as hj . Also, we regard the node representations other than hnk and hj among {hk}Kk=0 as negative node representations, denoted as hneg. Note that our negative representations contain intra- and inter-graph node representations simultaneously.

We aim to maximize the semantic consistency between the anchor and positive node representations while minimizing the semantic consistency between the anchor and negative node representations. Therefore, the node-level consistency representation learning loss is defined as follows:

where s(, ) is a metric function to measure the semantic similarity between different node representations, the I indicates an indicator function and τ1 is a temperature parameter. In this paper, we choose cosine similarity as the metric function following the previous works [Zhu et al., 2020].

Graph-Level Semantic Consistency Learning. With the help of node representations, we can employ a READOUT function to obtain perturbed graph representations:

where the Hk ∈ Rd2 represents the graph representation for the kth perturbation. We adopt global average pooling as the READOUT function.

To maintain the semantic consistency of the model at the graph level, we can directly adopt a similar learning approach to the node level (Equation 2) for our purpose. Differently, the anchor is represented as a graph representation Hk, and the positive representations are the other K - 1 perturbed graph representations from the same original graph, marked as Hj . In addition, we refer to all perturbed graphs from other original graph representations within the mini-batch as negative representations, marked as Hneg. We can derive the optimisation objective of the graph-level semantic consistency:

where M indicates the number of graph in a mini-batch .

As observed in Equation 4, the unsupervised contrastive learning adopted can push away all other negative graph representations, regardless of whether they have the same gold label as the anchor representation. We refer to these incorrectly pushed-away representations as false negative representations. We argue that this indiscriminate learning mechanism would disrupt the semantic representation learning of the graph encoder. Note that, the above case exists only for graph-level consistency representation learning. The negative representations of the anchor representation come from one mini-batch, which may contain graph representations with the same label as the anchor. In contrast, the negative representations of a node all come from an original graph and share the same label. Considering this, we introduce an instance weighting method to avoid the detriment caused by false negative representations for graph-level consistency learning.

More specifically, given an anchor graph representation Hk, and its label yk, one of anchor's negative graph representations hneg, the negative representation's label yneg, we can depend on their gold labels to generate the weight of this negative representation:

This way, if a graph representation is the anchor's false negative representation, its weight will be set to 0. Conversely, the weight is set to 1. To this point, the Equation 4 can be rewritten as follows:

where wneg represents the weights of negative representations.

Eventually, our instance-view consistency representation learning loss can be defined as follows:

3.5 Class-ViewDiscriminative Representation Learning

In this section, we describe our class-view discriminative representation learning approach. It utilizes a prototype-driven class distance optimization technique to adjust intra- and inter-class distances, thereby mitigating the confidence collapse issue of graph classification models.

To be specific, given K perturbed graph representations {Hk}Kk=0 , we first map these representations to the class space through a nonlinear projection network, a one-layer MLP followed by an activation function:

where the zk ∈ Rd3 represents the class space embedding for the kth perturbed graph representation and d3 denotes the dimension of the representation.

Then, with the help of the gold label, we can directly obtain the class prototypes by averaging the representation vectors with the same label, which can be denoted as:

where yi and Ni denote the label and number belonging to class i respectively. yk denotes the gold label of zk.

Since the calculation of class prototypes involves all graph representations, resulting in huge computational expense. Therefore, we introduce a moving average method [Ge et al., 2023] to update the class prototypes. This method can reduce the computational cost while stabilising training:

where ci,t denotes the prototype of class i in the t step and the λ ∈ (0, 1) is the moving average coefficient.

To optimize intra- and inter-class distances, we urge graph representations from the same class to approach the class prototype and move away from other class prototypes. We can learn a more compact class representation by reducing intraclass distances and increasing inter-class distances. The loss function can be defined as follows:

where zi belongs to class i, ci,t is the class prototype of class i, co,t denotes a class prototype of other class, d(·) denotes the distance measure function, |C| is the number of class and τ3 denotes the distance scaling factor.

Intuitively, we can adjust the class distances for discriminative representation learning via Equation 11, where the denominator represents the distance between class representations zi and its corresponding class prototype ci,t, and the numerator represents the distance between class representations zi and other class prototypes co,t. Generally, the distance between the zi and co,t is much larger than the distance between the zi and ci,t, resulting in ed(zi,co,t) (denominator) being significantly smaller than ed(ci,t,zi) (numerator). In particular, as training proceeds, the former keep decreasing and the latter keep increasing to approach e. This causes the gradient signal produced by Equation 11 to maintain decreasing, slowing or even ending training.

To handle this issue, we design prototype centres, the average vector of the two class prototypes, as virtual representations. The prototype centre is closer to the class representations than other class prototypes co,t, so it can provide more gradient signal for class distance optimisation.

Intuitively, we can directly replace the co,t in Equation 11 by the prototype centre. However, we experimentally find that utilizing the prototype centre to adjust the prototype distance can achieve better results. Therefore, we further adjust the inter-class distance by the following prototype centre loss:

where ci,j cen denotes the prototype centre of class prototype ci,t and class prototype cj,t Since the cosine function is used as the distance metric function, we minimize the Equation 12 equivalent to maximizing the distance between the prototype centre ci,j cen and the prototype ci,t.

Eventually, our loss for class-view discriminative representation learning can be defined as follows:

where α3 and α4 are trade-off parameters.

3.6 Training Objective

To the end, we obtain prediction results for the graph classification task employing a forward propagation network, which consists of a layer of perceptrons, and the standard crossentropy loss can be defined as:

where the y denotes the gold label.

Finally, the whole objective training loss for our model can be expressed as follows:

where θ denotes the parameters of the model.

4 Experiment

4.1 Datasets

We evaluate our approach on eight benchmark datasets from TUDataset [Morris et al., 2020], including four social networks datasets, such as IMDB-BINARY (IB), IMDBMULTI (IM), COLLAB (CO) and REDDIT-BINARY (RB), and four bioinformatics datasets, such as PROTEINS (PR), DD, NCI1 (NC) and Mutagenicity (MUT). The details of the statistical results of the datasets are shown in Table 1.

4.2 Implementation Details

We adopt accuracy as an evaluation metric for the graph classification task and employ GCN [Kipf and Welling, 2017] as the graph encoder. We randomly select 80% of the data for the training set, 10% for the validation set, and the remaining 10% for the test set. During the practical implementation, the batch size is set to 32, the dropout is set to 0.5, the moving average coefficient λ is set to 0.0001, and the α1, α2, α3, α4 are set to 0.001, 1, 1, 1, respectively. We utilize a grid search technique to select the other best hyper-parameters with the learning rate selected from {0.01, 0.05, 0.001, 0.005}, the temperature parameter τ1, τ2 and τ3 selected from {0.07, 0.1, 0.3, 0.5, 0.7, 0.9} and the K selected from {2,4,6,8,10}. We train our model using the Adam optimizer [Kingma and Ba, 2015] and adopt the validation set to perform model tuning. For comparison with related works, we leverage the opensource implementation of their original papers. We use PyTorch to implement our model on a Linux machine with a GPU device Tesla V100 SXM2 32 GB. All experimental results are from an average of five runs.

4.3 Performance Comparison

To evaluate the effectiveness and robustness of our model, we select some related works for comparison, including three graph representation learning approaches: MVGRL [Hassani and Ahmadi, 2020], InfoGraph [Sun et al., 2020], GraphCL [You et al., 2020], one graph structure learning approach VIB-GSL [Sun et al., 2022], one graph causal learning approach CAL [Sui et al., 2022] and three graph data augmentation approaches CSSL [Zeng and Xie, 2021], NodeSam [Yoo et al., 2022] and G-Mixup [Han et al., 2022].

As observed in Table 2, our MGRL shows the best results on most datasets, e.g. 2.84% higher accuracy than the competitive method GraphCL on the CO dataset and 2.14% higher accuracy than the model G-Mixup on the PR dataset, respectively. GraphCL develops four graph-specific data perturbation strategies to improve the model's semantic encoding capability. Unfortunately, these perturbation strategies may corrupt the original semantic structure of the graph. GMixup employs a mixup method for data expansion but ignores class-level representation learning. In contrast, our approach not only maintains the semantic integrity of the graph data but also learns the class discriminative representations through a prototype-driven distance optimisation technology.

4.4 Robustness Studies

Robustness to semantic consistency. As shown in Figure 3, our model MGRL demonstrates a strong advantage in semantic consistency under various perturbation scenarios. For instance, when removing edges, the perturbed graph representations of the NodeSam and G-Mixup models completely deviate from the original samples since they only focus on the overall metric of the data. In contrast, our MGRL utilises multi-granularity contrastive learning to obtain a consistent representation from the instance perspective.

Robustness to confidence variation. To assess the model's ability to stabilise confidence, we compare the absolute value of the difference in prediction confidence between the original and perturbed graphs for different models. As shown in the Figure 4, MGRL has the smallest jitter in prediction confidence for the two different perturbation scenarios compared to the other three models. The reason is that our prototype-driven class distance optimisation method learns a class-view discriminative representation, thereby improving the tolerance of the model to perturbed samples. This can effectively mitigate the confidence collapse issue.

Robustness to overall data metric. We compare the change in model performance against the perturbations to assess its robustness to the overall data metric. As shown in Table 3, our method achieves the best results when faced with different proportions of perturbations. For example, when removing 20% of the edges, our method obtains an accuracy of 73.40%, while the two different models NodeSam and GMixup yield an accuracy of 71.60% and 72.40%.

4.5 Ablation Studies

To explore the effectiveness of each model component, we design five variants of MGRL, including: (1) MGRLw/o nodel level removes the nodel-level semantic consistency learning, (2) MGRLw/o graph level cuts out the graph-level semantic consistency learning, (3) MGRLw/o instance view does not use the instance-view representational learning module, (4) MGRLw/o prototype center deletes prototype center loss and (5) MGRLw/o class view removes class-view representation learning module.

As observed in Table 4, the removal of any model component results in a decrease in model performance. For example, compared to MGRL, the accuracy of MGRLw/o nodel level and MGRLw/o graph level decrease by approximately 0.56% and 0.92% on CO dataset respectively, with MGRLw/o instance view drops even more. We observe the largest degradation for MGRLw/o class view, suggesting that class-view representation learning is more important to the classification performance of the model.

4.6 Graph-Level Representation Studies

To demonstrate that our model can learn discriminative representations, we visualize the graph embedding obtained by the graph encoders of different models on the CO dataset. As observed in Figure 5, the different class representations of CSSL, NodeSam and G-Mixup have almost no spacing, and the representations within the same class are more dispersed. In contrast, the graph representations learned via MGLR are more compact and have greater inter-class distances, indicating that our model can achieve more discriminative representations. The reason is that we use the prototype-driven class distance optimization approach to simultaneously force samples away from other class prototypes and different class prototypes away from each other.

5 Conclusion

This paper proposes the MGRL to improve the performance and robustness of graph classification models simultaneously. MGRL utilizes an instance-view consistency representation learning method and a class-view discriminative representation learning method to alleviate semantic bias and confidence collapse issues. The experiments on eight benchmark datasets illustrate the effectiveness of our MGRL.

Acknowledgements

We sincerely thank the reviewers for their insightful comments and valuable suggestions. Thanks for the computing infrastructure provided by Beijing Advanced Innovation Center for Big Data and Brain Computing.