Multi-view Unsupervised Graph Representation Learning

1 Introduction

Unsupervised graph representation learning (UGRL) can output discriminative representations using both structural (e.g., the relationship of data points) and non-structured (e.g., node feature ) information of the data, making it easily extract useful information to benefit downstream tasks [Wang et al., 2020a; Gan et al., 2021]. To achieve this, the objective function of the UGRL is often designed to maximize mutual information (MI) between the input and its related information. However, the MI maximization is popular transferred to the Jensen-Shannon divergence maximization, as its computation cost is expensive [Liu et al., 2021]. This results in contrastive learning based UGRL (CL-UGRL). Recently, CLUGRL shows its superiority in unsupervised representation learning, and has been widely used in various learning tasks, such as clustering and community detection [Jaiswal et al., 2021].

∗ Corresponding author (seanzhuxf@gmail.com).

The CL-UGRL method mainly includes three key components, i.e., data augmentation, graph convolutional network (GCN) encoder, and contrastive loss. Specifically, data augmentation aims at creating rational data for contrastive learning by applying certain transformation [Qiu et al., 2020]. GCN encodes the topology structure and node features of the data. The contrastive loss is related to the definitions of anchors, positive samples and negative samples. Moreover, it pushes the anchor embedding similar to positive emedding and dissimilar to negative embedding [Tian et al., 2021]. Although current CL-UGRL methods have achieved success, due to the complexity of the data, previous CL-UGRL methods still have many limitations.

The commonly used graph data augmentation methods are random perturbation, including node dropping, feature masking, subgraph and edge perturbation. For example, to conduct data augmentation, [Zhang et al., 2020] employed node dropping and subgraph, while [You et al., 2020] employed edge perturbation and feature masking. However, random perturbation might destroy the inherent structure and property of the graph, a�ecting the model performance. To address this issue, [Hassani and Khasahmadi, 2020] used the di�usion method to generate augmented data, which avoids the destruction of the data structure. However, such a method is independent of contrastive learning, and thus fails to adaptively consider the impacts of input graphs.

Compared to unsupervised representation learning methods, the UGRL maintains the graph structure of data points during representation learning, thereby outputting more discriminative representations [Goyal and Ferrara, 2018]. Therefore, the quality of the graph is very important to the performance of the UGRL method. In current UGRL methods, the topology graph and the feature graph are two popular graph structures to represent the relationship between data points. The topology graph usually comes from the real world to reflect the connection of data points, such as social networks and citation networks [Yang et al., 2019]. The feature graph is constructed by the kNN method or the �-graph method to represent the similarity of data points in the feature space [Liu et al., 2020]. These two kinds of graphs contain di�erent information, which may provide complementary inforamtion to benefit the UGRL. However, existing UGRL usually cannot take full advantage of the information. Therefore, the fusion of two graphs is a way to improve the UGRL.

In this paper, we propose a multi-view CL-UGRL method to address the aforementioned issues by two key components, i.e., adaptive data augmentation and multi-view contrastive learning. Our adaptive data augmentation first generates the feature graph from the feature space and then designs a deep graph learning model to jointly update the feature graph and the new representation based on the original representation. We further combine the feature and topology graphs with the original and new representations to form multi-view information, which is fed into two GCNs to generate multiview embedding features. Two kinds of contrastive losses are designed on multi-view embedding features to explore the complementary information between the topology and feature graphs. Di�erent from traditional data augmentation independent on contrastive learning, we embed adaptive data augmentation and multi-view contrastive learning in a framework to form an end-to-end model, which jointly conducts data augmentation and multi-view contrastive learning. As a result, data augmentation is iteratively updated by the adjusted multi-view contrastive learning and vice versa.

Compared to previous methods, the main contributions of our method are summarized as follows.

• � First, we propose a new data augmentation method, which generates a new graph and new representation, aiming at preserving the intrinsic structure of the data to generate multi-view information for contrastive learning. Furthermore, it is dependent on our multi-view contrastive learning as both of them are embedded in the same framework.
• � Second, we propose a new multi-view contrastive learning framework, including two kinds of contrastive losses, which provide complementary information between the feature and topology graphs to strength multiview contrastive learning.
• � Third, we embed adaptive data augmentation and multiview contrastive learning in a unified framework so that each of them can be iteratively adjusted by the other's update. Extensive experiments on benchmark data sets clearly demonstrate that our method outperforms the state-of-the-art methods on di�erent downstream tasks.

2 Method

In this paper, G1 = (X;A) consists of A and X, where X ∈ R N×d and A ∈ R N×N represent the feature matrix and the topology graph, respectively. N is the number of nodes and d is the dimensions of node features. Moreover, ai j = 1 indicates that there is an edge between the i-th node and the j-th node, otherwise ai j = 0.

The proposed framework is shown in Figure 1. Its key components include adaptive data augmentation and multiple contrastive learning. Specifically, given X and A, our adaptive data augmentation adaptively learns both the new representations eX and the feature graph S of X, followed by generating multi-view information, i.e., G1 = (X;A), e G1 = (eX;A), G2 = (X; S) and e G2 = (eX; S). In particular, two-view information with the same graph, e.g., G1 and e G1 (or G2 and e G2), is input into the same encoder to output new embedding features Zt and eZt (or Zf and eZf ), respectively. Based on resulted multi-view embedding features, we further design the intra-graph contrastive loss and the inter-graph contrastive loss to conduct multi-view contrastive learning.

2.1 Adaptive Data Augmentation

Existing data augmentation strategies are popularly used for Euclidean data, but are di�cult for the graph data [Chen et al., 2020]. In the literature, graph contrastive learning usually relies on the contrast between node embedding features in di�erent views [Rai et al., 2021]. To achieve this, data augmentation usually corrupts original data structures to obtain multi-view information by random perturbation, including node dropping, edge perturbation, feature masking, etc. However, random perturbation might destroy the intrinsic structure of the data. For example, either the removal or addition of edges might drastically change the identity or even the compound validity of bio-molecule data [You et al., 2020]. Thus, data augmentation should preserve the intrinsic structures of the data, which can help learnt embedding features insensitive to the perturbation on unimportant nodes and edges. Additionally, previous data augmentation is independent on contrastive learning. If these two processes are integrated in the same framework, each of them can be alternatively adjusted by the other. In this way, the weakness of data augmentation can further be improved by contrastive learning, while the updated data augmentation can adjust contrastive learning. However, few literature focused on this.

In this paper, we develop a novel adaptive data augmentation method to dynamically preserve the intrinsic structures of the data in this section and then integrates it with multiview contrastive learning in a unified framework (Section 2.4). More specifically, the adaptive data augmentation includes two parts, i.e., node feature augmentation and graph structure augmentation, while multi-view contrastive learning is used to explore complementary information between the feature and topology graphs.

Node Feature Augmentation

The method of randomly masking node features recovers masked node features using their remaining features, and is a widely used method of data augmentation [Qiu et al., 2020]. However, it might destroy the intrinsic structure of the data, thereby a�ecting the generation capability of the model. Hence, an e�ective data augmentation method is demand.

In traditional machine learning methods, feature selection is widely used to remove redundant features. For example, [Nie et al., 2010] employed a joint '2;1-norm regularization to select important features while [Wang et al., 2020b] utilized sparse subspace learning to learn feature weights. Motivated by this, we propose to design a new data augmentation method to keep significant feature unchanged while perturbing trivial features. Specifically, we propose to learn the significance of features and mask trivial features. To achieve this, we first define the following objective function:

where P ∈ R d×d is the weight matrix of features. We focus on corrupting the original feature graph at its feature levels. Specifically, we define an indicator vector m ∈ {0; 1} d×1 to assign zeros to the features with the least weights in the weight matrix P. As a result, the generated new node representation eX by the proposed node feature generation is:

where ◦ is the element-wise multiplication.

Graph Structure Augmentation

Edge perturbation perturbs the connectivity of the graph by randomly adding or dropping certain ratio of edges. However, deleting useful edges will reduce the discriminative ability of embedding features, and thus it has been proven to be unfriendly to biological and chemical data [You et al., 2020]. Additionally, the graph structure is invariant during the process of data augmentation. However, the feature graph S is constructed by the original representations X, which might contain noise or redundancy to a�ect the quality of S. Eq. (1) is used to output better representation than X, while the new representation can be used to further update S, so that the updated S can really reflect the intrinsic structure of the data. Graph learning has been designed to dynamically adjust the graph structure, thereby learning the intrinsic structure of the data. For example, [Zhou et al., 2020] designed to use dynamic graph learning for unsupervised feature selection while [Chen et al., 2019] proposed to embed representation learning with graph learning in a unified framework. Inspired by this, we design a graph learning method to highlight the intrinsic structures of the data as follows.

where 1 represents the all-one-element vector and � is a tuning parameter. In this work, Eq. (1) and Eq. (3) update the weight matrix P and the graph S, respectively. Hence, we integrate Eq. (1) and Eq. (3) in a unified framework as follows:

where �1 and �2 are tuning parameters and P is a trainable parameter. Based on [Jiang et al., 2019], we update S by:

where ReLU(:) is the activation function and q ∈ R h×1 is a trainable parameter.

Given the new representation eX and feature graph S, multiple-view information (i.e., e G1 = (eX;A), G2 = (X; S) and e G2 = (eX; S)) is generated by our proposed adaptive data augmentation method. Di�erent from previous data augmentation, our method iteratively updates both S and eX, thereby avoiding the drawbacks of previous methods, e.g., sensitive to perturbation on trivial nodes and edges, and destroying the structures of the data.

2.2 GCN Encoder

Given multi-view information (i.e., G1, G2, e G1 and e G2), two GCN encoders are utilized to generate multi-view embedding features as di�erent graphs might provide complementary information to each other [Kang et al., 2020]. For example, the topology graph A contains the similarity from the real world while the feature graph S gathers the similarity from the feature space. Specifically, given the topology graph A, we input both G1 and e G1 into GCN1 to output their corresponding embedding features Zt and eZt, aiming at simultaneously updating the new representation eX and the embedding features. In particular, GCN1 is used for both G1 and e G1 as they share the same topology graph, with which it is easy to output a high quality eX. Similarly, we input both G2 and e G2 into GCN2 to output embedding features Zf and eZf , as well as to update both eX and S.

where ˆA = DA - 1 2 (A + I)DA - 1 2 , ˆS = DS - 1 2 (S + I)DS - 1 2 , DA and DS are the degree matrix of A + I and S + I, respectively, MLP(:) represents the Multi-Layer Perception.

2.3 Multi-viewContrastive Learning

The key idea of contrastive learning is to define semantically similar (positive) and dissimilar (negative) pairs, encouraging the embedding features of similar pairs (x; x + ) to be close, and those of dissimilar pairs (x; x - ) to be far away. Specifically, the contrastive learning is to achieve the following objective:

where f (:) represents encoder and sim(:) measures the similarity of embedding features of two nodes. As our proposed adaptive data augmentation generates multi-view information, we design two kinds of contrastive losses (i.e., intragraph contrastive loss and inter-graph contrastive loss) to explore rich contrastive relations among embedding features, i.e., Zf , Zt, eZf , and eZt.

To do this, we employ two GCNs to generate multi-view embedding features. A intra-graph contrastive loss for the embedding features derived from the same encoder is then built, aiming at separating positive embeddings from negative embeddings within the same graph structure. Meanwhile, an inter-graph contrastive loss for the embedding features from di�erent graph structures can also be built, aiming to separate positive embeddings from negative embeddings across graphs.

Inter-graph Contrastive Learning

The inter-graph contrastive learning is to contrast the embedding features from di�erent graph structures, i.e., the topology graph and the feature graph. Specifically, an inter-graph contrastive loss is designed to pull the embedding features of the same node from di�erent views across graph structures close while pushing the embedding features of one node far away from the embedding features of other nodes from different views across graph structures, i.e.,

where vi represents the i-th node, sim(a; b) = exp( a T b ‖a‖·‖b‖ )=#, # represents the temperature factor that controls the concentration level of the distribution. Besides, z vi f ∈ Zf , z vi t ∈ Zt, ez vi f ∈ Zf ,ez vi t ∈ Zt. Combining the above two losses, the intergraph contrastive loss is defined as follows:

Intra-graph Contrastive Learning

Intra-graph contrastive loss is used to build the contrast from the same graph structure, which is popular in CL-UGRL [Xie et al., 2021]. In this paper, the intra-graph contrastive loss is designed to pull the embedding features of the same node from di�erent views in the same graph structure close while pushing the embedding features of one node far away from the embedding features of other nodes from both the same view and di�erent views, which come from the same graph structure, i.e.,

However, the node shares the same graph structure but is with di�erent views, which might cause the embedding features in di�erent views similar by Eq. (12). To make the embedding features of the same node in di�erent views distinguishable, we consider adding the diversity constraints into the intra-graph contrasitive loss.

where � = P N j=1 exp(sim(z vi f ; z vj f )). Similarly, we define L 2 v (vi) between G2 and e G2 as:

Combining Eq. (13) with Eq. (14), the intra-graph contrastive loss Lv is:

Finally, combining the inter-graph loss with the intra-graph loss, the objective function of the proposed multi-view contrastive learning is defined as follows.

where � is a non-negative tuning parameter.

The intra-graph contrasitive loss is defined in di�erent views of the same graph structure, while the inter-graph contrastive losses is defined in di�erent views across di�erent graph structures. Hence, Eq. (16) considers the individual graph structures (e.g., the feature graph or the topology graph) by inter-graph contrastive losses as well as the connection between two graphs by inter-graph losses, so that it easily obtains complementary information among di�erent graph structures.

2.4 Overall Objective Function

In order to train an end-to-end CL-UGRL model, we jointly consider multi-view contrastive learning and adaptive data augmentation to have our final objective function as follows.

where � is a tuning parameter. As a result, adaptive data augmentation is updated by multi-view contrastive learning and vice versa. Eq. (17) can be optimized by the standard gradient descent algorithm. We list the pseudo of our method in Appendix. In the downstream task, the mean function is employed to aggregate embedding features outputted by GCN1 and GCN2, i.e., Z = (Mean(Zt; eZt)||Mean(Zf ; eZf )), where || denotes the concatenation operator.

3 Experiments

Data Sets

The used data sets include citation networks data (i.e., Citeseer, Cora), reference network data (i.e., Wiki-CS), and networks data (i.e., Computers).

Comparison Methods

The comparison methods include unsupervised learning methods (i.e., Deep Graph Infomax (DGI) [Veliˇckovi´c et al., 2018], Graphical Mutual Information Maximization (GMI) [Peng et al., 2020], Multi-View Graph Representation Learning (MVGRL) [Hassani and Khasahmadi, 2020] and Deep Graph Contrastive Representation Learning (DGCRL) [Zhu et al., 2020]) and supervised learning methods (i.e., GCN [Kipf and Welling, 2016] and Graph Attention Networks (GAT) [Veliˇckovi´c et al., 2017]).

3.1 Result Analysis

We report the results of node classification of all methods in Table 1. Our method achieves the best performance, followed by MVGRL, DGCRL, GMI and DGI, in terms of unsupervised representation learning methods. Specifically, our method improves by on average 1.02%, 2.28%, 2.95%, and 3.17%, compared to four unsupervised comparison methods, in terms of classification accuracy, on four data sets. Compared to semi-supervised methods (i.e., GCN and GAT) which adopt the label information in the learning process, our method also achieve the superior performance. This demonstrates that our method can output representations that are beneficial for downstream learning tasks. Moreover, it is feasible to generate auxiliary information to mine the information hidden in the graph data through contrastive learning. The reasons is that our adaptive data augmentation method produces important information of the graph data from both the structure and feature levels, while all unsupervised comparison methods adopt the random perturbation method to generate augmented data. Moreover, our method makes full use of the topology and feature information of the graph data.

3.2 Ablation Study

E�ectiveness of Aata Augmentation

To verify the e�ectiveness of our data augmentation method, we propose three newmethods, i.e., Proposed-R-F, ProposedR-T and Proposed-R-DA, based on our method. Specifically, Proposed-R-F removes the first term of Eq. (4), i.e., without considering the adaptive data augmentation of node features. Proposed-R-T removes the second and third terms of Eq. (4), i.e., without considering the adaptive data augmentation of the graph structure. Proposed-R-DA employs random perturbation to conduct data augmentation, i.e., removing the adaptive augmentation module from our method.

We report the results of four methods in Table 2. First, our method improves by on average 1.92%, compared to Proposed-R-DAmethod, on all data sets. This illustrates that our adaptive augmentation method is e�ective in preserving the intrinsic structure and significant features of the data. Second, Proposed-R-F and Proposed-R-T improve by on average 1.03% and 0.93%, respectively, compared to Proposed- R-DA, on four data sets. This contributes to the fact that both the node feature and the topology structure are important for the UGRL. This clearly demonstrates the feasibility of the adaptive augmentation of either the node feature or the topology structure.

E�ectiveness of Contrastive Losses

To validate the e�ectiveness of contrastive losses, we conduct experiments to compare our method with its two variants, i.e., Proposed w/o n and Proposed w/o v. Specifically, Proposed w/o n denotes our method in Eq. (4) with only Lv, i.e., removing inter-graph contrastive loss Ln in Eq. (4). Proposed w/o v denotes our method in Eq. (4) with only Ln. The results are presented in Table 3. The performance of our method degrades without one of them on all data sets. This demonstrates the e�ectiveness of our contrastive losses. Specifically, our method improves by on average 1.24% and 1.42%, respectively, compared to Proposed w/o n and Proposed w/o v, in terms of node classification. This improvement can be attributed to our multi-view contrastive learning scheme. It also indicates that it is feasible to extract the feature structure and the topology structure of the data.

3.3 Parameters Sensitivity Analysis

We evaluate the e�ectiveness of our method at di�erent parameters' settings, i.e., �1 and �2 in Eq. (4) and summarize the classification results of our method at di�erent settings (e.g., �1 and �2 ∈ {10 -2 ; 10 -1 ; :::; 10 2 }) in Figure 2. Obviously, our method is not sensitive to the setting of either �1 or �2 as di�erent settings make little changes on the classification accuracy, e.g., ≤ 1% in our experiments. This shows that our method easily achieves good performance by setting the ranges of �1 and �2 as [0.001,0.1].

4 Conclusion

In this paper, we designed a novel CL-UGRL method to embed data augmentation and multi-view contrastive learning in a unified framework. To do this, this paper first proposed an adaptive data augmentation method to maintain the intrinsic structure of the data, and then designed contrastive losses to explore complementary information among two di�erent graph structures. As a result, data augmentation and multiview contrastive learning are iteratively adjusted to preserve the intrinsic of the data as well as extract complementary information. Experimental results on four real data sets demonstrated the e�ectiveness of our method, compared to state-ofthe-art comparison methods.

Acknowledgments

This work was partially supported by the Natural Science Foundation of China (Grants No: 61876046); the Guangxi "Bagui" Teams for Innovation and Research; the Sichuan Science and Technology Program (No. 2019YFG0535), and the China Scholarship Council (CSC).