Proximity Enhanced Graph Neural Networks with Channel Contrast

1 Introduction

Graph neural networks (GNNs) have emerged as a powerful tool for many graph analytic problems such as node classification, graph classification and link prediction [Kipf and Welling, 2017; Ying et al., 2018; Zhang and Chen, 2018]. Most of these tasks are semi-supervised, requiring a certain number of labels to guide the learning process. In reality, the label information is sometimes difficult to obtain, or may not conform to the overall distribution of the data, resulting in inaccurate models. Recently, self-supervised learning (SSL) on graphs, which aims to construct self-supervision signals from the input data itself without using any external labels, has gained increasing interest in the community due to its strong performance on various downstream tasks.

∗Corresponding author.

Most GNNs [Kipf and Welling, 2017; Veliˇckovi´c et al., 2018] depend on the assumption that similar nodes are likely to be connected and belong to the same class. Through local feature smoothing by neighborhood aggregation, these models tend to generate node embeddings that preserve the proximity of nodes from the original graph. Such an approach, however, faces challenges in some real-world networks, where neighbor relationship does not necessarily mean similarity [Ma et al., 2021], and sometimes truly similar nodes are far apart. For example, in social networks, the unique attributes of celebrities would be diluted if aggregated with their followers. To enhance their unique attributes, the nodes need to aggregate messages from those with similar attributes, regardless of distance. This requires the GNNmodel to preserve the proximity of nodes in the feature space instead of in the original graph. Two nodes' proximity in the feature space can be reflected by the similarity between their own features, termed self-features, or by the similarity between the features of their neighborhoods. The latter form is well known in matchmaking agency [Chen et al., 2021] who aims to connect two people on a blind date, based on a careful assessment of similarities of their close friends, rather than directly comparing features between the opposite genders.

In still other cases, it is found that structural similarity plays a particularly important role for graph learning. For example, for catalysts in the protein-protein interaction network of a cell [Ribeiro et al., 2017], nodes with similar structural features are defined to be similar, establishing a notion of proximity in the topology space.

With supervision signals or prior knowledge, most GNNs use only specific types of proximity information. For example, AM-GCN [Wang et al., 2020] tries to leverage proximity in the feature space and the original graph; Struc2vec [Ribeiro et al., 2017] and RolX [Henderson et al., 2012] preserve proximity in topology space, while ignoring proximity information in other forms.

In this paper, we argue that the three measures of proximity, namely proximity in the original graph, in the feature space, and in the topology space, provide complementary views of the graph, and thus combining all of them in a proper way can significantly improve the robustness and adaptability of GNNs. By exploiting the inherent consistency between different graph views, we are able to learn useful embeddings without label information. Our method follows the self-supervised contrastive learning (CL) approach [Chen et al., 2020]. By contrasting positive pairs against negative pairs in different views of the graph, CL has demonstrated clear advantages on unsupervised representation learning [Veliˇckovi´c et al., 2019; Hassani and Khasahmadi, 2020] and pre-training [You et al., 2020] tasks on graphs.

Our Contributions We present Proximity-Enhanced Graph Contrastive Learning (PE-GCL), a novel selfsupervised graph representation learning framework. First, we reconstruct the graph in the feature space and topology space to generate three kinds of graph views, in which similar nodes in terms of self-features, neighborhood features or local structures are linked. In the contrastive stage, we adopt an alternate training strategy to accelerate the training process, where our model alternately maximizes an agreement between the representations of a generated view and the original graph, with positive and negative pairs at the channel level instead of the node level. Compared with the state-of-the-art methods [Zhu et al., 2020a; Zhu et al., 2020b], which requires computation complexity O(N 2 ), where N is the number of nodes, PE-GCL reduces the number of contrastive pairs to O(d2 ), where d is the output dimension.

Our theoretical analysis from the perspective of maximizing mutual information confirms its rationality. We conduct experiments node classification task on a total of 9 realworld benchmark datasets including 6 assortative graphs and 3 disassortative graphs, and demonstrate that PE-GCL outperform representative unsupervised graph learning methods and sometimes defeat semi-supervised methods.

2 RelatedWork

Self-supervised learning on graphs aims to construct supervision signals from the graph itself without external labels. Earlier methods based on shallow neural networks construct supervision signals from graph structures to enforce the representations of nodes in the same local context to be similar, where local context can be random walk sequences [Perozzi et al., 2014; Grover and Leskovec, 2016], community members or specific order neighbors [Tang et al., 2015]. With the success of graph neural networks, some methods use multilayer graph autoencoder to learn to reconstruct certain parts of the graph, where the parts of the graph can be the adjacency matrix [Kipf and Welling, 2016], node features [Park et al., 2019] or both nodes and edges [Hu et al., 2020]. Recently, Contrastive learning (CL) has been successfully applied in graph representation learning. The key idea is to contrast views by maximizing agreement between positive pair and disagreement between negative pairs, where contrastive pairs can be subgraph-graph pairs [You et al., 2020], node-graph pairs [Veliˇckovi´c et al., 2019; Peng et al., 2020] or cross-view identical node pairs [Zhu et al., 2020a; Zhu et al., 2020b].

Proximity preserving is a means for graph learning models to retain and learn from the similarity and relation between nodes. Based on different similarity measures, the models may be classified into three categories: 1) models that preserve proximity in the graph structure. Network embedding methods such as DeepWalk [Perozzi et al., 2014], LINE [Tang et al., 2015], and node2vec [Grover and Leskovec, 2016] follow this approach by maximizing the co-occurrence probability of nodes and their neighborhoods. Message passing GNNs [Veliˇckovi´c et al., 2018; Hamilton et al., 2017] realize this by local feature smoothing; 2) models that preserve proximity in feature space, where nodes with similar features are brought into proximity. To this end, AM-GCN [Wang et al., 2020] constructs a kNN graph based on the feature matrix, and then input the generated kNN graph together with the original graph to jointly train the GNN model; 3) models that preserve proximity in topology space, where nodes with similar local structures become close to each other. Struc2vec [Ribeiro et al., 2017] uses a hierarchy to capture structural similarity at different scales. RolX [Henderson et al., 2012] recovers a soft-clustering of nodes into a specific number of distinct roles using recursive structural feature extraction.

3 Preliminaries

Notation Let G = (V,E,X) denotes an unweighted attributed graph, V = {vi}Ni=1 is a set of nodes with |V | = N, E ⊆ V × V is a set of edges, and X = {xi}Ni=1 is a set of feature vectors, where xi ∈ RF is a F-dimensional vector for node vi. Each column of X is a attribute vector, denoted as X:,k ∈ RN . The adjacency matrix A ∈ RN×N : if eij = (vi, vj) ∈ E, then Aij = 1, otherwise, Aij = 0.

Proximity Graph We reconstruct the graph in both feature and topology spaces to generate augmented views. As shown in Figure 1, the augmentation pool P(·|G, k) consists of three view generators, i.e., Φ(X, k), Ψ(A, k) and Θ(A,X, k). In the feature space, Φ(X, k) aims to build a self-feature proximity graph (FPG) where each node only links to the k nodes with most similar feature vectors, denoted Gf = (Af ,X). Θ(A,X, k) aims to generate a neighborhood feature proximity graph (NPG), where each node links to k most similar nodes in terms of attribute distribution of the feature sets of neighborhoods, denoted Gn = (An,X). In the topology space, Ψ(A, k) builds a topology proximity graph (TPG) where each node links to the top k nodes with most similar local topology, denoted Gt = (At,X). As such, Gf , Gt and Gn are three views of the input graph G. Note that the similarity ranking may be asymmetric so the generated views are directed. In addition, proximity graphs are structural views, therefore the augmentation is only applied to the structure of the graphs rather than the initial node features. Ψ(A, k) means the TPG generator only leverages the structure information A from G, so as other generators.

4 Views Generation

In this section, the instantiation of the generators Φ(X, k), Ψ(A, k) and Θ(A, X, k), is presented.

4.1 Instantiation of Ψ(A, k) for TPG

Figure 1 shows an example where nodes u, v and w have similar local topology: they all have degrees 4, and are connected to 2 triangles, therefore they should be connected in the topology space to construct TPG Gt. To this end, we first encode the structural roles of each node as a vector in the topology space. Although several structural role embedding methods [Ribeiro et al., 2017; Henderson et al., 2012] can do the job, they require separate back-propagation or enumeration operations, which are too heavy as a preprocessing step. Here, we give an alternative method based on graph kernel.

We first extract local subgraphs for all nodes, where the subgraphs can be r-hop egonets (i.e., the induced subgraph surrounding each node) or induced by a set of random walk (RW) sequences starting from each node. We have found that different choices do not affect the results significantly, except that RW which we use here is more efficient for dense graphs. Let S = {S1, · · · , SN} be the set of extracted local subgraphs for each node, we adopt the Weisfeiler-Lehman (WL) subtree kernel [Shervashidze et al., 2011] on the subgraph set S. Then, we can obtain a symmetric positive semi-definite kernel matrix K ∈ RN×N , where Kij denotes the similarity between local subgraphs Si and Sj , which can be seen as the local topological similarity between vi and vj. Then, we can directly factorize K to obtain a low-dimensional representation of each subgraph Si, and this vector is the local topological representation of vi. However, matrix factorization leads to a high computational complexity of O(N 3 ). To solve this problem, we use Nystr¨om approximation [Nikolentzos et al., 2018] to reduce the cost to O(m2N), where m ≪ N. The Nystr¨om method only uses a small subset of m columns of K, such that K ≈ RR⊤ where R ∈ RN×m. Finally, each row of R represents the coordinate of the corresponding node in the graph's topology space, and therefore serves as the local topology representation of the node. Figure 2 shows an experiment on a barbell graph B(6, 2). Based on R, we construct a directed graph where each node links to top-k most similar nodes using outgoing edges, denoted Gt = (At,X). Note that R is only used to generate the structure At of Gt, and node features are still X.

4.2 Instantiation of Φ(X, k) for FPG

Φ(X, k) is defined in the feature space. It generates a selffeature proximity graph (FPG) Gf = (Af , X) directly based on node features X, where each node only links to the k most similar nodes in terms of node features. Figure 1 shows nodes with similar colors (features) are connected to build Gf .

4.3 Instantiation of Θ(A,X, k) for NPG

To capture more complex graph properties which rely on both local topology and node features, we introduce the concept of neighborhood feature proximity graph (NPG): the existence of an edge between two nodes is based on the similarity of their local neighborhoods' features. The rationale behind our idea is that FPG only considers the relationship between individual node features, which is not robust to feature noise. Moreover, in some graphs such as disassortative graphs, harnessing neighborhoods' features while ignoring the self-features can significantly improve the discriminative power of the learned representations [Ma et al., 2021].

Although MPNN-based GNNs obtain a node representation that encodes a rooted subtree around the center node by iteratively aggregating neighborhoods, such a representation does not well reflect the characteristics of a node's neighbors, since it is a mixture of neighbors' features without considering the distribution of each attribute. Hence, Θ measures the similarity between nodes by the distance between the attribute distributions of the feature sets of the nodes' neighborhoods. Given every node's neighborhoods ˆ S = {ˆ Si}Ni=1 , where ˆ Si = Si/ vi is the neighbor set of vi excluding itself. From each ˆ Si, we sample m′ nodes uniformly with replacement to generate a neighborhood feature matrix Xi ∈ Rm′×F for vi, where the k-th attribute is Xi:,k . Then we assume the attribute is a discrete random variable whose realizations depend on the specific pair-wise nodes. Specifically, for vi and vj, the realizations of the k-th attribute are a set of all different values that appear in Xi:,k and Xj:,k , denoted as ωk. We denote the set of all different values in Xi:,k as ωik , then ωik ∪ ωjk = ωk where |ωik |, |ωjk | ≤ m′ . We can vectorize the attribute distribution of Xi:,k as an |ωik |-dimensional vector πik , where πik (t) is the frequency of an attribute value ωik (t) occurs in Xi:,k , and ∑t πik (t) = ∑t πjk (t) = m′ . Then, we measure the neighborhood feature similarity between vi and vj with Wasserstein distance (WD) between the corresponding attribute distributions as D(vi, vj) = ∑Fk=1 W k (vi, vj), where:

where T ∈ R|ωik |×|ωjk | is a flow matrix. In this way, the smaller value of D(vi, vj) indicates the higher similarity between the attribute distributions of the feature sets of two nodes' neighborhoods. As shown in Figure 1, u, w and r have similar neighborhood feature distributions (3 orange and 1 grey), so they are more likely to be connected. Based on the distance measure D(vi, vj), the generator Θ links top-k most similar nodes for each node to build a neighborhood feature proximity graph (NPG) as an augmented view, denoted as Gn = (An,X). Eq. (1) fits the form of WD, which has been well-studied and fast specialized algorithms [Pele and Werman, 2009] have been proposed to solve it.

Summary. FPG and TPG are kNN graphs that use cosine distance to measure node feature similarity and structural similarity, while NPG is a kNN graph based on Wasserstein distance, as we are interested in distribution similarity instead of vector similarity of neighborhoods. These three views are reconstructions of the original graph structure to preserve the high-order interactions between nodes with respect to selffeatures, local topology and distributions of neighborhoods. Now we decide the augmentation pool as P(·|G, k): feeding G and an integer k into the pool, it outputs a view from one of the generators. In the views generation stage, the computational cost mainly comes from calculating the vector similarity for the kNN graphs, which is O(N 2 ). Fortunately, fast approximation and parallelization methods have been well studied to construct kNN graphs efficiently. We are able to construct kNN graphs with local sensitive hashing [Zhang et al., 2013] in O(ln(F + logN)).

5 Proximity Enhanced GCL

We describe each component of PE-GCL in order as depicted in Figure 1.

(1) Alternate training strategy. In each training epoch, we randomly set a k and generate a view Ga ∼ P(·|G, k), where the generator is alternately set to Φ, Ψ or Θ. In other words, each training epoch involves only one augmented view Ga, which is taken turns to set to FPG (Gf ), TPG (Gt) or NPG (Gn). In this way, multiple types of proximity information can be preserved. As compared to joint training with three kinds of views in one epoch, the alternate training strategy reduces computational cost and accelerates the training process. Moreover, the randomness of k helps improve the diversity of views, so that proximity at different scales can be preserved. This also helps to achieve a trade-off among different types of proximity measures. However, repeatedly constructing kNN graphs in each training step is expensive. To address this issue, we limit k to a maximum value kmax, i.e., k ≤ kmax. We only need to find top-kmax neighbors for each proximity graph before training, which is a single-time effort. Then in each training step, an FPG, NPG or TPG with k can be easily obtained from the preprocessed kmax neighbors by masking a certain number of neighbors at the end of the similarity ranking. In practice, kmax is a hyperparameter and kmax < 10 is found to work well.

(2) Shared GNN encoder and projection head. The view Ga ∼ P(·|G, k) together with G are fed into a shared GNN encoder, where each node aggregates the messages propagated from its in-neighbors. We use GAT as the encoder; other MPNN-based GNNs can also be used without much difference in performance. Following the GNN encoder, we use a shared MLP g(·) as a projection head [You et al., 2020] to enhance the expressive power of the output representations. All trainable parameters are shared between two graphs during training.

(3) Channel-level contrastive objective. Unlike most previous methods that construct contrastive pairs between augmented views, PE-GCL constructs contrastive pairs between the original graph and a view. Specifically, the shared MLP outputs two representation matrices H,Ha ∈ RN×d for G and its augmented view Ga respectively, which can be seen as two signals with d channels on two graphs. Some works [Zhu et al., 2020b; Zhu et al., 2020a] focuses on node-level contrasting, which suffers from high computation cost. We offer an alternative method that generates contrastive pairs at the channel level. As shown in the right part of Figure 1, the i-th channels of two output signals, denoted ci and cai , are the ith columns of H and Ha, respectively. Then the contrastive objective aims to maximize the consistency between two representation matrices, such that the same channels ci and cai as inter-graph positive pairs (red solid double-headed arrows) are to be pulled together, and different channels ci and caj,j̸=i as negative pairs (gray dashed double-headed arrow) to be pushed away. Hence, the loss function of pair-wise channels (ci, cai ) is defined as:

where τ is the temperature hyper-parameter to scale the cosine similarity ϕ(·, ·). [Chen et al., 2020] finds that an appropriate τ can help the model learn from hard negatives. Considering all d channels, the total contrastive loss function is:

Prediction. The GNN encoder and MLP are all trained in an unsupervised manner. After the training stage, the original graph G is fed into the GNN encoder to generate the resulting node embeddings Z, as shown in Figure 1. Then we can apply Z to downstream prediction. See Appendix A for a description of the algorithm.

5.1 Theoretical Analysis

We create three graphs Gf , Gt and Gn as augmented views of G, and then maximize the agreement between the output embedding matrices of G and a view. By this means, the GNN encoder is able to capture richer semantic information (i.e., self-feature similarity, neighborhood feature similarity and structural equivalence), which may boost the subsequent prediction tasks. In other words, minimizing the contrastive

objective in Eq. (3) aims to maximize the "correlation" between the input graph G and its proximity graph Gf , Gt or Gn. However, what does "correlation" precisely mean? In the following, we theoretically discuss the connection between the loss function of PE-GCL and InfoNCE [Kong et al., 2019] to answer this question, and prove that minimizing our proposed channel-level contrastive objective can be viewed as a kind of maximizing a lower bound of the Mutual Information (MI) between G and its views.

Let fθ(c, ca ) = ϕ(g(GNN(G)), g(GNN(Ga)))/τ be the contrastive loss of a specific pair-wise channels c and ca , where θ is parametrized by τ, the GNN encoder GNN(·) and the MLP g(·), we can arrive at the following theorem.

Theorem 1. The contrastive loss in Eq. (3) can be transformed to the InfoNCE form and lower bounded by the MI between H and Ha as:

The proof is in Appendix B. Thus, minimizing Lcont is equivalent to maximizing a lower bound of MI between H and Ha.

6 Experiments and Analysis

6.1 Datasets, Evaluation and Baselines

Datasets. We conduct node classification experiments on the following commonly used six assortative graphs (Cora, CiteSeer, PubMed, WikiCS, ACM and CoauthorCS) [Kipf and Welling, 2017; Mernyei and Cangea, 2020; Sinha et al., 2015]

and three disassortative graphs (Texas, Chameleon and Actor) [Pei et al., 2019]. The statistics of datasets are in Table 1.

Evaluation Protocol. For assortative graphs, we use ten splits of the nodes into 10%/10%/80% for train/validation/test nodes over 10 random seeds, and run 10 times for each split to report average accuracy with standard deviation. For disassortative graphs, we use the same setting as [Pei et al., 2019]. The model is firstly trained in an unsupervised manner. In the test stage, the original graph is directly fed into the GNN encoder and output a node representation matrix Z ∈ RN×d′ , then the embeddings of the training set are used to train an ℓ2-regularized logistic regression classifier and give the results of classification on the test nodes.

Baselines. PE-GCL 1 is compared with three categories of unsupervised methods: (1) Random walk based methods, including DeepWalk [Perozzi et al., 2014], node2vec [Grover and Leskovec, 2016]. (2) Auto-encoder (AE) based methods, including GAE and VGAE [Kipf and Welling, 2016]. (3) Contrastive learning (CL) based methods, including MVGRL [Hassani and Khasahmadi, 2020], DGI [Veliˇckovi´c et al., 2019], GRACE [Zhu et al., 2020a], GMI [Peng et al., 2020], GraphCL [You et al., 2020]. We also compare PE-GCL with semi-supervised GNNs, including GCN [Kipf and Welling, 2017], GAT [Veliˇckovi´c et al., 2018], AMGCN [Wang et al., 2020], and DMGI [Park et al., 2020].

6.2 Results and Analysis

Overall performance. From Table 2 we can make several observations. (1) Our PE-GCL achieves the best results on five out of the six datasets. Only on CiteSeer dataset, PEGCL is slightly weaker than MVGRL. Considering that MVGRL needs 4 GNN encoders at the same time per epoch, this method suffers from high parameter complexity. By contrast, every component is shared across graphs in PEGCL, so only a single GNN encoder is needed. (2) Although without labels to guide training, PE-GCL still outperforms most semi-supervised methods on most datasets. (3)

1 The source code, hyper-parameter settings and complexity analysis are available at https://github.com/JhuoW/PE-GCL

GRACE and GraphCL both construct augmented views by imposing random perturbations on the graph, and MVGRL constructs views with diffusion. These methods and our PEGCL achieve comparable or even better results than semisupervised methods. It shows the superiority of CL models on graph learning tasks. (4) Traditional network embedding methods like DeepWalk and node2vec with negative sampling can also be seen as a kind of CL-based method, yet they do not perform well, because node features are ignored, and the number of negative samples is usually very small.

Table 3 shows that PE-GCL consistently yields the best performance on disassortative graphs, where most connected nodes belong to different classes. It demonstrates that our augmented views can help the model capture long-distance correlations between nodes.

Gain fromviews. We propose FPG, TPG and NPG as augmented views to help representations integrate high-order similarity information. However, it is not intuitive howmuch gain is generated by each type of view. To answer this question, we introduce three PE-GCL variants, each of which has one type of view removed. We choose Cora, Texas and Chameleon to conduct the ablation experiments. The results are shown in Table 4. We see that considering FPT and NPG can improve the model accuracy by 1.09%, 1.17% on Cora, while the gain from TPG is small. For Texas, FPG and TPG is important, while NPG may bring about some negative effects. However, for Chameleon, NPG plays a non-negligible role. Thus, our PE-GCL combines these proximity graphs that help achieve balanced gains for general graphs.

Transferrability. We also show that our proposed three types of views and alternate training strategy can be transferred to other CL-based GNNs. For example, we replace the graph augmentation scheme of GRACE which uses random corruption with proximity graphs, and adopt alternate training strategy, called PE-GRACE. The objective function is still NT-xent [Zhu et al., 2020a] used in the original GRACE. We compare the accuracy of GRACE and PE-GRACE to verify the proximity graphs' adaptability, see Figure 3(a). The results show that proximity graphs can also benefit GRACE and perform better than using random corruption.

Sensitivity analysis. Figure 3(b) shows the impact of hidden layer sizes of the GNN encoder and the MLP on the classification results of PE-GCL on Cora. With the increase of dimensionalities, the performance increases first and starts to stabilize when the dimensionalities reaches 256.

6.3 Link Prediction

For link prediction, we conduct experiments on Cora and CiteSeer, removing 5% existing edges and the same-size nonexisting edges for validation, 10% existing edges and the same-size non-existing edges for testing. We compute the predicted adjacency matrix by Apred = Sigmoid(ZZ ⊤) and report the AUC and AP in Table 5. The PE-GCL achieve the highest prediction scores among baselines indicating that PE-GCL has better reasoning ability for network connectivity and can better retain similarity information between nodes.

7 Conclusion

In this work, we propose PE-GCL, a self-supervised representation learning framework based on contrastive learning. PE-GCL aims to preserve multiple types of proximity information based on different definitions of node similarity. In PE-GCL, we propose proximity-based graph views based on self-feature similarity, local topology similarity and neighborhood feature similarity. Then we use a contrastive objective to maximize the agreement of the views and the original graph. In particular, we construct contrastive pairs on a channel level, by which we can greatly reduce the number of negative pairs. Experimental results show that PE-GCL is competitive against state-of-the-art methods.

A The Pseudocode of PE-GCL

The details of PE-GCL are described in Algorithm 1.

B Proof of Theorem 1

Proof. Given two view sets A and B, both with the same cardinality n, there exists a bijection π : A → B such that each view of one set is positively paired with exactly one view of the other set. Any other inter-set pairs are negative. Under

Algorithm 1 PE-GCL

• 1: Input: GraphG = (V,E,X); Upper limit of the number of neighbors kmax; Number of iterations T ; Temperature hyper-parameter τ; Dimension of the projection head d; Dimension of the GNN encoder d′
• 2: ̂ Af ← Φ(X, kmax); ̂ At ← Ψ(A, kmax); ̂ An ← Θ(A, X, kmax)
• 3: for t = 1, 2, · · · , T do
• 4: Random sample k ≤ kmax
• 5: Ω = Alternately sample a generator from {Φ,Ψ,Θ}
• 6: Aa =Mask(Ω(A, kmax), kmax -k) ▷ Mask the kmax - k least similar neighbors for each node
• 7: Ga = (Aa,X)
• 8: Z ←GNN(G), Za ←GNN(Ga)
• 9: H ← g(Z), Ha ← g(Za)
• 10: Lcont ←Eq.(3)
• 11: end for

this condition, InfoNCE which is upper bounded by MI, can be defined as:

where Q = ((a1, b1), · · · , (an, bn)), in which each pair as an element in Q represents a positive pair, fθ(·) ∈ R is a function parameterized by θ, P(Q) is the distribution of positive pairs, which is uniform. Then our contrastive objective Eq. (3) can be rewritten as the expectation form:

where Ha = g(GNN(Ga)), dimension d is a constant. To fit Eq. (4), we set A = H, B = Ha, then (ci, cai ) is a realization of the random variable Q. Here we consider H and Ha as two view sets, and each column of H and Ha, i.e., ci and cai are regarded as the views they are in. Thus, Q = ((c1, cai ), · · · , (cN, caN )), where (ci, cai ) is a positive pair. ci and caj form a negative pair, where i ̸= j. Hence, P(ci, cai ) = P(Q) is a discrete uniform distribution of size d, so P(ci, cai ) = P(cai ) = P(ci). Let fθ(c, ca ) = ϕ(g(GNN(G)), g(GNN(Ga)))/τ, where θ parametrized by τ, the GNN encoder and the MLP g(·). Then Eq. (5) can be transform to the InfoNCE form in Eq. (4) as:

here we omit the constant 2 log d, which concludes the proof.

Acknowledgements

This work is supported in part by Shenzhen Baisc Research Fund under grant JCYJ20200109142217397.