Universal Graph Self-Contrastive Learning

1 Introduction

Graph Self-Supervised Learning (GSSL) has made significant advancements in learning discriminative representations without relying on expensive hand-labeled nodes (samples) [Hendrycks et al., 2019]. As a representative architecture of GSSL, Graph Contrastive Learning (GCL) aims to maximize the agreement between the representations of two augmented views from the same input while minimizing the agreement between the representations from different augmented views. It has achieved remarkable performance on various node-level downstream tasks, such as node classification [Ma et al., 2023]. Depending on the different information contained in contrastive objects, existing GCLs for node-level tasks can be categorized into: 1) local-global contrast [Velickovic et al., 2019; Hassani and Khasahmadi, 2020] and 2) local-local contrast [Zhu et al., 2020b; Thakoor et al., 2021; Zhuo et al., 2024a; Zhuo et al., 2024c]. Given that the fundamental objective is to contrast nodes, these two categories of GCLs can be collectively termed as Node-Node Contrast.

∗Corresponding Author

Unfortunately, existing node-node graph contrastive models still face critical issues in the graph augmentation and positive/negative sampling due to the lack of supervision information. First, graph augmentation not only faces the challenge of maintaining the semantic integrity of the graph during the augmentation process but also inevitably increases computational costs [Thakoor et al., 2022; Trivedi et al., 2022]. Despite some methods successfully working without graph augmentations by selecting positive samples from neighboring nodes [Zhang et al., 2022; Xiao et al., 2022], challenges persist in accurately identifying TRUE positive and negative samples for nodes. On the one hand, positive samples, whose representations will be pulled together, can not be easily identified. Neighborhoods are from the same classes with a high probability on homophilic graphs, and thus can be treated as positive samples. However, this is not feasible on heterophilic graphs. On the other hand, it is more difficult to locally identify negative samples. Negative samples should be the nodes in the different classes. It is commonly accepted that nodes with long-distance may belong to different classes. Therefore, this compromises the model's universality to graphs with diverse structural characteristics.

To solve the aforementioned drawbacks, this paper seeks to introduce a universal augmentation-free GCL framework. Motivated by the Self-contrastive Learning [Bae et al., 2023], which leverages a multi-exit architecture to generate multiple representations from a single image without relying on data augmentation, this paper aims to explore selfcontrastive learning on graphs. To this end, a novel Graph Self-contrastive Learning framework named GRAph learning via Self-contraSt (GRASS) is proposed. It enables selfcontrast between nodes and attributes, thereby eliminating the reliance on augmentation and circumventing the challenges associated with sample selection. GRASS first treats attributes as another type of node, i.e., attribute nodes, and generates initial representations for each type of node separately. The representations of nodes and attribute nodes are respectively derived from the adjacency matrix, which describes the link relationships between nodes, and the Positive Pointwise Mutual Information (PPMI) matrix, which captures the cooccurrence relationships of attributes. Then, it identifies positive pairs via inclusion relationships and negative pairs via non-inclusion relationships in the attribute matrix for representation learning. In theory, the framework can implicitly capture the node-node co-inclusion relationship (a high-hop relationship), which guarantees its universality.

The main contributions of this paper are as follows:

• We investigate the challenges faced by existing Graph Contrastive Learning (GCLs) in terms of graph augmentation and positive/negative sample selection.
• We introduce a novel GCL framework named GRAph learning via Self-contraSt (GRASS), with a nodeattribute self-contrastive mechanism.
• We provide a theoretical analysis to justify the effectiveness and universality of GRASS.
• We conduct extensive experiments on twelve wellknown benchmark datasets with various homophily degrees to demonstrate the performance of GRASS.

2 RelatedWorks

Contrastive Learning. Contrastive Learning (CL) has become highly popular in self-supervised visual representation learning, aiming to learn discriminative representations by contrasting positive and negative sample pairs. Specifically, SimCLR [Chen et al., 2020] maximizes the consistency between representations of different augmented views of the same image; BYOL [Grill et al., 2020] removes negative pairs, using an online network to predict the representations produced by a target network. More recently, CMSCGC [Guan et al., 2024] has constructed two views by extracting the pixel neighborhood texture information and spatial-spectral information from hyperspectral images.

Augmentation-based Graph Contrastive Learning. CL has also been successful on graphs. GRACE [Zhu et al., 2020b] generates augmented graphs by randomly dropping edges or nodes and maximizes the consistency of the node representations from the two views; GCA [Zhu et al., 2021] uses an adaptive augmentation method to perturb the unimportant information; MVGRL [Hassani and Khasahmadi, 2020] generates the multiple views via graph diffusion and learns both node-level and graph-level representations. GREET [Liu et al., 2023] introduces a discriminator to assess the homophily of edges and implements random augmentation. AGCL [Yu and Jia, 2024] generates multiple views via graph topology and graph diffusion, capturing local and global information. GOUDA [Zhuo et al., 2024b] makes utilization of augmentation-centric vectors to simulate attribute variations in node neighborhoods, thereby accomplishing a unified augmentation framework.

Augmentation-free Graph Contrastive Learning. Some Graph Contrastive Learning (GCL) models without augmentation are given below. SimGRACE [Xia et al., 2022] takes the original graph as input and Graph Neural Network (GNN) model with its perturbed version as two encoders to obtain two correlated views for contrast. AF-GCL [Wang et al., 2022] leverages the features aggregated by GNN to construct the self-supervision signal instead of augmentations. DSSL [Xiao et al., 2022] imitates a generative process of nodes and links decoupling the potential semantics of different neighborhoods. GCFormer [Chen et al., 2024] develops a new token generator to generate both positive and negative token sequences for each node.

3 Notations and Preliminaries

3.1 Notations

Let G = (V, E) denote a graph with node set V = {v1, ..., vn} and edge set E, where n denotes the number of nodes. The adjacency matrix A = [aij] ∈ {0, 1}n×n represents the graph topology, where aij = 1 if and only if there exists an edge eij = (vi, vj). The degree matrix D is a diagonal matrix with diagonal element di = ∑ni=1 aij as the degree of node vi. X ∈ Rn×f denotes the collections of node attributes with the ith rows, i.e., xi ∈ Rf corresponding to node vi, where f stands for the dimension of the attribute.

3.2 Homophily

Typically, the graph edge homophily ratio [Pei et al., 2020; Zhu et al., 2020a] is defined as the proportion of edges connecting nodes with the same labels. Mathematically, this can be represented by the following equation:

Homophilic graphs have a high edge homophily ratio, i.e., h tends to 1, while heterophilic graphs correspond to a small edge homophily ratio, i.e., h tends to 0. This paper particularly examines graphs with varying degrees of homophily.

3.3 Positive Pointwise Mutual Information

Positive Pointwise Mutual Information (PPMI) is a common tool for measuring the association between two words in computational linguistics [Church and Hanks, 1990]. PPMI between words wi and wj is defined by:

Here, [p]+ denotes max(p, 0). P(wi) is the occurrence probability of the word wi, and P(wi,wj) is the co-occurrence probability of two words. If two words are completely independent, P(wi,wj) = P(wi)P(wj), and the value of PPMI is 0. In contrast, if there is a strong relationship between two words, they will usually co-occur frequently and thus share a high PPMI value, indicating a close association. Given this property, PPMI is employed in our framework to represent the correlations among attributes.

3.4 Contrastive Learning Loss

The key to contrastive learning (CL) is maximizing mutual information (MI) [Tsai et al., 2021]. To this end, an information theory viewpoint is adopted to construct the Graph Contrastive Learning (GCL) objective. Let I(X; Y ) be the MI between random variables X and Y , which is commonly represented by the Donsker-Varadhan variational representation [Donsker and Varadhan, 1983; Polyanskiy and Wu, 2014] as follows:

where IΘ(X, Y ) is the neural information with parameters θ ∈ Θ, which is precisely given by the following detailed and comprehensive definition:

Based on this variational lower bound, the contrastive loss is formulated by directly optimizing IΘ(X, Y ):

4 Methodology

This section first introduces a novel graph contrastive learning (GCL) framework named GRAph learning via Self-contraSt (GRASS). Subsequently, it theoretically analyzes the effectiveness of this framework.

4.1 GRASS

As shown in Figure 1, the architecture of GRASS consists of three key modules: (1) View Construction, which extracts two types of contrastive objects from the input graph. (2) Sampling Matrix Construction, which determines positive samples for self-contrast based on the node-attribute inclusion relationship (represented by a bipartite graph). (3) Graph Selfcontrastive Learning Loss, which guides the feature update process for these two types of objects.

View Construction. As discussed in the Introduction, traditional GCL methods rely on perturbation-based augmentations to construct two views, which may result in expensive training costs and compromised semantic integrity. The proposed view construction module regards nodes and attributes as two intrinsic views of the input graph, thus avoiding the need for perturbation-based augmentations. To be specific, for the input graph G(A,X), this module generates representations for attributes in addition to nodes, represented as Z ∈ Rn×d and B ∈ Rf×d .

The initial node representations are derived through the embedding of the graph topology matrix, while the initial attribute representations are obtained via the embedding of the attribute co-occurrence matrix. The graph topology matrix is selected as the adjacency matrix A. The attribute cooccurrence matrix is constructed through the utilization of Positive Pointwise Mutual Information (PPMI), which is defined as Mi,j = PPMI(X:,i, X:,j ).

After obtaining these matrices, the initial representations are computed by applying Multi-Layer Perceptrons (MLPs) to transform the graph topology matrix and the attribute cooccurrence matrix into feature embeddings, that is:

where MLPA and MLPM stand for two distinct Multi-Layer Perceptron (MLP). In this paper, the attributes are regarded as another type of node, referred to as attribute nodes.

SamplingMatrix Construction. It is commonly assumed that the attributes associated with nodes can accurately reflect the node categories [Lim et al., 2021]. Based on this assumption, this module aims to identify positive samples for nodes and attribute nodes by constructing a node-attribute bipartite graph G ′ using the attribute matrix X.

To ensure symmetry, the adjacency matrix of the bipartite graph, i.e., A′ ∈ R(n+f)×(n+f) , is defined as:

where 0 denotes an all-zero matrix. The adjacency matrix indicates the node-attribute inclusion relationship, which is regarded as the basis for identifying positive samples. Specifically, for a given type of node, the positive samples are identified as the other type of nodes where the corresponding weight in the adjacency matrix is non-zero.

Graph Self-contrastive Learning Loss. In general, contrastive losses aim to amplify feature similarities between positive sample pairs while reducing them between negative sample pairs [Xiao et al., 2022; Zhu et al., 2020b]. In the proposed framework GRASS, connected node pairs are regarded as positive sample pairs, while all unconnected node pairs are regarded as negative sample pairs. Therefore, it is entirely feasible and rational to construct the overall objective function in the following comprehensive and detailed manner:

where cθ(·) represents the similarity between node pairs calculated using a neural network, specifically a single MLP. zv is the vth row of Z, and bu is the uth row of B. PVU denotes the connection probability of positive sample pairs on the bipartite graph, i.e., node pairs (zv, bu) that are connected by an edge, where V is the set of original graph nodes and U is the set of attribute nodes. Meanwhile, h denotes any node, h′ denotes all nodes except h on the bipartite graph. PV ⊗PV denotes the product of marginal distribution of independently sampling two nodes from the bipartite graph node set V = V ∪ U, which covers all node pairs.

4.2 Theoretical Analysis

This subsection aims to provide a theoretical understanding of the proposed GCL framework. The framework can implicitly capture the node-node co-inclusion relationship (a high-hop relationship), which is important for its universality to both homophilic and heterophilic graphs.

Theorem 1. Let N2(v) denote the set of two-hop neighbors of zv on the bipartite graph G ′ . Minimizing the GRASS objective in Equation (8) is approximately minimizing the following alignment loss between two-hop neighbors:

Proof. By removing the second loss term and retaining only the positive sample pair loss, Equation (8) can be formally elaborated as follows:

where zv and bu are one-hop neighbors of each other on the bipartite graph. |V | denotes the number of nodes on the bipartite graph and N(v) denotes the set of neighbors of node zv. Expanding Equation (9), the following equation is subsequently and precisely obtained:

Let Pv = √dvzv be the vth row of the matrix P ∈ R(n+f)×d , and Qu = √dubu be the uth row of the matrix Q ∈ R(n+f)×d . ˜A′ = ˜D′-1/2A′ ˜D′-1/2 , where A′ denotes the adjacency matrix of the bipartite graph G ′ . dv and du denote the degree of zv and bu. Then Equation (10) can be specifically formulated as:

For Equation (11), the inequality is utilized: tr(PQ) ≤ 12 (‖P‖22 + ‖Q‖22 ) for any two matrices P ∈ Rm×k and Q ∈ Rk×t . Then, the property of the trace tr(X⊤X) = ‖X‖22 is used to obtain the following equation:

Equation (12) is extended using the cyclic invariant property of traces to derive the following equation:

Removing the ‖Q⊤‖22 term and expanding the formula for the trace in Equation (13) can be found:

where zu2 is the two-hop neighbor of zv. Expanding the adjacency matrix in Equation (14) obtains:

Combining Equation (15) and Equation (13), the final conclusion can be obtained:

Theorem 1 shows that the optimization objective of GRASS is equivalent to minimizing the distance between two-hop neighbors on the constructed bipartite graph. Since these two-hop neighbors correspond to node pairs that share attributes on the original graph, the framework effectively encourages attribute-similar nodes to learn consistent representations. Besides, two-hop neighbors on the bipartite graph are brought close, which effectively captures the high-hop relationship on the original graph. Compared to traditional nodenode contrast, the proposed node-attribute contrast can more accurately and comprehensively capture node-node relationships, thereby achieving universality for both homophilic and heterophilic graphs.

5 Experiments

In this section, to begin with, the proposed framework GRASS is validated by empirically evaluating its performances on the node classification task. Next, an in-depth understanding of the efficacy of this framework is provided through several experiment analyses.

5.1 Experimental Settings

Datasets. Experiments are conducted on twelve widely used benchmark datasets with various homophily. The homophilic graph datasets include Cora, CiteSeer, PubMed, Wiki-CS, Amazon-Computers (abbreviated as Computers), and Amazon-Photo (abbreviated as Photo). The heterophilic graph datasets include Chameleon, Squirrel, Actor, Cornell, Texas, and Wisconsin. The statistics of these datasets are summarised in Table 1. Cora, CiteSeer, and PubMed [Sen et al., 2008] are three citation network datasets where nodes represent papers and edges indicate citation relationships between papers. Wiki-CS [Mernyei and Cangea, 2020] is a hyperlink network constructed based on Wikipedia. Nodes represent articles in computer science and edges are hyperlinks between articles. Computers and Photo [Shchur et al., 2018] are co-purchase networks from Amazon. In these networks, nodes represent goods and edges represent two goods being frequently bought together. Chameleon and Squirrel [Pei et al., 2020] are two Wikipedia networks where nodes denote pages and edges denote links between pages. Actor [Pei et al., 2020] is an actor co-occurrence network where nodes denote actors and edges denote two actors co-occurring in the same film. Cornell, Texas, and Wisconsin [Pei et al., 2020] are networks of web pages from computer science departments of diverse universities, where nodes are web pages and edges are hyperlinks between web pages.

Baselines. To verify the superiority and effectiveness of the proposed GRASS, we compare it with three categories of graph learning methods: (1) semi-supervised GNN models for node classification tasks, including vanilla GCN [Pei et al., 2020] and GAT [Veliˇckovi´c et al., 2018]; (2) unsupervised graph learning methods, including DeepWalk [Perozzi et al., 2014] and Node2Vec [Grover and Leskovec, 2016]; (3) self-supervised graph learning methods, including DGI [Velickovic et al., 2019], GMI [Peng et al., 2020], GRACE [Zhu et al., 2020b], MVGRL [Hassani and Khasahmadi, 2020], GCA [Zhu et al., 2021], BGRL [Thakoor et al., 2022], AF-GCL [Wang et al., 2022], HLCL [Yang and Mirzasoleiman, 2023], MaskGAE [Li et al., 2023], GraphACL [Xiao et al., 2023], SGCL [Sun et al., 2024], and GCIL [Mo et al., 2024].

Experimental Details. For reproducibility, the detailed settings of the experiments are described below. The experiments are performed on Nvidia GeForce RTX 3090 (24GB) GPU cards. Furthermore, the representations are obtained using a 1-3 layer MLP, with the hidden dimension of each layer being subject to hyperparameter tuning. Based on the representations, we train a Logistic classifier to perform downstream tasks. In all the experiments, we use the Adam optimizer. The training epoch is 200 with full batch training. For hyperparameter settings, the learning rates are tuned in the range {0.1, 0.05, 0.01, 0.005, 0.001}. Besides, the weight decay is tuned from {0.0, 0.001, 0.005, 0.01, 0.1}. Finally, the representation dimension is tuned in the range {256, 512, 1024, 2048, 4096}. For homophilic graphs, all nodes are randomly divided into three parts, 10% nodes for training, 10% nodes for validation and the remaining 80% nodes for testing. The performance on heterophilic graph datasets is evaluated on the commonly used 48%/32%/20% training/validation/testing.

5.2 Experiment Results

Results on Homophilic Graphs. The comparison of accuracy between GRASS and the baselines on six homophilic graphs is shown in Table 2. First, it can be observed that GRASS achieves the optimal performance on four of the six datasets, which illustrates the superiority of GRASS for processing homophilic graphs. GRASS even surpasses all supervised comparison models on all datasets. To be specific, on the Cora dataset, GRASS outperforms the second-best selfsupervised model (i.e., GCIL) by 1.13%, and on the WikiCS dataset, GRASS outperforms the second-best model (i.e., SGCL) by 1.50% in classification accuracy. This highlights the effectiveness of GRASS in learning consistency between node representations and attribute representations.

Results on Heterophilic Graphs. Table 3 shows the experimental results on heterophilic graphs. It can be observed that GRASS outperforms the baselines on five of the six datasets. Specifically, contrastive strategies with augmentations (e.g., GRACE, GCA, and BGRL) can not work well on heterophilic graphs compared to homophilic graphs. In contrast, augmentation-free models such as GRASS, AFGCL, and GraphACL achieve consistent performance advantages across five heterophilic benchmark datasets. Existing studies have shown that graph augmentations preserve the low-frequency components of the graph while potentially perturbing the high-frequency components in heterophilic graphs [Lee et al., 2022]. Therefore, achieving better performance without graph augmentations may benefit from the preservation of high-frequency components. Additionally, GRASS outperforms the second-best model (i.e., AF-GCL and GraphACL) by 5.49%, 7.17%, and 8.71% on the Cornell, Texas, and Wisconsin datasets, respectively. Compared to GraphACL, the performance advantage of GRASS stems from its ability to identify more accurate and numerous node neighbors than GraphACL, which relies on two-hop neighbors on the graph. As shown in Figure 3, even in heterophilic graphs, the identified two-hop neighbors have over 90% similarity, which is significantly higher than the two-hop monophily reported in GraphACL.

Visualization. This experiment aims to intuitively demonstrate the representation ability of the proposed GRASS. For this purpose, the t-SNE [Van der Maaten and Hinton, 2008] method is exploited to perform feature reduction and visualization of the trained representations. Figure 2 exhibits the experiment results (scatter plots) on four benchmark datasets (i.e., CiteSeer, Computers, Photo, and Actor), where colors stand for the classes of nodes. It can be observed that compared to all the comparison models (i.e., GRACE and GraphACL), the proposed GRASS framework is enabled to produce the most informative representations. As exemplified by the CiteSeer and Actor datasets, compared to the self-supervised models, GRASS achieves more compact clusters in node embeddings. Specifically, the embeddings of the same class are closer, while the embeddings of different classes exhibit more significant differences. The experiment results emphasize the effectiveness of GRASS in enhancing representation ability. Furthermore, GRASS is able to address the uneven distribution in the dataset due to class imbalance. As exemplified by the CiteSeer dataset, GRASS can achieve a more balanced distribution of nodes across different classes, thereby mitigating the distribution disparity among classes compared to GRACE.

5.3 Similarity Analysis

Figure 3 plots the pairwise cosine similarity distribution of two-hop neighbor pairs on the bipartite graph. It can be observed that in both homophilic and heterophilic graphs, the similarity is mostly concentrated in the range of 0.75 to 1, indicating that GRASS is able to efficiently identify pairs of nodes with similar attributes in the original graph. This phenomenon coincides with Theorem 1. In homophilic graphs, connected nodes tend to belong to the same class, i.e., they share similar attributes. The two-hop neighbors in the bipartite graph are connected by the same attribute, thus they are inclined to be similar. Besides, nodes shared attributes (i.e., two-hop neighbors on the bipartite graph) often correspond to the high-hop relationship on the original graph. In heterophilic graphs, directly connected nodes are usually from different classes. Therefore, the high-hop relationship is more likely to uncover nodes of the same class.

5.4 Hyperparameter Study

To analyze the sensitivity of the proposed model, experiments have been conducted with various hyperparameters.

Representation Dimension. In Figure 4, the effect of the representation dimension on performance is analyzed. The results show that higher representation dimensions generally lead to better performance, especially on datasets with large initial attribute dimensions such as CiteSeer and Squirrel. This indicates that preserving more feature information contributes to improved performance on both homophilic and heterophilic graphs.

Learning Rate. In Figure 5, the impact of the hyperparameters lr1 and lr2 on node classification accuracy across different graphs is analyzed. lr1 and lr2 control the learning rates for the topology part (MLPA) and the attribute part (MLPM) of the model, respectively. In the experiments, when lr1 is adjusted, lr2 is kept fixed, and vice versa. The results show that in homophilic graphs, the performance is relatively stable with respect to both learning rates, which reflects the insensitivity to learning rates on homophilic graphs. However, in heterophilic graphs, results show that adjusting lr2 leads to more significant accuracy improvements, while tuning lr1 has a relatively smaller effect. This indicates that the attribute representation plays a more important role in handling heterophilic structures, with the model relying more on attributes to distinguish node categories.

6 Conclusions

This paper proposes a novel Graph Contrastive Learning (GCL) framework named GRAph learning via Self-contraSt (GRASS) to overcome the limitations of traditional GCL methods, including the reliance on complex graph augmentations and the difficulty of selecting TRUE positive and negative samples. GRASS adopts a node-attribute self-contrast mechanism that enhances the similarity between nodes and their included attributes while suppressing the similarity with non-included attributes. This framework implicitly ensures accurate node-node contrast by capturing high-hop coinclusion relationships. The theoretical analysis and empirical results on multiple benchmark datasets validate the efficiency and universality of GRASS across both homophilic and heterophilic graphs.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (No. U22B2036, 62261136549, 62376088, 62272340, 92370111, 62272020), in part by the Hebei Natural Science Foundation (No. F2024202047, F2024202068), in part by the Science Research Project of Hebei Education Department (BJK2024172), in part by the Guangxi Key Laboratory of Machine Vision and Intelligent Control (2023B03), in part by the Hebei Yanzhao Golden Platform Talent Gathering Programme Core Talent Project (Education Platform) (HJZD202509), in part by the Post-graduate's Innovation Fund Project of Hebei Province (CXZZBS2025036), and in part by the Tencent Foundation and XPLORER PRIZE.