GraphSculptor: Sculpting Pre-training Core Sets for Graph Self-supervised Learning

1 Introduction

Graph self-supervised learning (SSL) has become a cornerstone for representation learning on graph-structured data, achieving strong performance across domains from molecular science to social network analysis [You et al., 2020; Xia et al., 2022a; Hou et al., 2022]. Recent progress is often driven by scaling up unlabeled pre-training datasets, implicitly assuming that more pre-training graphs will reliably translate into better representations. In this work, we revisit this assumption and show that large graph datasets can contain substantial redundancy, motivating a more compute-efficient, data-centric alternative.

A simple diagnostic (Figure 1) shows that graph SSL is empirically robust to uniform random pruning: removing 50% of pre-training graphs preserves over 96% of downstream transfer performance. This indicates diminishing returns of data scaling and substantial redundancy, raising the question: can we identify a much smaller subset that remains representative of the full pre-training dataset?

To answer this question, we take a data-centric view of graph SSL: we aim to improve pre-training efficiency by selecting informative pre-training graphs instead of increasing data scale. Prior efforts to reduce training data often rely on training-time signals such as gradients or task supervision to identify "important" examples [Xu et al., 2023a; Zhang et al., 2024]. However, in graph SSL pre-training, labels are typically unavailable, and computing gradient-based criteria can introduce additional optimization overhead and may be tied to specific model/optimizer choices, potentially limiting the achievable wall-clock savings and generality. To this end, we introduce GraphSculptor, a label-free and modelagnostic framework that extracts an information-rich coreset from a large pre-training collection. GraphSculptor measures diversity from two complementary views-structure via intrinsic graph statistics and semantics via graph-to-text followed by a frozen text encoder-and then selects a coreset that preserves joint structural-semantic diversity under a target budget. Our objective is to substantially reduce pre-training cost while maintaining downstream transfer performance comparable to full-data pre-training.

The core idea of GraphSculptor is to curate pre-training data using a dual-view notion of diversity. We characterize each graph from two complementary perspectives: structure and semantics. For the structural view, we compute a compact vector of intrinsic, multi-scale graph statistics and aim to cover diverse topological regimes, rather than over-representing the most frequent patterns. At the same time, structure alone may be insufficient: in many domains, graphs with similar topology can correspond to different domain properties (e.g., molecules sharing similar scaffolds may have different biological activities; see Figure 2). To capture such domain-level variation, we map each graph to a textual description (graph-to-text) and encode it with a pre-trained language model, yielding a semantic embedding that serves as a complementary proxy beyond topology-based summaries.

Given the fused structural and semantic embeddings, GraphSculptor selects a budgeted coreset in a cluster-aware manner. Specifically, it clusters graphs in the joint space and then selects representative graphs across clusters, prioritizing clusters that are both diverse internally and well separated from others, to preserve joint structural-semantic diversity under the target budget. The selection in GraphSculptor is performed once per pre-training dataset as an offline preprocessing step: we compute structural/semantic embeddings for all graphs and derive a fixed coreset that can be reused across pre-training runs. This one-time cost is amortized when the same dataset is used to pre-train different models or under different SSL objectives, and it avoids any training-time gradient-based scoring, making the procedure label-free, model-agnostic, and easy to cache and deploy. Our main contributions are:

• We propose GraphSculptor, a label-free and model-agnostic coreset construction framework for graph SSL that jointly considers structural statistics and semantic embeddings.
• We provide a loss-gap bound for coreset pre-training versus full-data pre-training, supporting our selection principle.
• Experiments show strong efficiency-accuracy trade-offs: using 10% of the pre-training graphs retains up to 99.6% of full-data performance while reducing pre-training time by nearly 90%.

2 RelatedWork

Graph Pretraining. Self-supervised graph pre-training has become a standard approach for leveraging unlabeled graphs and mitigating label scarcity across domains [Xia et al., 2022b]. Existing methods broadly fall into two paradigms: contrastive pre-training, which aligns representations across augmented views to learn invariances [You et al., 2020; Xia et al., 2022a], and generative pre-training, which learns by reconstructing masked components of graphs [Li et al., 2023; Liu et al., 2026; Liu et al., 2024]. The latter line, represented by GraphMAE and its variants [Hou et al., 2022; Hou et al., 2023], is often adopted due to its relatively simple training pipeline without explicit negative sampling. Subsequent work extends the reconstruction targets to richer structural signals (e.g., edges and degrees) [Li et al., 2023; Tan et al., 2023; Shi et al., 2023; Liu et al., 2025] or combines reconstruction with contrastive objectives [Liu et al., 2023; Zheng and Jia, 2024; Wang et al., 2024; Shen et al., 2025]. While these approaches differ in objectives and architectures, they typically pre-train on the full available dataset and benefit from larger-scale data when feasible. This practice can incur substantial computational cost and limits the iteration speed and scalability of graph pre-training. In contrast, our work takes a data-centric perspective and asks how to curate a compact yet informative pre-training subset that preserves the utility of full-dataset pre-training. Therefore, our method is orthogonal to the choice of pre-training objective and can be applied on top of existing graph pre-training frameworks.

Graph Data Reduction. The high computational cost of learning on large-scale graph datasets has motivated research on reducing training data, broadly including graph dataset distillation/condensation and graph selection. Distillation methods such as GCond [Jin et al., 2022b] and DosCond [Jin et al., 2022a] aim to synthesize a compact surrogate dataset, often formulated as bilevel optimization to match training dynamics (e.g., gradients). While effective in some settings, these formulations can be computationally demanding and sensitive to modeling choices, prompting more tractable alternatives: KIDD [Xu et al., 2023b] replaces gradient matching with kernel-based objectives, SGDD [Yang et al., 2023] emphasizes preserving structural properties, and MIRAGE [Gupta et al., 2024] condenses representative substructures with improved model-agnosticism. Graph selection, which directly chooses a subset from the original dataset, is closer to our setting. Many existing selection approaches adopt dynamic in-training strategies that iteratively update the subset using model-dependent signals, such as training loss [Chen et al., 2024], gradient norms [Zhang et al., 2024], or predictive uncertainty [Xu et al., 2023a]. Although these signals can be informative, the resulting procedures are tightly coupled to specific training dynamics and typically require repeated evaluations over training, increasing overhead and reducing their suitability as a one-shot, model-agnostic preprocessing step [Li et al., 2025]. In contrast, GraphSculptor performs one-shot, offline coreset selection without training-time signals, enabling label-free and model-agnostic preprocessing before pre-training.

3 Method

GraphSculptor is a data-centric framework for constructing a compact pre-training coreset from a large unlabeled graph dataset. As illustrated in Figure 3, it follows two stages: (1) Holistic Graph Representation, which computes for each graph a fused representation that combines structural and semantic views; and (2) Principled Coreset Curation, which selects a budgeted subset via cluster-based selection in the fused space to preserve structural-semantic diversity. In particular, the semantic view is obtained by mapping each graph to a textual description (graph-to-text) and encoding it with a frozen text encoder, providing signals complementary to topologybased statistics. The entire procedure is performed offline as a one-time preprocessing step before SSL pre-training.

Preliminaries. We consider an unlabeled graph dataset D = {Gi}Mi=1 , where each graph Gi = (Vi, Ei) is associated with an adjacency matrix Ai and node features Xi. Our goal is to select a coreset Dcore ⊂ D with a fixed budget |Dcore| = Mtarget = ⌊ptarget M⌋, where ptarget ∈ (0, 1] denotes the retention ratio. We construct Dcore to remain representative of D in the proposed fused (structural-semantic) representation space under this budget, i.e., to preserve the diversity of graphs as measured in that space.

3.1 Holistic Graph Representation

We represent each graph by jointly capturing (i) intrinsic topology and (ii) semantic context. Concretely, for each Gi we compute a structural feature vector zS,i ∈ RdS and a semantic embedding zT,i ∈ RdT , where dS and dT denote their respective feature dimensions. We then combine them into a fused representation used for downstream clustering and coreset selection.

Structural Characterization

To capture structural diversity, we characterize each graph using a compact set of graph-level descriptors. The design goal is to balance scalability and discriminative power: the descriptors should be inexpensive to compute at dataset scale while still reflecting salient structural variations relevant to data curation. We therefore adopt a modular structural feature extractor that can incorporate domain-specific descriptors when beneficial, while using a lightweight default configuration in our main experiments.

For each graph Gi, we define a structural feature map ψS : G → RdS that outputs a fixed-dimensional descriptor zS,i. Specifically,

where each ϕj : G → R computes a scalar structural statistic. Concretely, ψS is instantiated with a streamlined set of descriptors, engineered to strike an optimal balance between discriminative expressivity and computational scalability: (i) Basic Topological Invariants. We explicitly encode the fundamental scale and sparsity of each graph via its node count |Vi|, edge count |Ei|, and average degree 2|Ei|/|Vi|. (ii) Global Positional Encodings. To transcend local aggregation metrics and capture long-range structural dependencies, we incorporate Positional Encodings. Specifically, we leverage random-walk based positional identifiers [Ying et al., 2021], which provide a compact yet informative signature of global connectivity, ensuring the structural feature vector remains distinct across topologically diverse graphs.

The resulting zS,i serves as the structural view used in the fused representation. We use a lightweight default configuration in our main experiments for efficiency, and ψS can be extended with additional domain-specific descriptors when they are available and useful.

Semantic Contextualization

Beyond structure, many graph datasets carry domain semantics that may not be fully captured by structural descriptors alone. For molecular graphs, compounds with highly similar scaffolds can exhibit markedly different biochemical effects (e.g., Aspirin vs. Salicylic Acid). Similarly, in citation networks, two papers may have nearly identical local citation neighborhoods (e.g., comparable in-/out-degree and co-citation patterns) while belonging to entirely different research topics. To incorporate this complementary signal, we leverage naturallanguage descriptions by mapping each graph to text and then encoding the text with a pre-trained text encoder, yielding a semantic embedding that is orthogonal to purely structural statistics. We define a semantic feature map as a two-stage composition:

which maps each graph to a dense semantic embedding.

Graph-to-Text conversion (fG2T). We map each graph Gi to a short textual descriptor ti = fG2T(Gi) using domainspecific instantiations. For molecular graphs, we leverage the available SMILES strings and employ a biomedical text generation model (e.g., BioT5 [Pei et al., 2023]) to generate a concise title/description. For social-network graphs, where no canonical textual field is provided, we first serialize the graph into an adjacency-list (edge-list) representation with a fixed, deterministic ordering (e.g., sorting nodes and neighbors by index to reduce permutation effects). We then prepend a fixed dataset-level background description that specifies only the domain context (without labels or task information), and use a general-purpose LLM (e.g., ChatGPT or Gemini) to produce a bounded-length, sociology-oriented title. We use a fixed prompt template and deterministic decoding, and cache the generated texts to ensure reproducibility.

Text-to-Embedding encoding (fT2E). We encode ti with a pre-trained text encoder to obtain a semantic embedding. In our main experiments on biochemical graphs, we use SciBERT [Beltagy et al., 2019] as the encoder; for general text, standard encoders such as BERT [Devlin et al., 2019] can be used. Formally, given a text encoder Encoder(·), we use the final-layer hidden state of the special classification token [CLS] as a pooled representation:

During coreset construction, the text encoder is kept frozen to avoid introducing any training-time signals and to keep the procedure lightweight. The resulting embeddings provide a semantics-aware proxy derived from natural language, complementing structural descriptors by capturing descriptive similarity that may not be reflected by topology alone.

Unified Representation Space

To enable holistic coreset selection, we project the distinct structural and semantic signals into a unified metric space. Direct fusion of the structural descriptor zS,i ∈ RdS and the semantic embedding zT,i ∈ RdT is non-trivial due to their disparate modalities and potentially incommensurable magnitudes. To prevent high-variance features from dominating the distance computation, we employ feature-wise standardization (z-score normalization) across the dataset, yielding normalized descriptors ˆzS,i and ˆzT,i. We then synthesize the holistic representation hi via concatenation:

In this unified space, the Euclidean distance aggregates both structural similarity and semantic affinity, preventing modal dominance and ensuring that coreset selection targets joint structural-semantic diversity.

3.2 Principled Coreset Curation

We construct the coreset via a cluster-aware selection scheme in the fused space. We first partition graphs by K-means, then prioritize clusters using a diversity score, and finally select representative graphs within each cluster under a fixed budget.

• ➊ Unsupervised Clustering in the Fused Space. Given fused representations {hi}Mi=1 , we apply K-means clustering and obtain clusters C = {C1, . . . ,CK} with centroids {µk}Kk=1 by minimizing the within-cluster squared Euclidean distance:

➋Diversity Scoring. An effective coreset should balance mass coverage (so that high-density regions of the empirical distribution are not under-represented) and diversity to avoid collapsing onto frequent but low-informative patterns. To combine these factors in a simple and scale-robust manner, we define the cluster score in the log domain:

where πj = |Cj |/M captures the cluster mass and w ∈ [0, 1] controls the trade-off between within-cluster dispersion and cross-cluster separation. This log-domain formulation ensures numerical stability and balances the magnitude differences between density and structural metrics. We next normalize these weights to obtain cluster-wise sampling proportions.

• ➌ Cluster-wise Selection. We convert {Ωj} into clusterwise sampling proportions:

where we use τ = 0.5 by default. We then assign each cluster a quota nj such that ∑Kj=1 nj = Mtarget; in implementation, we set nj = ⌊Mtarget ˜Ωj⌋ for j = 1, . . . ,K - 1 and allocate the remainder to the last cluster.

Within each cluster Cj , we perform centroid-biased sampling to obtain Sj . For Gi ∈ Cj , we define

and sample nj graphs without replacement fromCj according to {pi|j}:

where WRS(·) denotes weighted random sampling without replacement, i.e., drawing nj samples fromCj with probabilities proportional to {pi|j}. Since global coverage is mainly ensured by the cross-cluster budget allocation (Eq. (6)), this within-cluster step favors typical instances near the centroid while retaining randomness. The temperature σj controls the sampling sharpness, interpolating between uniform sampling (σj →∞) and hard nearest-to-centroid selection (σj →0).

3.3 Theoretical Analysis

We provide theoretical motivation for our cluster-aware allocation by analyzing optimization under a standard NTK-style linearization around the initialization W0. Here, W denotes the parameters of the pre-training model and W0 is their initialization. Concretely, we consider the linearized per-sample loss ˜ ℓ(W; G) = ℓ(W0; G) + ⟨∇ℓ(W0; G), W -W0⟩, so that the optimization dynamics are governed by initialization gradients. Under L-smoothness and bounded-gradient assumptions, we bound the expected pre-training loss gap between training on the selected coreset and training on the full dataset. The bound highlights the role of within-cluster gradient variability (heterogeneity), suggesting that clusters with larger variability should receive higher sampling proportions.

Theorem 1 (Expected Pre-training Loss Gap under Linearization). Let L(W) = 1M ∑Mi=1 ℓ(W;Gi) be an L-smooth loss over D = {G1, . . . ,GM}. Assume ∥∇ℓ(W0;G)∥F ≤ Bg for all G ∈ D and the linearized regime defined by ˜ ℓ(·; ·) above. Suppose D is partitioned into K disjoint clusters {Ck}Kk=1 with proportions πk = |Ck|/M. Let Mtarget be the coreset size, and let Mk be the number of selected samples from cluster Ck, inducing qk = Mk/Mtarget (with qk > 0 and ∑k qk = 1). Define the within-cluster gradient variance at W0 as

Let W (T ) full andW (T) core denote the parameters after T gradient steps with learning rate η when optimizing on the full dataset and on the sampled coreset, respectively. Then the expected pre-training loss gap satisfies:

Remark. Minimizing the upper bound in Eq. (10) subject to ∑Kk=1 qk = 1 yields q⋆ k ∝ πkVk, i.e., larger clusters with higher gradient variability should receive more budget.

Discussion. In practice, Vk is not available without trainingtime gradient evaluations. We therefore adopt offline, representation-based proxies computed once in our fused structural-semantic space. Under a mild regularity condition that initialization gradients vary smoothly with the fused representation (e.g., ∥∇ℓ(W0; Gi) -∇ℓ(W0; Gi′ )∥F ≤ Lh∥hi - hi′∥2), the within-cluster gradient variability can be controlled by within-cluster dispersion in the fused space, motivating the mass-intra component log(πk) + w log(dintra,k). We further incorporate inter-cluster separation dinter,k as a practical correction to discourage allocating excessive budget to highly overlapping clusters, leading to our score Ωk (Eq. (6)). Sampling proportions are then obtained via temperature-scaled softmax, implementing the mass-variability principle suggested by the theory while remaining label-free and avoiding training-time signals.

4 Experiments

4.1 Experimental Setup

Datasets and Tasks. Our evaluation covers two learning paradigms. Pre-training and Transfer Learning. For graph SSL, we pre-train on ZINC, which contains ∼250,000 commercially available molecules [Sterling and Irwin, 2015]. We then transfer the pre-trained models to eight binary classification tasks from MoleculeNet [Wu et al., 2018]. For all MoleculeNet tasks, we use the standard scaffold splits to evaluate generalization. Supervised Learning. To assess the generality of our selection strategy beyond SSL pre-training, we additionally conduct supervised graph classification on large-scale OGB molecular benchmarks, OGBG-MOLHIV and OGBG-MOLPCBA [Hu et al., 2020], following the official data splits provided by OGB. We further include two social graph benchmarks from TUDataset [Morris et al., 2020], COLLAB and IMDB-BINARY, using the standard evaluation protocol. We benchmark GraphSculptor against 13 static data selection baselines, covering Random sampling, classical coreset methods (k-Center, Herding, CD), and representative label-informed pruning approaches (Influence, DP). .

Implementation Details. For SSL pre-training on ZINC, we employ GraphMAE [Hou et al., 2022] and evaluate via linear probing on MoleculeNet. For supervised learning on OGB, we train GraphGPS [Rampasek et al., 2022] models from scratch on selected coresets. GraphSculptor performs graph-to-text translation and semantic embedding in an offline manner. For molecular graphs, we use BioT5-base [Pei et al., 2023] for graph-to-text translation and SciBERT [Beltagy et al., 2019] as the frozen text encoder. For social-network datasets from TUDataset (COLLAB and IMDB-BINARY), we use a general-purpose LLM (e.g., Gemini) for graph-to-text translation and a BERT as the encoder. The number of clusters K is treated as a hyperparameter. All reported results are the mean and standard deviation over 5 random seeds.

4.2 Main Result in Graph SSL

We evaluate whether a curated subset can match full-data pretraining for graph SSL. Table 1 reports transfer results on eight MoleculeNet benchmarks (ROC-AUC). GraphSculptor remains competitive across budgets and achieves the highest average ROC-AUC in Table 1 under each reported ratio: 74.1 at 70% remaining (vs. 73.7 for full-data pre-training), 73.8 at 50% remaining, and 73.4 at 10% remaining (vs. 72.372.6 for classical coreset baselines such as Herding, K-Center, and CD). Two observations stand out. First, GraphSculptor matches or slightly exceeds full-data pre-training at moderate budgets, suggesting that removing redundant graphs may sharpen the effective pre-training signal. Second, its advantage becomes more pronounced under aggressive compression, consistent with our theory that the loss-gap bound scales with 1/Mtarget and depends on the allocation across clusters, and with our design that uses fused structural-semantic clustering and cluster-aware budget allocation to better cover diverse, non-redundant regions.

4.3 Generality in Supervised Learning

We further evaluate the generality of GraphSculptor on supervised graph classification. On OGBG-MOLHIV and OGBG-MOLPCBA, GraphSculptor performs label-free selection while several strong baselines leverage labels for pruning; nevertheless, GraphSculptor remains competitive and often yields better accuracy under the same budget. For example, on MOLHIV it matches or slightly exceeds the full-data result at the 70% budget, and at 50% it achieves 77.0 on HIV and 25.3 on PCBA, outperforming Random subsets even at larger retention ratios. Beyond molecular graphs, we also test social-network benchmarks from TUDataset (Table 3). On COLLAB, GraphSculptor consistently improves over Random and classical coreset baselines across budgets, reaching 81.0/81.3 at 50%/70% and closely tracking the full-data reference (81.4). On IMDB-BINARY, the gains are more modest, which is expected given the small dataset size and limited redundancy. Nonetheless, GraphSculptor is most effective in the low-budget regime, outperforming Random and achieving the best (or tied-best) accuracy at 20%-30% retention, while at higher budgets it becomes comparable to classical coreset methods such as K-Center and Herding.

4.4 Further Discussion

Ablation Study. We study the contribution of the two diversity views in GraphSculptor by comparing the full method with two ablations: Structure-only, which performs clustering and selection using only structural descriptors zS, and Semantics-only, which uses only semantic embeddings zT . As shown in Figure 4, the full model achieves the best performance across all budgets, while removing either view leads to a consistent degradation, indicating that the two views provide complementary signals for coreset curation. In our setting, dropping the semantic view typically incurs a larger performance drop than dropping the structural view, suggesting that language-based semantics can capture information not well reflected by structural statistics alone. Overall, combining both structural and semantic cues yields a more reliable criterion than either single view.

Efficiency Analysis. Figure 5 summarizes the end-to-end efficiency of GraphSculptor. The method incurs a one-time preprocessing cost for graph-to-text and text encoding (about

30 minutes in our setup), but this cost is amortized across pretraining runs and is substantially lower than several trainingsignal-based pruning baselines (5-25× faster in wall-clock time). More importantly, coreset pre-training provides a direct reduction in training time proportional to the retained data budget: for example, using a 10% coreset cuts pre-training time by nearly 90% compared to full-data pre-training, while maintaining comparable downstream performance (and occasionally yielding small gains). Overall, GraphSculptor offers a favorable performance-efficiency trade-off.

5 Conclusion

We revisit the prevailing practice of scaling unlabeled graph corpora for self-supervised pre-training and observe notable redundancy in our benchmark evaluation. We propose GraphSculptor, a label-free and model-agnostic framework for constructing compact pre-training coresets by combining a structural view based on intrinsic graph statistics with a complementary semantic view derived from graph-to-text and a frozen text encoder. We further provide a loss-gap bound under a standard linearized optimization regime, offering theoretical motivation for cluster-aware budget allocation. Empirically, GraphSculptor achieves favorable performance-efficiency trade-offs across molecular and social graph benchmarks. Overall, our results suggest that diversity-driven, offline data curation can serve as a practical preprocessing step for more efficient scaling of graph SSL. As future work, we will explore more reliable semantic abstractions for text-scarce graphs and strengthen the theoretical analysis in training dynamics.

Acknowledgments

This work was supported in part by the Natural Science Foundation of China (No. 62502065, 62476203), the National Key Research and Development Program of China (2023YFC2705705), the Key Project of Traditional Chinese Medicine Joint Fund of Hubei Provincial Natural Science Foundation (No. 2025AFD47), the Hubei Province Science and Technology Innovation Plan Project (No. 2025BCB035), the Guangdong Provincial Natural Science Foundation General Project (No. 2025A1515012155), the Shenzhen Natural Science Foundation Project (No. JCYJ20250604122534006, JCYJ20230807090211021), the Hubei Jiangxia Laboratory Development Project (No. 2025JXKF034), the New Cornerstone Science Foundation through the XPLORER PRIZE, and the Innovative Research Group Project of Hubei Province under Grants 2024AFA017.