A Systematic Survey of Chemical Pre-trained Models

1 Introduction

Extracting vector representations for molecules is critical to applying machine learning methods to a broad spectrum of molecular tasks. Initially, molecular fingerprints are developed to encode molecules into binary vectors with rulebased algorithms [Consonni and Todeschini, 2009]. Subsequently, various Deep Neural Networks (DNNs) have been employed to encode molecules in a data-driven manner. Early attempts exploit sequence-based neural architectures (e.g., RNNs, LSTMs, and transformers) to encode molecules represented in Simplified Molecular-Input Line-Entry System (SMILES) strings [Weininger, 1988]. Later, it is argued that molecules can be naturally represented in graph structures with atoms as nodes and bonds as edges. This inspires a line of works to leverage such structured inductive bias for better molecular representations [Kearnes et al., 2016]. The key advancements underneath these approaches are Graph Neural Networks (GNNs), which consider graph structures and attributive features simultaneously by recursively aggregating node features from neighborhoods [Kipf and Welling, 2017]. Recently, another line of development of GNNs for molecular representations models 3D geometric symmetries of molecular conformations, considering the molecules are in a constant motion in 3D space by nature [Sch¨utt et al., 2017]. However, the majority of the above works learn molecular representations under supervised settings, which limits their widespread application in practice for the following reasons. (1) Scarcity of labeled data: task-specific labels of molecules can be extremely scarce because molecular data labeling often requires expensive wet-lab experiments; (2) Poor out-of-distribution generalization: learning molecules with different sizes or functional groups requires out-of-distribution generalization in many real-world cases. For example, suppose one wishes to predict the properties of a newly synthesized molecule that differs from all the previous molecules in the training set. However, models trained from scratch cannot extrapolate to out-of-distribution molecules well [Hu et al., 2020a].

Pre-trained Language Models (PLMs) have been a potential solution to the above challenges in the Natural Language Processing (NLP) community [Devlin et al., 2019; Zheng et al., 2022; Hu et al., 2022]. Inspired by their success, as shown in Fig. 1, Chemical Pre-trained Models (CPMs)

have been introduced to learn universal molecular representations from massive unlabeled molecules and then fine-tuned over specific downstream tasks. Initially, researchers adopt sequence-based pre-training strategies on string-based molecular data such as SMILES. A typical strategy is to pre-train the neural encoders to predict randomly masked tokens like BERT [Devlin et al., 2019]. This line of works include ChemBERTa [Chithrananda et al., 2020], SMILES-BERT [Wang et al., 2019], Molformer [Ross et al., 2022], etc. More recently, the community explores pre-training on (both 2D and 3D) molecular graphs. For example, [Hu et al., 2020a] propose to mask atom or edge attributes and predict the masked attributes. [Liu et al., 2022] pre-train GNNs via maximizing the correspondence between 2D topological and 3D geometric structures.

Although CPMs have been increasingly applied in molecular representation learning, this rapidly expanding field still lacks a systematic review. In this paper, we present the first survey for CPMs to assist audiences of diverse backgrounds in understanding, using, and developing CPMs for various practical tasks. The contributions of this work can be summarized from the following four aspects. (1) A structured taxonomy. A broad overview of the field is presented with a structured taxonomy that categorizes existing works from four perspectives (Fig. 2): molecular descriptors, encoder architectures, pre-training strategies, and applications. (2) Thorough review of the current progress. Based on the taxonomy, the current research progress of pre-trained models for molecules is systematically delineated. (3) Abundant additional resources. Abundant resources including open-sourced CPMs, available datasets, and an important paper list are collected and can be found at https://github.com/junxia97/ awesome-pretrain-on-molecules. These resources will be continuously updated on a regular basis. (4) Discussion of future directions. The limitations of existing works are discussed and several promising research directions are highlighted.

2 Molecular Descriptors and Encoders

In order to feed molecules to DNNs, molecules have to be featurized in numerical descriptors. Various descriptors are designed to describe molecules in a concise format. In this section, we briefly review these molecular descriptors and their corresponding neural encoder architectures.

Fingerprints (FP). Molecular fingerprints describe the presence or absence of particular substructures of a molecule with binary strings. For example, PubChemFP [Wang et al., 2017] encodes 881 structural key types that correspond to the substructures for a fragment of compounds in the PubChem database.

Sequences. The most frequently used sequential descriptor for molecules is the Simplified Molecular-Input Line-Entry System (SMILES) [Weininger, 1988] owing to its versatility and interpretability. Each atom is represented as a respective ASCII symbol. Chemical bonds, branching, and stereochemistry are denoted by specific symbols. Transformers [Vaswani et al., 2017] are a powerful neural model for processing sequences and modeling the complex relationships among each token. We can split sequence-based molecular descriptors into a series of tokens denoting atoms/bonds at first and then apply transformers on top of these tokens [Chithrananda et al., 2020; Wang et al., 2019].

2D graphs. Molecules can be represented as 2D graphs naturally, with atoms as nodes and bonds as edges. Each node and edge can also carry feature vectors denoting the atom types/chirality and bond types/direction for instance [Hu et al., 2020a]. Here, GNNs [Kipf and Welling, 2017; Xu et al., 2019] can be used to learn 2D molecular graph representations. Some hybrid architectures of GNNs and transformers [Rong et al., 2020; Ying et al., 2021] can also be leveraged to capture the topological structures of molecular graphs.

3D graphs. 3D geometries represent the spatial arrangements of atoms of the molecule in the 3D space, where each atom is associated with its type and coordinate plus some optional geometric attributes such as velocity. The advantage of using 3D geometry is that the conformer information is critical to many molecular properties, especially quantum properties. In addition, it is also possible to directly leverage stereochemistry information such as chirality given the 3D geometries. A number of approaches [Sch¨utt et al., 2017; Satorras et al., 2021; Du et al., 2022a] have developed message-passing mechanisms on 3D geometries, which enable the graph representations to follow certain physical symmetries, such as equivariance to translations and rotations.

3 Pre-training Strategies

In this section, we elaborate on several representative selfsupervised pre-training strategies of CPMs.

3.1 AutoEncoding (AE)

Reconstructing molecules with autoencoders (Fig. 3a) serves as a natural self-supervised target for learning expressive molecular representations. The prediction in molecule reconstructions is (partial) structures of the given molecules such as the attributes of a subset of atoms or chemical bonds. A typical example is SMILES transformer [Honda et al., 2019], which leverages a transformer-based encoder-decoder network and learns the representations by reconstructing molecules represented by SMILES strings. More recently, unlike conventional autoencoders with the same types for the input and output data, [Lin et al., 2022] pre-train a graph-to-sequence asymmetric conditional variational autoencoder to learn molecular representations. Although autoencoders can learn meaningful representations for molecules, they focus on single molecules and fail to capture inter-molecule relationships, which limits their performance in some downstream tasks [Li et al., 2021].

3.2 Autoregressive Modeling (AM)

Autoregressive Modeling (AM) factorizes the molecular input as a sequence of sub-components and then it predicts the sub-components one by one, conditioned on previous sub-components in the sequence. Following the idea of GPT [Brown et al., 2020] in NLP, MolGPT [Bagal et al., 2021] pre-trains a transformer network to predict the next token in the SMILES strings in such an autoregressive manner.

For molecular graphs, GPT-GNN [Hu et al., 2020b] reconstructs the molecular graph in a sequence of steps (Fig. 3b), in contrast to graph autoencoders that reconstruct the whole graph at once. In particular, given a graph with its nodes and edges randomly masked, GPT-GNN generates one masked node and its edges at a time and maximizes the likelihood of the node and edges generated in each iteration. Then, it iteratively generates nodes and edges until all masked nodes are generated. Analogously, MGSSL [Zhang et al., 2021] generates molecular graph motifs instead of individual atoms or bonds autoregressively. Formally, such autoregressive modeling objectives can be written as

where Ci, C<i are the attributes of i-th component and the ones generated before the index i in the moleculeM, respectively. Compared with other strategies, AM allows CPMs to perform better at generating molecules its training procedure resembles that of molecule generation [Bagal et al., 2021]. However, AM is more computationally expensive and requires a preset ordering of atoms or bonds beforehand, which may be inappropriate for molecules because the atoms or bonds do not present inherent orders.

3.3 Masked Component Modeling (MCM)

In language domains, Masked Language Modeling (MLM) has emerged as a dominant pre-training objective. Specifically, MLM randomly masks out tokens from the input sentences, where the model can be trained to predict those masked tokens using the remaining tokens [Devlin et al., 2019]. Masked Component Modeling (MCM, Fig. 3c) generalizes the idea of MLM for molecules. Specifically, MCM masks out some components (e.g., atoms, bonds, and fragments) of the molecules and then trains the model to predict them given the remaining components. Generally, its objective can be formulated as

where m(M) denotes the masked components from the moleculeMandM\m(M) are the remaining components. For sequence-based pre-training, ChemBERTa [Chithrananda et al., 2020], SMILES-BERT [Wang et al., 2019], and Molformer [Ross et al., 2022] mask random characters in the SMILES strings and then recover them based on the output of the transformer of the corrupted SMILES strings. For molecular graph pre-training, [Hu et al., 2020a] propose to randomly mask input atom/chemical bond attributes and pre-train the GNNs to predict them. Similarly, GROVER [Rong et al., 2020] attempts to predict the masked subgraphs to capture the contextual information in the molecular graphs. Recently, Mole-BERT [Xia et al., 2023] argues that masking the atom types could be problematic due to the extremely small and unbalanced atom set in nature. To mitigate this issue, they design a context-aware tokenizer to encode atoms as chemically meaningful discrete values for masking.

MCM is especially beneficial for richly-annotated molecules. For example, masking atom attributes enables GNNs to learn simple chemistry rules such as valency, as well as potentially other complex chemistry descriptors such as the electronic or steric effects of functional groups. Additionally, compared with the aforementioned AM strategies, MCM predicts the masked components based on their surrounding environments as opposed to AM which merely relies on preceding components in a predefined sequence. Hence, MCM can capture more complete chemical semantics. However, as MCM often masks a fixed portion of each molecule during pre-training following BERT [Devlin et al., 2019], it cannot train on all the components in each molecule, which results in less efficient sample utilization.

3.4 Context Prediction (CP)

Context Prediction (CP, Fig. 3d) aims to capture the semantics of molecules/atoms in an explicit, context-aware manner. Generally, CP can be formulated as

where t = 1 if neighborhood components M1 and surrounding contexts M2 share the same center atom and otherwise t = 0. For example, [Hu et al., 2020a] use a binary classification of whether the subgraphs in molecules and surrounding context structures belong to the same node. While simple and effective, CP requires an auxiliary neural model to encode the context into a fixed vector, adding extra computational overhead for large-scale pre-training.

3.5 Contrastive Learning (CL)

Contrastive Learning (CL, Fig. 3e) pre-trains the model by maximizing the agreement between a pair of similar inputs, such as two different augmentations or descriptors of the same molecule. According to the contrastive granularity (e.g., molecule- or substructure-level), we introduce two categories of CL in CPMs: Cross-Scale Contrast (CSC) and Same-Scale Contrast (SSC).

Cross-ScaleContrast (CSC). Deep InfoMax is a representative CSC model that is originally proposed for learning image representations by contrasting a pair of an image and its local regions against other negative pairs [Hjelm et al., 2019]. For molecular graphs, InfoGraph [Sun et al., 2020] follows this idea by contrasting molecule- and substructure-level representations, which can be formally described as

where N is a set of negative samples, C is a substructure of M, C- is a substructure of the other molecule, and s(·, ·) denotes a similarity metric. Follow-up work MVGRL [Hassani and Ahmadi, 2020] performs node diffusion to generate an augmented molecular graph and then maximizes the similarity between original and augmented views by contrasting atom representations of one view with molecular representations of the other view and vice versa.

Same-Scale Contrast (SSC). Same-Scale Contrast (SSC) performs contrastive learning on individual molecules by pushing the augmented molecule close to the anchor molecule (positive pairs) and away from other molecules (negative pairs). For example, GraphCL [You et al., 2020] and its variants [You et al., 2021; Sun et al., 2021; Suresh et al., 2021; Xu et al., 2021a; Fang et al., 2022b; Wang et al., 2021; Xia et al., 2022b; Wang et al., 2022a] propose various augmentation strategies for molecule-level pre-training represented by graphs. Additionally, some recent works maximize the agreement between various descriptors of identical molecules and repel the different ones. For example, SMICLR [Pinheiro et al., 2022] jointly leverages a graph encoder and a SMILES string encoder to perform SSC; MM-Deacon [Guo et al., 2022] utilizes two separate transformers to encode the SMILES and the International Union of Pure and Applied Chemistry (IUPAC) of molecules, after which a contrastive objective is used to promote similarity of SMILES and IUPAC representations from the same molecule; 3DInfoMax [St¨ark et al., 2022] proposes to maximize the agreements between the learned 3D geometry and 2D graph representations; GeomGCL [Li et al., 2022b] adopts a dual-view Geometric Message Passing Neural Network (GeomMPNN) to encode both 2D and 3D graphs of a molecule and design a geometric contrastive objective. The general formulation of the SSC pre-training objective is

whereM′ is the augmented versions or other descriptors of the moleculeM, and N is the set of negative samples.

Although CL has achieved promising results, several critical issues impede its broader applications. Firstly, it is difficult to preserve semantics during molecular augmentations. Existing solutions pick augmentations with manual trial-anderrors [You et al., 2020], cumbersome optimization [You et al., 2021], or through the guidance of expensive domain knowledge [Sun et al., 2021], but an efficient and principled way to design the chemically appropriate augmentation for molecular pre-training is still lacking. Also, the assumption behind CL that pulls similar representations closer may not always hold true for molecular representation learning. For example, in the case of molecular activity cliffs [Stumpfe et al., 2019], similar molecules hold completely different properties. Therefore, it remains unsolved which pre-training strategies can better capture discrepancies between molecules. Additionally, the CL objective in most CPMs randomly chooses all other molecules in one batch as the negative samples regardless of their true semantics, which will undesirably repel the molecules of similar properties and undermine the performance due to the false negatives [Xia et al., 2022a].

3.6 Replaced Components Detection (RCD)

Replaced Components Detection (RCD, Fig. 3f) proposes to recognize randomly replaced components of the input molecules. For example, MPG [Li et al., 2021] splits each molecule into two parts, shuffles their structure by combining parts from two molecules, and trains the encoder to detect whether the combined parts belong to the same molecule. This objective can be written as

where t = 1 if the two parts M1 andM2 are from the same molecule M and t = 0 otherwise. While RCD can uncover intrinsic patterns in molecular structures, the encoders are pretrained to always produce the same "non-replacement" label for all natural molecules and "replacement" label for randomly combined molecules. However, in downstream tasks, the input molecules are all natural ones, causing the molecular representations produced by RCD to be less distinguishable.

3.7 DeNoising (DN)

Inspired by the success of denoising diffusion probabilistic models [Ho et al., 2020], DeNoising (DN, Fig. 3g) has been recently adopted as a pre-training strategy for learning molecular representations as well. A recent work [Zaidi et al., 2023] adds

,

noise to atomic coordinates of 3D molecular geometry and pre-trains the encoders to predict the noise. They demonstrate that such a denoising objective approximates learning a molecular force field. A concurrent work, Uni-Mol [Zhou et al., 2023] adds noise to atomic coordinates motivated by the fact that masked atom types can be easily inferred given 3D atomic positions. More recently, GeoSSL [Liu et al., 2023] proposes a distance denoising pre-training method to model the dynamic nature of 3D molecules. Generally, the pre-training objective of denoising can be formulated as

where ϵ denotes the added noise, ˜ M denotes the input moleculeMwith noise added, and fθ(·) denotes the encoders that predict the noise.

3.8 Extensions

Knowledge-enriched pre-training. CPMs usually learn general molecular representations from a large molecular database. However, they often lack domain-specific knowledge. To improve their performance, several recent works try to inject external knowledge into CPMs. For example, GraphCL [You et al., 2020] first points out that bond perturbations (adding or dropping the bonds as data augmentations) are conceptually incompatible with domain knowledge and empirically not helpful for contrastive pre-training on chemical compounds. Therefore, they avoid adopting bond perturbations for molecular graph augmentation. More explicitly, MoCL [Sun et al., 2021] proposes a domain knowledge-based molecular augmentation operator called substructure substitution, in which a valid substructure of a molecule is replaced by a bioisostere which produces a new molecule with similar physical or chemical properties as the original one. More recently, KCL [Fang et al., 2022b] constructs a chemical element Knowledge Graph (KG) to summarize microscopic associations between chemical elements and presents a novel Knowledge-enhanced Contrastive Learning (KCL) framework for molecular representation learning. Additionally, MGSSL [Zhang et al., 2021] first leverages existing algorithms [Degen et al., 2008] to extract semantically meaningful motifs and then pre-trains neural encoders to predict the motifs in an autoregressive manner. ChemRL-GEM [Fang et al., 2022a] proposes to utilize molecular geometry information to enhance molecular graph pre-training. It designs a geometry-based GNN architecture as well as several geometry-level self-supervised learning strategies (the bond lengths prediction, the bond angles prediction, and the atomic distance matrices prediction) to capture the molecular geometry knowledge during pre-training. Although knowledge-enriched pre-training helps CPMs capture chemical domain knowledge, it requires expensive prior knowledge as guidance, which poses a hurdle to broader applications when the prior is incomplete, incorrect, or expensive to obtain.

Multimodal pre-training. In addition to the descriptors mentioned in Sec. 2, molecules can also be described using other modalities including images and biochemical texts. Some recent works perform multimodal pre-training on molecules. For example, KV-PLM [Zeng et al., 2022b] first tokenizes both the SMILES strings and biochemical texts. Then, they randomly mask part of the tokens and pre-train the neural encoders to recover the masked tokens. Analogously, following the replace corrupted spans task of T5 [Raffel et al., 2020], MolT5 [Edwards et al., 2022] first masks some spans of abundant SMILES strings and biochemical text descriptions of molecules and then pre-train the transformer to predict the masked spans. In this way, these pre-trained models can generate both the SMILES strings and biochemical texts, which is especially effective for text-guided molecule generation and molecule captioning (generation of the descriptive texts for molecules). [Zhu et al., 2022b] propose to maximize the consistency between the embeddings of four molecular descriptors and their aggregated embedding using a contrastive objective. In this way, these various descriptors can collaborate with each other for molecular property prediction tasks. Additionally, MICER [Yi et al., 2022] adopts an autoencoderbased pre-training framework for molecular image captioning. Specifically, they feed molecular images to the pre-trained encoder and then decode the corresponding SMILES strings. The above-mentioned multimodal pre-training strategies can advance the translations between various modalities. Also, these modalities can work together to create a more complete knowledge base for various downstream tasks.

4 Applications

The following section takes drug discovery as a case study and showcases several promising applications of CPMs (Tab. 1).

4.1 Molecular Property Prediction (MPP)

The bioactivity of a new drug candidate is influenced by various factors in real life, including solubility in the gastrointestinal tract, intestinal membrane permeability, and intestinal/hepatic first-pass metabolism. However, such labels for molecules can be extremely scarce because wet-lab experiments are often laborious and expensive. CPMs offer a solution that can exploit the massive unlabeled molecules and serve as powerful backbones for downstream molecular property prediction tasks [Wang et al., 2022b; Zaidi et al., 2023]. Furthermore, compared with the models trained from scratch, CPMs can better extrapolate to out-of-distribution molecules, which is especially important when predicting the properties of newly synthesized drugs [Hu et al., 2020a].

4.2 Molecular Generation (MG)

Molecular generation, a long-standing challenge in computeraided drug design, has been revolutionized by machine learning methods, especially generative models, that narrow the search space and improve computational efficiency, making it possible to delve into the seemingly infinite drug-like chemical space [Du et al., 2022b]. CPMs, such as MolGPT [Bagal et al., 2021], which employs an autoregressive pre-training approach, has proven to be instrumental in generating valid, unique, and novel molecular structures. The emergence of multi-modal molecular pre-training techniques [Edwards et al., 2022; Zeng et al., 2022b] has further expanded the possibilities of molecular generation by enabling the transformation of descriptive text into molecular structures. Another crucial area where CPMs have demonstrated their prowess is the generation of three-dimensional molecular conformations, particularly for the prediction of protein-ligand binding poses. Unlike conventional approaches based on molecular dynamics or Markov chain Monte Carlo, which are often hindered by computational limitations, especially for larger molecules [Hawkins, 2017], CPMs based on 3D geometry [Zhu et al., 2022a; Zhou et al., 2023] exhibit remarkable superiority in conformation generation tasks, as they can capture some inherent relationships between 2D molecules and 3D conformations during the pre-training process.

4.3 Drug-Target Interactions (DTI)

Predictive analysis of Drug-Target Interactions (DTI) is a vital step in the early stages of drug discovery, as it helps to identify drug candidates with binding potential to specific protein targets. This is particularly important in drug repurposing, where the goal is to recycle approved drugs for a new disease, thereby reducing the need for further drug discovery and minimizing safety risks. Also, obtaining sufficient drug-target data for supervised training can be challenging. CPMs can overcome this issue by providing molecular encoders with good initializations. To achieve accurate DTI predictions, it is essential to consider both molecular encoders and target encoders, predict binding affinities, and co-train both for the DTI prediction task [Nguyen et al., 2021]. Previous works such as MPG [Li et al., 2021] follow these principles to advance DTI predictions.

4.4 Drug-Drug Interactions (DDI)

Accurately predicting Drug-Drug Interactions (DDI) is another crucial stage in drug discovery pipelines as such interactions can result in adverse reactions that can harm health and even cause death. Moreover, accurate DDI predictions can also assist in making informed medication recommendations, making it an essential part of the regulatory investigation prior to market approval. From the machine learning perspective, DDI prediction can be regarded as a classification task that determines the influence of combination drugs as synergistic, additive, or antagonistic. To achieve effective DDI prediction, expressive molecular representations are required, which can be obtained using CPMs. MPG [Li et al., 2021] is a representative example that has demonstrated the usefulness of CPMs by adopting DDI prediction as a downstream task.

5 Conclusions and Future Outlooks

In conclusion, this paper provides a comprehensive overview of Chemical Pre-trained Models. We start by reviewing the widely-used descriptors and encoders for molecules, then present the representative pre-training strategies and evaluate their advantages and disadvantages. We also showcase various successful applications of CPMs in drug discovery and development. Despite the fruitful progress, there are still several challenges that warrant further research in the future.

5.1 Improving Pre-training Architectures

While remarkable advancements have been achieved in analyzing the learning capabilities of neural architectures, such as the WL-test for GNNs, these analyses lack specificity in determining the optimal design for highly structured molecules. The ideal featurizations and architectures for CPMs remain elusive, as evidenced by conflicting results such as the negative impact of Graph Attention Networks (GATs) [Velickovic et al., 2018], which are widely adopted in graph learning, on downstream performance in previous studies [Hu et al., 2020a; Hou et al., 2022]. Furthermore, there is a pressing need to explore ways of seamlessly integrating message-passing techniques into transformers as a unified encoder to accommodate pre-training of large-scale molecular graphs. Additionally, as discussed in Sec. 3, the pre-training objectives still leave much room for improvement, with the efficient masking strategy of subcomponents in MCM being a prime example.

5.2 Building Reliable and Realistic Benchmarks

Despite the numerous studies conducted on CPMs, their experimental results can sometimes be unreliable due to the inconsistent evaluation settings employed (e.g., random seeds and dataset splits). For instance, on MoleculeNet [Wu et al., 2018] that contains several expensive datasets for molecular property prediction, the performance of the same model can vary significantly with different random seeds, possibly due to the relatively small scale of these molecular datasets. It is also crucial to establish more reliable and realistic benchmarks for CPMs that take out-of-distribution generalization into account. One solution is to evaluate CPMs through scaffold splitting, which involves splitting molecules based on their substructures. In reality, researchers must often apply CPMs trained from already known molecules to newly synthesized, unknown molecules that may differ greatly in properties and belong to divergent domains. In this regard, the recently established Therapeutics Data Commons (TDC) [Huang et al., 2021] offers a promising opportunity to fairly evaluate CPMs across a diverse range of therapeutic applications.

5.3 Broadening the Impact of CPMs

The ultimate goal of CPMs is to develop versatile molecular encoders that can be applied to various downstream tasks related to molecules. Compared to the progress of PLMs in the NLP community, there remains a substantial disparity between methodological advancements and practical applications of CPMs. On one hand, the representations produced by CPMs have not yet been extensively used to replace conventional molecular descriptors in chemistry, and these pre-trained models have not yet become standard tools for the community. On the other hand, there is limited exploration into how these models can benefit a more extensive range of downstream tasks beyond individual molecules, such as chemical reaction prediction, molecular similarity searches in virtual screening, retrosynthesis, chemical space exploration, and many others.

5.4 Establishing Theoretical Foundations

Even though CPMs have demonstrated impressive performance in various downstream tasks, a rigorous theoretical understanding of these models has been limited. This lack of underpinnings presents a hindrance to both the scientific community and industry stakeholders who seek to maximize the potential of these models. The theoretical foundations of CPMs must be established in order to fully comprehend their mechanisms and how they drive improved performance in various applications. For example, a recent empirical study [Sun et al., 2022] has questioned the superiority of certain selfsupervised graph pre-training strategies over non-pre-trained counterparts in some downstream tasks. Further research is necessary to gain a more robust understanding of the effectiveness of different molecular pre-training objectives, so as to provide guidance for optimal methodology design.

Contribution Statement

Jun Xia, Yanqiao Zhu, and Yuanqi Du contribute equally to this work. Stan Z. Li is the corresponding author.

Acknowledgements

This work was supported by the National Key R&D Program of China (2022ZD0115100), the National Natural Science Foundation of China (U21A20427), the Research Center for Industries of the Future (WU2022C043), and the Competitive Research Fund (WU2022A009) from the Westlake Center for Synthetic Biology and Integrated Bioengineering.