A Semi-supervised Molecular Learning Framework for Activity Cliff Estimation

1 Introduction

The Similar-Structure, Similar-Property Principle (SSP) [Bender and Glen, 2004] has been a central premise in medicinal chemistry that similar molecules tend to exhibit similar properties, which can be physical (e.g., boiling points) or biological (e.g., activity). This fundamental assertion is validated by long experience suggesting that rules of thumb such as phenethylamines are likely to have activity in the central nervous system and that β-lactams frequently possess antibacterial activity [Martin et al., 2002]. Computational chemists have also exploited this premise in their analysis of the molecular diversity of compound libraries and the selection of compounds for high-throughput screening (HTS) [Johnson et al., 1989].

This concept of chemical similarity (or molecular similarity) rationalizes the thriving deployment of machine learning (ML) models in discovering and designing new drugs with expected functions. Noticeably, ML algorithms usually work by recognizing the statistically important patterns and assigning close values to neighbor samples in the highdimensional feature space [Mitchell and Mitchell, 1997]. They learn motifs that appear repeatedly in the training set, cluster molecules that carry the same motif, and make similar predictions for them [Wu et al., 2023b]. Therefore, SSP lays down a solid scientific foundation for and highly aligns with the inner operation logic of ML mechanisms.

Despite the triumph that ML has achieved in forecasting molecular properties, one particular exception to this principle holds great insights into the underlying structure-activity (or structure-property) relationships [Stumpfe et al., 2020]. This exception is constituted by activity cliffs [Maggiora, 2006], which are pairs of structurally similar molecules that exhibit a large difference in their biological activity. These outliers can impose a detrimental effect on ML models by misguiding them to seriously mispredict the activity of certain molecules, even with an overall high model predictivity

(see Figure 1). Although numerous studies have focused on defining activity cliffs and investigating their influence [van Tilborg et al., 2022], how to ameliorate this obstacle to the development of ML remains unexplored.

In particular, the quantity and quality of the data are one of the most pivot factors in the success of deep learning (DL) algorithms, but the evaluation of DL methods on activity cliff estimation is always in a low-data scenario. Meanwhile, the graph research community for molecule modeling has been trying to replicate the victory of self-supervised pretraining in NLP and claims substantial improvements by pretraining on large-scale datasets [Rong et al., 2020; Zhou et al., 2022]. Consequently, it provokes the question that "how much benefit self-supervised learning can bring to estimate activity cliffs?". As an answer, we observe a negligible reward gained by pretraining in many cases. Our first important finding is that self-supervised graph pretraining does not always have statistically significant advantages over non-pretraining methods with an average improvement ratio of merely 6.35%. Secondly, although several recent studies leverage the 3D spatial information of molecules during the pretraining stages [Fang et al., 2022], the incorporation of 3D geometry does not exhibit considerable advantages in recognizing molecular cliffs. Third, the design of the backbone GNNs has a larger impact on the accuracy of activity cliff estimation than the choice of pretraining mechanisms.

This phenomenon inspires us to explore a more efficient schema to employ the abundant unlabeled molecules and relieve the data scarcity in activity cliff identification. Towards this direction, we present a novel and effective semi-supervised learning (SSL) mechanism dubbed SemiMol, which leverages predictions on unannotated data as supervised signals for subsequent training. However, pseudolabels can be unreliable because labeled and unlabeled samples are usually drawn from different data distributions and DL models often struggle to detect this discrepancy. Besides that, existing pseudo-labeling approaches require probabilistic outputs to reveal the model's confidence and fail to be applied in regression tasks [Rizve et al., 2021; Wu et al., 2023a]. To surmount this barrier, we introduce an additional instructor model to evaluate the accuracy and trustworthiness of proxy labels. These confidence scores then instruct the target molecular model to selectively focus on different data points in a progressive way. In SemiMol, rather than using fixed thresholds or manually tuning thresholds with percentile scores [Cascante-Bonilla et al., 2021], we propose a selfadaptive criterion for curriculum learning to determine which subset of pseudo-labeled samples should be incorporated into the training set. With these delicately designed controllable learning paces, the target molecular model can take advantage of the unlabeled data assuredly without suffering from overreliance on labeled data and the severe inaccuracy of some pseudo-labels.

We conducted extensive experimentation and visualization on MoleculeACE [van Tilborg et al., 2022] to verify the effectiveness of our method. It can be found that our SemiMol significantly promotes the representation ability of graph-based DL models in low-data scenarios and it outperforms state-ofthe-art (SOTA) pretrained GNNs, descriptor-based ML models, and sequenced-based language models as well as prevailing SSL methods on all 30 activity cliff datasets. These results provide encouraging evidence that SemiMol can accurately capture adequate chemical and biological information to recognize cliffs in bioactivity. Our work positions itself in a broader movement within the graph-learning community for bioinformatics and aims to explore the optimal solution for low-data molecular property prediction systematically.

2 Method

2.1 Preliminaries and Background

Task formulation. Assume that there exists a set of accessible molecular data D⋆ = {(x⋆i , y⋆ i )}Ni=1 , which consists of labeled data points x⋆i . They can be in different formats such as 1D sequences, 2D graphs, and 3D structures, and y⋆ i can be any discrete (e.g., toxicity, and drug reaction) or continuous properties (e.g., water solubility, and free energy). Here we only consider the continuous property to align with the setting of the activity cliff estimation problem. Still, our approach can be easily extended to binary or multi-label circumstances. The target molecular model f : X → Y aims to capture the mapping of molecules and their properties. It can be any type of architecture such as Transformers, GNNs, or geometric neural networks. We also have another set of unseen data Dtest to evaluate the model's performance. Typically, D⋆ is divided into the training and validation sets, denoted as Dtrain = {(xtrain i , ytrain i )}N1 i=1 and Dval = {(xval i , yval i )}N2 i=1 , respectively. Furthermore, there is a set of unlabeled data points D∗ = {x∗i }Mi=1 , where the number of unlabeled data M is much larger than that of labeled data N (e.g., 100 or 10000 times larger).

Semi-supervised learning. SSL has attracted increasing interest in overcoming the need for large annotated datasets. Existing semi-supervised algorithms can be roughly separated into three sorts: consistency regularization, proxy-label methods, and generative models. The consistency regularization is based on the simple concept that randomness within the neural network (e.g., with dropout) or data augmentation transformations should not modify model predictions given the same input and imposes an auxiliary loss. This line of research includes π-model [Laine and Aila, 2016], temporal ensembling [Laine and Aila, 2016], and mean teachers [Tarvainen and Valpola, 2017] The proxy-label methods regard proxy labels on unlabeled data as targets and consist of two groups: self-training [Yarowsky, 1995], where the model itself produces the proxy labels, and multiview learning [Zhao et al., 2017], where proxy labels are produced by models trained on different views of the data. The generative models rely on variational autoencoders (VAE) [Ehsan Abbasnejad et al., 2017] and generative adversarial networks (GAN) [Odena, 2016] to capture the joint distribution p(y|x) more accurately.

2.2 Motivation of SemiMol

Proxy-labeling with confidence. Labeled data D⋆ and unlabeled data D∗ in most cases follow significantly different data distributions, i.e., P(x⋆i , y⋆ i ) ̸= P(x∗i , y∗ i ). However, on the one hand, many existing semi-supervised algorithms [Yarowsky, 1995; Laine and Aila, 2016; Sohn et al., 2020; Xie et al., 2020] assume that there is no distributional shift between D⋆ and D∗ . They are likely only to reinforce the consistent information in the labeled data D⋆ to unlabeled examples D∗ instead of mining auxiliary information from D∗ , without exception for proxy labeling [Lee and others, 2013]. On the other hand, despite the versatility and modality-agnostic of proxy labeling, it achieves relatively poor performance compared to recent semi-supervised algorithms [Rizve et al., 2021]. This arises because some pseudolabels ˆ y∗ i can be severely incorrect during training due to the poor generalization ability of classic DL models [Hendrycks et al., 2021]. If we directly utilize pseudo-labels that are predicted by a previously learned model for subsequent training, the conformance-biased information in the proceeding epochs could increase confidence in erroneous predictions and eventually lead to a vicious circle of error accumulation [Arazo et al., 2020]. The situation can be even worse when the labeled data D⋆ contain noises due to unavoidable experimental errors. Consequently, it is necessary to understand the quality and reliability of pseudo-annotations and intelligently select a subset to diminish the noise present in training [Rizve et al., 2021].

Confidence for regression tasks. Even though confidence is crucial for pseudo-label selection and confidence-based selection reduces pseudo-label error rates, the poor calibration of neural networks renders this solution insufficient. Explicitly, incorrect predictions in poorly calibrated networks can also have high confidence scores (i.e., ˆ y∗ i → 0 or ˆ y∗ i → 1) [Rizve et al., 2021]. More importantly, the majority of existing approaches such as uncertainty-aware pseudo-label selection (UPS) [Rizve et al., 2021] resort to the target model's output ˆ yi as the indicator of confidence. They tend to produce hard labels by y∗ i = 1 [ˆ y∗ i ≥ γ1] or y∗ i = 1 [ˆ y∗ i ≤ γ2], where γ1 and γ2 are pre-defined confidence thresholds. However, this selection mechanism becomes inapplicable if Y is a continuous label space because networks no longer output class probabilities. In other words, for regression tasks, ˆ y∗ i does not disclose any confidential information. Noticeably, a great number of biochemical problems are regression-based, including molecular property prediction [Wu et al., 2018], 3D structure prediction [Jumper et al., 2021], and binding affinity prediction [Wang et al., 2005]. This presents an unavoidable but essential challenge for probabilistic output-based proxylabeling algorithms [Rizve et al., 2021].

According to these two motivations, instead of depending on the output of the model ˆ y∗ i to judge the reliability of the proxy labels, we accompany the target molecular model f with an additional instructor model g. It plays the role of a critic and predicts label observability, i.e., whether the label is true or fake. The introduction of g disentangles the confidence prediction and the property prediction and can greatly reduce the noise introduced by the pseudo-labeling process.

2.3 Framework of SemiMol

Instructor-guided SSL. To circumvent the obstacle of over-reliance and reduce the noise of proxy labels, we present an instructor-guided semi-supervised framework, dubbed SemiMol (see Figure 2), to intelligently select a subset of pseudo-labels with less noise. SemiMol is made up of two components: one target molecular model f and one instructor model g. The former f is responsible for predicting the properties ˆ y. In contrast, the latter g plays a role in measuring the reliability p of supervised signals, which can be interpreted as the confidence probabilities of pseudo-labels. We separate the integral procedure of SemiMol into two phases. In the first step, we retain pseudo-labels {ˆ y∗ i }Mi=1 . There are several approaches to creating proxy labels, such as label propagation via neighborhood graphs [Iscen et al., 2019]. Here we follow [Lee and others, 2013] and require the molecular model f to directly annotate samples in the unlabeled dataset D∗ . Then in the following step, we construct a new dataset with both labeled and pseudo-labeled samples as D′ = D⋆ ∪ {(x∗i , ˆ y∗ i )}Mi=1 and proceed training the target molecular model f and the instructor model g based on this new set. These two operations are iteratively repeated until f reaches the optimal performance on the validation set Dval .

To be specific, the instructor model g : (X ×Y ×Hf) → P forecasts the confidence measure pi (0 ≤ p ≤ 1) of whether the given label y′i belongs to the ground-truth label set {y⋆ i }Ni=1 or the pseudo-label set {y∗ i }Mi=1 . It digests three items: the data sample with its label (x′i , y′i ) ∈ D′ and an additional loss term Hf(f(x′i ), y′i ), where Hf is traditionally selected as a root-mean-squared-error (RMSE) loss or a mean absolute error (MAE) loss for regression tasks and cross-entropy (CE) loss for classification problems. Here we regard Hf (.) as the ingredient of g's input to provide more information about the main molecular property prediction task. Finally, the instructor model g is supervised via a binary CE loss (BCE) as

where ci ∈ [0, 1] is an integer and represents the observability mask. It indicates whether yi is pseudo-labeled (ci = 0) or not (ci = 1).

Curriculum-based pseudo-labeling. Meanwhile, the target molecular model f receives discriminative information {pi}N+M i=1 from the instructor model g and uses it to efficiently select pseudo-labeled samples to backpropagate its gradient. Remarkably, the criterion leveraged to determine which subset of unlabeled samples D∗ to be incorporated into the training in each round is vital to the success of pseudolabeling. Various uncertainty metrics have been explored in the previous literature. For instance, several approaches choose instances with the highest-confidence [Zhu, 2005] or retrieve the nearest samples in the feature space [Shi et al., 2018]. Afterward, some studies adopt techniques of fixed thresholds, tuning thresholds manually, or using percentile scores [Cascante-Bonilla et al., 2021]. Here, we propose a self-adaptive curriculum learning algorithm to enable pseudoannotated samples to enter or leave the new training set.

Specifically, we first initialize a percentage threshold γ and use it to produce hard labels and constitute the hybrid set D′′ = D⋆ ∪ {(x∗i , ˆ y∗ i ) |p∗i ≥ γ} to train the target model f.

Then, as the training progresses, we adaptively adjust the confidence cut-off γ based on the model's performance in the validation set. If the model is assessed to be more powerful

Algorithm 1 SemiMol Algorithm

• 1: Input: target model f, instructor model g, labeled data D⋆ , unlabeled data D∗ , pseudo-label update frequency k, threshold percent γ, stepping threshold percent ∆γ
• 2: Initialize and pretrain a target model f0 and an instructor model g0
• 3: for epochs n = 0, 1, 2, ... do
• 4: if n mod k == 0 then
• 5: ˆ y∗ i ← f(x∗i ) ∀x∗i ∈ D∗ ▷ Iteratively assign pseudo-labels every k epochs
• 6: end if
• 7: D′ ←D⋆ ∪ {(x∗i , ˆ y∗ i )}Mi=1
• 8: ˆ y′i ← f(x′i ) ∀x′i ∈ D′
• 9: pi ← g(x′i , y′i ,Hf(.)) ∀(x′i , y′i ) ∈ D′ ▷ Predict the confidence scores
• 10: Lg (D′ , {ˆ y′i }N+M i=1 ) ← Equation 1
• 11: D′′ ←D⋆ ∪{(x∗i , ˆ y∗ i ) |p∗i ≥ γ} ▷Build the training set
• 12: Lf (D′′ ) ←Equation 2
• 13: Update the parameters of f and g based on Lg(.) and Lf (.)
• 14: Evaluate f on the validation set and attain performance s
• 15: if n > 0 and s < s′ then
• 16: γ ← γ - ∆γ ▷ Turn down the threshold automatically
• 17: end if
• 18: s′ ← s ▷ Record the validation metric of previous epoch
• 19: end for

(e.g., the evaluation metric like RMSE becomes smaller), we propose to lower the threshold as γ = γ -∆γ, where ∆γ is the stepping percent. This leads to a larger hybrid set D′′ and the loss of the target model f can be written as:

Our self-adaptive curriculum learning strategy discourages concept drift or confirmation bias since it can prevent erroneous annotations predicted by an undertrained network during the early stages of training to be accumulated [CascanteBonilla et al., 2021]. The pseudo-code of our SemiMol is depicted in Algorithm 1.

Tips for implementing SemiMol. Before executing SemiMol, it is natural to first obtain a well-trained molecular target model f0 through regular supervised learning on the labeled dataset D⋆ and then initialize an instructor model g0 by discriminating pseudo-labels generated by that f0. This is empirically proven to achieve higher training stability and robustness. Moreover, we choose to assign pseudo-labels every k epochs, where a proper setting of k is critical to the success of SemiMol. If pseudo-labels are updated too frequently, the training procedure tends to be volatile.

3 Experiments

3.1 Data and Experimental Setups

All our evaluations in this section are performed on datasets from MoleculeACE (Activity Cliff Estimation), which is an open-access benchmarking platform and available on Github https://github.com/molML/MoleculeACE. It contains more than 35,000 molecules over 30 macromolecular targets, where each target corresponds to a dataset. In particular, 12 of 30 datasets in MoleculeCAE have no more than 1K molecules in the training set, indicating a standard lowdata regime. In addition, we take Graph Isomorphic Network (GIN) [Xu et al., 2018] along with a Graph Multi-set Transformer (GMT) [Baek et al., 2021] to aggregate node features as the backbone. More experimental details and dataset statistics are elaborated on Appendix.

3.2 Does GNN Pretraining HelpMitigate the Activity Cliff Problem?

Data quantity and quality are always regarded as one of the most pivot factors in the success of DL algorithms. However, assessing DL methods on activity cliffs is in a low-data scenario. It is worth noting that the past few years have witnessed the prevailing of self-supervised learning across multiple areas [Devlin et al., 2018; Brown et al., 2020;

Dosovitskiy et al., 2020], and the graph research community in molecule modeling has been trying to replicate its success in natural language processing. Various works attempt to excavate information on large-scale unlabeled molecules [Rong et al., 2020; Fang et al., 2022; Wang et al., 2022; Zhou et al., 2022], and claim significant performance improvements [Rong et al., 2020; You et al., 2020; Zhu et al., 2021; Zhou et al., 2022]. Nevertheless, the efficacy of those pretraining frameworks is only validated on datasets of nonactivity cliff molecules and it remains unknown whether they are also powerful in alleviating the data scarcity trouble in activity cliff estimation.

Here, we investigate several SOTA pretraining GNNs, containing GROVE [Rong et al., 2020], MolCLR [Wang et al., 2022], and GEM [Fang et al., 2022], and examine their effectiveness in activity cliff datasets. To be explicit, GROVE adopts a Transformer-style [Vaswani et al., 2017] architecture, significantly enlarging the representation ability and application scope of molecular representation learning schemes. MolCLR applies data augmentation to molecular graphs at both node and graph levels and uses a contrastive learning strategy to generalize GNNs to a more giant chemical space. GEM proposes a geometry-based GNN with dedicated geometry-level self-supervised learning techniques to capture molecular geometry knowledge. Figure 3 exhibits the empirical results. It can be found that pretraining generally benefits the activity cliff estimation problem in all three selected pretraining frameworks. It leads to average improvements of 8.57%, 3.92%, and 6.54% for GROVE, MolCLR, and GEM on all 30 datasets. However, the improvements are not always convincing and can be negligible or even negative. For instance, GROVE performs worse with pretraining in two datasets, MolCLR becomes less competitive with pretraining in six datasets, and GEM is negatively influenced with pretraining in three datasets.

This phenomenon is consistent with the conclusion in one recent study [Sun, 2022] that self-supervised graph pretraining does not always behave statistically more powerfully over non-pretraining methods in many settings. Besides, the local and global molecular 3D structures may not be the key to tackling our problem since GEM has incorporated them for self-supervised learning but shows little benefit. Finally, we compare the performance of different GNNs before and after self-supervised learning. GROVE reaches an average RMSE of 1.3107 and 1.1871 before and after pretraining. MolCLR attains an average RMSE of 1.0556 and 1.0071 before and after pretraining. GEM realizes an average RMSE of 0.9699 and 0.9035 before and after pretraining. The design of GNNs' architecture is a more foundational factor in achieving a better performance in our activity cliff problem. GEM without pretraining can easily outpace pretrained MolCLR and pretrained GROVE.

Why is GNN pretraining not Beneficial? We fail to reproduce the gain of pretraining in conventional molecular datasets like MoleculeNet due to several potential reasons. Firstly, the target labels of most unsupervised pretraining are not appropriately aligned with or even contradictory to our activity cliff estimation task. For instance, GROVE [Rong et al., 2020] requires GNNs to decide whether a molecule contains a motif. However, these 85 sorts of pre-defined motifs might be deceptive in estimating the property of activity cliff molecules. Secondly, as suggested by [Sun, 2022], pretraining is found to typically help when high-quality hand-crafted features are absent. In our experiments, we inject rich features into GNNs, including the additional node features such as hydrogen acceptor match, acidic match, and bond features such as ring information. Last but not least, some pretraining methods, such as masked node label prediction, might be too easy to transfer enough knowledge for our activity cliff estimation task.

3.3 Does SemiMol Improve the Activity Cliff Estimation?

Overall performance. We compare SemiMol with diverse ML and DL algorithms on MoleculeACE and document the results of activity cliff molecules and all molecules in Figure 4 and Appendix, separately. It can be observed that our SemiMol outperforms other baselines with the lowest RMSE on 30 activity cliff datasets. More essentially, we discover an average improvement of 26.53% brought by SemiMol, which is significantly larger than the benefits of SOTA pretraining approaches. This phenomenon declares that GNNs empowered by SSL can be a more influential toolkit than existing algorithms to overcome the obstacle of activity cliffs and accelerate the process of new drug discovery.

Among all approaches, non-pretrained GNNs, including MPNN [Gilmer et al., 2017], GAT [Veliˇckovi´c et al., 2017], GCN [Kipf and Welling, 2016], and AFP [Baek et al., 2021], generally perform worse than other methods. Meanwhile, the performance of pretrained GNNs varies dramatically. For instance, GEM outweighs GROVE by a large margin. However, as analyzed in Section 3.2, the benefit of pretraining is proven limited. This difference is more related to the capability of the backbone GNNs than to the pretraining strategies. Additionally, sequence-based models such as LSTM [Hochreiter and Schmidhuber, 1997] and Transformer [Vaswani et al., 2017] are conspicuous for their outstanding performance. They are empirically a better choice for addressing the activity cliff problem than graph-based and descriptor-based models if no additional unlabeled data are provided in the finetuning stage.

Comparison with other semi-supervised methods. Diverse SSL algorithms have been invented to employ the abundant unlabeled resources in a task-specific manner. Approximately, they can be divided into three categories: consistency regularization, proxy-label methods, and generative models. Here, we pick up one baseline instance from each kind and investigate their improvements for molecular property prediction. Notably, random node dropping and edge perturbation are biochemically improper, and data augmentation may alter the semantics of molecular graphs [Sun et al., 2021]. Thus, we employ the π-model [Laine and Aila, 2016] for comparison, which applies different stochastic transformations (i.e., dropout) to the networks instead of the input graphs. Moreover, we select Semi-GAN [Odena, 2016], which introduces a discriminator to classify whether the input is labeled or not. As mentioned previously, UPS [Rizve et al., 2021] leverages the prediction uncertainty to guide the pseudo-label selection procedure and is therefore inapplicable in our situation. We select five datasets from MoleculeACE for comparison and use GIN as the backbone architecture. As shown in Table 1, SemiMol outperforms all other semi-supervised methods in overall RMSE. In addition to that, SemiMol overcomes the drawbacks of UPS and can be used for regression tasks. All the above evidence clarifies that SemiMol is a cuttingedge proxy labeling algorithm in comparison to existing SSL mechanisms with stronger generalization and broader applications.

Ablation study. We also implement an extra experiment to illustrate the effectiveness of our self-adaptive curriculum learning mechanism and report the results in Table 2. It can be found that our self-adaptive schedule achieves stronger performance in overall RMSE than a fixed threshold and percentile score-based curriculum learning paradigm [CascanteBonilla et al., 2021].

Performance for classification tasks. Notably, datasets in MoleculeACE are all regression tasks, and one major advantage of our SemiMol over other proxy-labeling algorithms is that SemiMol can deal with continuous property space. In spite of that, SemiMol can also be employed for classification tasks. In order to better demonstrate its superiority, we use a classification dataset, CYP3A4 [Rao et al., 2022], and examine its performance. CYP3A4 consists of the property cliffs in the Cytochrome P450 3A4 inhibitions experimentally measured by [Veith et al., 2009], including 3,626 active inhibitors/substrates and 5,496 inactive compounds. We report the corresponding performance of different approaches in Table 3, where the best performance is in bold and the second best is underlined. It can be observed that SemiMol outperforms all semi-supervised baselines, which rely on MPNN as the backbone architecture.

4 Conclusion

The past decade has witnessed the increasing employment of machine learning techniques for drug discovery, but the topic of activity cliffs (AC) receives little interest from the scientific community. In this work, we identify the best practice to enhance models' predictivity in the presence of activity cliffs and propose an effective semi-supervised learning (SSL) method named SemiMol. It introduces an instructor model to understand the confidence of proxy-label and exploits a self-adaptive curriculum training paradigm to move the target model toward hard samples more efficiently. We demonstrate that graph-based DL models can hold up against simpler ML algorithms for drug discovery even in low-data scenarios. We envision more efficient and better-suited SSL methods to propel the frontier of AC identification.