Zero-shot Learning for Preclinical Drug Screening

1 Introduction

Improving the efficiency of preclinical drug candidate screening is a long-standing core challenge in the field of drug discovery [Li et al., 2022]. It takes an average of 10 to 15 years and more than $2 billion for a new drug to reach the pharmacy shelf [Siqueira-Neto et al., 2023; Pandey et al., 2022; Berdigaliyev and Aljofan, 2020]. Historically, natural products have been the main source of new drug entities; in recent years, however, there has been a shift towards highthroughput screening (HTS) techniques [Sadybekov et al.,

∗Corresponding author

2022; Engels and Venkatarangan, 2001]. HTS drug screening methods can screen chemical libraries to identify the most promising compounds that have the desired effect on specific biological targets. Notably, the conventional libraries employed in HTS and virtual ligand screening [Irwin and Shoichet, 2016] (VLS) are constrained to less than 10 million accessible compounds, representing a mere fraction of the vast existing chemical space encompassing an estimated 1020 to 1060 novel compounds [Ertl, 2003]. This limitation of standard HTS and VLS decelerates the pace of drug discovery [Stein et al., 2020; Lyu et al., 2019], frequently yielding compounds with moderate affinity, limited selectivity, and initial hits displaying absorption, distribution, metabolism, excretion, and toxicity (ADMET) factors profiles [Wu et al., 2022] that necessitate extensive testing.

In recent years, driven by the rapid advancement of AI technology, virtual screening based on deep learning (DL) is poised to emerge as a swifter and more cost-effective approach to drug discovery [Sadybekov and Katritch, 2023;

Stärk et al., 2022]. Exhaustive catalogs of somatic mutations in various cancer types have been created [Forbes et al., 2017; Lawrence et al., 2014] and major oncogenic mutations identified [Stratton et al., 2009]. As a result, the establishment of cancer drug sensitivity databases, such as the Genomics of Drug Sensitivity in Cancer (GDSC) [Yang et al., 2012], has made a large amount of drug response data newly available to researchers. Many approaches leverage these databases to design and validate a diverse array of models that aim to elucidate connections between genomic data and drug responsiveness in the context of drug discovery [Chu et al., 2023; Nguyen et al., 2022].

Conventional drug response prediction (DRP) methods typically rely on supervised learning using labeled response data from the drugs, as depicted in Figure 1(a). Predicting the responses of unlabeled novel compounds constitutes a zero-shot learning challenge [Pourpanah et al., 2023] since these novel compounds must not have been encountered during training. Furthermore, the drug responses of novel compounds in practical preclinical drug screening applications are unknown. The distribution of data between known drug response data and that of novel compounds is often inconsistent, rendering supervised learning-based drug response prediction methods ineffective for novel compounds [Nguyen et al., 2022], as illustrated in Figure 1(b).

To underscore the shortcomings of supervised learning methods in practical preclinical drug screening, we test two state-of-the-art (SOTA) DRP methods (GraphDRP [Nguyen et al., 2022], GratransDRP [Chu et al., 2023]) on three drugs (5-Fluorouracil, Pelitinib, Alectinib), as shown in Table 1. These methods typically achieve correlation metrics exceeding 90% in a supervised learning experimental setup but exhibit a significant drop to only 20-40% correlation in a zeroshot learning scenario. The experimental results indicate that these methods are unable to accurately predict novel compounds in the practical application of preclinical drug screening. How, then, can the drug responses of previously unseen novel compounds be effectively predicted? This zero-shot learning problem has become the focus of DRP tasks in preclinical drug screening.

As shown in Figure 1(a), each drug can interact with numerous known cell lines, yielding corresponding response data. We define the collection of response data for a single drug across multiple cell lines as a single drug domain. The response data for one drug interacting with one cell line is recorded as one sample for that drug. DA aims to maximize the performance of an unknown domain (the target domain) by leveraging knowledge from known domains (source domains). This approach aligns well with the requirements of zero-shot learning in the context of the DRP task. This is because DA is an adaptive approach, proposed with the assumption that it allows the model access to the samples in the target domain [Bai et al., 2023]. This aligns with the conditions of zero-shot learning during the testing phase of the DRP task, wherein the DRP model must handle novel compounds without access to any prior samples.

In this paper, we propose a zero-shot learning solution for the DRP task in preclinical drug screening. Within this solution, we present the Multi-branch Multi-Source Domain Adaptation Test Enhancement Plug-in (MSDA), designed to enhance the effectiveness of DRP model predictions for novel compounds. During the inference phase, the MSDA identifies several drug domains as the source domains with the strongest correlation to the target domain. It guides the pre-trained model to acquire invariant features from these domains, facilitating adaptation to the target domain. The MSDA consists of two modules. The first module, a multi-source domain selector, employs the Wasserstein distance metric [Panaretos and Zemel, 2019] on drug features to identify the most relevant drug domains from a large training dataset, treating them as multi-source domains. Unlike conventional multi-source domain methods, the number of source domains in the MSDA is not the regular two or three, but several tens or hundreds. The second module is a multi-branch drug domain adaptor. This module comprises two distinct prediction branches: the first is the prediction branch of the pre-training model itself, and the second is the target domain adaptation branch responsible for transferring multi-source domains to the target domain. The latter aligns cell line types from various domains before computing the Maximum Mean Discrepancy (MMD) [Li et al., 2020].

To showcase the substantial performance improvements attained with the MSDA, we conduct comparative studies with other SOTA methods on the GDCSv2 and CellMiner [Reinhold et al., 2012] datasets. In the inference phase, the MSDA provides average improvements of 26.1%, 15.8%, 16.7%, 9.3%, 9.8% on GDSCv2 and 26.2%, 15.6%, 1.5%, 10.2%, 11.8% on CellMiner across four metrics for each of the five DRP methods, respectively. We further perform ablation studies on various aspects of the MSDA, including the strategy for the selection of source domains and the design of the target domain adaptation branch. Additionally, we conduct a series of hyperparameter experiments to investigate the influence of expanding the number of domain adaptation branches on performance. These experiments affirm that the MSDA plug-in in our solution is plug-and-play, highly versatile, significantly enhances generalization, and can potentially facilitate higher performance when given adequate computational resources. The significance of this zero-shot learning solution resides in its capacity to improve drug candidate evaluation and facilitate successful drug discovery.

2 RelatedWork

There are many research areas related to zero-shot learning, such as domain generalization, meta-learning, transfer learning, covariate shifting, and so on. Recently, Domain Adaptation (DA) has gained popularity in the machine learning field due to its demonstrated effectiveness in enhancing model prediction performance, especially when dealing with unlabeled test samples. This applicability extends to domains like computer vision [Zhou et al., 2023], as well as natural language processing [Han et al., 2023]. Nevertheless, there are still challenges associated with effectively enhancing the zeroshot learning capabilities of DRP models for preclinical drug screening through the utilization of DA methods. These challenges include:

1. Insufficient theoretical research on domain adaptation methods for regression tasks. Numerous effective DA methods have been devised for classification tasks [Fang et al., 2022; Liang et al., 2021]. To the best of our knowledge, however, there is a paucity of methods specifically tailored to regression tasks [Ou et al., 2023; Yang et al., 2022]. One perspective [Jiang et al., 2021] posits that domain alignment may widen the margins between classes in the target domain, thereby enhancing model generalization. However, it is essential to note that the regression space is typically continuous; this can be contrasted with the classification space, in which clear decision boundaries exist. The other perspective [Chen et al., 2021] asserts that classification is robust to feature scaling but regression is not, and that aligning the distributions of deep representations would alter the feature scale and impede domain adaptation regression. To summarize, it can be concluded that domain adaptation methods pose greater challenges when applied to regression tasks compared to classification tasks.
2. Numerous open-source domains, not limited source domains. Considering a single drug domain as the source domain for data mining results in a significant decrease in model generalization accuracy due to data availability limitations and the demands of practical application [Liu et al., 2018]. Each drug's response data should be treated as an independent drug domain, which results in an exceptionally large number of drug domains. Furthermore, the conventional single-domain adaptation method will fail when confronted with substantial distribution differences between the source and target domains [Ben-David et al., 2010]. Therefore, the efficient construction of source domains from a pool of over 20,000 known drug domains presents a challenge due to the presence of inter-domain shifts in these source domains [Peng et al., 2019].
3. Complex distribution patterns exist between the source and target domains. Conventional DA methods require the alignment of only one type of input (e.g., feature maps of images [Jaritz et al., 2022], sentence embeddings [Wang et al., 2022]). However, the input data for

the DRP task comprises two distinct types: drugs and cell lines. The types of cell lines in the source domains may not perfectly correspond to those in the target domain [He et al., 2022]. Additionally, the distributions of combined features from different drugs and cell lines are inconsistent. Hence, the effective alignment of complex source and target domains is a challenging task.

3 Methods

3.1 ProblemDefinition

In the DRP task, the input data is the drug d and the cell line c, and the output is the half maximal inhibitory concentration (IC50 scores) of this drug on one cell line. A drug domain D is defined as a set of response data for a specific drug d, across multiple cell lines, denoted as {c1, c2, . . . , cn}. In Figure 1(a), one column of the matrix corresponds to a drug domain D. The drug response datasets are divided into drug domains based on drug types. These drug domains are divided into a source domain (defined as S) and a target domain (defined as T ) according to the drug domain as the smallest unit, regarded as the training set and the test set, respectively. The source domain S is denoted as {(Di, Yi)}Ni=1 , and the target domain T is denoted as {(Dj , Yj)}Mj=1 . The set {(di, cj)}|Di| j=1 denotes the inputs of the drug domain Di, and Yi = {yj}|Di| j=1 is the corresponding IC50 scores. N is the size of the source domain, while M is the size of the target domain. In general, M is a finite number in our experimental setting; in the real world, M would be enormous.

In the field of drug discovery, the core requirement of the DRP task is as follows: the model trained on S can achieve high accuracy on T to verify that the model already has high generalization ability, which ultimately serves the practical application of drug discovery.

3.2 Drug and Cell Line Feature Extractors

The drug feature representation branch and the cell line gene feature representation branch share weights derived from pretrained DRP models, which are not incorporated into gradient calculations. This paper does not provide a detailed exposition of the computational methods employed in these two data representation branches. Broadly, we categorize feature extraction techniques for drugs and cell lines into Convolutional Neural Networks (CNNs), Multi-Layer Perceptrons (MLPs), Graph Neural Networks (GNNs), Transformers, and Transformer-based GNNs.

The input types for the drug extractor are the SMILES sequence or molecular graph. We uniformly describe the process of branching feature representation of a drug di as follows:

where, Φdrug denotes the drug extractor with shared weights, while Di is the representation of the drug di. Similarly, cj is the input of the cell line extractor which can be described as follows:

where, Φcell denotes the cell line extractor with shared weights, while Cj is the representation of the cell line cj, j ∈ [1, |Di|).

3.3 Multi-source Domain Selector

Due to the vast number of drugs with known responses, utilizing the complete set of drug domains as source domains when implementing the MSDA for domain adaptation presents significant computational and time-related challenges. Therefore, a practical approach to mitigate computational resource constraints and enhance inference speed involves the selection of partially effective drug domains as source domains.

The structure of the domain selector is illustrated in Figure 2(b). In the initial stage of the MSDA, we design the multi-source domain selector to select multiple drug domains that are similar to the target domain from the training dataset; these are defined as the source domains. The set of drugs in the target domain T is denoted by Dt = {di}Mi=1 . Next, for a drug di from Dt , the set of drugs Ds→t i from S are selected as shown below:

Ds→t i = TopK(Wij), where j ∈ [1,N], (3) where TopK(·) denotes the operation of sorting from smallest to largest order and fetching the first K elements. The Wasserstein distance Wij is a distance function defined between probability distributions on a given metric space, and its 1-order form is formulated as follows:

where Γ = Π(Dsj , Dti ) denotes the set of all joint distributions γ(x, y) whose marginals are respectively Dsj = Φdrug(dj)

and Dti = Φdrug(di); here, Φdrug(·) is the shared drug feature extractor referred to in Section 3.2. The distributions of various drug features exhibit significant dissimilarity, often with minimal overlap.

3.4 Multi-branch Drug Domain Adaptor

The multi-branch drug domain adaptor includes three functions: the fusion of drugs and cell line features, the adaptation from the multi-source domains to the target domain, and the predictions of drug responses in the target domain.

The inputs of the drug and cell line feature fusion branches are the outputs of the shared drug feature extractor Φdrug and the shared cell line feature extractor Φcell, respectively (refer to Section 3.2 for details). Specifically, this module has a fusion branch for multi-source domains and several fusion branches for the target domain. Denote the fusion branch for the target domain as the target domain adaptation branch. The general fusion branch can be expressed as follows:

where, Fs→t = [Ds→t ,Cs→t ], and Φfusion denotes the fusion branch that can be replaced by different methods. The fusion branch of the multi-source domains uses pre-trained model parameters and is not involved in the gradient computation.

For example, in the case of the target domain Dt i , each target domain fusion branch is constrained by three conditions, which are regression loss and ranking loss in multi-source domains Ds→t i and feature-invariant consistency based on the MMD distance between Ds→t i and Dt i . The objects of the feature-invariant constraint are FC(Fs→t i ) and FC(Fti ), which are activated by the fully connected layer FC(·) once. The formulas for these constraints are as follows:

where, MSELoss(·) denotes the L2 norm loss. The label value yi is subtracted from the model output (estimate) f (xi), after which the square is calculated to obtain the L2 norm loss, which is expressed as follows:

The ranking loss Lrank is computed using MarginRankingLoss (MRLoss). For data (x1, x2, r) containing N samples, x1 and x2 are the two inputs given to be ranked, and r represents the true ranking labels. The ranking loss Lrank is computed as shown below:

where margin denotes the margin value. If this value is larger, it means that the expectation x1 is further away from x2 (i.e. the margin is larger). The MMD is one of the most widely used loss functions in transfer learning, especially in domain adaptation, and is mainly used to measure the distance between two different but related distributions. The distance between two distributions is defined as follows:

where, ϕ(·) denotes a mapping from the original space to Hilbert space H. Hilbert space is the extension of Euclidean space that is no longer restricted to the finite-dimensional situation. The dimensions of the drug and cell line features encoded by the different methods are different. The fact that the calculation of the distance between two distributions is not limited by the dimension of the sampled features is one of the keys to the functioning of the MSDA as a general plugin. The distribution distances of invariant features from between multi-source domains Ds→t i and the target domain Dt i are constrained by MMD distances as follows:

where φalign(·) denotes the operation in which the feature vectors of two distributions are first fused through a fully connected layer, after which a union set is taken according to the cell line types, and eventually, the features are aligned according to their cell line types.

Finally, the overall loss of each target domain adaptation branch is obtained by weighted summation, as follows:

where, α ∈ [0, 1] and β ∈ [0, 1] are the weights of the loss function. The computation of the loss of each target domain adaptation branch, the back-propagation, and the updating of the gradient are independent and serial.

4 Experiments

4.1 Evaluation Strategies andMetrics

We evaluate the performance of the MSDA plug-in on two publicly available datasets: GDSCv2 [Yang et al., 2012] and CellMiner [Reinhold et al., 2012]. The drug domain is then randomly partitioned into source and target domains in an 8:2 ratio. The DRP methods without the MSDA are trained and validated on the source domains and tested on the target domains. The response data is clustered by drug type and then randomly partitioned into source and target domains, with the type of drug serving as the criterion for division. As illustrated in Figure 2(c), it should be noted here that a single drug domain represents the smallest unit of division.

To comprehensively evaluate the impact of the MSDA, we employed several evaluation metrics: Root Mean Square Error (RMSE) to gauge deviation, Pearson correlation coefficient (PCC) for assessing linear correlation, Spearman's Rank Correlation Coefficient (SRC) to measure monotonicity, and Margin Ranking Loss (Rank) to evaluate ranking performance.

4.2 Overall Experiment

Our proposed zero-shot learning solution aims to mitigate data limitations in drug response prediction, thereby enhancing the efficiency and accuracy of drug discovery and evaluation. Our goal is to validate the effectiveness of MSDA as a test-time enhancement plug-in for implementing this solution using various datasets and diverse DRP methods. In the overall experiment, we selected five publicly available DRP methods and utilized two drug response databases. The prediction results, including those that both do and do not integrate our proposed MSDA plug-in, are presented in Table 2. The prediction results of the original models are denoted by Original, while results after integrating MSDA are labeled +MSDA. The comparative analysis demonstrates that MSDA significantly enhances the performance of DRP methods in predicting responses for unknown drug domains. Across the drug response databases GDSCv2 and CellMiner, the average PCC metrics exhibit improvements of 22.4% and 1.3%, respectively, while the average RMSE metrics indicate enhancements of 31.7% and 18.2%, respectively. This indicates that MSDA can bolster the assessment of preclinical drug candidates by offering more dependable predictions, a crucial factor for progressing potential drugs through the development pipeline.

4.3 Ablation Study

In this study, we introduce the MSDA plug-in as a key component of our proposed solution, which aims to enhance the performance of drug response prediction in the context of zero-shot learning. This plug-in can be seamlessly integrated with publicly available DRP methods. The MSDA creates multiple target domain adaptation branches by replicating the fusion prediction branches from the pre-trained DRP model while preserving the parameters of the original model prediction branches. Finally, the MSDA combines the predictions from the target domain adaptation branches with those of the original branches, using summation and averaging, to generate the fine-tuning results specific to the target domain. Therefore, in the design of the structure of the MSDA, two crucial aspects necessitate verification through ablation experiments conducted on the GDSCv2 public dataset:

A1 Whether the average adaptive fine-tuning of multiple target domain adaptation branches is better than that of just one single branch for K drug domains as source domains filtered by the domain selector.

A2 Whether the results of the target domain adaptation branches n need to be merged with the results of the original branch predictions.

For A1, to investigate the structural performance disparities between multi-branching and single-branching within the MSDA framework while maintaining an equal number of source drug domains, we fixed the number of source domains K at 10 and 15, representing two and three times the original number of units (5), respectively. In the single-branching experiments, we set K to 10 and 15, respectively. In the multibranching experiments, the number of source domains in each branch is fixed at 5, while n is set to 2 and 3, respectively. Table 3 (B) and (C) show the results of the experiments with an overall number of source domains of 15 and 10, respectively. These results indicate that a rational division of the K domains into Kn sub-source domains, followed by fine-tuning using n target domain adaptation branches, is a more effective strategy.

For A2, we discuss the effectiveness of integrating the prediction branch of the original method with the target adaptation branch. The K is set to 70 and n is 2. We again select five representative DRP methods. As indicated in Table 3(A), the performance is enhanced when the original prediction branches of each DRP model are integrated. This is because, during the inference stage of domain adaptation, the fine-tuned branch lacks access to real results from the target domain, making it challenging to gauge the extent of domain adaptation adjustments. Integrating the domain adaptation branch with the original branch stabilizes the prediction results for the target domain. Furthermore, it is observed that when the original method's effectiveness (e.g., tCNNs) is notably subpar, the mandatory fusion of the original branch and the domain adaptation branch diminishes the improvements brought about by the MSDA.

4.4 Hyperparameter Experiment

The MSDA acts as a multi-branch multi-source domain adaptation test-time plug-in that can be integrated into general DRP methods, incorporating two critical hyperparameters:

• B1 The number of drug domains K selected by the domain selector while maintaining a fixed model structure.
• B2 The number of target domain adaptation branches n with the number of drug domains remaining fixed for each branch adapted to the target domain.

For B1, we perform a sensitivity analysis with K values, considering 14 parameters selected at uneven intervals from 0 to 50, utilizing the TransEDRP method within the GDSCv2 dataset. The trend in Figure 3(a) illustrates the substantial impact of varying the value of K on the four metrics. Beyond K = 20, improvements either remain stable or exhibit a decreasing trend. This indicates that if the similarity between drugs in the source domains and those in the target domains is relatively low, confusion may be introduced into the pertinent information.

For B2, we conduct three sets of experiments, each with a varying number of target domain adaptation branches (n ∈ [1, 2, 3]), while using five drug domains as the source domains on each branch. Additionally, a blank control group is included. The experimental results on GDSCv2 for five existing published methods in Figure 3 (b-f) illustrate that as the number of target domain adaptation branches increases, the performance improvement of the models becomes more substantial for a specific number of drug domains that can be learned within each target domain adaptation branch.

5 Conclusion

In this paper, we introduce a zero-shot learning solution tailored to address the DRP task within preclinical drug screening. Our proposed solution MSDA is seamlessly integrated into conventional DRP methods, enabling the learning of invariant features from previous response label information and thereby augmenting the model's real-time prediction capabilities. We conduct a series of experiments on GDSCv2 and CellMiner with the same division standard. Among the five most representative and high-performing DRP methods, the incorporation of the MSDA significantly enhances their predictive performance in the zero-shot learning scenario. To deal with the actual needs of the industry in real-world applications, the MSDA can adaptively select different parameters to balance the performance and inference speed requirements.

Acknowledgments

The work of Wenbin Hu was supported by the National Key Research and Development Program of China (2023YFC2705700). This work was supported in part by the Natural Science Foundation of China (No. 82174230), Artificial Intelligence Innovation Project of Wuhan Science and Technology Bureau (No. 2022010702040070), Natural Science Foundation of Shenzhen City (No. JCYJ20230807090211021).