GLPocket: A Multi-Scale Representation Learning Approach for Protein Binding Site Prediction

1 Introduction

Binding sites on the surface of 3D proteins are usually in the form of deep grooves or tunnels to accommodate small molecule drugs or ligands. Binding sites are also regarded as binding pockets, or binding cavities . Given a binding site, small molecule drugs can be designed and bound to the protein, so as to change the characteristics and biological functions of the protein. Thus, accurate detection of binding sites is the premise of drug discovery and drug design [Anderson, 2003; Patani and LaVoie, 1996].

It is a very challenging problem to detect the binding sites. The binding sites on proteins are much smaller than proteins themselves. The protein surface are very bumpy and may lead to many false positive sites. Examples of protein-ligand pairs are given in Fig. 1. We quantitatively calculate the ratio of the largest diameter of ligand to the largest diameter of protein. From diameter ratios below each figure, we can see that the pocket where the ligand is located is much smaller than protein, since the length of some ligands is about 1/10 of the protein length. It is very difficult to accurately predict such small pockets for a given large protein structure.

Many methods have been proposed to tackle protein binding site detection problem. Traditional methods can be divided into three categories, i.e., geometric-based methods, template-based methods, and energy-based methods. Geometric-based methods, including Fpocket [Le Guilloux et al., 2009], Ligsite-series [Hendlich et al., 1997; Huang and Schroeder, 2006] and CriticalFinder [Nguyen et al., 2017], predict binding sites according to geometric characteristics of protein surface, and then sort the candidate sites according to their druggable ability. Template-based methods, such as FindSite [Brylinski and Skolnick, 2008], search for the most similar protein from a database, and then assign the binding site of the hit protein to the query protein. These methods require a large number of proteins in the database and known locations of the binding sites for each protein. Energy-based methods [Ravindranath and Sanner, 2016; Ngan et al., 2012; Hernandez et al., 2009] are to find ligands that require minimal interaction energy to bind to proteins. These methods require multiple matches and massive ligand templates.

In the past few years, as a tremendous amount of 3D protein structures become available, machine learning models have been developed to learn from the data to predict the site location with promising performance. Prank [Kriv´ak and Hoksza, 2015] is a typical machine learning approach. It firstly uses Fpocket and concavity [Chen et al., 2011] tools to assign and label pocket points according to physical and chemical properties. Random Forest is adopted to calculate the binding score of each pocket. Pockets with high scores are regarded as predicted sites.

In recent years, Deep Learning methods have been proposed for binding site detection with promising performance. DeepSite [Jim´enez et al., 2017] is the first work to adopt a 4layer Convolutional Neural Network (CNN) as feature extractor to predict the location of binding sites. Protein is firstly voxelized into multi-channel 3D grids, and then the 3D grids are divided into multiple subgrids, which are used as the input of the CNN to obtain ligandability scores. DeepSite clusters the scores to screen out the likely ligandability regions, which are combined as the predicted site. DeepSurf [Mylonas et al., 2021] takes local subgrids centered in each voxel point on the protein surface as network input to obtain ligandability scores. The scores are clustered to determine the final pockets. The classifications by DeepSite and DeepSurf are both made on local subgrids, not accurate enough to voxel level.

To do per-voxel classification, Kalasanty [StepniewskaDziubinska et al., 2020] is the first method to employ 3D UNet [Ronneberger et al., 2015] as backbone, with the whole protein as input to predict a 3D binary mask. DeepPocket [Aggarwal et al., 2021], RecurPocket [Li et al., 2022] and RefinePocket [Liu et al., 2023] take the local small-scale regions of a protein rather than the whole protein as input to predict pockets. They utilize Fpocket which predicts many candidate pockets with high recall but low precision, to generate candidate centers of each pocket. Then, they take subgrids centered in each candidate center as the input of 3D U-Netbased network for binary mask classification. The difference between RecurPocket (τ >1) and DeepPocket is that it builds a feedback link from decoder to encoder to predict pockets in a circular and progressive manner. Fig. 2 summarizes the network structures of the above mentioned methods.

Although the large-scale input can help Kalasanty to have a global view and do end-to-end binary mask predictions, it is difficult to focus on local regions of protein because the target is relatively very small, as shown in Fig. 1. As a result, it may lead to less discriminative features of small-scale regions and loss of details. On the contrary, DeepSite, DeepSurf, and DeepPocket take each small-scale subgrid of protein as input to do prediction, which brings the benefit that the networks are more inclined to notice small potential sites. However, this modeling doesn't consider the surrounding environment of the target and the global information of the protein structure. Thus the choice of the scale of the input to the model is a trade-off between the local attention and the global consideration. The larger the scale, the greater amount of neighborhood information, but making it difficult to concentrate on small regions. The smaller the scale, the more attention will be paid to potential small sites, resulting in insufficient auxiliary features of the neighborhood morphology to determine the location and shape of pocket.

The Least mean square error reconstruction (Lmser) [Xu, 1993; Xu, 2019] was initially proposed by folding the autoencoder along the center hidden layer. It establishes forward skip connections and feedback connections between the paired layers of the encoder and decoder. Later, it evolved into a 2D or 3D CNN-based Lmser structure and has been proven to be better than U-Net in various tasks [Li et al., 2019; Li et al., 2022].

In this paper, we propose GLPocket based on Lmser architecture to capture the multi-scale representations of proteins to predict binding sites. In the encoding stage, we take the whole protein as network input to extract global representations. In order to make the predictions more concentrated and precise, a Target Cropping Block (TCB) is developed to extract local patterns of interest for targeted prediction. In the decoding stage, to fully capture the local context semantic information for refined prediction, Transformer Block (TB) is utilized to establish dependency relationships of patches within small-scale regions. Along with the global information transmitted from encoder, predictions for local regions are further refined.

Our contributions can be summarised as follow:

• A multi-scale Lmser network structure was proposed for binding site prediction on the surfaces of 3D protein structures. It learns global and local representations for targeted prediction.
• We devised a Target Cropping Block (TCB) to capture local features of interest for accurate details of the binding pockets. Moreover, TCB greatly reduces volume of the feature maps to be calculated of 82% without introducing additional parameters. We also built a Transformer Block (TB) to establish dependence relationships of patches within the local region, enriching the context semantic information.
• We demonstrated the effectiveness of GLPocket on four benchmark datasets. GLPocket outperforms related state-of-the-art methods in terms of various metrics.

2 Materials andMethods

2.1 Data Preparation

In our experiments, we adopt five publicly available datasets to evaluate our method. We use scPDB [Desaphy et al., 2015] as training set, COACH420, HOLO4k, PDBbind [Wang et al., 2005], SC6K as testing sets. We ensure that there is no intersection between the training set and the test sets. The details are summarized as follows.

ScPDB (v2017) is one of the largest protein-ligand pairs datasets. It is widely used as the training set for binding site prediction tasks [Jim´enez et al., 2017; Mylonas et al., 2021; Stepniewska-Dziubinska et al., 2020; Aggarwal et al., 2021]. This dataset contains 16, 612 structures with 17, 594 binding sites. It also provides all-atom description of the proteinligand pairs and their binding sites. We divide the dataset into ten parts and use one of them as validation dataset. PDBbind (v2020) contains 5, 316 protein-ligand complexes. After removing 18 proteins which is too large to be loaded, and 1, 405 complexes appeared in the training set, the remaining 3, 893 complexes were used for testing. COACH420 and HOLO4K have been used as testing sets to evaluate the performance of P2Rank [Radoslav and David, 2018]. We removed the proteins with invalid cavities, leaving 207 and 2, 752 structures in COACH420 and HOLO4K, respectively. SC6K dataset was added to the PDB (Protein Data Bank) from January 1st, 2018, until February 28th, 2020, for binding site detection. We used 2, 378 proteins screened by DeepPocket as testset.

2.2 The Architecture of GLPocket

An overview of the proposed GLPocket network is shown in the Fig. 3(a). The architecture applies Lmser as backbone, including Encoder Module, Decoder Module, Target Cropping Block (TCB) and Transformer Block (TB). Encoder Module contains one Pre block and four encoder blocks (Ei, i ∈ [1, 2, 3, 4]). Decoder Module contains four decoder blocks (Di, i ∈ [1, 2, 3, 4]) and one Post block. The details of TCB and TB are shown in the Fig. 3(b) and 3(c). Network structure details can be seen in the Fig. 3(d).

Encoder Module

Given an voxelized protein data x ∈ RC×H×W×D , we firstly employ Pre block to extract preliminary characteristics of protein to prepare for subsequent encoder blocks. We utilize E1 to further extract compact protein features, then apply TCB to choose interested region to focus on target proposal. Finally, E2, E3, E4 are employed to extract proposal features to get g4. The process is formulated as below:

where ˜ f2 is the local proposal feature maps obtained from global feature maps. As a result, the proposal representation integrates the global structural information and neighborhood information outside the region without expanding its size. Actually, it reduces the volume of feature maps to 18% of the original size.

Target Cropping Block (TCB)

Although the encoder is able to encode the pocket location in a rough manner from a global perspective, it is difficult for the decoder to predict the boundary details of a relatively small pocket on the protein surface. So we propose a Target Cropping Block (TCB) to capture small and interested local regions from the global feature maps as pocket proposals.TCB ensures more neighborhood information to enhance proposal representation without increasing its size.

The details of TCB are shown in Fig. 3(b). Given protein feature maps fi (i ∈ [1, 2]), TCB cuts a cube centered on the candidate center (in yellow) from fi as a local proposal ˜ fi, and then passes ˜ fi to subsequent block to extract features more intensively. In order to keep scale invariance, we set a cropping ratio for feature maps with different sizes. In this work, the cropping ratio of ˜ fi to fi is 0.56 for width, height and depth respectively. TCB does not introduce additional network parameters, and it also performs a 82% reduction of volume of feature maps by selecting regions of interest.

As shown in Fig. 3(a), the input of multi-channel 3D protein grid x is processed by Pre and E1 blocks to get f1 and f2, later f2 is processed by a TCB, and then ˜ f2 is passed to next encoder block. Like vanilla U-Net structure, encoder blocks need to pass multi-scale information to the corresponding decoder blocks for aggregating features of different semantic layers. Here, we transfer the proposal feature maps rather than the global protein feature maps, in order to help decoder make accurate local predictions. So f1 and f2 are processed by TCB and then passed to corresponding decoder blocks, while f3 and f4 are directly passed to corresponding decoder blocks.

Decoder Module

Decoder Module is responsible for predicting pocket of part of a protein rather than the whole protein, which is more purposely focused on small binding site. The input of Decoder Module includes not only forward propagation representation but also information transmitted from each encoder block. Following Eq. 3 & 4, Di gradually predicts binding pocket by concatenating with the information passed from corresponding encoder blocks.

In addition, we utilize Transformer Block (TB) to model the long distance dependency relationship in proposal space after D4 according to Eq. 3. Prediction result p can be get from Post block following Eq. 4 and 5.

where p is the predicted result and represents the probability that each voxel belongs to binding pocket.

Transformer Block (TB)

When processing the global feature maps of protein, Transformer Block (TB) will fall into a dilemma of high computational complexity due to long token sequence. To solve this problem, we put TB after Encoder Module to process feature representation with small size in the hidden space, so that the complexity can be reduced. But in high-level space, proposal representation is too compact and abstract to establish the long-distance dependencies of patches. Therefore, as shown in Fig. 3(a), after the first decoder block (D4), where the resolution is increased, we apply TB for volumetric space and depth modeling.

The details of TB is shown in Fig. 3(a) and 3(c). Given the feature maps g3 ∈ RK×h×w×d , we flat and permute it into z ∈ RN×K as patch embeddings, where N is the number of tokens (N = h × w × d). Since spatial location relationship between different patches is the important information for comprehensive representation, we add learnable position embeddings (PE) to patch embeddings as follows:

where z0 ∈ RN×K is the feature embeddings and is used as the input of TB, cpe denotes position encoding.

The TB is composed of Transformer Layer repeated for L times. The yellow area in the Fig. 3(c) is Transformer Layer, which contains LayerNorm (LN), Multi-Head Attention (MHA), and Feed Forward Network (FFN). The result of each Transformer Layer zl (l ∈ [1, 2, 3, ...,L]) can be get through the following equation,

where zl is the output of l-th Transformer Layer. Finally, we reshape zL to get volumetric data ̂ g3.

2.3 Implementation Details

GLPocket is implemented in PyTorch and trained for 30 epochs with a batch size of 12 on 3 A100 GPUs. SGD optimizer was applied to train the model. The learning rate is set to 0.001 and remains the same. The binary cross entropy loss is employed to optimize our network.

2.4 EvaluationMetric

We use three metrics to evaluate the performance of models. The metrics include:

• DCA is the distance between the predicted pocket center and closest ligand atom. When distance is less than threshold, the prediction is considered to be correct, otherwise is wrong.
• DCC is the distance between the centers of predicted pocket and label. Prediction with DCC less than threshold is considered successful.
• DVO is the ratio of the overlap of predicted pocket and corresponding label to the union of their volumes.

In our experiments, we set threshold to 4 ˚A as in related works. Both DCA success rate and DCC success rate are the ratio of the number of successfully predicted pockets to the total number of pockets. Under the DCC metric, the DVO is calculated for the correctly predicted pockets, otherwise, the DVO is set to 0 for the unpredicted pockets.

3 Results

3.1 Ablation Study

To analyze the efficiency of two blocks, Target Cropping Block (TCB) and Transformer Block (TB), we perform several ablation experiments. Models are evaluated under four test datasets.

Target Cropping Block

Target Cropping Block (TCB) is to select interested local feature from global feature representation for targeted prediction. Specifically, in encoding stage, according to the given candidate center and crop ratio, the protein feature maps are cropped for targeted predictions. This facilitates subsequent decoder blocks to extract local features more finely.

To further analyze TCB, we visualized the middle feature maps of GLPocket with and without TCB, as shown in Fig. 4. We can see that GLPocket without TCB introduces more artifacts in the output. GLPocket with TCB has a cleaner and more concentrated output with clearer boundaries. We can observe that the protein regions of with TCB have higher activate values in f1 and f2 than those without TCB, which leads to the features of two regions (empty region and protein region) to be more discriminative than those without TCB. This is mainly because GLPocket without TCB has limited local view. It is difficult to distinguish between proteins and empty areas. This problem is solved by TCB by selecting interested features from the global representation.

Furthermore, comparing f3 of GLPocket with and without TCB, we can see that although they are the feature maps extracted from the same region of a protein, the prediction of with TCB is closer to the label than that without TCB. This is because the proposal integrates the global distribution information, making GLPocket with TCB more accurate to predict the shape of the pocket.

We also calculate the DCC and DVO of GLPocket with and without TCB, as shown in Tab. 1. Comparing the first two rows, TCB further improves prediction performance of model without increasing additional parameters.

Transformer Block

We also conduct comparative experiments to verify the effectiveness of Transformer Block (TB). TB aims to capture the spatial dependency within proposal to enrich local context semantic information.

The quantitative results are presented in Tab. 1. Comparing the first and third rows, GLPocket with TB, with only a few network parameters added, shows great superiority in both DCC and DVO metrics with significant improvements. This is mainly because the dependency between patches within proposal can be captured early in the decoding phase by using attention mechanism of TB. The subsequent decoder blocks can perform step-by-step targeted refinement on the local proposal based on the enhanced features. The result clearly reveals the benefits of TB to model semantic relationships of small-scale representation.

As shown in the last row of Tab. 1, GLPocket with TCB obtains further improvements in DCC and DVO under four test datasets. This is mainly because the network not only adopts TCB for global and local information modeling but also uses TB to learn the semantic correlation between patches within the proposals. Since PDBbind dataset has various compounds pairs, like protein-nucleic acid pairs etc., where the sites are relatively large. TCB in GLPocket is to remove surrounding information interference and do local refined prediction, paying more attention on the small-site detection. However, detecting larger sites needs a model with more consideration of neighborhood information in some extent. So the GLPocket without TCB delivers a superior performance on PDBbind.

3.2 Comparison to State-of-the-art Methods

We make a quantitative comparison of GLPocket with the state-of-the-art (SOTA) methods on four different test sets, as shown in Tab. 2. Note that Kalasanty [StepniewskaDziubinska et al., 2020], DeepPocket [Aggarwal et al., 2021] and RecurPocket [Li et al., 2022] are the latest work to do site prediction tasks. It is found that GLPocket achieves the best performance for both DCC and DVO values. Kalasanty takes the whole protein as input, which makes it difficult to focus on small segmentation targets. DeepPocket and RecurPocket pay more attention to local proposals so their performances are better. RecurPocket builds feedback links from decoder to encoder to guide the representation learning in a recurrent and progressive manner, so it works better than DeepPocket. GLPocket takes both the global and local information into account with an appropriate trade-off, and establishes the semantic relationship between smaller patch blocks within the local proposal, so it achieves the best results.

We also report DCA on these test sets in Tab. 3. DCA Top-n mainly measures two aspects, one is the sorting ability of the prediction results, and the other is location ability for pocket. Top-n is the success rate of the first n predictions according to the priority of prediction results, where n is the number of annotated pockets for a given protein. Top-(n+2) is the same. Kalasanty, DeepPocket and RecurPocket are the latest best models in DCA [Aggarwal et al., 2021], and they have different ways of ranking predictions. Kalasanty sorts prediction results according to the segmentation density. It selects the first n and n + 2 to do DCA test. DeepPocket trains a classification network and a segmentation network. The classification network is dedicated to scoring and ranking pockets predicted by Fpocket. Our work mainly solves the location and shape prediction of pockets. For a fair comparison, as in RecurPocket, models are tested in two steps. First, according to the ranking results of DeepPocket classification network, the first n and n + 2 candidate pockets are selected to calculate candidate centers. DeepPocket, RecurPocket and GLPocket take the candidate centers as priori information of segmentation network, to predict pockets which are used to calculate DCA.

From Tab. 3 we see that GLPocket outperforms previous related works on four datasets with good generalization performance. The performance of GLPocket is 2.49% higher than RecurPocket on Top-n under COACH420 testset. Such improvements demonstrate the effectiveness of GLPocket, especially on this challenging task with such small pockets. GLPocket is slightly inferior to RecurPocket on PDBbind, but it is comparable with RecurPocket, indicating the efficiency of our methods.

We visualize examples of predictions in Fig. 5. The predictions by Kalasanty are often incomplete, discontinuous, and even far out-of-bounds, while those by DeepPocket and RecurPocket are more accurate and smooth. In general, the results by GLPocket are the closest to the real pockets.

4 Discussion and Conclusion

In this paper, we proposed a Lmser-based network structure, called GLPocket, to capture the multi-scale representations of objects to predict binding sites. We devised Target Cropping Block (TCB) to select interested local features from the global representation for targeted prediction. TCB makes the local region representation more discriminative by integrating the global distribution information, which helps to make the prediction more concentrated and precise. TCB also greatly reduces the volume of feature maps to be calculated by 82%, without introducing additional parameters. To fully capture the spatial dependency within local regions, we utilize Transformer Block (TB) to establish dependency relationship within proposal for enriching local context semantic information. Experiments show that GLPocket outperforms the state-of-the-art methods with a few parameters increase. This method can be extended to other detection or segmentation models easily, especially to detect or segment small targets from large-scale samples.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (grants No. 62172273), and Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102).