Improving Paratope and Epitope Prediction by Multi-Modal Contrastive Learning and Interaction Informativeness Estimation

1 Introduction

An antigen (Ag) is a typical protein that elicits an immune response in the human body, while antibodies (Ab) are flexible Y-shaped proteins [Graves et al., 2020]. The antigen interacts with the antibody and then initiates immune responses. As shown in Figure 1, these binding residues on antibodies and antigens are known as paratopes and epitopes [Garc´ıaS´anchez et al., 2023], respectively. The identifying epitopes help to construct epitope-based antibodies while confirming paratopes allows precise mutation of binding regions, optimizing properties like affinity [Rangel et al., 2022]. Traditionally, researchers employed wet experiments such as X-ray crystallography, nuclear magnetic resonance spectroscopy, and cryoelectron microscopy for identifying paratopes and epitopes [Schotte et al., 2003; Bax and Grzesiek, 1993; Zhou, 2008]. However, these approaches are often costly, time-consuming, and struggle to cope with the scale of rapidly emerging high-throughput data [Dutta et al., 2015].

Due to the special structure of antibodies, CDR regions are more prone to conformational changes, resulting in more complex interaction patterns, and the general methods for proteins cannot perform well on paratope and epitope prediction. In recent years, plenty of computational methods have been developed for paratope and epitope prediction. These methods are roughly classified into three categories: machine learning-based methods, deep learning-based methods, and pre-trained language model-based methods. The machine learning-based methods [Olimpieri et al., 2013; Daberdaku and Ferrari, 2018] extracted the hand-crafted features of antibodies and antigens, and fed the features into classifiers (e.g., SVM, random forest) to predict paratopes and epitopes. The deep learning-based methods [Deac et al., 2019; Xu and Zhao, 2022] utilize multi-layer neural networks, including convolutional neural networks (CNNs), and equivariant graph neural networks (EGNNs), to extract representations from input data. Currently, pre-trained language models (PLMs) like ESM-1b, and ProtTrans are trained on extensive protein sequences across various downstream tasks, including the paratope and epitope prediction [Clifford et al., 2022].

The existing methods have made great progress in paratope and epitope prediction. However, given the specificity of antibodies, the prediction of paratopes and epitopes still faces many challenges. In particular, the scarcity of labeled paratopes and epitopes is always a problem to be solved, posing a challenge in extracting abundant information from limited data. Additionally, paratope and epitope prediction contains three tasks: single paratope/epitope prediction, and joint paratope-epitope prediction. However, existing methods usually focus on one of them and do not consider all three tasks.

Multi-modal information from sequences and structures could provide richer complementary knowledge on limited data. Effectively leveraging the multi-modal information of antibodies and antigens may improve paratope and epitope prediction, but still remains challenging. Antibodies (antigens) have multiple paratopes (epitopes), leading to potential binding with different counterparts. However, existing methods extract representations solely from antibodies (antigens) to predict paratopes (epitopes), but they fail to exploit the information on how epitopes are specifically bound to paratopes, especially in single paratope/epitope prediction. Therefore, incorporating the antibody-antigen interaction information into models could further enhance the performances of paratope and epitope prediction.

To address these challenges, we propose a novel Multimodal contrastive learning and Interaction informativeness estimation-based method for Paratope and Epitope prediction, named MIPE. Specifically, we introduce multi-modal contrastive learning (CL) to effectively integrate the modal information from sequences and structures. Within each modality, we deploy the intra-CL to maximize representations of binding and non-binding residues; between different modalities, we deploy the inter-CL to align uni-modal representations towards effective modal representations, reducing the impact of noise. Furthermore, we devise a knowledgedriven module named interaction informativeness estimation, which utilizes multi-head attention layers to compute the attention matrices for antibodies and antigens, aiming to capture the specific interaction patterns between them.

In summary, the main contributions of this paper are described as follows:

• We introduce a multi-modal contrastive learning strategy that learns the representation of paratopes and epitopes by applying intra-CL and inter-CL to multi-modality sequence and structure data.
• We devise an interaction informativeness estimation to approximate the estimated matrix to the actual antibodyantigen interaction map, which guides the model to learn the specific patterns of the antibody-antigen interactions.
• Extensive experiments show that MIPE achieves competitive performance in paratope and epitope prediction, and also results in promising performance even utilizing AlphaFold2-generated structures as substitutes for real structures which are unavailable.

2 RelatedWork

For the paratope and epitope prediction, existing computational methods focus on three tasks: single paratope prediction, single epitope prediction, and joint paratope-epitope prediction. For instance, Parapred [Liberis et al., 2018] employs the CNN and BiLSTM networks and integrates features from both the local residue domain and the entire antibody sequence for paratope prediction. Paragraph [Chinery et al., 2022] utilizes EGNNs to extract 3D representations of the antibody's CDR regions for paratope prediction. For single epitope prediction, NetBCE [Xu and Zhao, 2022] extracts sequence features of antigens and employs BiLSTMs and CNNs to learn the representations to predict epitopes. BepiPred-3.0 [Clifford et al., 2022] leverages ESM-2 to obtain sequence representation of antigens and subsequently employs various classifiers (e.g., FFNN and CNN) for epitope prediction. For joint paratope-epitope prediction, [Honda et al., 2020] proposed a cross-attention network to encode both the antigen and antibody sequences. Epi-EPMP [Del Vecchio et al., 2021] designed a graph attention neural network, incorporating the encoding of the structural graphs of both the antibody and antigen. In general, an ideal prediction method is deemed to perform well on all three tasks.

Although existing methods produce good performances, they typically concentrate on uni-modal data (sequences or structures) while overlooking the information across multimodal data. Studies have shown that multi-modal data can offer complementary information [Rahate et al., 2022], and such multi-modal based approaches have been successfully applied in various fields, e.g., drug-target interaction and protein-protein interaction [Zhang and Kabuka, 2019; Xia et al., 2023; Wang et al., 2024]. To fill the gap, we propose a multi-modal contrastive learning strategy for the sequence-structure modality fusion to learn better representation for the paratope and epitope prediction.

As mentioned above, the antibody-antigen interactions reflect how paratopes and epitopes are bound, which could benefit the paratope and epitope prediction, but such explicit interaction information hidden in the known antigenantibody complexes has not been exploited in the previous studies [Honda et al., 2020; Del Vecchio et al., 2021]. Moreover, the lack of explicit binding information from structures may undermine the performance of the prediction model and also hinder the model from proficiently analyzing the interaction patterns. To address this issue, we present an interaction informativeness estimation strategy to facilitate the model's learning of interaction patterns, guided by the actual antibody-antigen interaction map.

3 Methodology

In this section, we first formulate the problem of paratope and epitope prediction. Subsequently, we introduce the components of the proposed method MIPE, which includes multimodal encoders, multi-modal contrastive learning, and interaction informativeness estimation, as illustrated in Figure 2.

3.1 ProblemFormulation

An antibody or antigen in the antibody-antigen complex can be represented as a sequence-structure tuple (S, C). S = [si]Mi=1 stands for the residue arrangements where si is the type of residue i, and C = [ci]Mi=1 stands for the backbone coordinates where ci is a Cartesian coordinate of the Cα atom in residue i. Binding residues are determined by identifying residues within a Euclidean distance threshold between the antibody and the antigen, and paratopes/epitopes refer to binding residues on the antibody/antigen.

Given a set of known antibody-antigen complexes, we can obtain the dataset containing sequence and structure data of antibodies and antigens, the annotated paratopes and epitopes, and the interaction map. The objective of our study is to train a model using the dataset and then apply it to predict paratopes and epitopes of new antibodies and antigens.

3.2 Multi-Modal Encoders

To build our model, the first step is to characterize the sequences and structures of antibodies and antigens, by using a sequence encoder and a structure encoder.

For sequences, we initialize them as residue-type features (one-hot encoding with 20 dimensions): xs ∈ RM×20 . Then, we implement the sequence-based encoder to obtain the embedding from xs , formalized as follows:

where Fs consists of a pre-trained network[Rives et al., 2021; Olsen et al., 2022], a dilated convolutional neural network[Yu and Koltun, 2016], and a recurrent network[Huang et al., 2015] consecutively.

For structures, we build the structure encoder as follows. First, we construct a 3D graph based on residue coordinates and add three different types of edges to the graph. If the sequential distance between the i-th residue and the j-th residue is below a predefined threshold dseq, the edge between these two residues is considered a sequential edge. If the Euclidean distance between two residues is smaller than a threshold dradius, the edge between them is regarded as the radius edge. To ensure the density of spatial edges is comparable among different graphs, for each residue, si, the k residues with the smallest Euclidean distances to it are considered its K-nearest neighbors, and the edges between them are considered as K-nearest neighbor edges. Herein, we set dseq = 3, dradius = 10 ˚A, K = 8. For each node in the graph, we construct an initial 62-dimensional physicochemical feature: xc ∈ RM×62 (see details in Appendix 1). Following [Zhang et al., 2022], we employ the equivariant graph convolutional layers to learn and update the graph:

where l denotes the number of layers. h = {hc1 , ..., hcM } represents the set of node embeddings, which are initialized with xc . s = {s1, ..., sM} is the set of residue coordinates. E = (eij), eij denotes the feature of the edge between node i and j. Finally, we apply an MLP over the updated embeddings of all nodes to produce the output embeddings for the residues of antibodies and antigens.

3.3 Multi-Modal Contrastive Learning

After encoding sequences and structures, it becomes crucial to fuse their embeddings for the paratope and epitope prediction. Mutual information (MI) offers an effective measure for assessing nonlinear dependencies between variables, which compels the alignment of modal data and facilitates the sharing of crucial information [Liu et al., 2023]. Along this line, we bring in contrastive learning (CL) to align sequences and structures by maximizing their MI in the embedding space [Zhu et al., 2023]. Specifically, we devise the multi-modal CL strategies with intra-CL and inter-CL, as follows:

Intra-CL. To better discriminate residues between binding and non-binding in both sequences and structures, it is desirable for the binding residues to cluster together while being distant from non-binding ones in the embedding space. To do so, in a given antibody/antigen with length M, we define the binding residue i as the anchor, another binding residue j as the positive instance, and the non-binding residue k is regarded as a negative instance. Then, we maximize the MI between binding residues while minimizing the MI between the binding and non-binding residues at the sequence and structure levels. The objective of intra-CL is to minimize the following InfoNCE-based [Chen et al., 2020] loss function:

where hsi /hci represent the sequence/structure representation of residue i (refers to the anchor), j means the residue indices consistent with the label of residue i; 1[k̸=i] ∈ {0, 1} is an indicator function evaluating to 1 if k ̸= i; E(·, ·) is the cosine similarity function with the temperature coefficient to measure the MI score between two variables.

Inter-CL. To acquire dependable multi-modal representations of antibodies/antigens, it is preferable to reduce the prediction bias in different modalities and strengthen their consistency. Inspired by [Zou et al., 2023], we bring in an inter-CL to encourage modalities with both correct uni-modal predictions to share a stronger correspondence. Specifically, for the learned sequence and structure representations of residues, we utilize two pre-trained probabilistic discriminators for calculating their pseudo labels:

where f s /f c is the sequence-based/structure-based discriminator (see details in Appendix 1), psi /pci stands for the obtained pseudo labels of residue i.

If the obtained labels of both sequence and structure (psi , pci ) are consistent with the original labels, corresponding hsi and hci are regarded as positive pairs, forming the set of all positive pairs (denoted as P). If the obtained labels of both sequence and structure are inconsistent with the original labels, they are defined as negative pairs, denoted as N. Additionally, when only one of the predicted binding labels is consistent with its original label, these two modal representations are defined as a semi-positive pair S. The objective is to minimize the following loss function:

where B = P ∪ S ∪ N. In addition to N, different modal embeddings of distinct residues need to separate their representations; therefore, when i ̸= j, they are also considered negative pairs.

Above inter-CL can optimize the similarity of positive and semi-positive pairs toward a higher value while optimizing the similarity of negative pairs toward a smaller value. The reason behind this design is twofold: (1) for samples with incorrect predictions in both modalities, we aim to enhance the dissimilarity between their uni-modal representations to obtain more complementary representations; (2) for samples with mutually exclusive predictions, we encourage the alignment of ineffective modality with the effective one by leveraging uni-modal predictions as supervision signals.

Furthermore, we fuse the sequence and structure representations through point-wise addition:

3.4 Interaction Informativeness Estimation

How epitopes are specifically bound to paratopes is a crucial question overlooked by existing research. Therefore, incorporating the interaction between antibodies and antigens could further enhance the performances of paratope and epitope prediction. Here, we design an interaction informativeness estimation (IIE) to incorporate the interaction maps into the model prediction.

First, we deploy the transformer encoder to estimate interaction maps between antibodies and antigens from the learned representations [Vaswani et al., 2017]. The estimated matrix A can be defined by the following cross-attention layer:

where the antibody (or antigen) representations of sequence and structure are integrated as the query Q = hab (or Q = hag), the antigen (or antibody) representations serve as the key and value K = V = hag (or K = V = hab), d denotes the representation dimension of residues. After that, we can obtain the antibody-oriented and antigen-oriented attention matrices: Aab ∈ RM×N and Aag ∈ RN×M , where Aab(i, j)/Aag(i, j) represents the binding score between the i-th residue on the antibody/antigen and the j-th residue on the antigen/antibody.

Both Aab and Aag encapsulate the estimated antibodyantigen interaction matrix. Our objective is to approximate these two estimated matrices towards the actual antibodyantigen interaction map, facilitating the model in capturing specific interaction patterns and thereby enhancing overall performance. This process enables the model to learn and represent the complexities inherent in antibody-antigen interactions more effectively. With the actual antibody-antigen interactionmap ˆ A ∈ RM×N , we minimize their distance by the following BCE loss:

3.5 Prediction andModel Training

For paratope and epitope prediction, the embedding of each residue on the antibody and antigen (hab and hag) derived from the transformer encoders are individually inputted into MLP-based predictors Fab and Fag. These predictors assess the probability of a residue being the paratope/epitope:

The supervised loss of the paratope prediction can be formulated as follows:

where yabi indicates the ground-truth label of the i-th residue of the antibody. Similarly, the loss for the epitope prediction is defined as Lag.

By combining the loss functions for Intra-CL, Inter-CL, IIE, the paratope prediction, and the epitope prediction, the overall objective function for model training is defined as:

where α, β, γ represents a hyperparameter to balance the contributions of different tasks.

4 Experiments

4.1 Experimental Settings

Datasets. From the publicly accessible Structural Antibody Database (SAbDab), we collected a total of 7571 antibodyantigen complexes, with the sequence data in FASTA format and structural data in PDB format. Following previous studies [Pittala and Bailey-Kellogg, 2020], we used CD-HIT [Li and Godzik, 2006] to remove high-homology antibody and antigen sequences with the thresholds of 95% and 90% sequence identity, respectively. Subsequently, we excluded antibodies and antigens with any residue type rather than 20 naturally occurring types. Finally, we compiled a dataset consisting of 626 binding antibody-antigen pairs, including their sequences, structures, and corresponding interaction maps. Noteworthy, antibodies primarily bind to antigens through their CDR regions. Most researchers use Euclidean distance to define paratopes and epitopes, and we follow the usual way in our dataset: within the CDR regions/antigen, a residue is labeled as a paratope/epitope if the Euclidean distance between its backbone atom and any backbone atom on the other antigen/CDR regions is less than 4.5 ˚A.

Baselines. We compared our MIPE with the following methods for the paratope and epitope prediction at three different tasks. For the single paratope prediction, the Parapred [Liberis et al., 2018], AG-Fast-Parapred [Deac et al., 2019], PECAN [Pittala and Bailey-Kellogg, 2020] and Paragraph [Chinery et al., 2022] are listed as the baselines. For the single epitope prediction, we took BepiPred-3.0 [Clifford et al., 2022], NetBCE [Xu and Zhao, 2022] and PECAN [Pittala and Bailey-Kellogg, 2020] as the baselines. PesTo [Krapp et al., 2023] is a method used to predict protein binding sites and is applicable to both the single prediction task for paratopes and epitopes. Additionally, for the joint paratope-epitope prediction, we leveraged the [Honda et al., 2020] and Epi-EPMP [Del Vecchio et al., 2021] as the compared methods.

Evaluationmetrics. We randomly split the dataset, allocating 90% binding antibody-antigen pairs for cross-validation (CV) and reserving the remaining 10% for an independent test set. For the CV set, we implemented the 5-fold CV. Furthermore, we performed independent testing, in which the model was trained on the CV set and evaluated on the separate and previously untouched independent test set. Following previous studies, we employed three evaluation metrics to evaluate the performance of paratope and epitope prediction. Due to the class imbalance in binding and nonbinding residues, the area under the precision-recall curve (AUPR) is the primary metric for measuring the predictive performance. We also reported the metrics, such as the area under the receiver operating characteristic curve (AUC) and the Matthews correlation coefficient (MCC), consistent with prior research, to facilitate the comparisons. Notably, the reported values are aggregated across five random seeds. Details about the datasets, baselines, metrics, and implementations can be found in Appendix 2. Our code, data, and appendix are available on GitHub (https://github.com/ WangZhiwei9/MIPE)

4.2 Performance Comparison

Table 1 shows the performances of MIPE and baselines in three tasks of the paratope and epitope prediction. To implement different tasks, we simultaneously train these components and two distinct predictors for joint paratope-epitope prediction, while separately training these components and a predictor for single paratope/epitope prediction. The results indicate the following observation: (1) For the single paratope prediction, our MIPE outperforms these baselines by an average of up to 7.1% in terms of AUC and 2.0%-12.9% in AUPR and produces slightly lower MCC than PECAN. (2) For the single epitope prediction, we observed a decline in overall performance compared to paratope prediction. Indeed, compared to antibodies, sequences and structures of antigens are more variable, and make the epitope prediction much more difficult [Akbar et al., 2021]. Nevertheless, our method still produces satisfying results. Compared to the sequence-based approaches BepiPred-3.0 and NetBCE, MIPE surpasses them by 0.7% and 7.8% in AUC, respectively. Compared with the structure-based methods PECAN and PesTo, MIPE improves AUC scores by 1.2% and 21.5%. (3) For the joint paratope-epitope prediction, our method produces the best performance in terms of all metrics, particularly excelling in predicting epitopes. Compared to [Honda et al., 2020] and Epi-EPMP, our MIPE exceeded their performance in the epitope prediction by 9.5% and 22.2% in AUC, 16.8% and 18.2% in AUPR, and 12.1% and 15.8% in MMC. But joint paratope-epitope prediction underperforms compared to single paratope/epitope prediction, which is because the joint prediction needs the simultaneous training loss of two predictors designed for paratopes and epitopes, while single paratope/epitope prediction only one loss needs to be considered. Simultaneously constraining two distinct types of losses poses a challenge for the model, leading to performance degradation. (4) It's worth noting that compared to the general protein binding site prediction method, PesTo, MIPE achieved better performance in both single paratope and single epitope predictions. In summary, MIPE demonstrates superiority over the baseline methods for all three tasks, showing great promise for paratope and epitope prediction.

In practical use, the structures of antibodies and antigens are not always available, primarily due to the costs of experimental acquisition, which hinder the model from predicting binding residues for the new antibodies and antigens. Fortunately, recent years have witnessed the success of AlphaFold2, which can generate structures from sequences. Herein, we use the AlphaFold2-generated structures in the prediction process. As illustrated in Table 1, MIPE (AlphaFold2) with the generated structures still produces competitive performances in terms of various metrics compared to the baselines for three tasks and performs slightly lower than the vanilla MIPE, indicating that our model can be applicable to new antigens and antibodies without structures.

4.3 Ablation Study

To illustrate the necessity of each module, we conducted several comparisons between MIPE and its variants:

• MIPE (w/o CL) that removes the multi-modal contrastive learning module.
• MIPE (w/o IIE) that removes the interaction informativeness estimation module.
• MIPE (w/o CL & IIE) that removes both the multimodal contrastive learning and interaction informativeness estimation modules.
• MIPE (w/o Sequence) that removes the sequence input.
• MIPE (w/o Structure) that removes the structure input.

Figure 3 indicates the comparisons of MIPE and its variants in joint paratope-epitope prediction (additional ablation processes can be found in Appendix 3). When removing the multi-modal contrastive learning (w/o CL), we observed a decline in all metrics. Similarly, the exclusion of both the sequence and structure encoder (w/o Sequence and w/o Structure) also resulted in a decrease across all metrics. These findings underscore the significance of leveraging multi-modal data and aligning them through contrastive learning. Additionally, the results from the variant MIPE (w/o IIE) indicate that utilizing the interaction informativeness estimation module to differentiate between binding and non-binding residues significantly contributes to the predictive performance, even in the absence of complex information. As anticipated, the performance of MIPE experienced a substantial drop when both the multi-modal contrastive learning and informativeness estimation modules were removed (w/o CL & IIE). Overall, the above results emphasize the critical role played by these components in enhancing the model's effectiveness.

To assess the efficacy of our proposed multi-modal CL and IIE, we performed t-SNE visualizations. Previous experiments have shown that our model exhibits a favorable distinction between binding and non-binding residues on antibodies/antigens, which is also reflected in the ability to discriminate these sites on each antibody/antigen effectively. Here, we randomly select an antigen and utilize t-SNE to visualize its embedding after the sequence encoder, structure encoder, multi-modal CL, and IIE. We utilize the Silhouette Coefficient (SC) to assess the clustering effectiveness of different embeddings. As shown in Figure 4, initially scattered embeddings after sequence and structure encoders gradually differentiate between binding and non-binding residues after multi-modal and IIE. Meanwhile, the enhanced SCs also affirm the modules' capability to extract reliable representations and discern binding and non-binding residues.

4.4 Antibody-Antigen Binding Analysis

To further test our model's capability of capturing effective antibody-antigen interaction patterns, we randomly selected two antigen-antibody complexes (PDB: 3KR3, PDB: 7KQ7) from the PDB database as references, and employed MIPE to predict the corresponding binding residues. We visualized the actual and predicted antibody-antigen interaction maps using PyMOL. As shown in Figure 5, the left panel represents the results of the Fab antibody and insulin-like growth factor-II receptor [Dransfield et al., 2010], with 39 experimental validated binding pairs of paratopes and epitopes, and MIPE could correctly predict 24 pairs, achieving a recall score of 64%. Additionally, the right panel denotes the results of the Fab antibody and the Interleukin-21 receptor [Campbell et al., 2021]. MIPE predicts 23 pairs out of 38 true pairs, yielding a recall score of 61%. The results show that MIPE can effectively capture the interaction patterns between antibodies and antigens, providing reliable information for the prediction.

5 Conclusion

In this paper, we propose a multi-modal contrastive learning and interaction informativeness estimation-based method MIPE, which uses both sequence and structure data for paratope and epitope prediction. Specifically, the multimodal contrastive learning strategy is designed to maximize representations of binding and non-binding residues within each modality and meanwhile align uni-modal representations towards effective modal representations. Moreover, the interaction informativeness estimation module incorporates the interaction maps from the known complexes into the model and learns the specific patterns of interactions to enhance the performances. Experimental results demonstrate the effectiveness of MIPE and the importance of the abovementioned components.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (62372204, 62072206, 61772381, 62102158); Huazhong Agricultural University Scientific & Technological Self-innovation Foundation; Fundamental Research Funds for the Central Universities (2662021JC008, 2662022JC004). The funders have no role in study design, data collection, data analysis, data interpretation, or writing of the manuscript.

Contribution Statement

Wen Zhang, the corresponding author of this paper, supervised this project and advised on all parts of this paper. Zhiwei Wang and Yongkang Wang took the lead in designing and writing the manuscript, which should be considered to equally contribute to this work. Zhiwei Wang carried out the computational experiments, and Yongkang Wang provided help in the analysis and discussion of results. All authors reviewed and approved the final version of the manuscript. Note: in the author's list, † indicates these authors contributed equally, and * indicates the corresponding author.