Cross-View Contrastive Fusion for Enhanced Molecular Property Prediction

1 Introduction

In modern drug research and development (R&D), accurately predicting molecular properties is crucial for speeding up drug launch and enhancing R&D efficiency. Molecular Property Prediction (MPP) technologies offer an innovative approach for estimating properties based on molecular structural data, thereby refining the screening and validation of candidate drug molecules.

*Corresponding author.

Existing MPP approaches can be broadly categorized into single-view and multi-view MPP methods. Single-view MPP methods uncover molecular traits from singular data perspectives,e.g., molecular knowledge graphs [Fang et al., 2022], molecular fingerprints [Kearnes et al., 2016], SMILES representations [Honda et al., 2019], and 2D molecular graphs [Hu et al., 2020]. These data capture different angles and features of molecules to achieve specific downstream tasks, e.g., activity prediction, and drug-target prediction. In contrast, multi-view MPP methods [Zhu et al., 2021; Liu et al., 2021; Ma et al., 2022; Zhou et al., 2023], considering more comprehensive contextual information, have garnered increasing attention in this field. However, existing multi-view MPP approaches generally have not considered quality variations across views, lacking consistency in learning and interaction for the same molecule. Low-quality information could easily misguide the learning of common semantics. For instance, [Zhu et al., 2022] concatenate all views to obtain global features, where the inconsistency between the latent feature spaces is ignored.

To this end, we implement a two-stage feature learning framework for predicting molecular properties. Specifically, we consider different types of molecular data, including biophysical (e.g., BACE) and physiological (e.g., ClinTox) datasets. Biophysical datasets focus on moleculebiomolecule interactions, emphasizing physical traits. On the other hand, physiological datasets spotlight bioactivity, toxicity, and metabolic processes in biological systems. They comprehensively reveal bond types and molecular connectivity, which are particularly critical when certain toxicities link to specific structures and bonds.

Balancing these aspects is complex, as detailed atomic interactions can become intricate, especially in larger molecules, impacting model training. Focusing solely on chemical bond embeddings might inadvertently gloss over granular atomic details, an omission that might be detrimental for biophysical datasets that rely heavily on atomic interaction insights. To bridge this gap, inspired by [Ma et al., 2022], we introduce a novel approach which synergizes the topological and spatial perspectives of molecules in the first stage, which crafts a holistic portrayal that encompasses both its backbone and shape attributes, thereby streamlining the representational domain while ensuring the nuanced requirements of both dataset types are met.

In the second stage, we design a dual learning loss to utilize view complementarity to refine the feature representation, where the reliable semantic information between different views are shared. Furthermore, we implement crossview contrastive fusion to obtain the global features. In this module, the mutual information between global features and view-specific refined features is maximized. Specifically, two linear feature transformation layers are constructed for different views to yield their refined features. Then, a nonlinear MLP is used to fuse all views' features to generate global features, thereby exploring view consensus information from different views.

The main contributions of our work are as follows:

• We emphasize the comprehensive learning of molecular information through integrating multiple views, fully considering molecular sequence features, atomic features along with their neighboring attributes, and chemical bond angular features.
• We introduce a multi-view contrastive fusion method, which jointly conduct cross-view interaction and consistency learning on latent features across views, thereby addressing inconsistency in multi-view molecular data while maintaining consistent shared semantics.
• Extensive experiments demonstrate the approach outperforms other state-of-the-art methods on multiple molecular benchmark datasets.

2 RelatedWork

This section provides a concise overview of molecular machine learning methods, including single-view MPP and multi-view MPP.

2.1 Single-ViewMolecular Property Prediction

For molecular prediction work, RDKit [Tosco et al., 2014] can process SMILES strings [Anderson et al., 1987], subsequently converting them into feature vectors or molecular fingerprints. These vectors or fingerprints can be used as inputs for machine learning models. Sequence-based MPP methods [Chithrananda et al., 2020; Honda et al., 2019; Ross et al., 2022] decompose SMILES into a series of tokens representing atoms/bonds and then apply deep models on these tokens. In addition to the traditional use of sequence data, graph-based MPP models are one of the dominant models. [Hu et al., 2020] propose a graph-based pretraining model, which pre-train expressive GNNs at the level of individual nodes and entire graphs. [Zhuang et al., 2023] emphasize the relation between each molecule and its multiple properties, and construct a meta-learning framework with scheduling subgraph sampling by contrastive loss. [Xiong et al., 2020] propose to learn molecular characterization from relevant drug discovery datasets, and employ a graph neural network architecture for molecular representation. [You et al., 2020] propose Graph Contrastive Learning (GraphCL), which is a general framework for learning node representations. On this basis, [You et al., 2021] proposes a unified bi-level optimization framework to automatically select data augmentations when performing GraphCL on specific graph data. [Wang et al., 2022] present a self-supervised learning framework for large unlabeled molecule datasets. [St¨ark et al., 2022] uses 3D pre-training to provides significant improvements for different properties.

2.2 Multi-ViewMolecular Property Prediction

Based on the diversity of molecular data, such as molecular fingerprints, SMILES representation, 2D molecular graph and 3D molecular graph, each view captures different angles and properties of molecules, making the technology utilizing multi-view properties gain more attention in this field. [Guo et al., 2022] propose a multilingual molecular embedding generation approach, incorporating molecular SMILES and InChI for pretraining. [Zhu et al., 2021] design a new pretraining algorithm, dual-view molecule pre-training, which constructs an auxiliary contrast loss using molecular map features and sequence features. [Liu et al., 2021] employ selfsupervised learning to enhance 2D molecular graph encoders. The enhancement is through correspondence and consistency between 2D topological structures and 3D geometric views. [Fang et al., 2022] combine chemical knowledge graphs with molecular graphs. By integrating domain knowledge into graph semantics, the model can consider the correlation between atoms with shared properties, thereby capturing the features of molecules in the representation of the molecular graph more accurately. [St¨ark et al., 2022] utilize existing 3D molecular datasets to pre-train models. It infers the geometry of molecules in advance and then employs a selfsupervised learning approach, maximizing mutual information of 3D summary vectors for downstream tasks. [Yang et al., 2023] propose a gradient perturbation-based contrastive Learning to generate positive and negative molecules to optimize the exposure bias problem in translational molecule optimization. [Zhu et al., 2022] present a novel multi-view contrastive learning approach, and a unique pretraining framework which is learned from four molecular representations. In response to molecules are treated as 1D sequential tokens or 2D topology graphs in most MRL methods, [Zhou et al., 2023] propose a universal MRL framework named Uni-Mol. All these valuable works have demonstrated the superior performance of multi-view learning in molecular prediction.

3 ProposedMethod

3.1 Multi-ViewMolecule Feature Learning

Embedding SMILES Strings

First, we use the SMILES to represent molecular sequences, which offers a concise and expressive format. In this way, each atom is uniquely represented using its ASCII symbol and employs distinctive symbols to articulate chemical bonds, branches, and stereochemical configurations. The proposed MolFuse begins by dissecting the SMILES molecular descriptors into a series of tokens ei. The molecular sequence S is an ordered list of ei, expressed as S =<

e1, e2, e3, · · · , en > (1 ≤ i ≤ n). Subsequently, a GRU is applied to these tokens to obtain the molecular embedding Hseq in sequence view:

where the embedding layer ϕ encodes the molecular sequence S, the ⊕ operator symbolizes matrix concatenation, and the gseq (·) signifies the bidirectional gated recurrent unit (GRU) [Chung et al., 2014]. This design ensures that the model recognizes and integrates information from both backward and forward states in the sequence, employed to capture contextual information within the molecular sequence. In particular, in the first stage, we train our GRU units under the constraints of the training dataset labels. In order to simplify the computational process and stabilize the pre-trained model, we strategically freeze the GRU units in the second stage.

Embedding Topological and Spatial Graph

Considering the molecular structure represented by the topological graph T = (V, E), the individual nodes v ∈ V are emblematic of distinct atoms, while each edge (u, v) ∈ E encapsulates the chemical bond formed between atoms u and v. We denote the atom attribute as xv for the atom v, and the bond attribute as x(u,v) for the bond (u, v).

Diving deeper into the spatial intricacies, the molecular spatial graph M = (E,A) is presented. Within this depiction, each node (u, v) ∈ E is indicative of the chemical bond established between atoms u and v. The edge auvw ∈ [0, π] captures the bond angles formed between two consecutive chemical bonds (u, v) and (v, w), unified by the shared atom v. This delineation provides an insight into both the molecular configuration and spatial orientation. Specifically, we conduct a comprehensive graph representation by considering both topological graph and spatial graph.

In topological graph T , we leverage the graph neural network (GNN) to get the graph representation. Specifically, the embedding of atom v is initialized as h(0) v = xv, and them(k) v represents the hidden state of atom v during the k-th iteration. The GNN produces the embedding h(k) v for atomv at the k-th iteration, which can be described by:

where F (·) and C (·) denote aggregating messages from the neighbours and updating the node representation in the GNN module, respectively. The N(v) indicates the neighbouring atoms of v at the k-th iteration. Particularly, h(k-1) (v,w) denotes the embedding of bond (v, w) learned at the (k - 1)-th iteration in a molecular spatial graphMas Eq. (5).

Similarly, in spatial graphM, the embeddings of chemical bonds (u, v) and (v, w) are also initialized as h(0) (u,v) = x(u,v) and h(0) (v,w) = x(v,w), undergo training via GNN. Let m(k) (u,v) denote the hidden state of the chemical bond (u, v) during the k-th iteration. Finally, the GNN produces the embedding h(k) (u,v) for this bond at k-th iteration:

where N(u) and N(v) denote the neighbouring atoms of u and v, respectively. Similarly, the F (·) and C (·) denote aggregating messages from the neighbours and updating the node representation in GNN module.

To achieve an advanced molecular representation that adaptively combines both topological and spatial views, we employ READOUT function to make it integrate local node embeddings into a coherent global graph representation:

where Hgra is the learned graph representation, and the K is the number of iterations. Notably, with restrictions of the training dataset's label, we can get the optimized molecular representation under the graph view, which will stop gradient propagation in the second stage.

3.2 Cross-ViewRepresentation Fusion

After the first learning stage, we obtain both optimized molecular representations under the sequence view and graph view. We design a novel contrastive fusion framework for learning more comprehensive joint representation from two views. Specifically, we propose a dual learning module, where view-specific features acquire reliable knowledge from another view to refine their own unclear feature expression. Another contrastive fusion module is designed to learn more comprehensive global features via maximizing consistency with view-specific molecular representations. Notedly, in our designed contrastive fusion framework, the consistency and complementarity under different views are fully explored.

Dual Learning Module

Let hseq i and hgra i denote the i-th molecular representations under the sequence view and the graph view, respectively. Generally, the diversity of molecular features brings inconsistency in molecular representation. We consider that two representations of the same molecule should be consistent in the latent space. To do this, we first stack two view-specific MLPs on the features hseq , and hgra respectively, to project them into a consistent feature space. Specially, for a molecule i under the different views, the transformation is defined as:

where Zseq and Zgra denote the refined molecular representations for the sequence view and the graph view, respectively. The Ws, bs, Wg, bg are the learnable parameters in the MLPs, and the σ is an activation function.

Considering that the characteristics and qualities of different views generally vary widely, we hope to utilize view complementarity to refine the feature expression of the lowquality view. Simply put, the low-quality view acquires reliable knowledge from other views. To this end, we design a dual learning module, which aims to achieve 1) different views maintain consistency in latent feature space and 2) reliable semantic information is delivered to each other from different views. Specifically, we construct two linear feature transformation layers G (1) and G (2) , where the feature Zseq is projected to maintain consistency with the feature Zgra , vice versa. Then the reconstructed features ˆZgra and ˆZseq can be described as:

whereW(1), b(1), andW(2), b(2) denote the parameters on the feature transformation layers G (1) and G (2) , respectively.

In this module, We employ a dual learning strategy, where the consistency objective is conducted by forcing viewspecific features Zseq and Zgra to be consistent with their reconstruction objective ˆZseq and ˆZseq . In this way, crossview reliable information is delivered to each other. Then, the dual learning loss can be formulated as:

Contrastive Fusion Module

In this module, we conduct cross-view contrastive fusion to obtain more comprehensive global features from two views. To leverage the global discriminative information, we first concatenate all origin embedding, i.e., hseq and hgra . Then, a multi-layer perceptron (MLP) is applied to fuse the sequence view representations with the graph view representations adaptively. Then, the global fused feature Zfuse is obtained by:

where ⊕ denotes the matrix concatenation, σ denotes the activation function, and Wfuse, bfuse denote the parameters on fusion MLP.

Motivated by the insight that the representations of the same molecule from different views are typically similar, we conduct contrastive fusion between view-specific features, i.e., Zseq and Zgra , and global fused feature Zfuse to implement the fusion objective. In this respect, taking the sequence view as an example, we denote {Zfuse i , Zseq j }j=i as positive

feature pairs, and the rest {Zfuse i , Zseq j }j̸=i are negative feature pairs. By contrastive learning, the model could achieve that the similarities of positive pairs are maximized, and negative pairs are minimized. Firstly, the cosine distance is used to measure the similarity of feature pairs:

where ⟨·, ·⟩ is the dot product operator. Then, the contrastive loss under the sequence view can be expressed as:

where τ is a temperature coefficient. Similarly, we can also compute the contrastive loss under the graph view:

Then, we combine the contrastive losses under the two views, and the final contrastive fusion loss is:

After cross-view contrastive fusion, we use the global fused feature Zfuse as the final molecular representation to predict the molecular property. Specifically, the probability ˆ yi of the molecule i classified as a positive instance can be calculated as follows:

where σ is an activation function, and the Wpred, bpred are learnable parameters on classifier MLP. The supervised loss Lsup based on cross entropy is defined as follows:

where yi is the true label of the molecule i. In the end, we jointly optimize the supervised loss, the contrastive fusion loss, the dual learning loss. Specifically, the α and β are hyperparameters, and the overall loss function is:

4 Experiments

In this section, we focus on the following three research questions to validate the effectiveness of MolFuse. Due to space limitation, appendix B offers more comprehensive analysis and more details on our experiments.

• RQ1: Can MolFuse surpass state-of-the-art on molecular property prediction task?
• RQ2: Does MolFuse enable high-quality view guidance through dual learning?
• RQ3: Does the multi-view fusion module explore the consistency and complementarity of molecules under different views?

4.1 Experimental Setup

Dataset Description

MoleculeNet [Wu et al., 2017] is a large scale benchmark for molecular machine learning. It curates multiple public datasets and establishes metrics for evaluation. We use seven of these datasets, BBBP, BACE, Tox21, ClinTox, SIDER, ToxCast, HIV, for this experiment. The statistics of these datasets are summarized in Table 2.

Comparison Algorithms

We used three different types of baseline comparisons.

GNN Classifier. We employ a standard classifier that consists of a 5-layer GNN complemented by a single fully connect layer. This encompasses three distinctive GNN architectures: GCN [Kipf and Welling, 2016], GAT [Veliˇckovi´c et al., 2018], and GIN [Xu et al., 2018].

Single-ViewMMPMethods. We select the representative single-view MPP pretraining models, include AttrMask [Hu et al., 2020], ContextPred [Hu et al., 2020], GraphCL [You et al., 2020], along with the recently introduced methods JOAO [You et al., 2021] and MolCLR [Wang et al., 2022].

Multi-View MMP Methods. We adopt multi-view MPP benchmarks, include GraphMVP [Liu et al., 2021], 3D InfoMax [St¨ark et al., 2022], MEMO [Zhu et al., 2022], and the newly revealed Uni-Mol [Zhou et al., 2023].

Dataset Splitting

In addition to dataset selection, how to split the dataset rationally is another important factor in training the model. The dataset is usually split into training set, validation set and test set for benchmarking using random splitting, which is considered an effective and simple method in typical machine learning. However, due to the uniqueness of the molecular data, using random split molecular data is not an appropriate method. We used a splitting method called scaffold splitting, to cope with this situation, which is an attempt to split molecules with different structures into different sets [Bemis and Murcko, 1996]. And in this experiment, we used the dataset splitting method recommended in MoleculeNet.

Evaluation Metrics

Considering that the seven datasets in MoleculeNet used are all related to the classification task, we use the area under the curve of the receiver operating characteristic curve (ROCAUC) to quantify the classification performance. We use the mean metric value as the final performance results for the datasets with multi-labels.

Implementation Details

Flowing [Ma et al., 2022] and [Wang et al., 2022], each model was trained for up to 100 epochs, with the training procedure halting if there was no increase in the validation ROC-AUC over 15 consecutive epochs. A 1-layer BiGRU is employed as the backbone to extract sequence features and two 5-layer graph isomorphism networks with edge features as the foundation for the graph view representation encoder. All modules undergo training using the Adam optimizer. A grid search based on the validation ROC-AUC was conducted to seek the optimal hyperparameter configuration. Crossentropy loss was implemented as the classification loss. All simulations are implemented using PyTorch 1.7.1, and the original code of this method will be provided later.

4.2 Experimental Results and Analysis

Overall Performance (RQ1)

The performance of seven molecular property prediction tasks is summarized in Table 3. Generally, it can be found from the table that MolFuse shows strong empirical performance across all seven downstream datasets, delivering six out of seven state-of-the-art results and acquiring an 2.2% absolute improvement on average. The outstanding results validate the superiority of our proposed model. We further visualize the performance changes during the model's learning process to provide an intuitive understanding in Fig. 2. We observe that as the number of training epochs increased, MolFuse's performance on the BACE, HIV, SIDER, ToxCast, and BBBP datasets first improved and then stabilized. However, for the Clintox and Tox21 datasets, the performance initially improves but subsequently begins to decline. This indicates that MolFuse requires an appropriate number of training epochs to facilitate optimal model iterations.

Molecule Clustering Analysis (RQ1)

We take the BBBP and BACE dataset as the case studies to visualize molecular embeddings at different views and stages. Fig. 3 (a) illustrates that molecules derived solely from the single view SMILES information do not have relatively indistinct separations. This suggests an absence of clear chemical information and atom-level differences. Fig. 3 (b) shows that the molecular differentiation boundaries obtained from 2D information differ from those in Fig. 3 (a), also do not have relatively indistinct separations. This disparity arises due to the inconsistencies in the latent feature space across different views. However, Fig. 3 (c) demonstrates that our MolFuse model distinctly separates different types of molecules with clearer boundaries. This indicates that the comparative fusion strategy of our model effectively deploys the consistency and complementarity across the views.

Ablation Studies (RQ2 & RQ3)

Table 4 investigates the effect of removing different modules on the model. We observe that bypassing the cross-view fusion module and simply concatenating the views produces suboptimal results. This underscores the importance of harnessing the consistency across views. Concurrently, we observe that MolFuse still outperforms baseline methods on the BBBP dataset under study, attesting to the robust capability of the dual learning module in handling view alignment. When the model is without both module, it reduces to a GNN Classifier, and its performance closely mirrors that of the GNN Classifier. Across all tasks, we note that combining the dual learning module with the cross-view fusion module yields superior representations.

Figure 4: (a) Parameter sensitivity analysis of α and β on SIDER. (b) Parameter sensitivity analysis of τ on three datasets.

Parameter Sensitivity Analysis (RQ1)

In the parameter sensitivity analysis, we investigate two hyperparameters that balance the loss components, specifically: L = Lsup + αLdua + βLcon. Additionally, we examine the sensitivity of the temperature coefficient τ in Eqs. (15) and (16).

Fig. 4 (a) displays the average ROC-AUC across 100 independent runs. As depicted in the figure, our method exhibits robustness to the choices of α and β. Selecting both hyperparameters at 0.5 leads to optimal model performance.

The ROC-AUC initially increases with the growth of τ. However, when it surpasses 0.6, there is a noticeable performance drop on the SIDER dataset. This decline might be attributed to the increased entropy causing greater disorder and subsequently deteriorate MolFuse's performance. Overall, MolFuse exhibits relative stability.

5 Conclusion

In this paper, we introduce a novel model MolFuse that amalgamates various molecular features. Particularly, all view information is integrated into a cross-view contrastive fusion framework, where the quality of each view is fully considered. Firstly, the intricate molecular semantics and structures from both sequence and graph views are extracted. Then, we design the dual learning loss to refine low-quality views, which acquire reliable semantic information from different views. Finally, to fully explore view consistency and complementary, we conduct the fusion object on all views' features, where the mutual information between global feature and view-specific refined features is maximized. As evidenced by the results, MolFuse model produces precise chemical representations for a diverse range of molecules, ensuring accurate molecular property predictions.

Acknowledgments

This work is supported in part by Shenzhen Science and Technology Program under grants JCYJ20230807115959041 and JCYJ20230807120010021. Lifang He is partially supported by the NSF grants (MRI-2215789, IIS-1909879, IIS2319451), NIH grant under R21EY034179, and Lehigh's grants under Accelerator and CORE.