Enhancing Multimodal Knowledge Graph Representation Learning through Triple Contrastive Learning

1 Introduction

A knowledge graph (KG) is a complex semantic network that includes various entities, each with unique attributes, linked by edges representing different semantic relationships [Ji et al., 2021]. These graphs play a vital role in many contemporary applications, especially in recommendation systems [Guo et al., 2020] and natural language-based question answering [Chen et al., 2020b]. With the rapid development of graph management tools and analysis methods, knowledge graphs are increasingly used in scientific research, with notable applications in fields such as genomics, proteomics, and systems biology.

However, most existing knowledge graphs are represented with pure symbols in the form of natural text or identifiers, which does not align with how humans perceive and process knowledge in the real world [Zhu et al., 2022]. Human cognition stores knowledge in a variety of non-symbolic modalities. For example, when a person observes an apple, they do not retrieve a symbolic identifier but rather access a rich array of sensory and contextual information, including the apple's taste, smell, color, and texture. This multimodal integration is crucial for holistic understanding. Grounding entities in a knowledge graph with these diverse modalities can enable it to mirror human-like cognition more accurately during representation learning.

Current knowledge graph embedding (KGE) models tries to represent all entities and relations with low-rank continuous vectors that preserve the graph's inherent structure and capture diverse contextual information [Wang et al., 2017]. This makes them easier to work with and apply to various machine learning and deep learning tasks [Mohamed et al., 2021]. Common KGE methods includes translation models like TransE [Bordes et al., 2013], RotatE [Sun et al., 2019], etc., semantic match models like DisMult [Yang et al., 2014], SimplE [Kazemi and Poole, 2018], etc., and neural network models like ConvE [Dettmers et al., 2018], ER-MLP [Dong et al., 2014], etc. Recently, researchers have extended KGE methods to multimodal knowledge graphs. They proposed a series of multimodal KGE methods, ranging from conceptual frameworks [Lu et al., 2022; Cao et al., 2022; Wang et al., 2019] to practical applications [Yao et al., 2023; Li et al., 2023]. Despite these advancements, the integration of more than two modalities' information remains a challenge.

Recent advances in foundation models offer an ideal solution for representing different modalities. For example, for text or natural language modalities, semantic vector representations can be obtained through large language models (LLMs) such as GPT-4 [Achiam et al., 2023], LLama [Touvron et al., 2023], BERT [Devlin et al., 2018], and others. For image modalities, vector representations containing image information can be acquired through Vision Foundation Models (VFMs) like MAE [He et al., 2022], ImageBind [Girdhar et al., 2023], and similar models. Additionally, there has been a surge in domain-specific foundation models, such as CNBERT [Lu et al., 2023a], ChemBERTa-2 [Ahmad et al., 2022] and scGPT [Cui et al., 2024]. Additionally, contrastive learning has been proven to be an effective method for aligning information from different modalities. In the face of representations from foundation models in various domains, the CLIP model [Radford et al., 2021] was the first to use contrastive learning to bring closer the representations of different modalities that convey the same semantics, and to distance the representations of different semantics. For multimodal information in knowledge graphs, using contrastive learning is also an effective approach [Zhang et al., 2023; Yang et al., 2022; Liang et al., 2022].

In this paper, we propose a multimodal representation integration model for knowledge graph representation learning (KG-MRI). This approach aims to help incorporate different modalities' representations from foundation models with knowledge graph embeddings. To further optimize alignment among these varied representations, we employ a triple contrastive learning (TCL) module and apply a dual-phase training strategy. Comprehensive comparison experiments with various KGE models demonstrate the superiority of the KGMRI algorithm. In addition, we conducted an empirical analysis on a non-alcoholic fatty liver disease (NAFLD) realworld clinical cohort, illustrating the practical application of KG-MRI in clinical decision-making and removing batch effects. Altogether, our contributions are three-fold:

• We proposed a multimodal representation integration method in knowledge graph representation learning to fully integrate different information of an entity.
• We introduce a triple contrastive learning module and a dual-phase training strategy in aligning multimodal representation. We demonstrate the efficacy of KG-MRI in comparison with other KGE methods.
• We test the KG-MRI's practical utility using a private NAFLD cohort from Jidong Hospital, China. Experimental results indicate that the multimodal embedding enhances diagnostic accuracy and mitigates batch effects.

2 RelatedWork

2.1 Multimodal Knowledge Graph Embedding

Contrary to earlier unimodal KGE methods, multimodal KGE takes advantage of the extensive knowledge derived from multiple modalities [Zhu et al., 2022]. Recent advancements in multimodal KGE have introduced various models, with the objective of combine diverse data modalities for enhanced representation learning. Models like MMKRL [Lu et al., 2022] and CapEnrich [Yao et al., 2023] leverage multi-source knowledge to improve the semantic understanding of entities and relations in knowledge graphs, focusing on enriching traditional KGE methods that primarily rely on triplet facts, he2019integrating presents MK-BERT, a model that integrates BERT with multimodal data to enrich entity and relation embeddings in knowledge graphs. Likewise, the TransAE model [Wang et al., 2019] combines a multimodal autoencoder with TransE model, addressing challenges in unifying multimodal data for better representation learning. These models underscore a shared goal to harness multimodal information to provide more robust representations of realworld entities and relations. They have demonstrated promising results in link prediction tasks and knowledge graph completion benchmarks.

In our work, we propose a multimodal representation integration algorithm, aiming to better align the different representation of entities through contrastive learning and KGE methods, thus improving the representation capability of the whole knowledge graph.

2.2 Contrastive Learning in Knowledge Graph

Contrastive learning, as a self-supervised learning method, has proven highly effective in leveraging the inherent properties and structures of knowledge graphs (KGs) to enhance the quality and utility of graph embeddings. Liang2022Relational utilizes the relational symmetrical structures in KGs to construct positive pairs for contrastive learning, significantly boosting the discriminative capacity of the embeddings. Similarly, the CrossScale Contrastive Graph Knowledge Synergy approach by Zhang2023Contrastive leverages hierarchical graph views to promote knowledge sharing and enhance the generalization ability of graph embeddings. Additionally, yang2022knowledge addresses the challenges of noise and sparsity in KG-enhanced recommendation systems through a novel contrastive learning framework KGCL, which uses a knowledge graph augmentation schema to mitigate noise and employs a cross-view contrastive learning paradigm to leverage unbiased user-item interactions effectively. These methods share a unified objective to harness structural and relational data inherently present in KGs, fostering more accurate and robust embeddings.

However, current contrastive learning-based KG embedding methods are primarily limited to dual modalities and tend to focus predominantly on graph structures, often overlooking the potential to integrate richer multimodal information on entities.

3 Preliminaries

In this section, we first define the concepts of a Knowledge Graph (KG) and Multimodal Representation Integration (MRI) as used in our study.

3.1 Definition of Knowledge Graph

A Knowledge Graph, denoted as G = (E,R, T ), consists of entities and relationships forming a network.

Specifically, E represents the set of entities in the graph G, R denotes the set of relationships connecting pairs of entities, and T = {(h, r, t) | h, t ∈ E, r ∈ R} is the set of triplets. Each triplet (h, r, t) indicates that a relationship r exists between the head entity h and the tail entity t. In this work, we aim to enhance Knowledge Graph Embedding (KGE) methods by using multimodal representations of entities. To this end, we retrieve two different modalities for each entity e using the queries (e, has modality 1, ?) and (e, has modality 2, ?).

3.2 Definition of MRI

The Multimodal Representation Integration (MRI) algorithm is designed to enhance the representation of entities by integrating their representations from different modalities.

Specifically, for an entity e ∈ E with an initial representation eemb ∈ Rd , and two other modalities' representations em1 ∈ Rd1 and em2 ∈ Rd2 , the MRI algorithm integrates these representations to produce a refined multimodal embedding e′emb ∈ Rd through a triple contrastive learning module and a dual-phase training strategy. This approach aims to improve the representational capability of the existing knowledge graph embedding methods.

4 Methods

The overall framework of MRI is shown in Figure 1. In general, the MRI algorithm consists of the following modules: multimodal representation acquisition, triple contrastive learning, and dual-phase training. First, the multimodal representation acquisition module utilizes two distinct large language models to produce the representations of two different representations of entities. Then, the triple contrastive learning models aligns different representations of the same entity into a fixed-dimension vector space while maintaining the richness of information from each modality. Finally, the dualphase training approach guarantees a holistic and in-depth learning experience by concentrating on refining the KG representations.

4.1 Multimodal Representation Acquisition

A multimodal knowledge graph primarily stores different modalities related to a entity e. Yet, a comprehensive understanding of each entity necessitates the integration of different modalities' information. With the advancements in visual foundation models (VFMs) and large language models (LLMs), it's now feasible to obtain different representations through specially tailored LLMs.

We initially extract one modality s from a entity through knowledge graph query (e, has modality 1, ?). We use a foundation model FM1 with a specified cut-off length of 128 to get the corresponding representation semb.

where fFM1 is the encoder of the first foundation model, s1:128 is the truncated or padded information, and the first element of the encoder's output is the global vector of the entity's representation of modality s.

Similarly, we retrieve the second modality t of the same entity e via a knowledge graph query (e, has modality 2, ?). We use a corresponding foundation model FM2 with a specified cut-off length of 256 to get the corresponding representation temb.

where fFM2 is the encoder of the second foundation model, t1:256 is the truncated or padded, and the first element of the encoder's output is the global vector of the entity's representation of modality t.

4.2 Triple Contrastive Learning

With two 768-dimensional representation vectors semb and temb, we consider to integrate them into the entity's representation eemb ∈ Rd calculated by the base KGE model. We first project semb and temb to the same dimension of eemb by an average-pooling network.

where es[i] and et[i] is the i-th element after pooling and k = 768 d is the pooling stride.

With 3 same-dimension representation vectors eemb,es and et, we propose a triple contrastive learning module to learn an integrate vector for multiple entity representations. This module brings together the distances between the representations eemb, es, et of all modalities (3 in our experiment) of the same entity while pulling away the representations e′emb , e′s , e′t of other samples:

where n is the number of samples, τ is a temperature parameter, e1, e2, e3 represents eemb, es and et respectively. K is the number of negative samples. We then calculate the average vector of eemb, es and et

as the integrated multimodal representation for entity e.

Different from the traditional contrastive learning loss function calculated between two elements, such as InfoNCE loss in SimCLR [Chen et al., 2020a], triple contrastive learning [Lu et al., 2023b] seeks to concentrate the representations from three modalities of the same entity while ensuring their representations remain distinct from the other entities. An intuitive illustration of this concept is available in Figure 2. Both InfoNCE loss and its triple version cannot well distinguish the three-modality scenario.

4.3 Dual-phase Training

To better integrate the representations of multiple modalities, we designed a two-stage training process, namely the multimodal training phase and the combined training phase. Here, we use the RotatE [Sun et al., 2019] KGE method as an example for explanation.

We first initialize all the entity and relation embeddings E ∈ Rnum entites×d and R ∈ Rnum relations×d . Then, given a triplet (h, r, t), the head entity embedding is rotated by the relation embedding using element-wise multiplication:

where ⊙ denotes element-wise multiplication. We then compute the distance score between the rotated head entity embedding and the tail entity embedding as

and we applied margin ranking loss LRL as the total loss as the supervisory signal for KGE methods.

During the multimodal training phase, the gradient is updated using sum of LRL and LTCL. However, during the combined training phase, only LRL is utilized to ensure a more consistent update of the integrated representation. In our experiments, each training phase is set to run 500 epochs. Upon the completion of training, we extract the embeddings of E and R as the representations of all entities and relations.

5 Experiments

5.1 Dataset

We constructed a biomedical knowledge graph named HMKG from the Human Metabolome Database (HMDB, https://hmdb.ca/, [Wishart et al., 2022]), which is a comprehensive electronic online database offering intricate details on small molecule compounds present in the human body. HMDB presents chemical, clinical, and molecular biological specifics of over 200,000 compounds. Each compound is cataloged under a distinct MetaboCard, encompassing 130 data fields; roughly two-thirds pertain to chemical and clinical details, while the remaining one-third focuses on enzymatic or biological data. Many data fields are hyperlinked to other databases, which provide a high extensibility in the future. All the information in HMDB are storaged in the XML format. Based on this, we designed a specially-tailed parser to extract all the biological information in HMDB in the form of triplets.

5.2 Data Preprocessing

Regarding the data preprocessing process of HMKG, we incorporated data de-duplication, data alignment, and data disambiguation methods to enhance data quality. In the process of entity standardization, we eliminated duplicates caused by variations in letter case and special symbols. Entities representing the same concept are linked using the "the same as" relationship. Additionally, we preprocessed all numerical information in the knowledge graph, removing missing values, outliers, and erroneous data to ensure data accuracy and reliability. Furthermore, we perform standardization of the normal concentration values of compounds in the human body. By comparing these values with concentration values in patients with specific diseases, we establish triplets indicating upregulation or downregulation relationships between compounds and diseases. This enables us to identify compound expression changes related to diseases accurately.

5.3 Multimodal Information Representation

In our experiment, we first extract all the SMILES sequences of each compound. SMILES is a notation employed to depict chemical structures in textual form, with an average sequence length of 64. We utilize a recently developed language model named ChemBERTa-2 as the encoder for SMILES. ChemBERTa-2 is pretrained on a vast dataset comprising the SMILES sequences of 77 million compounds sourced from PubChem.

Then we extract all the text descriptions stored in HMDB of each compound and opt for SciBERT to encode these text descriptions. SciBERT is pretrained on a large multi-domain scientific publications corpus of 1.14M papers from Semantic Scholar and is widely used in biomedical natural language processing tasks.

5.4 Experiment Settings

In our experiment, we meticulously partitioned the dataset into training, validation, and testing sets following an 8:1:1 ratio. This division was aimed at ensuring a robust evaluation framework. The hyperparameters were chosen to strike a balance between the model's efficiency and reliability. Both the entity and relation embeddings were initialized with a dimensionality of 128. The learning rate was established at 1.0×10-3 . The training was conducted over 1000 epochs on a single Tesla A100 GPU. The total computation time varied between 3 and 60 hours, contingent on the base KGE model employed.

To uphold the reproducibility of our experimental outcomes, we utilized a fixed random seed of 42. During training, we applied AdamW [Loshchilov and Hutter, 2018] and CosineAnnealingLR [Loshchilov and Hutter, 2016] as our default optimizer and scheduler. This meticulous setup was designed to balance computational efficiency with model fidelity, thereby ensuring robust and reproducible results.

For reporting the experimental results of HMKG, common metrics including Hits@1, Hits@3, Hits@5, Hits@10, MR (Mean Rank), and MRR (Mean Reciprocal Rank) were utilized. The higher values of Hits@n and MRR and the lower values of MR explain the better performance of the method. These metrics are extensively used in evaluating the performance of models in knowledge graph embedding and information retrieval tasks.

6 Results

6.1 Comparison with Other KGEMethods

In HMKG, different patterns exist among various relationships. Some relationships are symmetric, like "the same as" relationship, while others are asymmetric, like the "is a" relationship. Additionally, some relationships are inversions of each other. For instance, if Entity A has a "has sub class" relationship with Entity B, it typically implies that Entity B has a "has father class" relationship with Entity A. Moreover, some relationships exhibit properties of composition and transitivity. For example, if Compound A "has disease" Disease B, and Disease B is "related to" Gene C, then Compound A and Gene C may also have a "related to" relationship.

Since different knowledge graph embedding (KGE) methods may have varying effects in knowledge graphs composed of different patterns, we selected a group of KGE methods to compare their vector representation learning capabilities within HMKG. We broadly categorized our selected KGE methods into three groups. Firstly, we considered translation model-based KGE models such as TransE [Bordes et al., 2013], TransD [Ji et al., 2015], TransH [Lin et al., 2015], and RotatE [Sun et al., 2019], etc. Secondly, we explored semantic matching models like Dismult [Yang et al., 2014] and SimplE [Kazemi and Poole, 2018]. Lastly, we delved into neural network-based models including ER-MLP [Dong et al., 2014] and NodePiece [Galkin et al., 2021]. The comparison results are shown in Table 1.

In Table 1, we bolded the best results, and underlined the second-best results. Among all KGE methods, RotatE [Sun et al., 2019] achieved the best performance in Hits@n and MRR, and significantly outperformed other methods, such as (Hit@1 was 5% higher than the second place DisMult model [Yang et al., 2014], and MRR was 0.08 higher than the second place DisMult model as well). However, TransD [Ji et al., 2015] and ER-MLP [Dong et al., 2014]models performed somewhat better on the MR evaluation metric. We have also tested the results of other KGE methods like ConvE [Dettmers et al., 2018], KG2E [He et al., 2015], ConvKB [Nguyen et al., 2017] and RGCN [Schlichtkrull et al., 2018], etc., but their performance is not as good as the methods listed in Table 1, so we choose not to display these results.

Based on the above results, we chose RotatE [Sun et al., 2019] as the base KGE model in our MRI algorithm, and we integrate the chemical embedding of compounds' SMILES sequences and semantic embedding of compounds' text descriptions to enhance the representation capability of HMKG. We report the performance of our MRI-RotatE KGE methods in the last row of Table 1. Not surprisingly, our method notably outperformed on various metrics, encompassing all Hit@n and MRR values. Furthermore, it achieved an MR of 550, marking a substantial advancement over RotatE [Sun et al., 2019] and surpassing ER-MLP [Dong et al., 2014] (MR=644), while nearing the performance of TransD [Ji et al., 2015] (MR=462). The comparison results strongly demonstrate the effectiveness of the MRI algorithm.

6.2 An Empirical Study in NAFLDDiagnosis

In demonstrating the practical application of KG-MRI for disease diagnosis and health assessment, we carried out a study on a non-alcoholic fatty liver disease (NAFLD) cohort (160 NAFLD patients and 150 Normal Control) from Jidong Hospital, Hebei, China. Blood sample of each patient was collected after fasting and followed by medical follow-up. Detailed statistics of this cohort can be found in Table 2.

In clinical scenario, using compound information for health assessments is a crucial concern. Traditional machine learning methods have been successful in diagnosing certain single diseases. However, the challenge of integrating data from different batches, due to variations in devices and conditions, hinders the reuse of pre-trained models. In order to overcome this problem, it's essential to decouple the fixed number of compounds in patient clinical test data from predictive models. Our compound embedding through KG-MRI model allows the handling of compound data regardless of the number of compounds. Moreover, using information about individual compounds from knowledge graphs for disease prediction can provide a more detailed and thorough description of compounds. This enhances the model's generality and interpretability.

As illustrated in Figure 3, our analytical pipeline starts by sampling all normal population samples to establish reference ranges for each compound. Next, for each sample under study, the model classifies the compounds into three categories, that is, upper-regulation, normal, and lowerregulation, based on these reference ranges. Within each category, we select 50 most relevant compounds and look up into HMKG to obtain their vector representations. These vectors are then multiplied by their respective expression levels to create a comprehensive patient-level matrix. If the number of compounds in a given category is fewer than 50, zero-padding vectors are used to fill up the shortfall.

We then used the MultiLayer Preceptron (MLP) neural network to classify this NAFLD cohort using the aforementioned patient-level matrix under the cross-validation settings, achieving an average AUC of 0.87, an average F1 of 0.84, and an average accuracy of 0.83, which surpasses the classification performance attained when directly applying traditional machine learning models to classify patients' different compounds expression data (a fixed-length vector that cannot generalize to a varying lengths due to factors like batch effects of samples and disparities among different clinical test machines). The full comparison results with other machine learning models [Su et al., 2012; Rish and others, 2001; Hearst et al., 1998; Breiman, 2001] are shown in Table 3.

7 Conclusion

In this study, we proposed a knowledge graph multimodal representation (KG-MRI) learning model using triple contrastive learning and a dual-phase training strategy. This model uniquely integrates various modalities associated with a single entity in a knowledge graph, enhancing the multimodal coherence and utility of the derived embeddings. Our exhaustive comparative analysis reveals that KG-MRI outperforms traditional KGE models across multiple metrics. Further in-depth evaluations were conducted using a biomedical knowledge graph specifically tailored to analyze NonAlcoholic Fatty Liver Disease (NAFLD). By applying our KG-MRI model to this domain, we were able to significantly reduce batch effects, which are common issues in traditional classification models used in biomedical contexts. This reduction in batch effects, coupled with our model's enhanced performance metrics, underscores the practical and theoretical benefits of integrating multimodal data in knowledge graph representations.