Variational Multi-Modal Hypergraph Attention Network for Multi-Modal Relation Extraction

1 Introduction

Relation Extraction (RE) is a fundamental task focusing on determining the relationships between pairs of entities within a given context [Xue et al., 2022; Cao et al., 2023]. Over recent years, there has been a growing interest in advancing this task through multi-modal relation extraction (MMRE), which integrates textual information with visual data. By leveraging complementary insights from both modalities, MMRE addresses the limitations of text-only approaches, providing enriched semantic understanding and contextual grounding. MMRE plays a pivotal role in a variety of cross-modal applications. For instance, it unifies information across text and images to enable more comprehensive knowledge representation [Liang et al., 2023; Ge et al., 2021]. These advancements highlight MMRE's potential to bridge the gap between textual and visual information, enabling innovative solutions for complex cross-modal challenges.

∗Corresponding Author.

Previous studies [Chen et al., 2022c; Liang et al., 2022; Chen et al., 2022d; He et al., 2023] have made significant progress in advancing multi-modal relation extraction (MMRE) by effectively leveraging visual information to complement textual data. Visual content plays a crucial role in bridging semantic gaps, providing additional evidence to enhance the extraction of relationships. Despite their successes, these approaches primarily focus on aligning relations between objects and text, overlooking challenges posed by multiple entity pairs within a single sentence that share the same contextual information (e.g., identical text and image). This limitation makes it difficult to differentiate relationships between distinct entity pairs, leading to errors in scenarios where nuanced distinctions are critical. Addressing this gap remains an essential step toward advancing MMRE.

As illustrated in Figure 1, a single text-image pair can involve multiple entity pairs, each associated with distinct relationships. The most important objects for the entity Evan Massey and the Lori Blade are the objects framed in blue, which are helpful to the task. Effectively associating entities with distinct sets of objects allows the model to extract meaningful semantic information that is critical for accurate relational classification. Additionally, a single entity may participate in multiple relationships across different pairs, introducing further diversity in the relations. For instance, Evan Massey appears in two distinct entity pairs, where some objects in the image contribute to multiple relationships. This overlap increases the complexity of learning specific representations, as the model must discern shared and unique contextual cues for each relation. This phenomenon underscores the importance of establishing accurate associations between entities and objects in the image for effective multi-modal relation classification. However, existing approaches often fall short of capturing correlations beyond simple pairwise interactions, especially when dealing with diverse entity relationships across modalities. Addressing this gap requires methods capable of modeling high-order correlations and diverse representations of entities and relations, a challenge that remains largely underexplored in the current landscape of MMRE research.

To address the challenges outlined above, we propose the variational multi-modal hypergraph attention Network (VMHAN) for MMRE. Unlike existing methods that rely heavily on pre-defined features or rigid contextual structures, our approach dynamically learns a joint representation of multiple modalities by leveraging hypergraphs to capture complex, high-order correlations across different modalities. We model each sentence and its corresponding image (along with objects in the image) as a multi-modal hypergraph. This representation enables the model to go beyond pairwise interactions, capturing intricate, high-order relationships between textual and visual data. Our model autonomously learns the edges and weights of the hypergraph, optimizing its structure to uncover the latent relationships between entities and their associated visual and textual contexts. To further improve generalization and robustness, we adopt a variational approach, transforming node representations into Gaussian distributions. By modeling node features as distributions rather than fixed vectors, VM-HAN captures the underlying variability of relationships, resulting in more accurate predictions. The main contributions are summarized as follows:

• We technically design a novel MMRE framework to capture complex and high-order correlations among different modalities.
• We construct a multi-modal hypergraph capturing highorder correlations between different modalities. We also design Variational Hypergraph Attention Networks to handle the diversity of entities and relations.
• Experimental results on benchmark datasets demonstrate the effectiveness of our approach, outperforming state-of-the-art methods.

2 RelatedWork

2.1 Multi-Modal Relation Extraction (MMRE)

Multi-modal relation extraction (MMRE) has garnered significant attention in recent years [Zhang et al., 2017; Wu et al., 2020b; Zheng et al., 2021a; Lu et al., 2022; Cao et al., 2021; Chen et al., 2022a]. MMRE aims to identify textual relations between two entities in a sentence by incorporating visual content [Zheng et al., 2021a; Zheng et al., 2021b; Chen et al., 2022c], which compensates for insufficient semantics and aids in relation extraction [Zheng et al., 2021b; Zheng et al., 2021a]. However, existing works ignore the multiple relations in different entity pairs in one sentence, which is caused by variations in entity pairs. In the task of multi-modal relation extraction, it is observed that images can provide valuable information. However, the potential of utilizing image information in a distinct manner for different entity pairs has not been fully explored. This paper aims to address this gap by capturing high-order correlations among entity pairs and the associated image objects and integrating intra-modal and inter-modal correlations that are truly useful for each entity pair.

2.2 Hypergraph Network

Hypergraph networks have gained attention in various fields for modeling complex relationships among nodes [Gao et al., 2023; Wu et al., 2020a]. In the realm of multimodal learning, [Kim et al., 2020] constructed a shared semantic space among different modalities by hypergraph and generated a joint representation by attentively integrating the modalities through a co-attention mechanism. MKHG [Zeng et al., 2023] presents a degree-free hypergraph solution that ingeniously addresses the challenges posed by heterogeneous data sources and modalities. The effectiveness of hypergraph networks lies in their ability to capture rich semantic dependencies among entities, which is crucial in tasks such as multi-modal relation extraction. In this paper, we propose a novel approach that integrates hypergraph networks with multi-modal relation extraction to exploit high-order correlations among entity pairs and associated image objects.

3 Preliminaries

Multi-Modal Relation Extraction (MMRE). MMRE involves identifying relationships between entities by leveraging both textual and visual information. Formally, given a text T = [w1,w2, · · · ,wl] associated with an image I, and an entity pair (h, t), the goal is to predict the relationship between h and t from a predefined set of relations R = {r1, r2, · · · , rk, none}. An MMRE model takes the input (h, t, T, I) and outputs the most likely relation ri ∈ R. The output relationship inherently incorporates visual cues provided by the image I, enabling a richer contextual understanding. The model assigns a confidence score p(ri | h, t, T, I) for each possible relation ri, determining which relation best describes the connection between the entities h and t in the given context.

Hypergraph. Ahypergraph is a specialized formof a graph that differs from simple graphs by containing hyperedges. These hyperedges can connect two or more nodes and are often used to represent high-order correlations [Feng et al., 2019]. A hypergraph is typically defined as G = (V, E), consisting of a node set V = {v1, v2, · · · , vn}, a hyperedge set E = {e1, e2, · · · , em}, and an optional diagonal weight matrix P ∈ Rm×m that represents weight of each hyperedge. The hypergraph G can be represented by an incidence matrix H ∈ {0, 1}n×m. In this way, each hyperedge ej connects all associated nodes vi, indicating correlations among them.

4 Framework

We propose the variational multi-modal hypergraph attention network (VM-HAN) framework, as illustrated in Figure 2.

4.1 Multi-Modal Hypergraph Construction

To effectively capture the high-order correlations across the modalities, we first construct a multi-modal hypergraph.

Multi-Modal Node Selection. The multi-modal hypergraph contains textual/visual nodes selected from the given text T and image I. Specifically, the entities h,t in text are initialized as textual nodes. We employ an LLM (LLama3) to extract the top m entities that are deemed useful for relation extraction. The prompt for this process is as follows:

Please identify and extract the top m entities and actions that are most relevant for relation extraction involving the given entity pair (h,t).

Furthermore, we use an LLM (e.g., LLaVa) to process the corresponding image I and extract the top n entities deemed relevant, which is guided by the following prompt:

Please identify and extract the top n entities and actions from the image that are most relevant to the given entity pair (h,t).

In addition, the visual nodes include the image itself and k most relevant objects to the given entity pair (h, t). We utilize another LLM (LLava), to select these k objects. The prompt for this process is as follows:

Please identify and select the top k objects in the image that are semantically most relevant to the given entity pair (h,t).

Global Hyperedge. To capture global correlations among all modalities, we first construct the global hyperedges connecting all nodes. Formally, global hyperedge is as follows:

Global hyperedge allows all nodes to propagate information to each other and get representations that contain global semantics (i.e., both textual and visual information).

Intra-Modal Hyperedge. For relations within a single modality, we construct two intra-modal hyperedges for each hypergraph, one connecting the textual nodes (i.e., head and tail entities) and the other connecting the visual nodes (i.e., image and top-k objects). The intra-modal hyperedges are defined as follows:

Intra-modal hyperedges focus on aggregating information within one single modality to obtain modal-specific semantics (i.e., textual/visual correlations).

Inter-Modal Hyperedge. In addition, to capture the relations across different modalities, we construct two different inter-modal hyperedges, one connecting the head entity with the visual nodes, and the other connecting the tail entity with the visual nodes. The inter-modal hyperedges are defined as:

Inter-modal hyperedges mainly help to learn the associations between modalities (i.e., text ↔ image). ehead-visual denotes the cross-modal hyperedge connecting the head entity with the image and top three objects, and etail-visual denotes the cross-modal hyperedge connecting the tail entity with the image and top three objects.

The advantages of our proposed multi-modal hypergraph construction module are twofold. First, by explicitly modeling the high-order correlations among different modalities, our module can capture more complex and fine-grained relations that might be overlooked by existing approaches. Second, by introducing hyperedges to connect multiple nodes simultaneously, our module can reduce the number of parameters needed to model the relationships among all nodes, which can reduce the risk of overfitting and improve the generalization performance of the model. The constructed multimodal hypergraph contains three types of hyperedges, including global, intra-modal, and inter-modal hyperedges. To further study the strength of associations in the constructed multi-modal hypergraph, we next design an attention-based hypergraph encoder using variational modeling approaches.

4.2 Variational Hypergraph Attention Network

To further improve the performance of MMRE on the constructed multi-modal hypergraph, we propose a variational hypergraph attention network (V-HAN), utilizing a variational approach to learn representations under Gaussian distributions. The V-HAN model consists of two main components: variational hypergraph representation and variational hypergraph attention. The Variational Hypergraph Representation is responsible for encoding the nodes in the hypergraph, while the Variational Hypergraph Attention module is responsible for propagating the node information through hyperedges and updating the node representations.

Variational Hypergraph Representation. The variational hypergraph representation component serves as the foundation of V-HAN. It takes the textual and visual nodes as input and outputs representations modeled as Gaussian distributions. This representation captures both the core semantic features (mean) and the uncertainty (variance) of each node, providing a robust framework for modeling complex multimodal relationships.

To obtain the node representations, we model the feature of each node vi ∈ V using a Gaussian distribution and compute the mean and variance of node features for transferring specific representation into Gaussian representation as follows:

Specifically, we model each node vi using a Gaussian distribution N (xi,µ, diag(xi,σ)), where x(0) i,µ and x(0) i,σ are the initial representations of the mean and variance of the node feature distribution, respectively. xi are the origin feature initialized in Section 4.1. Wµ,Wσ are learnable parameters. In this paper, we adopt the diagonal variance matrix, which is a common choice of previous works [Zhu et al., 2019; Petrov, 2022].

The variational representations preserve the original feature information in the mean vector and estimate the uncertainty of the original feature information in the variance vector. By modeling the node features as Gaussian distributions, this module provides a more informative and robust representation of each node, which helps for better multi-modal relation extraction performance.

Variational Hypergraph Attention. The variational hypergraph attention component updates node and hyperedge representations by leveraging attention mechanisms. This iterative process ensures that the model captures high-order correlations and dynamic associations between nodes and hyperedges. To adaptively learn the influences under the structure of multi-modal hypergraphs, we deploy a multi-head attention mechanism to compute the weight between hyperedge ej and a node vi connected with it as follows:

where xi,µ is the mean vector of node vi and ej,µ is the hyperedge representation updated below, and d is the dimension. The variational hypergraph attention mechanism updates the Gaussian distribution of hyperedge ej by aggregating information from all nodes vi ∈ ej. Specifically, the mean and variance of hyperedge ej in the (l+1)-th layer of the updated distribution are computed using the following equations:

where x(l) i,µ and x(l) i,σ are the mean and variance representation of node vi in l-th layer. In the variational framework, especially one that deals with Gaussian distributions, each node (or hyperedge) is represented by a mean (µ) and a variance (σ2 ). The mean represents the expected value of the feature, while the variance captures the uncertainty or spread of that feature's distribution. The double product in the updating of σ (variance) reflects the need to account for the uncertainty in the attention weights α(l) j,i themselves, in addition to the uncertainty in the node or hyperedge representations. This is particularly important in a setting where attention mechanisms are used, as the attention weights determine how much influence one node has over another. The squaring (or double product) of σ in this context ensures that the variance is scaled appropriately.

For node representation updating, similar to the hyperedges, we compute the weight between node vi and a hyperedge ej connected with it as follows:

The updated Gaussian distribution of node vi is computed by aggregating information from all hyperedges connected to it.

Specifically, the mean and variance of node vi in the (l+1)-th layer of the updated distribution are computed as follows:

where e(l) j,µ and e(l) j,σ are the mean and variance representation of hyperedge ej in l-th layer.

Through the iterative updating of attention weights and representations, V-HAN effectively captures high-order correlations across modalities, enabling the model to learn robust and informative representations for both nodes and hyperedges. By modeling these as Gaussian distributions, the module addresses challenges arising from ambiguity and variability in multi-modal data. Combined with the carefully designed joint optimization objectives presented in the next section, V-HAN significantly improves the performance of the proposed VM-HAN framework on MMRE tasks.

4.3 Joint Optimization Objectives

We design a joint loss function that evaluates the model's performance under the variational representation.

Relation Classification Constrain. The relation classification loss measures the error between the predicted relationship and the ground truth. This loss ensures that the model accurately classifies the relationship between entities in the multi-modal context, defined as follows:

where (xh,µ, xh,σ), (xt,µ, xt,σ) are the representations of the two entities, and p(r|xh,µ, xh,σ, xt,µ, xt,σ) is derived by concatenating four vectors followed by a classification layer.

Reconstruction Constrain. The reconstruction loss Lrec measures the difference between the predicted node representations and the actual node representations. It is defined as the mean squared error (MSE) between the predicted mean vectors xi,µ and the true mean vectors xi,h, and the MSE between the predicted variance vectors xi,σ and the true variance vectors xi,τ :

where n is the number of nodes.

Prior Constrain. The KL divergence loss LKL measures the difference between the predicted Gaussian distributions and a standard Gaussian distribution. It is defined as:

where DKL is the Kullback-Leibler divergence and N(µ,Σ) is the multivariate Gaussian distribution with mean µ and covariance matrix Σ. The KL loss plays a crucial role in regularizing the variational distribution, thereby enhancing the generalization ability of the model. The overall loss function of VM-HAN is a weighted sum of the three loss terms:

where λc, λrec, and λKL are hyperparameters that control the relative importance of each loss term.

5 Experiment

5.1 Experimental Setup

Dataset. To evaluate the performance of the proposed VMHAN framework, we use two widely recognized datasets for Multi-Modal Relation Extraction (MMRE): MNRE and MORE. (1) The MNRE dataset [Zheng et al., 2021b] is sourced from Twitter 1 . (2)To broaden the scope of our investigation, we incorporate the MORE dataset [He et al., 2023].

Comparision Methods. We compare VM-HAN against three categories of methods: text-based relation extraction (RE) models, BERT-based multi-modal RE (MMRE) models, and graph neural networks (GNNs) for the multimodal relation extraction. (1) Text-based RE models include Glove+CNN [Zeng et al., 2014], PCNN [Zeng et al., 2015], Matching the Blanks (MTB) [Soares et al., 2019]. (2) MMRE models include BERT+SG [Devlin et al., 2019], BERT+SG+Att, VisualBERT [Li et al., 2019], MEGA [Zheng et al., 2021a], HVPNet [Chen et al., 2022c], DGF-PT [Li et al., 2023], MKGformer [Chen et al., 2022b] , MOREformer [He et al., 2023], TMR [Zheng et al., 2023], HVFormer [Liu et al., 2024], CAMIM [Zhang et al., 2024]. (3) Graph-based approaches include GCN [Kipf and Welling, 2016], GAT [Velickovic et al., 2018], HGNN [Feng et al., 2018]. More details can be found in Appendix B.2.

Implementation Details. For text-based initialization, the textual embeddings were initialized using the bert-base-uncased model from HuggingFace 2 , with an embedding dimension of 768. Text inputs were either truncated or padded to a maximum sequence length of 128 tokens. For visual feature extraction, visual features were extracted using the VGG16 network 3 and the YOLOv3 [Redmon and Farhadi, 2018], widely recognized for their performance in image feature extraction. The dimensionality of visual object features was set to 4096, and the number of objects per image was limited to three to maintain consistency and computational efficiency. The AdamW optimizer [Loshchilov and Hutter, 2019] was employed, with a learning rate of 2e-5 and a weight decay of 0.01. A dropout rate of 0.6 was applied to prevent overfitting, ensuring the model's robustness across diverse scenarios. The training process used a batch size of 16. To fine-tune hyperparameters effectively, a grid search approach was adopted, conducting five trials to identify the optimal configuration based on the validation set performance. Node embeddings in the multi-modal hypergraph were learned using a Graph Convolutional Network (GCN).

1 The direct link to the Twitter data stream provided at https://archive.org/details/twitterstream.

2 https://github.com/huggingface/transformers

3 https://github.com/machrisaa/tensorflow-vgg

5.2 Main Results

To evaluate the effectiveness of our proposed model, we compared it against SOTA baselines, some results were sourced from the respective original publications [Zheng et al., 2021a; Chen et al., 2022c; Li et al., 2023; He et al., 2023] for consistent comparisons. From Table 1, we can observe that: 1) Our model outperformed all baseline methods, confirming its ability to effectively integrate multi-modal knowledge for improved performance. 2) Across the four evaluation metrics, our model consistently demonstrated superior performance. Notably, it achieved a minimum improvement of 1.08% in F1-score and 2.93% in Accuracy, underscoring its robustness and efficiency. 3) When compared to text-based RE methods, our model demonstrated clear advantages, highlighting its capacity to utilize visual information to enhance relational understanding. 4) Our model surpassed BERT-based MMRE models, emphasizing its ability to capture intricate structural features through hypergraph learning techniques. A particularly noteworthy observation is the significant performance gains on the MORE dataset, where our model achieved at least a 2.93-point improvement across the evaluation metrics.

This improvement can be attributed to the dataset's richness in visual objects, which enables the model to capture more fine-grained cross-modal relationships.

5.3 Ablation Study

From Table 2, we can observe that: 1) All of the core modules of VM-HAN demonstrate significant improvements in performance. 2) The multimodal hypergraph emerged as a particularly influential component, with its removal causing the most substantial performance drop among the tested variants. This can be attributed to its ability to model high-order relationships, thereby providing richer contextual clues for relation extraction. 3) Replacing the variational representations with fixed, specific representations led to noticeable performance declines. This suggests that the variational framework enables the model to better capture the underlying distributions of relationships. 4) Removing any one type of hypergraph (global, intra-modal, or inter-modal) resulted in a clear decrease in performance. This demonstrates that each hypergraph type captures distinct high-order relationships, all of which are essential for comprehensive multimodal reasoning.

5.4 Discussions for V-HAN

Entities Appearing in Multiple Categories. In this scenario, an entity is associated with multiple categories across different occurrences. We compared the performance of our proposed method against a baseline on datasets with varying numbers of entity repetitions, as shown in Figure 3. The results demonstrate that our method outperformed the baseline in both single and multiple repetition scenarios. This indicates that V-HAN effectively models the diversity of multisemantic entities, capturing nuanced semantic representations that enhance the accuracy of relation extraction.

Entities with Different Meanings Across Pairs. Another challenging scenario involves entities that take on different meanings when paired with various counterparts in the same sentence, leading to different relation types. We evaluated the performance of our method and baselines on sentences containing different numbers of entity pairs, as illustrated in Figure 4. Our model consistently achieved state-of-theart performance in both single-pair and multi-pair scenarios. This suggests that the hypergraph learning mechanism in VHAN successfully leverages inter-modal associations, effectively distinguishing the differences across entity pairs.

5.5 Effect of Visual Information

To evaluate the contribution of visual information, we conducted an experiment comparing its performance with and without image features. The results are shown in Figure 5. Specifically, we trained two versions of the VM-HAN model: one utilizing both textual and visual data, and the other relying solely on textual data. From the figure, we can observe that the results clearly indicate that the VM-HAN model incorporating visual features outperforms the text-only version across all evaluation metrics. By integrating visual features, MTB VBERT MEGA

Figure 6: Performance and efficiency.

VM-HAN effectively captures complex, high-order correlations between modalities, which allows for more precise relation extraction in a multi-modal context.

5.6 Efficiency

To evaluate the efficiency of VM-HAN, we conducted a comparative analysis against several baseline models, with the results presented in Figure 6. The results indicate that our method achieves significant improvements in training efficiency while maintaining state-of-the-art accuracy. In comparison to BERT-based models, which are widely adopted but computationally demanding, our approach demonstrates substantially reduced training time. Unlike traditional graph neural networks, our hypergraph-based design captures highorder features both within and across modalities, streamlining the learning process and enabling faster convergence. The dual advantage of accuracy and efficiency positions ours as a practical and effective solution for real-world applications.

Case Study. We also provide a case study to showcase the ability of MV-HAN to identify and utilize relevant visual information effectively in Appendix C.

6 Conclusion

In this work, we introduced the variational multi-modal hypergraph attention network (VM-HAN), a novel framework that effectively addresses the challenges of multimodal relation extraction (MMRE). By incorporating hypergraph structures and variational modeling, our approach captures complex, high-order correlations across modalities, enabling it to model intricate associations between entities and their relationships. The variational modeling technique employed by our variational hypergraph attention Networks (V-HAN) proves particularly effective in handling polysemous entities. Extensive experimental evaluations demonstrate that our proposed VM-HAN achieves state-of-the-art performance.

Acknowledgments

We thank the anonymous reviewers for their insightful comments and suggestions. The corresponding author is Cheng Ji. The authors of this paper were supported by the NSFC through grant No.62402054, No.62425203 and No.62032003, and the 76th batch of general grants from China Postdoctoral Science Foundation through grant 2024M760279, and supported by the Postdoctoral Fellowship Program and China Postdoctoral Science Foundation under Grant Number.