Representation Learning with Mutual Influence of Modalities for Node Classification in Multi-Modal Heterogeneous Networks

1 Introduction

In the real world, numerous online platforms can be characterized as heterogeneous networks [Sun and Han, 2012], which encompass multiple types of nodes connected with various relations, such as movie networks in Douban and product review networks in Amazon [Ni et al., 2019]. With the rapid evolution of Internet, besides textual contents, nodes of certain types also incorporate attributes from other modalities (e.g., images). That enriches the node information and forms multi-modal heterogeneous networks (MMHNs) [Wei et al., 2023; Kim et al., 2023; Chen and Zhang, 2020; Jia et al., 2022], including multi-modal knowledge graphs (MMKGs) [Kannan et al., 2020; Liang et al., 2022; Zhu et al., 2024; Wang et al., 2023b; Sun et al., 2020; Pezeshkpour et al., 2018]. As shown in Figure 1, a movie network comprises movies, actors and directors, with the movie nodes possessing both textual and visual attributes.

∗Corresponding author

It is significant to classify these nodes in the network into distinct categories to understand and analyze key entities, leveraging their multi-modal attributes as well as the network structure [Jangra et al., 2023]. Recently, more and more attentions have been paid to integrate multi-modal features into GNN-based representation learning methods. Some existing studies adopt an early fusion strategy in the encoding process aiming at initial features [Zhang et al., 2019], which may lose the characteristics of individual modalities, while late fusion models learn node embeddings separately for each modality and blend them only at the last layer [Jia et al., 2022].

For example in Figure 1, the two film nodes v1 and v2 belong to the same category "Comedy", but their textual descriptions differ greatly. Hence, when learning textual embeddings aggregated from neighbors, the similarity-based attentive (2-hop) propagation between them would be limited, potentially resulting in the assignments of them to different categories. Nevertheless, noticing that the images of them are very similar, if that can be acknowledged in determining the propagation weights, the two nodes may eventually own similar embeddings on both modalities and obtain the same label. While, maybe in another scenario, textual similarity plays a pivotal role in realizing a category-oriented aggregation.

Consequently, in order to harness the multi-modal information more effectively in representation learning on MMHNs, it is essential to consider the mutual influence of modalities during the information propagation process and learn it in an adaptive way. This poses three main challenges: Firstly, as the influence involves at least two modalities and two nodes, we need to choose the appropriate granularity to define and distinguish the cross-modal influence. Here, a tradeoff among expressiveness, intuitive interpretability and model complexity is supposed to be achieved. Secondly, in real-world scenarios, certain types of nodes may have missing attributes for specific modalities (e.g., actors and directors lack images), so it is required to handle these incomplete data to ensure a smooth feature propagation process for each modality. Thirdly, even for nodes with attributes across all modalities, some of these attributes may fail to accurately reflect node characteristics. This misalignment among modalities would import undesired noise during propagation, causing the learned representations to deviate from correct labels.

To address these issues, this paper proposes a novel model named Heterogeneous Graph Neural Network with InterModal Attention (HGNN-IMA). It aims to learn node representations in MMHNs via capturing the mutual influence of multiple modalities during the information propagation process, thereby supporting the node classification task. Specifically, after pre-processing to encode the attributes of each modality, we devise a heterogeneous network propagation module within the framework of heterogeneous graph transformer, to enrich the node features of each modality by aggregating neighbor information. A key innovation is the nested inter-modal attention mechanism integrated into the classical inter-node attention. When propagating neighbor embeddings to a current node for a specific modality, attention scores are computed as the weighted sum of the similaritybased attention in terms of each modality. The weights are also determined in an attentive and thus adaptive manner, which are further constrained by an attention loss to mitigate the effect of missing modalities. Moreover, these attention scores are necessarily modulated according to the similarity consistency among modalities, amplifying the contribution of modality-aligned nodes. Then, through a feature fusion module, enriched features of all modalities are mixed to generate the final node embeddings. Besides the cross-entropy loss on the fused features, uni-modal features are also incorporated to underline the respective effect of each modality.

In summary, this paper makes the following contributions:

• To the best of our knowledge, this is the first work to
• adaptively learn and leverage the mutual influence of multiple modalities for model fusion in GNN-based representation learning on MMHNs, tailored for the node classification task.
• A novel model, framed within the heterogeneous graph transformer architecture, is proposed to fulfill the core idea. It features a nested inter-modal attention mechanism on the inter-node attention, plus a modulation term based on similarity consistency to encourage modality alignment, and an additional loss function tackling the modality missing issue. This comprehensive approach accommodates the intricate nature of cross-modal interactions in heterogeneous networks and enhances the category discriminability of node representations.
• Extensive experiments on diverse real-world benchmark datasets demonstrate the effectiveness and stability of the model, achieving significant performance improvements over existing approaches for node classification.

2 RelatedWork

Heterogeneous networks widely exist in the real world. In the past decade, significant efforts have been devoted to learn node representations with various attention mechanism [Zhuo et al., 2023; Wang et al., 2023a; He et al., 2024; Li et al., 2023b]. For heterogeneous networks with multimodal attributes, such as texts, images and audios, the fusion of them is necessary. Most of existing models fall into two types, early fusion and late fusion [Jangra et al., 2023; Zhao et al., 2024; Wu et al., 2024; Huang et al., 2024].

Early fusion means the fusion is conducted just after the feature extraction from multi-modal attributes [Wang et al., 2020; Chen and Zhang, 2020]. Besides extending the typical heterogeneous network embedding methods mentioned above by combining initial attributes, HetGNN [Zhang et al., 2019] employs a Bi-LSTM model to encode the feature of each modality and fuse them with mean pooling before intratype and inter-type aggregations. As this kind of fusion appears prior to information propagation, all modalities of a specific node are merged into one feature, at the expense of losing the characteristics of individual modalities, especially neglecting their different roles in the aggregation process.

Conversely, late fusion refers to the fusion after the node representations in terms of each modality have been obtained, so the information propagation of each modality is separately executed [Wei et al., 2019; Tao et al., 2020; Cai et al., 2024; Cao et al., 2022]. As representatives, MHGAT [Jia et al., 2022] performs dual-level aggregations within individual modalities, followed by feature fusion using the modalitylevel attention mechanism, and FHIANE [Yang et al., 2023a] adds an early fusion module, taking advantage of the consistency and complementarity of multi-modal information. Although these models allow each modality to propagate independently, preserving its original properties, they fail to leverage the information from other modalities when computing the similarity-based attention for accurate aggregation weights. As a result, the learned embeddings may be inconsistent with category labels.

In addition to the two manners of multi-modal fusion above, XGEA [Xu et al., 2023] considers the influence between two modalities during the propagation process, but the role of the current modality itself is neglected and the characteristics of each node in the propagation cannot be distinguished. In contrast, our model explores to adaptively learn and combine the mutual influence among modalities for each node, through a novel nested attention mechanism to enable flexible aggregations.

Beyond that, IDKG [Li et al., 2023a] regards the knowledge graph as another modality. Its translation-based embeddings, along with visual and textual features, are fused into a unified representation, so is able to solve the node classification problem with a different usage of network structure.

3 ProblemFormulation

Definition 1 (Multi-Modal Heterogeneous Networks). A Multi-Modal Heterogeneous Network (MMHN) is defined as G = (V, E,M), where V, E and M respectively denote the set of nodes, edges and multi-modal attributes (e.g., texts, images and audios). There are also a node type mapping function ψ(v) : V → O and an edge type mapping function ϕ(e) : E → R, where O and R denote the set of pre-defined types of nodes and edges respectively, with |O| + |R| > 2.

Notice that although each node in V can contain multiple types of attribute information from different modalities inM., some modalities are not available for certain node types. For example, reviews in Amazon are probably not associated with an image. Thus, we define a mapping function f(o) : O → 2M to indicate the subset of modalities that each node type possesses. In this context, for each node vi ∈ V, its attribute of each modality m ∈ f(ψ(vi)) is denoted as xmi .

It is reasonable to assume there is only one target node type o∗ ∈ O required to be categorized, such as films in movie networks and items in product review networks. Given a predefined set of categories C, the node classification task on MMHNs aims to assign a category label yi ∈ C to each target node vi in a semi-supervised setting.

4 ProposedModel HGNN-IMA

In this section, we at first present the framework of the proposed model, and then elaborate on the key modules.

4.1 Framework

HGNN-IMA is structured into three modules: pre-processing module, heterogeneous network propagation module, and feature fusion module. Since most of real-world MMHNs have only text and vision attributes, we illustrate our model in Figure 2 with these two modalities, i.e., M = {T, V}.

As pre-processing, the feature for each modality of each node is extracted through a corresponding pre-trained encoder. Then, the core propagation module is designed to learn enriched representations of nodes in the framework of heterogeneous graph transformer (HGT) [Hu et al., 2020]. To capture synergetic effect of multi-modal attributes, we primarily innovate with a Cross-modal Influence Unit, which nests the inter-modal attention into modal-specific inter-node attentions, allowing the aggregation weights to be determined by all modalities together. Besides, an additional loss for intermodal attention and a modulation on inter-node attention are employed during propagation to mitigate the impact of nodes with missing and misaligned modalities respectively.

At last, the feature fusion module incorporates the representations from different modalities also in an attentive manner. For semi-supervised training, the cross-entropy loss on both multi-modal and uni-modal features, as well as the attention loss are combined with equal proportions.

4.2 Heterogeneous Network PropagationModule

In order to aggregate the embeddings of heterogeneous neighbors for the semantic enrichment of each node, we adopt HGT as the base architecture instead of dual-level ones [Hu et al., 2019; Wang et al., 2019] to uniformly handle various types of nodes and edges with the help of multiple type-dependent feature projection functions. That facilitates the characterization of complicated interactions among heterogeneous entities and their multi-modal attributes.

Considering the discrepancy among node types, for each node vi and its feature on a modality m, we initially make a linear function dependent on node type to get the embedding h(0),m i at the 0-th layer, as the starting point of aggregation.

where ¯h(0),m i is the output of the m-encoder in the preprocessing phase. Then, each node vi expects to receive information from its neighbor set, denoted as Ni ⊆ V.

Cross-modal Influence Unit

Like previous work of multi-modal fusion in GNNs, the propagation here is still executed for each modality m (e.g., text or vision) separately, but as exemplified in Figure 1, other modalities should also take effect in deciding the importance (weights) of neighbors. Thus, we need to compute these weights from the perspective of each influencing modality based on the similarity of the features on that modality, and unequally combine them in an adaptive way.

Specifically, for each influencing modality m′ ∈ M, given a current node vi and one of its neighbors vj ∈ Ni, the importance of vj for vi at the k-th layer can be calculated as the similarity of the two nodes on this modality, and realized by a bilinear transformation following HGT [Hu et al., 2020]:

where lK and lQ are two linear functions dependent on the node type similar to lI , and W NODE is a learnable matrix dependent on the edge type. Then, the inter-node attention score between vi and its neighbor vj at the k-th layer on the modality m′ is denoted as α(k),m′ and computed as follows:

ij

Here, a single attention head is used for simplicity, and is easy to be extended to multiple heads similar to HGT.

With these attention scores on each modality, it is crucial to determine which modalities are suited to play more important roles. That naturally depends on the features of the two nodes vi and vj, so should be learned adaptively. To this end, we design a nested attention mechanism. At first, the inter-modal attention score for the modality m′ is calculated in two steps:

Notice that we use the same features of the two nodes as the inter-node attention α, but different parameters W MODAL to characterize their correlations from the perspective of crossmodal influence, instead of similarity.

Next, we apply the inter-modal attention λ(k),m′ ij in Equation 5 on the modal-specific inter-node attention α(k),m′ ij in Equation 3, and obtain the combined inter-node attention β(k) ij reconciling the features and effects of all modalities:

It is worth noting that this attention is independent of the current (influenced) modality m. In other words, for a specific node, the features of all modalities are propagated according to unique weights from its neighbors. Although compromising the expressiveness to some extent, this simplification is reasonable to maintain a moderate number of parameters and highlight the essential idea of the cross-modal influence, which is certified via ablation study in Section 5.5.

In view of the complexity of multi-modal data, there exist the phenomenons of modality misalignment and even missing. Therefore, the inter-modal attention and the inter-node attention need to be adjusted accordingly to fulfill a smooth and category-oriented propagation.

Attention Loss for Missing Modalities

As formulated in Section 3, there exist some types of nodes not possessing all modalities originally, and their features on the missing modalities have to be completed in some way, so they should not take much effect compared to modalities with real attributes. To address this issue, we specially design a loss function to constrain the inter-modal attention scores computed by Equation 5 in such cases are not too large:

where K is the number of propagation layers.

Attention Modulation toward Modality Alignment

Even for those nodes possessing all modalities, the attributes of some modalities may not represent the correct meaning of nodes due to the irregularity of Internet. If a node suffering such misalignment propagates too much information to the current node, it is inevitable to import noises in the aggregation. Since the similarity of two nodes on each modality has been computed in Equation 2, we can utilize the consistency of these scores to imply the modality alignment degree of each neighbor, and thus get a new inter-node attention ¯ β for the aggregation weights as follows.

Then, through mixing the two inter-node attentions from different views into final aggregation weights ˜ β(k) ij , the embedding of each modality m at the k-th layer is computed as:

where lM is the fourth function dependent on node type and W MSG is the third learnable matrix dependent on edge type.

At last, to avoid over-smoothing, we introduce a residual connection to get the final output at the k-th layer, with the sigmoid function and another type-dependent function lA :

4.3 Feature FusionModule

Although the computation of embeddings for each modality has already taken cross-modal influence into accounts, they are still required to be fused to get final representations. We use standard modality-level attention to learn adaptive importance of each modality for classification at the last layer:

The final embedding Zi of node vi is then calculated, and the probability distribution Oi of vi for each category is obtained:

4.4 Training Objective

For semi-supervised classification, the cross-entropy loss is used. To embody the effect of each modality, we incorporate the losses of individual modalities into the fused one:

where V ∗L is the labeled target node set, and yi is the one-hot label vector for vi. Omi is the embedding for each modality m, computed similar to Oi in Equation 15, but replacing Zi by h(K),m i . Then, the whole loss function is expressed as the equal-weight sum of the two losses:

It can be inferred that the total computational complexity is O(|V|2 · |M|2 ), which demonstrates the scalability of the model when handling large graphs and multiple modalities. To save space, the detailed analysis is put in Appendix.

5 Experiments

In this section, we first introduce the datasets, baselines, and experimental settings. Subsequently, we demonstrate the superiority of our model over baselines on node classification, followed by ablation study and hyper-parameter analysis.

The experiments were performed on NVIDIA Tesla V100 32 GPUs, and implemented in Python 3.9 with PyTorch. 1

5.1 Datasets

We evaluate our model on five diversified real-world benchmark datasets. The statistics of them are shown in Table 1.

• DOUBAN 2 and IMDB 3 collect data from two online movie websites respectively. Movies (M), actors (A) and directors (D) compose the heterogeneous network with edge types AM, MA, MD and DM. Besides textual descriptions, each movie possesses a poster image, and can be categorized into "Action", "Comedy" and "Drama" ("Drama" only exists in IMDB).
• AMAZON 4 contains items (I) and user reviews (U) in Amazon, with three types of relations: UI, IU and II. Each item (product) has a text and an image. Three subcategories of appliances are used for classification.
• AMAZON-1 and AMAZON-2 are self-constructed larger datasets under the Electronics category in the AMAZON dataset 5 . Four types of relation between items are selected: also buy, also viewed, buy after viewing, and bought together. In AMAZON-2, the price of items is added as the third modality.

5.2 Baselines

For baselines, we choose representative models of the three manners regarding modalities: (1) typical heterogeneous graph neural networks without handling multi-modal attributes (HAN [Wang et al., 2019], SHGP [Yang et al., 2022], SeHGNN [Yang et al., 2023b], HERO [Mo et al., 2024]) and HGT [Hu et al., 2020], for which we employ both early fusion and late fusion strategies; (2) special heterogeneous graph neural networks tackling multi-modal attributes (HetGNN [Zhang et al., 2019] by early fusion , MHGAT [Jia et al., 2022] by late fusion with two versions of inter-node aggregation, and XGEA [Xu et al., 2023] considering fixed influence between modalities); (3) translation-based methods treating the network structure a new modality (IDKG [Li et al., 2023a]). The details are provided in Appendix.

1 The code is available at https://github.com/Jiafan-ucas/ HGNN-IMA

2 https://github.com/jiaxiangen/MHGAT/tree/main/douban

3 https://github.com/Jhy1993/HAN/tree/master/data/imdb

4 https://github.com/jiaxiangen/MHGAT/tree/main/amazon

5 https://nijianmo.github.io/amazon/index.html

5.3 Experimental Settings

As to AMAZON, IMDB and DOUBAN datasets, we directly use the encoded features provided by MHGAT [Jia et al., 2022]. For self-constructed datasets AMAZON-1 and AMAZON-2, texts and images are encoded using CLIP [Radford et al., 2021], while price is embedded through an FFNN. For missing visual attributes, we complete them with the text features after encoding, as the only available information.

We divide each dataset into the training set (20%), the validation set (10%), and the test set (70%). The model is trained using the Adam optimizer with a learning rate of 0.001, incorporating 3 layers of propagation (K = 3) and setting the maximum number of iterations as 300. Each layer's output is subjected to a dropout with a rate of 0.6, and the dimension d of all embeddings is standardized to 64. During propagation, we employ multi-head attention with the number of heads set to 8. To further prevent over-fitting, we employ an early stopping mechanism with a patience of 50, which activates when the validation loss exceeds any previously recorded values and its accuracy dips below the highest recorded one. Both Micro-F1 and Macro-F1 are used for evaluation. We report the average results of five executions with different seeds.

5.4 Overall Results

Table 2 shows the overall results on five datasets. It can be seen that our model consistently outperforms all the baselines across all datasets. For the first three datasets, there are 1.2%, 1.6%, and 0.3% gains in Macro-F1 compared to the strongest baseline. While, for the two larger datasets with more categories, the Macro-F1 value is promoted by around 2.5%. That indicates the model is able to achieve a more balanced performance improvement across all categories on larger datasets.

Specifically, HGNN-IMA is superior to typical HGNNs such as HAN, SHGP, HERO and SeHGNN to a great extent, no matter early pre-processing or late post-processing for multiple modalities, so the special treatment on multi-modal attributes in MMHNs is necessary to understand the categories of entities and thereby worth studying. The secondbest performances of HGT on most datasets are attributed to its use of type-dependent parameters to capture heterogeneous attentions over each edge from a global view. Our model further enhances HGT via effectively handling modalities.

Compared to HetGNN with explicit early concatenation of multi-modal attributes, as well as MHGAT employing late modality-level attention mechanism, the notable promotion of our model can naturally give credit to the innovative nested attention mechanism. It adaptively learns the cross-modal influence and determines the aggregation weights of each node to attain expected information propagation, rather than just blending multi-modal features before or after the propagation. Although XGEA considers the influence of another modality in propagation, it assumes a fixed influence relation and neglects the role of the current modality itself, so fails to identify the complicated synergy of modalities on categoryoriented propagation for each node. Also, the advantage over IDKG implies GNN-based propagation is essential to learn discriminative node embeddings through incorporating highorder correlations, especially facing multi-modal attributes.

Besides, we calculate the standard deviation (STD) of our model and sub-optimal HGT, and conduct a two-sample ttest between them, presented in Table 3. The results confirm the stability of our model. Additionally, the p-value is less than 0.05 in all datasets, indicating a significant difference between HGNN-IMA and baseline models. The intuitive comparison on learned embeddings is displayed in Appendix.

5.5 Ablation Study

Here, we design hierarchical variants of HGNN-IMA to inspect the importance of the key components in the model. Other ablative models are analyzed in Appendix.

• Removing or changing the Cross-modal Influence Unit
• HGNN-IMA-cross: It eliminates this unit and directly uses inter-node attention α(k),m′ ij to replace β(k) ij in Equation 6 for the aggregation.
• HGNN-IMA-adapt: It utilizes the mean of α(k),m′ ij on each modality to serve as β(k) ij in Equation 6, losing weight adaptability.
• HGNN-IMA+inf: It distinguishes the influenced modality m when computing the inter-modal attention to form λ(k),m′ ,m ij in Equation 5.
• HGNN-IMA-nei: It neglects the distinction among the neighbors vj when computing the intermodal attention to form λ(k),m′ i in Equation 5.
• Removing attention modulation for modality alignment
• HGNN-IMA-align: It directly uses inter-node attention β(k) ij to replace ˜ β(k) ij in Equation 10.
• Removing some part of loss functions
• HGNN-IMA-Latt: It removes the attention loss to disregard the modality missing issue.
• HGNN-IMA-Lind: It removes the individual modality part from the cross-entropy loss.

Table 4 proves that the Cross-modal Influence Unit is beneficial, and the weights of influence should be adaptively learned rather than pre-defined. We also find compared to the traditional attention score α(k),m ij , our nested one β(k) ij aligns more closely with node categories, which is visualized in Appendix. Regarding the specific design of the nested intermodal attention, when altering the granularity such as adding the influenced modality or removing the influencing node, the performance declines in varying degrees. Hence, while the core idea is simple, the tradeoff between expressiveness and complexity in defining the cross-modal influence is nuanced.

Then, when removing the modulation term, the model exhibits decreased performance for all datasets. That certifies the unconstrained propagation according to the crossmodal influence is susceptible to noises imported by inaccurate modality attributes. The modulation based on similarity consistency just weakens this awkward impact through focusing more on those nodes which realize modality alignment.

At last, the ablation for loss functions highlights their respective roles within the model. An exception appears when the single-modality loss is removed in the AMAZON-2 dataset. That can be explained as the introduction of the new modality, so it should be cautious to consider the contribution of individual modalities, which may bring a negative impact.

5.6 Hyper-parameter Analysis - Layer Number

We change the layer number K from 0 to 4, as shown in Figure 3. It can be observed that the trend of Macro-F1 scores is similar across all datasets and arrives an optimal value when K = 3. That is because adopting fewer than 3 layers prevents sufficient information exchange, whereas the continual increase of layers would lead to over-smoothing.

6 Conclusion

This paper delves into the intricate problem of node representation learning within multi-modal heterogeneous networks, characterized with complicated interactions of modalities and node/edge types. To overcome the limitations associated with early or late fusion of multi-modal features, we put the fusion inside the GNN-based propagation process, thereby prompting node representations to align closely with category labels. Notably, the innovative inter-modal attention acting on the modal-specific inter-node attention is proposed to enable adaptive modal fusion, based on the heterogeneous graph transformer framework. Moreover, another two critical factors in multi-modal data, modality alignment and missing, are also integrated into the model in a straightforward way to achieve significant improvements on node classification.

Future work will extend this method to more tasks demanding node representation learning with network structures.

Acknowledgements

This work is supported by CAS Project for Young Scientists in Basic Research (YSBR-040), National Natural Science Foundation of China (62373061), Beijing Natural Science Foundation (L232028), and the Project of ISCAS (ISCASJCMS-202401).