Continual Multimodal Knowledge Graph Construction

1 Introduction

The rise of multimodal data on social media platforms has sparked significant interest among knowledge graph and multimedia researchers in the domain of multimodal knowledge graphs [Liu and et al, 2019; Zhu et al., 2022; Zheng et al., 2023; Hu et al., 2023; Liang et al., 2024]. To address the limitations of relying on human-curated multimodal data and to systematically extract insights from vast multimedia repositories, the concept of Multimodal Knowledge Graph Construction (MKGC) has been proposed [Zhang et al., 2023; Liu et al., 2023]. MKGC leverages multimodal data as an additional information source to disambiguate polysemous terms and perform tasks like Multimodal Named Entity Recognition (MNER)[Lu et al., 2022] and Multimodal Relation Extraction (MRE)[Zheng et al., 2021a]. However, existing MKGC architectures [Zheng et al., 2021a; Chen et al., 2022b] primarily focus on "static" knowledge graphs, where entity categories and relations remain fixed We posit that this observation may decline in MKGC models during continual learning may stem from disparate convergence rates among different modalities [Wang et al., 2020], leading to two primary challenges: Challenge 1: How to alleviate the imbalanced learning dynamics across modalities to enhance the plasticity? The differential learning dynamics in multimodal settings can hinder the adaptability of MKGC models to new entity categories and relations, especially when employing replay strategies. This imbalance may result in inferior representations, undermining the addition of new knowledge. Challenge 2: How to reduce the forgetting in the process of multimodal interaction? Continual MKGC models face the unique problem of varying forgetting rates across modalities, unlike their unimodal counterparts. These disparities can disproportionately affect secondary modalities, increasing the risk of forgetting and jeopardizing the performance of prior tasks. Addressing these challenges necessitates the development of continual MKGC models that ensure uniform multimodal forgetting and robust modality integration to manage the retention and acquisition of knowledge.

∗ Corresponding author.

To overcome the highlighted challenges in continual MKGC, we introduce the Multimodal Stability-Plasticity Transformer (MSPT), a novel framework that advances the stability-plasticity trade-off through strategic multimodal optimization. Our method is distinguished by two pivotal modules: (1) Gradient Modulation for Balanced Learning: We propose a gradient modulation technique to address the imbalanced learning dynamics across modalities, thereby preserving the model's ability to learn new information. By adaptively tuning gradients according to each modality's optimization contribution, our approach ensures nuanced representation development for both modalities, enhancing plasticity. (2) Hand-in-Hand Multimodal Interaction with Attention Distillation: Deviating from traditional cross-attention multimodal interaction, MSPT calculates inter-modal self-query affinities against an external learnable key. This decoupling of fusion parameters allows for a more deliberate modulation of forgetting rates, promoting consistent knowledge retention. And the attention distillation is utilized to refine this process, leveraging the multimodal interaction outputs to preserve crucial attention patterns. The results of our thorough experiments demonstrate that MSPT outperforms both traditional MKGC and continual unimodal KGC models in various class-incremental settings, showcasing its potential in the field of continual MKGC 1 .

2 RelatedWorks

2.1 Advancements inMKGC

Multimodal Named Entity Recognition. Advancements in MNER have shifted from text-only approaches to also harnessing visual cues. Studies such as those by [Zhang et al., 2018; Lu and et al, 2018; Moon et al., 2018; Arshad et al., 2019] have introduced interactions between CNN-driven visual and RNN-based textual features. Others, UMT [Yu et al., 2020] and UMGF [Zhang et al., 2021], have suggested utilizing fine-grained semantic correspondences with a combination of transformer and visual backbones, taking into account regional image features to represent objects. The ITA [Wang et al., 2022b] model exploits self-attention to enrich text embeddings with image spatial context, showing superiority over text-centric models.

1 Our data and code are available at https://github.com/zjunlp/ ContinueMKGC

Multimodal Relation Extraction. Researchers have started exploring techniques to link entities mentioned in the textual content with corresponding objects depicted in associated images. Some examples include work done by [Zheng et al., 2021b], who presents an MRE dataset that can associate the textual entities and visual objects for enhancing relation extraction. Then [Zheng et al., 2021a] revises the MRE dataset based on [Zheng et al., 2021b] and utilizes scene graphs to align textual and visual representations. [Wan and et al, 2021] also collects and labels four MRE datasets based on four famous works in China to address the scarcity of resources for multimodal social relations.

2.2 Continual Knowledge Graph Construction

Continual learning addresses catastrophic forgetting in the following strategies: consolidation-based methods [Zenke et al., 2017; Liang et al., 2023] that adjust parameter updates through regularization, dynamic architectures [Rusu et al., 2016] that evolve with data, and rehearsal-based methods [Sprechmann et al., 2018; Chaudhry et al., 2019] using memory banks to preserve knowledge. The latter has exhibited superior performance in continual KGC [Monaikul et al., 2021]. To address the challenge of continual RE, memory interaction methods [Cui et al., 2021] have been proposed to effectively utilize representative samples. Additionally, prototype methods [Han et al., 2020; Cui et al., 2021] are increasingly employed to abstract relation information and mitigate overfitting. In the context of continual NER, the ExtendNER method [Monaikul et al., 2021] tackles classincremental learning by creating a unified NER classifier that encompasses all encountered classes over time. Moreover, approaches [Xia et al., 2022; Wang et al., 2022a] prevent forgetting of previous NER tasks by utilizing stored or generated data from earlier tasks during training. However, previous studies have focused on continual KGC and have not been readily applicable to MKGC due to the inherent challenges posed by multimodal data.

3 Preliminaries

3.1 Delineation of MKGC Tasks

DEFINITION 1. MNER. This subtask emphasizes the extraction of named entities from textual content and its associated images. Given a token sequence, denoted as xt = [w1, . . . , wm], and its affiliated image patch sequence xv , the principal goal of continual MNER is to consistently model the sequence tags' distribution, expressed as p(y|(xt , xv )). Within this context, for task Tk, the label sequence y is defined as y = [y1, . . . , ym] and integrates emergent entity types from the entity category set Ek.

DEFINITION 2. MRE. This subtask focuses on extracting relationships between designated entity pairs from token sequences. For a given task Tk, and provided with a token sequence xt and its corresponding image patch sequence xv , the goal is to infer the relationship of a specific entity pair, (eh, et), derived from xt . A key challenge lies in computing the probability distribution over possible relations r from the set Rk, expressed as p(r|(xt , xv , eh, et)). This is made more complex by the potential addition of novel relations to Rk.

3.2 Class-Incremental Continual Learning

We define a class-incremental continual learning scenario as a series of K separate tasks, each with its schema classes and MKGC corpus. Formally, the tasks are denoted as:

The k-th task Tk includes a distinct set of entity types Ek and relations Rk, along with an MKGC corpus C k which is divided into training, validation, and testing subsets Dk, Vk, and Qk, respectively. Each training instance in Dk consists of a textual input xt , a sequence of image patches xv-utilizing ViT encoding-and a corresponding label y, which is either an entity from Ek or a relation from Rk. Learners are restricted to use only the data from Dk during the training phase of task Tk, and to ensure non-overlapping classes between tasks, we enforce Ei ∩ Ej = ∅ and Ri ∩ Rj = ∅ for i ̸= j. This setup follows the convention of several benchmark methodologies [Masana et al., 2023]. In our classincremental MKGC setting, after training completes on Dk, the model undergoes evaluation across an aggregated test set ∪i = 1kQi, which includes all class categories up to the current task. This differs from task-incremental learning, where evaluation is confined to the specific task S k . The evaluation metrics are introduced as follows:

DEFINITION 3. Forgetting Metric (Ak): ** Measures the F1 score on aggregate test sets ⋃ki=1 Qi for tasks {Ti}ki=1 post-training on Tk. It indicates the model's ability to prevent catastrophic forgetting, especially in sequential data with new entity categories and relations.

DEFINITION 4. Plasticity Metric (Uk): ** Defined by the F1 score on the current task Tk, showcasing the model's capacity to learn new tasks while retaining existing knowledge, a critical aspect of continual learning.

4 Methodology

4.1 Framework Overview

As illustrated in Figure 2, our continual KGC framework adopts a dual-stream Transformer structure with the taskspecific paradigm, including:

(1) Structure. We incorporate a Visual Transformer (ViT) [Dosovitskiy et al., 2021] for visual data and BERT for textual data. Building on prior research [Clark et al., 2019; Chen et al., 2022a], which indicates that manipulating the upper layers of language models (LMs) more effectively leverages knowledge for downstream tasks, our framework engages in multimodal interactions and attention distillation within the top three layers of the Transformers.

(2) Task-specific paradigm. For the MRE task, we employ a task-specific approach by fusing the [CLS] token representations from both ViT and BERT models. This integrated representation enables us to derive the probability distribution over the relation set R for the given task.

where ht ∈ Rmt×dt and hv ∈ Rmv×dv represent the output sequence embeddings from BERT and ViT, respectively. In the context of MNER, for fair benchmarking against prior work, we employ a CRF function akin to that in the MSPT framework. For the entity tag sequence y = [y1, . . . , ym], we enhance the BERT embeddings with hvcls and positional embeddings Etpos to capture visual information. The probability of a tag sequence y within the predefined label set Y is computed using the BIO tagging scheme that follows in [Lample et al., 2016] as:

4.2 BalancedMultimodal Learning Dynamics

Modulating Optimization with Gradient. Diverse convergence rates across modality-specific parameters can lead to imbalanced learning dynamics during continual learning, potentially hampering current task performance. To address this, we propose a gradient modulation strategy to fine-tune the optimization of visual and textual encoders, depicted in Figure 2(b). Building upon concepts, we adapt these to the k-th task using the Stochastic Gradient Descent (SGD) algorithm:

where φ(θv(k) n ) = 1N ∑x∈Bn∇θv ℓ(x; θv(k) n ) is an unbiased estimation of the full gradient, Bn represent a random mini-batch with N samples at step n of optimization, and ∇θv ℓ(x; θv(k) n ) denotes the gradient w.r.t. batch Bn.

Drawing from [Peng and et al, 2022], to counteract imbalanced multimodal learning dynamics, we introduce an adaptive gradient modulation mechanism for visual and textual modalities. This is based on quantifying their respective contributions to the learning goal via the contribution ratio γn:

To dynamically assess the contribution ratio γ tn between textual and visual modalities, we introduce a modulation coefficient gtn that adaptively regulates the gradient, defined as:

where α is a hyper-parameter that adjusts the influence of modulation. , Gt(k) is the averaged modulation coefficient

• (a) Hand-in-hand Multimodal Interaction with Attention Distillation.

(b) Balanced Multimodal Learning Dynamics.

Figure 2: Overview of our MSPT framework.

of the model trained after the task k. We further propose to balance the multimodal learning rhythm by integrating the coefficient gtn into the SGD optimization process of task k in iteration n as follows:

Remark 1. Through gradient modulation in task 1 and using the average coefficient from the preceding k-1 tasks to influence training on the current k-th task, we ensure a smoother transition in balanced multimodal learning across tasks.

4.3 Hand-in-handMultimodal Interaction via Attention Distillation

Inspired by the concept of collaborative progress with the saying goes "hand in hand, no one is left behind", our method establishes a coherent framework for continual learning by integrating a dual-stream Transformer with attention distillation. As depicted in Figure 2(a), the model promotes uniform learning across various modalities, reducing unequal forgetting and enhancing multimodal resilience.

Hand-in-handMultimodal Interaction

The self-attention mechanism (SAM) [Vaswani and et al, 2017], central to Transformer-based architectures, derives attention maps through self-key and self-query similarity calculations. Our proposed multimodal interaction approach introduces a unique attention generation process using shared learnable keys (KW) and corresponding self-queries to enhance knowledge consolidation and retention. This method aims to counteract catastrophic forgetting by embedding previous task knowledge into the attention framework. It also promotes a tighter integration between visual and textual encoders, minimizing fusion bias and inconsistency associated with forgetting. Additionally, by regulating updates to KW, our strategy preserves knowledge from earlier tasks, safeguarding against information degradation during new task.

Applying linear transformations to the input tensors Xv and Xt, we obtain the visual self-query QXv = W qv Xv and self-value V Xv = W vv Xv, alongside the textual self-query QXt = W qt Xt and self-value V Xt = W vt Xt using the visual and textual encoders' parameters W qv , W vv , W qt , W vt , respectively. We introduce a shared external key Ks that supersedes the original self-key, generating updated attention maps for both modalities. For the k-th task, utilizing a ViT and BERT model, we denote the prescaled attention matrix at the l-th layer as A(k) l and the resulting SAM output as Z (k) l , prior to softmax activation. Note that details on multi-head attention and normalization are omitted for conciseness.

where L denotes the encoder's total number of layers, and Bv l and Bt l serve as bias terms for the vision and text attention maps, respectively. Nnote that the external key Ks l is not constrained by the current feature input, allowing for end-toend optimization and the integration of prior knowledge.

Core Attention Distillation

Accordingly, we introduce a method for refining the attention matrices within the dual-stream Transformer's interaction module. This involves stabilizing attention maps through a distillation function that leverages learnable shared keys Ks to mitigate forgetting. Considering the attention maps on thevisual side across consecutive steps k and (k - 1), we quantify the distillation loss in the width dimension as:

where H and W represent the height and weight of the attention maps. The total distance between attention maps a and b along the h or w dimension is represented by δ(a, b).

Our proposed attention distillation framework incorporates a crucial asymmetric distance function δ, diverging from typical continual learning approaches that use symmetric Euclidean distance for model outputs comparison across tasks k and (k - 1)[Wang et al., 2022a; Douillard et al., 2020]. Symmetric distances tend to equally penalize shifts in attention from both new and old tasks, potentially impeding learning by increasing the loss when attention to previous tasks is maintained. Although preserving past tasks' knowledge mitigates forgetting, over-penalization can inadvertently suppress newly acquired insights, creating a tension between preserving past knowledge and embracing new information. To address this, we suggest δ, an asymmetric distance measure that conserves prior knowledge while sustaining the model's adaptability, aligning with findings in computer vision[Pelosin et al., 2022]. The modified function, δW , is specifically crafted to balance the trade-off between plasticity and forgetting, illustrated as follows:

We employ Fasym, an asymmetric distance function, with ReLU [Nair and Hinton, 2010] integrated as Fasym in subsequent experiments. The attention distillation loss is:

Remark 2. This setup permits the development of new attention patterns during the k-th task without penalties, while attention absent in the current but present in the (k - 1)-th task is penalized, promoting targeted knowledge retention.

4.4 Training Objective

Our model leverages a cross-entropy loss (LCE) to effectively recognize entities and relations, while an attention distillation loss (LAD) mitigates the issue of catastrophic forgetting. We formulate the combined loss function as:

Here, λ serves as the weighting factor for the attention distillation loss. Additionally, we adopt the rehearsal strategy from PR-CRE to retain a concise memory set-merely six examples per task-for continual learning alignment, and optimizing memory footprint.

5 Experiments

5.1 Incremental MKGC Benchmarks

IMNER Benchmark. We utilize the established Twitter2017 MNER dataset, which consists of multimodal tweets from 2016-2017, containing examples with multiple entity categories. To simulate more realistic learning conditions and reduce labeling ambiguity, we transition to a classincremental framework, modifying the dataset such that each entity category is exclusive to a single task.

IMRE Benchmark. For our IMRE benchmark, we partition the dataset into 10 subsets for 10 distinct tasks. The original benchmark imposes two constraints that are at odds with the principles of lifelong learning: (1) clustering semantically related relations, and (2) excluding the "N/A" (not applicable) class. To rectify this, every task incorporates the "N/A" class, and relations are randomly sampled without bias, enhancing the benchmark's diversity and adherence to real-world lifelong learning conditions.

5.2 Compared Baselines

We benchmark our MSPT against SOTA multimodal baselines to demonstrate its effectiveness: 1) UMT [Yu et al., 2020]; 2) UMGF [Zhang et al., 2021]; 3) MEGA [Zheng et al., 2021a]; 4) MKGformer. Apart from previous multimodal approaches, we also compare MSPT with typical continual learning methods for a fair comparison as follows: 1) Vanilla fine-tunes a BERT model on new task data without memory, acting as a lower bound for catastrophic forgetting. 2) Joint Training retains all data in memory, retraining the MKGformer for each task, establishing an upperperformance limit. 3) EWC constrains critical parameter shifts to preserve performance on prior tasks. 4) EMR combines new task data with a memory of key past samples for incremental learning. 5) EMAR-BERT employs reconsolidation and activation techniques to address catastrophic forgetting. 6) RP-CRE represents the forefront in continual relation extraction, using stored relation samples to refine prototypes. 7) ExtendNER applies KD, leveraging an existing NER model to guide the learning of a subsequent model.

5.3 Performance on IMRE Benchmark

Experiments on the IMRE benchmark (Table 1) yield several insights: (1) Fine-tuning unimodal BERT (Vanilla approach) with new examples leads to performance degradation due to overfitting and catastrophic forgetting. Surprisingly, multimodal models, expected to outperform Vanilla, delivered inferior results, emphasizing the need for research in continual multimodal learning. (2) Our method MSPT outperforms all existing MKGC models. While other continual learning approaches, utilize memory modules and sampling strategies to reduce forgetting, they are outstripped by MSPT in the 10split IMRE benchmark, highlighting our method's effective use of multimodal interactions.

5.4 Performance on IMNER Benchmark

In this section, we thoroughly compare MSPT with baseline methods across two task orders, detailed in Table 2. The insights are as follows: (1) Overall performance: Despite variations in MKGC model performance, these models generally lag behind unimodal BERT in continual MNER tasks, highlighting unresolved challenges in multimodal continual learning. Yet, MSPT significantly outshines all competing methods on the IMNER benchmark, demonstrating its robustness and ability to overcome the limitations of previous MKGC approaches in continual settings. (2) Task order robustness: To test MSPT's robustness and order independence, we evaluate it on two entity-type permutations: "PER → ORG → LOC → MISC" and "PER → LOC → ORG → MISC". MSPT consistently tops baselines across permutations, indicating it is not bound to a particular order and can generalize effectively. This across-the-board superiority on the IMNER benchmark confirms the method's effectiveness. (3) The "M-[]" series methods surpass both RP-CRE and our MSPT but do not reach the performance levels of SOTA unimodal continual RE methods, suggesting that simple transferbased strategies are inadequate for optimal performance.

5.5 Ablation Study and Analysis

Effect of EachComponent. Table 3 reveals that each component generally enhances model performance. Specifically, the "MM" strategy boosts the average forgetting metric by 28.5%, resonating with evidence that rehearsal is effective for continual KGC. "GM" leads to a 22.2% increase in F1 score, suggesting the necessity of balancing learning rates across modalities to reduce forgetting. "AD" yields a 10.4% F1 score improvement, indicating that preserving attention patterns aids in retaining prior knowledge. "MI" shows a 5.7% F1 score gain, confirming its crucial role in consistent learning. Notably, omitting "MI" resulted in a temporary performance spike on the second task, potentially due to the self-attention mechanism's efficacy in short-term learning, enhanced by attention distillation. However, our findings suggest this approach is less suitable for longer task sequences. The performance declines across all tasks when the other components are removed, further validating the effectiveness of each proposed element.

Plasticity Assessment. Our evaluation of model plasticity, depicted in Figure 3, indicates that MSPT surpasses other models employing continual learning strategies like RP-CRE and EWC. We found that Joint Training exhibits the lowest plasticity due to its reliance on replaying all previous tasks' data, which hampers the model's ability to adapt to new tasks. The results highlight the superior plasticity of MSPT, which outperforms other continual learning approaches and competes with leading multimodal methods. Through attention distillation, MSPT strikes a balance between maintaining past knowledge and adapting to new information, thereby mitigating catastrophic forgetting effectively.

Imbalance Modulation Analysis. Our evaluation investigates our method's capability to mitigate training imbalances by monitoring the discrepancy ratio γnt , which reflects inter-modality disparity. Figure 4 demonstrates that our method successfully minimizes γnt , indicating its effectiveness in rectifying the common issue of modality imbalance in datasets. Through nuanced modulation, our approach ensures equitable learning across modalities, promoting a balanced contribution to the learning process.

Model Dependence on Rehearsal Size. The performance of rehearsal-based continual MKGC models is inherently linked to the rehearsal size, which governs the volume of training samples preserved. We assessed our model's robustness by evaluating its performance under varying rehearsal sizes. Our MSPT model consistently outperforms competing methods on the IMRE benchmark, regardless of the allocated rehearsal size, as depicted in Figure 5. This steadfastness highlights our method's capability to maintain per- Task ID

Figure 6: Sensitive Analysis on Task Numbers.

formance even when faced with constraints on rehearsal size. Remarkably, the superiority of our model becomes more apparent with smaller rehearsal sizes, showcasing its effective utilization of limited memory resources.

Sensitive Analysis on Task Numbers. We assess how the number of tasks impacts the MSPT model's performance, using the IMRE benchmark with 5, 7, and 10 tasks. All methods, including RP-CRE, MKGformer, and Vanilla, were tested under uniform experimental conditions: identical random seeds, hyperparameters, and task sequences. MSPT demonstrates superiority over RP-CRE and other baselines for all task quantities, showcasing consistent performance regardless of the number of tasks. This consistency confirms the robustness and adaptability of MSPT for continual MRE.

6 Conclusion and FutureWork

Our study introduces the novel concept of continual MKGC, addressing the critical and practical challenge of continuously recognizing new entity categories and relations within a knowledge graph. We present a benchmark for MKGC and propose a unique approach named MSPT, which adeptly combats the dual challenges of catastrophic forgetting and plasticity, central issues in continual learning. MSPT employs a harmonized multimodal training approach to improve the detection of novel patterns, alongside a synergistic multimodal interaction with attention distillation to effectively retain previous knowledge. Comprehensive experiments and analysis demonstrate the superiority of MSPT over existing techniques in the context of continual learning. Future work will aim to expand our approach to a broader range of MKGC and investigate rehearsal-free strategies for continual MKGC.

Acknowledgments

We would like to express gratitude to the anonymous reviewers for kind comments. This work was supported by the National Natural Science Foundation of China (No. 62206246), the Fundamental Research Funds for the Central Universities (226-2023-00138), Zhejiang Provincial Natural Science Foundation of China (No. LGG22F030011), Yongjiang Talent Introduction Programme (2021A-156-G), CCF-Baidu Open Fund, and Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.