HyLoVQA: Dynamic Hypernetwork-Generated Low-Rank Adaptation for Continual Visual Question Answering

1 Introduction

Visual Question Answering (VQA) is a central task in multimodal intelligence [Antol et al., 2015; Johnson et al., 2017]. Given an image (or video) and a natural-language question, a VQA model predicts the answer. Solving VQA requires grounding language in visual evidence and reasoning over the scene in a question-dependent way. This often calls for multi-step, compositional inference rather than direct recognition alone. Over time, the data distribution and question patterns may change, motivating robustness under shift and continual adaptation [Gama et al., 2014; Parisi et al., 2019; Hayes et al., 2020].

∗Corresponding author.

Continual learning (CL) offers a natural framework for continual VQA by learning from non-stationary data streams. It aims to acquire new knowledge while retaining prior capabilities. A central challenge in CL is the stability-plasticity trade-off: models must remain stable enough to avoid forgetting while staying plastic enough to rapidly integrate novel information. Classic CL methods include regularization, rehearsal, and architectural methods for unimodal learning. When applied to VQA, these ideas face added pressure from multimodal grounding and compositional reasoning. Shifts can arise not only in vision or language, but also in their cross-modal co-occurrences and required reasoning, which makes retention and updating more difficult than in unimodal classification. Recent continual VQA work adapts CL principles to multimodal grounding and compositional shifts; VQACL, for example, evaluates unseen skill-concept compositions [Zhang et al., 2023].

Despite recent progress, most continual VQA methods still follow classic CL recipes. They adapt by updating the main model and mitigate forgetting with regularization or rehearsal [Sun et al., 2023; Greco et al., 2019]. Regularization improves stability but can reduce plasticity under large shifts. Rehearsal depends on the memory budget and how well stored samples remain representative as the stream evolves [Kirkpatrick et al., 2017a; Li et al., 2019]. These strategies can retain performance on past data. However, most existing methods adapt by updating a largely shared parameter set across the stream. This often induces cross-level task interference, where previously learned tasks disturb the current one. They also struggle to dynamically adapt to the current task and object. Here, the task is the capability required by the question (e.g., count), and the object is the grounded entity in the input (e.g., a cat). In this paper, a sample refers to a VQA instance, which can be viewed as a combination of a textual task and a visual object. As illustrated in the top of Figure 1, these methods can introduce task and object ambiguity, leading to incorrect answers.

To address this limitation, we propose HyLoVQA for continual VQA. It ensures parameter efficiency, enabling the model to adapt to each task and object dynamically while reducing drift-induced interference, as shown in Figure 1. First, we maintain a Drift-Resilient Memory Anchor Bank. It stores compact anchors that capture the content of visual objects and textual tasks, and the anchors are updated using current input features to remain stable under non-stationary streams. Second, we introduce a Hypernetwork-Generated LoRA Module, where a hypernetwork generates lightweight LoRA adapters from the retrieved anchors. This ensures parameter efficiency and allows the model to dynamically adapt to each task and object, while mitigating interference from shared backbone updates. Third, we propose a SemanticFunctional Alignment loss that aligns semantic discrepancy (feature space) to functional change (parameter space). It avoids updates that deviate from the current task and object.

In this paper, we make the following contributions:

• We develop a drift-resilient memory bank that stores anchor representations of visual objects and textual tasks, and updates them online using current input features.
• We generate lightweight LoRA adapters using a hypernetwork conditioned on retrieved anchors, enabling parameter-efficient and dynamic adaptation to each task and object.
• We formulate an alignment objective that couples feature-space semantic discrepancies with parameterspace functional changes, thereby constraining LoRA adapters to stay focused on the current task and object.
• We evaluate HyLoVQA on VQA v2 and NExT-QA under both standard and novel compositional test settings. On standard tests, HyLoVQA achieves 45.41% AP / 2.82% AF on VQA v2 and 42.43% AP / 2.07% AF on NExT-QA.

2 RelatedWork

2.1 Visual Question Answering

Visual Question Answering (VQA) asks a model to answer a natural-language question about an image[Goyal et al., 2017]. Early pipelines paired CNN image encoders with RNN question encoders and used attention for visual grounding[Ren et al., 2015; Devlin et al., 2019]. Fusion evolved from concatenation to bilinear pooling, with Multimodal Compact Bilinear pooling as an efficient high-order design[Kim et al., 2018; Theeuwes, 2010]. Object-centric representations further advanced VQA via region-level attention, where Bottom-Up and Top-Down attention over detected regions became a widely used baseline[Chen et al., 2020; G´omez-P´erez and Ortega, 2020]. Recently, Transformer-based architectures and large-scale vision-language pretraining have dominated VQA, improving multimodal representation learning and reasoning in the pretrain-then-finetune paradigm[Liang et al., 2020].

However, as VQA applications expand to more complex and dynamic scenarios, the assumptions of a fixed task and a stationary data distribution become fragile. For traditional VQA models, high scores on standard benchmarks may not transfer to VQA settings with distribution shifts [Whitehead et al., 2021; Li et al., 2019]. Meanwhile, incomplete or noisy images and open-ended question formulations further amplify the limitations of fixed training setups. Therefore, VQA requires models that can adapt as tasks and data distributions keep changing, while avoiding forgetting previously learned knowledge. This has motivated continual learning for VQA to mitigate catastrophic forgetting and support longterm adaptation [Zhang et al., 2021].

2.2 Continual Learning in VQA

Continual Learning (CL) studies learning from sequential data streams whose distribution shifts over time [Wan et al., 2022a; Lopez-Paz and Ranzato, 2017]. A central challenge is the stability-plasticity trade-off [Kim et al., 2024]. Models must acquire new knowledge while retaining old knowledge. CL has been widely explored in vision and language, and recent efforts extend it to multimodal settings. To mitigate catastrophic forgetting, most methods fall into three categories: regularization, rehearsal, and architectural approaches. Regularization methods constrain updates to parameters that are important for past tasks, such as MAS and NPC [Aljundi et al., 2018; Paik et al., 2020]. Rehearsal methods replay stored samples to stabilize training, such as ER and DER [Chaudhry et al., 2019; Chaudhry et al., 2018; Wan et al., 2022b]. Architectural methods are commonly studied in unimodal continual learning[Rusu et al., 2016].

Recent continual VQA methods tackle evolving visual concepts and language. VQACL proposes a dual-level task stream and compositional testing. QUAD and ProtoGroup reduce memory via attention consistency and multi-prototype grouping. PROOF mitigates interference with expandable projections, while CLT-VQA targets long-tailed continual VQA via prototype balancing and feature alignment[Marouf et al., 2025; Zhang et al., 2025b; Zhou et al., 2025; Zhang et al., 2025a]. Yet these approaches typically keep a fixed backbone, so interference from shared parameters can still accumulate over time. In contrast, our work reduces cross-level task interference and improves accurate adaptation to the current task and object. By preserving past knowledge while maintaining parameter efficiency, it provides a more reliable and adaptive solution for long-term continual VQA.

3 Methodology

We propose HyLoVQA, a lightweight continual learning framework (Figure 2) for visual question answering, addressing dynamic adaptation, memory robustness under drift, and stable knowledge updates. By integrating a DriftResilient Memory Anchor Bank, a Hypernetwork-Generated LoRA Module, and a Semantic-Functional alignment loss, HyLoVQA enables parameter-efficient, task/object-specific adaptation while reducing interference on the shared backbone. Module details are described below.

3.1 Drift-Resilient Memory Anchor Bank

The Drift-Resilient Memory Anchor Bank (DMA) stores compact, modality-specific anchors that summarize visual objects and textual tasks. For each incoming sample, DMA retrieves the most relevant visual and question anchors to form an anchor-conditioned context, which then conditions the hypernetwork to generate LoRA adapters. This yields an explicit long-termmemory for continual VQA.

Multimodal representations are constructed from the input image and question as follows. Given an image I, N salient regions are extracted and each region is embedded into a d-dimensional token by linearly projecting its RoI feature fj,roi ∈ Rdroi , adding a projected normalized bounding box bj = [x1, y1, x2, y2] ∈ R4 (each coordinate in [0, 1]), and a learnable region-id embedding ej,id ∈ Rd . The resulting region token is denoted by vj ∈ Rd . Stacking all region tokens yields V ∈ RN×d , and a global visual summary is computed by mean pooling:

For a question Q with M tokens, a text encoder outputs token features Q ∈ RM×d , and a global question summary is obtained as

where Pool(·) denotes a pooling operator.

Within task i, a sample denotes an image-question pair. DMA updates the anchor bank online per sample, and the within-task index is omitted by using Pv,i and Pq,i to denote the current bank. At the beginning of task i, anchors are inherited from the previous task (for i=1, Pv,0 = {f (k) v }Kv k=1 and Pq,0 = {f (k) q }Kq

For each incoming sample, the most relevant anchors are retrieved by cosine similarity cos(a, b) = a⊤b ∥a∥2 ∥b∥2 :

Only the retrieved anchors are updated using momentum:

where α, β ∈ [0, 1) are sample-adaptive coefficients. For all non-retrieved anchors, they are kept unchanged, e.g., pv,k,i = pv,k,i-1 for k ̸= k∗ v and pq,k,i = pq,k,i-1 for k ̸= k∗ q , yielding the updated bank Pv,i and Pq,i.

Finally, the instance context is defined as

To further mitigate representation drift in continual learning, we introduce a memory consistency loss. We use Lmem to denote the aggregate of modality-specific memory terms:

In the next section 3.2, it is detailed how zinst,i conditions the hypernetwork to generate LoRA adapters.

3.2 Hypernetwork-Generated LoRAModule

The Hypernetwork-Generated LoRA Module (HyGen) enables continual VQA adaptation by generating lightweight LoRA adapters for a largely shared backbone. A hypernetwork produces these updates conditioned on the anchors retrieved from DMA, allowing sample-wise, dynamic adaptation under non-stationary streams. This design adapts the model by injecting Hypernetwork-Generated LoRA adapters into a shared backbone, maintaining parameter efficiency while reducing interference across tasks.

The construction starts from generating LoRA factors for each incoming sample within task i, conditioned on its instance representation zinst,i. Let Hϕ(·) denote a hypernetwork with parameters ϕ. To enable layer-specific generation, we additionally introduce a learned layer embedding eℓ ∈ Rdℓ . For each fusion layer ℓ ∈ {1, . . . ,L}, we generate LoRA factors for cross-attention projections as

where X ∈ {Q,K, V } denotes the projection type (query/key/value), Aℓ,X ∈ Rr×din , Bℓ,X ∈ Rdout×r , and the rank satisfies r ≪ min(din, dout) (typically din = dout = d).

Given the generated factors, the frozen cross-attention projections are augmented to obtain sample-adaptive projections. We represent each sequence as a matrix whose rows are token embeddings, and apply linear projections by rightmultiplication. Let Wℓ,base,X ∈ Rdin×dout denote the frozen base projection. The sample-adaptive projection is formed as

where λ ≥ 0 controls the adaptation strength.

In parallel to the parameter adaptation, the retrieved anchors are injected as memory-guided global context tokens by prepending them to the original sequences (concatenation along the sequence-length dimension). Here Q ∈ Rnq×d and V ∈ Rnv×d denote the input token sequences on the query side and value side of cross-attention, respectively (note that V here denotes a sequence, not the value projection). We compute

where pv,k∗ v ,i ∈ Rd and pq,k∗ q ,i ∈ Rd are the retrieved anchor embeddings for task i, and Wg ∈ Rd×d is learnable. Accordingly, gv, gq ∈ Rd are global context tokens.

With the augmented sequences and sample-adaptive projections in place, the model performs adaptive multimodal fusion and outputs the answer distribution. At layer ℓ, we compute cross-attention with adaptive projections as

where dk is the key dimension used for scaling (typically dk = d/h for h attention heads), and Softmax(·) is applied row-wise. For brevity, we omit the standard multi-head reshape and per-head projections in the notation.

After L fusion layers, we pool the fused sequence into eout and predict an answer distribution π ∈ [0, 1]C via an MLP with Softmax. We construct a soft target distribution t ∈ [0, 1]C from multiple human annotations.

We optimize the model by minimizing LCE.

3.3 Semantic-Functional Alignment loss

The Semantic-Functional (SF) Alignment loss stabilizes anchor-conditioned adaptation by enforcing consistency between semantic and functional changes. Specifically, it matches the semantic distance between the current input features and the retrieved anchors with the functional change measured in parameter space between the corresponding Hypernetwork-Generated LoRA adapters. The loss discourages inconsistent parameter updates and improves long-term robustness.

For task i, zraw,i is constructed by concatenating fv and fq. The semantic discrepancy between the anchor-conditioned context and zraw,i is measured using cosine distance:

To measure functional discrepancy, the hypernetwork outputs across all layers are vectorized, where vec(·) concatenates all generated matrices into a single vector:

where m is the total number of generated scalar parameters after vectorization, and a normalized ℓ2 distance in parameter space is defined as:

Dual-Space Consistency is used as an auxiliary objective: Lalign encourages agreement between the anchor-conditioned semantic shift (zinst vs. zraw) and the induced change in parameter space:

The overall objective is

where γ1 and γ2 control the relative weights of Lmem and Lalign, respectively. We minimize Ltotal by backpropagation, updating all trainable modules while keeping the base backbone projections frozen.

4 Experiments

To comprehensively evaluate HyLoVQA in continual VQA, we conduct experiments addressing six research questions:

• RQ1: How does HyLoVQA compare to baselines on Standard and Novel Composition tests, overall and finegrained?
• RQ2: What is the contribution of each component to overall effectiveness and forgetting?
• RQ3: How effective are HyLoVQA's sample-specific LoRA adapters, and what evidence supports this?
• RQ4: What is the impact of DMA capacity on retention and forgetting across tasks?
• RQ5: What is HyLoVQA's sensitivity to key hyperparameters such as anchor momentum?
• RQ6: What failure modes of prior methods are revealed by case studies, and how does HyLoVQA address them?

(a) t-SNE Visualization of CLT-VQA

(b) t-SNE Visualization of Ours

Figure 3: t-SNE visualization of sample feature representations on VQA v2 across task categories.

4.1 Experimental Settings

Datasets. We evaluate on VQA v2 (200k+ COCO images and 1.1M QA pairs [Lin et al., 2014]) and NExT-QA, a video QA benchmark requiring temporal and causal reasoning [Xiao et al., 2021]. Following the standard continual VQA protocol, VQA v2 is split into 10 question-type tasks (recognition, location, judge, commonsense, count, action, color, type, subcategory, causal), and NExT-QA into 8 tasks (CW, TN, TC, DL, DB, DC, DO, CH).

Evaluation Metrics. We report Final Average Performance (AP) and Average Forgetting (AF), which measure overall performance after continual learning and forgetting on earlier tasks, respectively[Lee and Toutanova, 2018].

Baselines. We compare HyLoVQA with sequential fine-tuning (Vanilla), EWC[Kirkpatrick et al., 2017b], MAS[Aljundi et al., 2018], ER[Chaudhry et al., 2019], DER[Chaudhry et al., 2018], VS[Wan et al., 2022b], VQACL[Zhang et al., 2023], QUAD[Marouf et al., 2025], ProtoGroup[Zhang et al., 2025b] and CLT-VQA[Zhang et al., 2025a]. All methods are evaluated under the same task stream, backbone, and metrics for fair comparison.

Implementation Details. Evaluation follows two paradigms: Standard testing on seen task-object combinations and Novel Composition testing on held-out (unseen) combinations. For visual tokens, we use a Faster R-CNN[Ren et al., 2015] pretrained on Visual Genome to extract N=36 RoI features per image on VQA v2, and an inflated 3D ResNeXt-101[Hara et al., 2018] to extract N=16 clip-level motion features on NExT-QA. We use Adam [Adam and others, 2014] with a learning rate of 3e-5, gradient clipping of 5, and a warmup ratio of 0.1; γ1 and γ2 are tuned from {0.1, 1, 10}. We implement our proposed method based on PyTorch.

4.2 RQ1: Main Results

As shown in Table 1, HyLoVQA achieves the best overall performance on both datasets under Standard and Novel Composition evaluation. On VQA v2, HyLoVQA attains 45.41% AP/2.82% AF (Standard) and 44.96% AP/2.62% AF (Novel), yielding about +4.72% AP and +4.96% AP over the strongest baseline, respectively, while maintaining low forgetting. On NExT-QA, HyLoVQA reaches 42.43% AP/2.07% AF (Standard) and 43.05% AP/3.10% AF (Novel), improving AP by about +6.66% and +7.19% over the best baseline in the two settings. Table 2 further breaks down the Novel Composition setting under the group-removed protocol in Fig.2 (AP only), where we hold out one object group Gi during training and evaluate Novel compositions involving Gi versus Seen compositions over the remaining groups. HyLoVQA achieves the best AP on both Novel and Seen splits (VQA v2: 44.96%/45.30%; NExT-QA: 43.05%/43.97%), confirming consistent gains across group splits and stronger compositional generalization.

4.3 RQ2: Ablation Study

Table 3 shows that each component consistently contributes to continual VQA under the Standard Test setting. The values in parentheses denote the change relative to the full model (DMA+HyGen+Align.). Starting from DMA only, performance drops and forgetting increases (41.77% AP/3.46% AF on VQA v2; 39.14%/3.79% on NExT-QA). Adding HyGen improves both AP and AF (44.96%/2.95% on VQA v2; 41.65%/2.52% on NExT-QA), confirming that anchor-conditioned, hypernetwork-generated LoRA adapters enhance instance-specific adaptation and alleviate interference. Adding Align. further yields the best trade-off (45.41%/2.82% on VQA v2; 42.43%/2.07% on NExT-QA) by keeping LoRA adapters aligned with the current task and object and reducing off-target adaptation.

4.4 RQ3: Representation Stability (t-SNE)

This section examines whether HyLoVQA can efficiently adapt to the current sample input under continual distribution shift, where each sample consists of a visual object and a textual task. As shown in Figure 3, we visualize sample features on VQA v2 and color points by task category (Count/Color/Action). Baseline features exhibit substantial overlap across categories, while HyLoVQA yields more compact intra-category clusters and clearer inter-category separation. This indicates reduced cross-task representation entanglement, which is closely associated with forgetting and negative transfer in continual VQA. We attribute this behavior to the DMA providing stable anchor references and HyGen performing anchor-conditioned adaptation, consistent with the higher AP and lower AF in Table 1.

4.5 RQ4: Memory Capacity Robustness

To evaluate memory capacity robustness, we vary the anchor memory bank size from 1k to 5k and report AP/AF under Standard testing on two datasets: VQA v2 and NExT-QA (Figure 4). HyLoVQA consistently outperforms strong baselines (VQACL, QUAD, and ProtoGroup) across all capacities, achieving a better AP-AF trade-off without requiring a large replay buffer. On VQA v2 (Figure 4a), increasing the memory from 1k to 5k improves AP from 39.94% to 45.41% while reducing AF from 4.84% to 1.82%. On NExT-QA (Figure 4b), AP increases from 39.55% to 42.43% and AF decreases from 3.08% to 2.07% as the memory grows from 1k to 5k. Overall, HyLoVQA benefits steadily from additional memory across datasets, indicating effective and robust utilization of the anchor memory bank.

4.6 RQ5: Hyperparameter Sensitivity

We evaluate the sensitivity of the modality-aware anchor update momentum on NExT-QA. Figure 5 reports AP (%) over a grid of visual and question momentum (α, β) and shows clear task-specific preferences, highlighting the flexibility of our DMA design. On Vision-Dominant tasks (Figure 5(a)), performance concentrates in the high-α region and peaks at (α, β) = (1.0, 0.8) with 45.22% AP. In contrast, QuestionDominant tasks (Figure 5(b)) favor larger β, achieving the best AP of 44.61% at (α, β) = (0.4, 1.0). Overall, the optimal region shifts consistently with the dominant modality, validating that DMA can adapt its anchor/prototype update dynamics to diverse reasoning needs via modality-aware momentum control.

4.7 RQ6: Qualitative Case Study

Figure 6 illustrates failure modes of prior methods and how HyLoVQA addresses them. CLT-VQA often forgets samplespecific constraints and falls back to generic shortcuts, e.g., ignoring color-conditioned counting, relying on static object priors in action queries, or merging fine-grained categories into coarse labels. HyLoVQA retrieves relevant anchors from DMA and uses HyGen to generate anchor-conditioned LoRA adapters that steer cross-attention toward task- and objectspecific cues: retrieving pv (green) to suppress distractors, retrieving pq (ride) to emphasize interaction regions for action recognition, and retrieving fine-grained visual anchors to preserve discriminative feather details. As a result, HyLoVQA preserves the intended constraints under drift and produces answers consistent with the queried attributes and interactions (e.g., one, riding, and parrot).

5 Conclusion

We propose HyLoVQA, a lightweight continual VQA framework that mitigates cross-level task interference while enabling dynamic adaptation to each incoming sample's task and object. With a drift-resilient memory anchor bank and a hypernetwork that generates task/object-specific LoRA adapters, HyLoVQA supports dynamic and parameterefficient adaptation and better preserves prior knowledge. A semantic-functional alignment loss further keeps updates focused on the current task and object. Overall, HyLoVQA highlights memory-guided parameter generation as a simple and general route to balance stability and plasticity in continual VQA under non-stationary and compositional shifts.

Acknowledgments

This work was supported in part by the Natural Science Foundation of Hubei Province of China (No. 2025AFB653), the Open Fund of Hubei Key Laboratory of Big Data Intelligent Analysis and Application, Hubei University (No. 2025BDIAA01), and the National Natural Science Foundation of China (No. 62207011, 62407013, 62377009).