Multi-Sourced Compositional Generalization in Visual Question Answering

1 Introduction

Compositional generalization refers to the ability of generalizing novel compositions from seen primitives. In visionand-language (V&L), the primitives of a composition come from either the linguistic modality or the visual modality, i.e., compositions are multi-sourced. As shown in Figure 1, in the context of visual question answering (VQA), "white (linguistic modality) + dog (linguistic modality)" is an novel composition, and "white (linguistic modality) + (visual modality)" and " (white + dog in visual modality)" are also novel compositions. However, prior works [Dankers et al., 2022;

† Corresponding author: Yuwei Wu.

Li et al., 2024b; Li et al., 2023b] only consider novel compositions of primitives from a single modality (e.g., "white + dog"). Whether the model's generalization ability to different modality primitives' novel compositions (i.e., multi-sourced novel compositions) remains unexplored. To generalize over multi-sourced compositions, a model needs to not only have the ability to understand individual primitives but also the ability to align primitives of different modalities.

To address the above issue, we propose a retrievalaugmented training framework. The basic idea of the framework is to learn similar representations for primitives from different modalities by retrieving semantically equivalent primitives, thereby maintaining consistent generalization ability for multi-sourced compositions. Specifically, the framework consists of three key components: retrieval database construction, feature retrieval, and feature aggregation. For the retrieval database, we construct separate primitive databases for the linguistic and visual modalities, where words with the same prototype are treated as the same linguistic primitive, and visual entities with the same label are treated as the same visual primitive. For the same primitive, both the linguistic/visual primitive databases contain multiple instances of that primitive in different contexts. For example, for the linguistic primitive "dog", the linguistic primitive database contains instances of "Is the dog black?", "How many dogs are there?", "Do you see a golden dog in this picture?" in different contexts. During training, for each primitive in the training sample, semantically similar primitives from both the linguistic and visual primitive databases are retrieved. The retrieved features are then aggregated with the original primitive features to optimize the model. Since the representations in the primitive databases are updated throughout the training, the model continuously refines its current representation by leveraging the retrieved relevant features to learn similar representations for the same semantic primitives across different modalities and contexts.

To quantitatively evaluate the generalization ability of VQA models for multi-sourced novel compositions, we construct the GQA-MSCG dataset based on the GQA dataset [Hudson and Manning, 2019]. We categorize compositions based on the modality of the primitives from the composition, resulting in three types of novel compositions: [Linguistic primitive, Linguistic primitive] (LL), [Visual primitive, Visual primitive] (VV), and [Linguistic primitive, Visual primitive] (LV). We construct three basic test splits, labeled LL, V V , and LV , each containing test samples of different types of novel compositions. To further explore how the co-occurrence of different types of novel compositions in samples affects model performance, we construct test splits, where each sample simultaneously contain two types of novel compositions: LL + V V , LL + LV , and V V + LV , as well as a test split, where each sample contain all three types of novel compositions: LL+ V V +LV . Experimental results demonstrate that the proposed framework significantly improves VQA models' generalization ability to multi-sourced novel compositions while maintaining their independent and identically distributed (IID) generalization ability.

To sum up, our contributions are as follows:

• We are the first to explore the multi-sourced compositional generalization in V&L, which is critical for crossmodal understanding.
• We propose a retrieval-augmented training framework that improves the multi-sourced compositional generalization ability of VQA models by learning similar representations for primitives from different modalities.
• We present a GQA-MSCG dataset to evaluate the multisourced compositional generalization ability of VQA models with different types of novel compositions.

2 RelatedWork

Compositional generalization has garnered significant attention in various research fields. In natural language processing (NLP), works [Li et al., 2021; Dankers et al., 2022] focus on improving the compositional generalization ability of models for tasks like machine translation. Additionally, Chai et al. [2024] used text generation models for data augmentation during training to enhance compositional generalization in multi-label text classification tasks. In computer vision (CV), works [Naeem et al., 2021; Jing et al., 2024; Li et al., 2024b] focus on improving compositional generalization for tasks like compositional zero-shot learning, specifically for novel compositions of [visual primitive, visual primitive]. In V&L, Pantazopoulos et al. [2022] constructed the first dataset to evaluate the compositional generalization ability of image captioning models. Works [Bahdanau et al., 2018; Akula et al., 2021; Yamada et al., 2024; Li et al., 2023a] focus on enhancing the compositional generalization ability of VQA models. Moreover, Li et al. [2022] constructed the Charades-CG and ActivityNet-CG datasets for temporal video grounding (TVG) and used variational cross-graph reasoning to improve the compositional generalization ability of TVG models. These works primarily focus on improving the generalization ability of models for novel compositions involving the same modality of primitives, e.g., works in NLP and V&L focus on [linguistic primitive, linguistic primitive] compositions, and works in CV focus on [visual primitive, visual primitive] compositions. In contrast, we explore the compositional generalization ability of V&L models for novel compositions involving different modality primitives, construct a new benchmark GQA-MSCG, and propose a retrieval-augmented training framework to improve the model's generalization ability for different types of novel compositions.

3 Framework

3.1 Overview

The overall framework of the proposed framework in the context of VQA is shown in Figure 2. For the training set Dt, the first step is to construct linguistic and visual primitive databases, Dq and Dv, respectively, for retrieval purposes. primitives with the same category (e.g., dog, cat) or attribute (e.g., blue, big) are treated as the same type of primitive. Both Dq and Dv contain multiple instances of each primitive in different contexts from Dt. During training, for each training sample (Q, V ), where Q represents the question and V represents the image, the retrieval module is used to retrieve similar primitives from Dq and Dv for the primitives in Q and V at feature level, respectively. The retrieved features are then aggregated with the original primitive features and used to replace the original features for training. Through this training process, the VQAmodel continuously refines its feature extractor by leveraging the retrieved relevant features, thereby learning similar features for the same semantic primitives across different modalities and contexts, thus improving the generalization ability of the model for multi-sourced novel compositions.

3.2 Retrieval Database Construction

To ensure fairness in the comparison, the retrieval database is constructed based on the training set Dt to avoid introducing external data during training.

Linguistic Primitive Database. For the linguistic primitive database, we first extract all words from the questions in Dt using the NLTK toolkit [Bird et al., 2009]. All words are lemmatized, and words with part-of-speech tags as nouns, verbs, adjectives, and adverbs are considered as linguistic primitives. The set of all unique linguistic primitives is denoted as Sq. For each linguistic primitive, Tq questions containing the primitive are sampled from Dt. Although the meaning of the primitive remains the same across these Tq questions, the different questions provide different contexts. In different contexts, the features of the same linguistic primitive, extracted using a recurrent neural network, will differ. Thus, it is necessary to sample different questions for each linguistic primitive. To preserve the different contexts of the linguistic primitives, the linguistic primitive database stores the original questions, not just the linguistic primitive itself.

The resulting linguistic primitive database is represented as Dq = ⋃p∈Sq fq(p), where fq(p) = {pi}Tq i=1 represents the Tq sampled questions containing the linguistic primitive p. Visual Primitive Database. The process of constructing the visual primitive database relies on the scene graph annotations, which contains information about object categories and attributes in the images. Objects in the images are treated as visual primitives, with category and attribute information serving as the labels of these primitives. After removing duplicates from the images in Dt, the resulting set of labels is denoted as Sv. Similar to linguistic primitives, even though different visual primitives may share the same label, they represent different visual expressions of objects with the same type/attribute. Therefore, for each label, Tv images containing the visual primitive with the label are sampled. Since different VQA models use different visual encoders, the input forms are not limited to object-level features. To address this, the visual primitive database stores the original images, not just the visual primitives. The resulting visual primitive database is represented as Dv = ⋃l∈Sv fv(l), where fv(l) = {p(l) i }Tv i=1 represents the Tv sampled images containing visual primitives with label l.

3.3 Feature Retrieval and Aggregation

To perform retrieval and aggregation at the feature level, both the training samples and all primitives in the two primitive databases (Dq and Dv) need to pass through the feature extractors of the VQA model. For a question Q, we obtain its feature hq = gq(Q) = {hiq }Ni=1 , where gq(·) denotes the linguistic feature extractor of the VQA model. Here, hiq represents the feature of the i-th word, and N is the number of words. For an image V , the visual feature extractor gv(·) typically produces two types of features: "object-level" features or "patch-level" features, which can be represented as hv = gv(V ) = {hiv }Mi=1 , where hiv represents the feature of the i-th object/patch, and M is the number of objects/patches.

After obtaining the primitive-level features (word-level, object-level, patch-level), for each primitive feature in the training sample, retrieval is performed on the primitive feature sets of the questions and images in Dq and Dv (processed by gq(·) and gv(·)). Specifically, for a primitive feature p in the training sample, the top Kq most similar primitive features {p(i) q }Kq i=1 are retrieved from the primitive feature set in Dq, and the top Kv most similar primitive features {p(i) v }Kv i=1 are retrieved from the primitive feature set in Dv. We use cosine similarity cos(·, ·) to measure the similarity between two primitive features. The features are aggregated using a weighted average

where the hyperparameters wq and wv control the contributions of the different modality primitive databases. We use the aggregated feature pa to replace the original primitive feature p for training.

3.4 Optimization

The proposed framework is applicable to different VQA baseline models, using the same optimization approach as the baseline model without introducing additional training losses or constraints. Therefore, for a VQA baseline model using the training loss L, for a training sample (Q, V ) with groundtruth A, the training loss for both the baseline model and the model with our framework is the same:

where P(Q, V ) represents the output of the VQA model (e.g., a distribution vector over the number of categories), and loss(·, ·) represents the training loss function, such as the cross-entropy loss used in the UpDn model [Anderson et al., 2018].

4 GQA-MSCGDataset

In order to quantitatively evaluate the generalization ability of VQA models to multi-sourced novel compositions, we construct the GQA-MSCG dataset based on the GQA dataset [Hudson and Manning, 2019]. The process of constructing the GQA-MSCG dataset consists of three main steps: composition extraction, sample filtering, and sample classification. Composition Extraction. We use the same steps as in Section 3.2 to extract the linguistic and visual primitives for all samples in the train balanced split Dt of the GQA dataset. The linguistic and visual primitives for a sample s ∈ Dt are represented by P q s = {pqi s }Ni=1 and P v s = {pvi s }Mi=1 , respectively, where N and M denote the number of linguistic and visual primitives in the sample. The compositions in sample s are represented as Cs = {[p1, p2]|p1 ∈ P q s ∪ P v s , p2 ∈ P q s ∪P v s , p1 ̸= p2}. Thus, the complete set of primitives in Dt can be expressed as PDt = ⋃s∈Dt P q s ∪P v s , and the complete set of compositions can be expressed as CDt = ⋃s∈Dt Cs. Similarly, the compositions in the val all split Dv of the GQA dataset can be processed to obtain all compositions. We denote the set of all compositions as CDv .

Sample Filtering. For a sample s ∈ Dv, if the sample satisfies the following conditions simultaneously, it is added to the candidate test sample set Dc:

• All primitives in the sample must appear in the training set - ∀p ∈ P q s ∪P v s , p ∈ PDt .
• The sample must contain at least one composition not seen in the training set (i.e., an novel composition) - ∃c ∈ Cs, c /∈ CDt .

Sample Classification. Based on the modality of the primitives in a composition, we categorize three types of novel compositions: [Linguistic primitive, Linguistic primitive] (LL), [Visual primitive, Visual primitive] (VV), and [Linguistic primitive, Visual primitive] (LV). Test samples in Dc are classified into seven categories: (1) LL: Samples containing only novel compositions of the type LL. (2) V V : Samples containing only novel compositions of the type VV. (3) LV : Samples containing only novel compositions of the type LV. (4) LL+V V : Samples containing both novel compositions of the type LL and VV. (5) LL + LV : Samples containing both novel compositions of the type LL and LV. (6) V V + LV : Samples containing both novel compositions of the type VV and LV. (7) LL+V V +LV : Samples containing novel compositions of all three types: LL, VV, and LV. Samples in LL, V V and LV are used to evaluate the multi-sourced compositional generalization ability, while samples in LL + V V ,

(b) QV split

(c) VV split

Figure 3: Level-1 samples in the GQA-MSCG dataset.

• Q: What type of furniture is to the right of the rope? (GT: chair)

LL+LV , V V +LV and LL+V V +LV are used to further evaluate the impact of the co-occurrence of different types of novel compositions on model performance.

For each category of test samples, we randomly sample 5,000 samples fromDc, resulting in a total of 35,000 samples for the GQA-MSCG dataset. Samples containing x types of novel compositions are referred to as Level-x samples, and the difficulty of the samples increases as x increases. For example, samples in LL, V V and LV are Level-1 samples, LL + V V , LL + LV and V V + LV are Level-2 samples, LL+V V +LV are Level-3 samples. As the level increases, the co-occurrence count of novel composition categories in the test samples continues to rise, making the difficulty of the test samples increasingly higher. The test samples of GQAMSCG are shown in Figure 3 and Figure 4, where the words or visual regions of the same color in a sample represent a pair of novel compositions in the sample.

5 Experiments

5.1 Experimental Setup

Dataset. We evaluate proposed frameworks on the VQA task, three datasets are selected to validate the effectiveness of the proposed frameworks: the GQA dataset [Hudson and Manning, 2019], the VQA v2 dataset [Goyal et al., 2017] and our GQA-MSCG dataset. The GQA dataset is a widely used large-scale dataset in VQA, containing a large number of template-based compositional questions. The VQA v2 dataset is an extended balanced version of the VQA v1 dataset [Antol et al., 2015], where questions are human-made. These two datasets are often used to test the generalization ability of VQA models to IID data as well as their ability for compositional reasoning. Our GQA-MSCG dataset is used to evaluate the VQA model's ability to generalize consistently to multi-sourced novel compositions, which refers to the VQA model's ability to generalize to novel compositions of primitives from different modalities.

Baseline Models. We use CFR [Nguyen et al., 2022] and Qwen-VL [Bai et al., 2023] as baseline models. The baseline models combined with the proposed framework are referred to as CFR+RAG and Qwen-VL+RAG, respectively. CFR is a representative small model in the VQA task with a parameter size less than 0.2B. Qwen-VL is a popular open-sourced multimodal large model (with more than 7B parameters), which can be applied to various vision-and-language tasks such as VQA, image captioning, visual dialogue, and others. For both CFR and Qwen-VL, we reimplemented them based on their officially released code.

Implementation Details. For experiments on all three datasets including GQA, GQA-MSCG and VQA v2, we finetune Qwen-VL and Qwen-VL+RAG with LoRA [Hu et al., 2022] with a maximum of 2 epochs. For CFR+RAG and Qwen-VL+RAG, we set wq = 0.6 and wv = 0.4.

For experiments on the GQA dataset and the GQAMSCG dataset, we fine-tune CFR, Qwen-VL, CFR+RAG, and Qwen-VL+RAG using the train balanced split of the GQA dataset and selected the best-performing model weights on the val balanced split of GQA. Using these model weights, we present the experimental results on the test-dev split of the GQA dataset and all seven test splits of our GQA-MSCG dataset. The maximum number of epochs for fine-tuning CFR and CFR+RAG was set to 12. The sampled number Tq and Tv for constructing Dq and Dq are set to 8 and 32, respectively. Moreover, we use the scene graph provided by GQA to construct Dv. The number of aggregated primitive Kq and Kv are set to 4 and 16, respectively. Distinctively, for experiments on the VQA v2 dataset, we set Tq = 1, Tv = 32, Kq = 4 and Kv = 4. For constructing Dv, we use the object categories detected by Faster R-CNN [Ren et al., 2016], ignoring the object attributes.

5.2 Multi-Sourced Compositional Generalization Performance

We evaluate the MSCG ability of VQA models on the proposed GQA-MSCG dataset. We compare with multimodal large models, including LLaVA-1.5 [Liu et al., 2023] and LLaVA-1.6 [Liu et al., 2024a], with the experimental results shown in Table 1. We can observe that: (1) As the cooccurrence count of novel composition categories in the test samples increases (Level-1 → Level-3), the model's performance gradually decreases. For example, LLaVA-1.5 has an accuracy of 70.22% in the LL split, 67.04% in the LL+ V V split, and 64.14% in the LL+V V +LV split. (2) Different VQAmodels exhibit significant differences in their ability to generalize to novel compositions of different modality primitives. For example, LLaVA-1.5 performs better on the V V split than on the LL split, while Qwen-VL shows the opposite. (3) Both small VQA models (e.g., CFR) and large VQA models (e.g., Qwen-VL) benefit significantly from the proposed framework, which enhances their generalization ability to multi-sourced novel compositions.

As a result, based on the experimental results, the following conclusions can be drawn: (1) Existing VQA models still have shortcomings in handling complex compositional questions. (2) It is necessary to specifically consider the multisourced compositional generalization ability for VQA models. (3) The proposed framework is highly versatile and can be applied to different baseline models, improving their generalization ability to novel compositions of primitives from different modalities.

5.3 Independent and Identically Distributed Generalization Performance

We verify the improvement of our framework in IID generalization performance on the test-dev split of GQA [Hudson and Manning, 2019] and the val split of VQA v2 [Goyal et al., 2017]. For GQA, we compare to five different types of VQA models, including attention-based MAC [Hudson and Manning, 2018], graph-based LCGN [Hu et al., 2019], neural modular network based (NMN-based) MMN [Chen et al., 2021], zero-shot pretrain-based BLIP-2 [Li et al., 2023c], MiniGPT-4 [Zhu et al., 2023], LLaVA-1.5 [Liu et al., 2024b], LLaVA-1.6 [Liu et al., 2023], and fine-tuned pretrain-based CFR [Nguyen et al., 2022], Qwen-VL [Bai et al., 2023]. For VQA v2, we compare to VQA models with less than 0.3B parameters (DLR [Jing et al., 2020], CF-VQA [Niu et al., 2021], CLS [Mao et al., 2024], ASS [Li et al., 2024a], KDAR [Peng and Wei, 2024]) and VQA models with more than 7B parameters (PNP-VQA [Tiong et al., 2022], BLIP-2 [Li et al., 2023c], Qwen-VL [Bai et al., 2023]).

Experimental results on the test-dev split of the GQA dataset are shown in Table 2. From the table, we can observe that: (1) Pretrain-based models outperform other types of models (such as attention-based, graph-based, and NMNbased frameworks) in terms of IID generalization performance. (2) The proposed framework further enhances the IID generalization performance of baseline models (e.g., 1.43%

and 1.13% absolute performance gains in accuracy for CFR and Qwen-VL, respectively). These experimental results demonstrate that the proposed framework is applicable to different task scenarios, including multi-sourced compositional generalization scenario and independent and identically distributed generalization scenarios.

Experimental results on the val split of the VQA v2 dataset are shown in the Table 3. The results demonstrate that our framework is inoffensive for IID generalization. The reason why the performance gains of the proposed framework on VQA v2 are less than on GQA is that the questions in GQA are more compositional and thus are more suitable to be improved by our framework. Such experimental results further prove the effectiveness of the proposed framework for improving the IID generalization ability of VQA models.

5.4 Ablation Studies

The results of ablation studies on GQA-MSCG using CFR as the baseline model are shown in Table 4. From the experimental results, the following conclusions can be drawn: (1) Conducting retrieval only on the question retrieval database Dq or the image retrieval database Dv during training can also improve the baseline model's performance, especially for novel compositions of specific modal primitives. For example, when retrieval is performed only on Dv, improvements are more evident for compositions of visual modal primitives (the VV split). (2) Overall, using both the question and image retrieval databases simultaneously results in the best performance (highest average accuracy across all splits). These ablation study results confirm the necessity of multi-sourced retrieval during training and demonstrate that the linguistic primitive database Dq or visual primitive database Dv are complementary, helping to enhance the baseline model's generalization ability across different compositions.

5.5 Parameter Analysis

We analyze the influences of wq and wv on the MSCG ability of our framework, which denote the explicit contributions of the linguistic primitive database and the visual primitive database, respectively. Figure 5 shows the performance variations of the proposed framework with changing values of wq and wv. First, we conduct an initial analysis of wq and wv by setting them to the same value. The experimental results are shown in Figure 5 (a). It can be seen that as the values increase, the performance of the proposed framework peaks when wq = wv = 0.4, and further increasing wq and wv does not improve the performance. Furthermore, we analyze the effectiveness of the proposed framework with wq ̸= wv by

• (a) Experimental results with wq = wv.

Figure 6: The qualitative comparison between CFR+RAG (Ours) and CFR on the GQA-MSCG dataset.

fixing wq or wv at 0.4 and adjusting the other parameter (i.e., wv or wq), as shown in Figures 5 (b) and (c). We can observe that: (1) The performance of CFG+RAG fluctuates obviously when adjusting either wq or wv. (2) CFR+RAG performs best with setting wq = 0.6 and wv = 0.4 simultaneously. Based on the above experimental results, we set wq to 0.6 and wv to 0.4 for all experiments.

5.6 Qualitative Analysis

For each split of Level-1, we provide two qualitative examples of CFR+RAG (Ours) and CFR (the baseline model) on the GQA-MSCG dataset in Figure 6. In the figure, novel compositions of linguistic primitives are highlighted in red text, and visual primitives are enclosed in red boxes. It can be observed that for test samples with novel compositions composed of primitives from different modalities, the proposed framework helps the baseline model make more accurate predictions. For example, in the first sample from the GQA-MSCG dataset's LL split, the question is "Are the jeans different in color than the flowers?", which includes the novel composition of linguistic primitives "jeans + flowers". The baseline model CFR gives an incorrect answer, "no". In contrast, after incorporating CFR into the proposed framework, it correctly predicts "yes". Moreover, for test samples in the V V and LV splits, the proposed framework still provides accurate answers, thanks to the alignment of primitives with the same semantics across different contexts and modalities. These qualitative examples demonstrate that the proposed framework enhances the baseline model's generalization ability to multi-sourced novel compositions, proving the effectiveness of the framework.

6 Conclusion

In this paper, we explored the multi-sourced compositional generalization ability of models in the context of VQA. We have presented a retrieval-augmented training framework to encourage VQA models to learn unified representations for the same semantic compositions by aligning semantically equivalent primitives across different modalities at the feature level. The proposed framework can be seamlessly incorporated into existing VQA models to improve their multisourced compositional generalization ability. We extend the GQA dataset to construct a GQA-MSCG dataset, which enables the quantitative evaluation of the multi-sourced compositional generalization ability for VQA models. Experimental results demonstrate that our framework can improve not only the multi-sourced compositional generalization ability, but also the IID generalization ability.

Acknowledgements

This work was supported by the Shenzhen Science and Technology Program under Grant No. JCYJ20241202130548062, the Natural Science Foundation of Shenzhen under Grant No. JCYJ20230807142703006, and the Natural Science Foundation of China (NSFC) under Grants No. 62176021 and No. 6217204.

Contribution Statement

Chuanhao Li and Wenbo Ye contributed equally to this work.