Answer Mining from a Pool of Images: Towards Retrieval-Based Visual Question Answering

1 Introduction

Question Answering (QA) over textual as well as visual data has been an active area of research [Hsu et al., 2021; Guo et al., 2023]. In text-based QA, the research focus has recently shifted from highly-explored QA on a single paragraph such as SQuAD [Rajpurkar et al., 2018] to a setting where mining answers from a huge corpus of documents is a requirement [Ahmad et al., 2019; Hsu et al., 2021]. On the contrary, visual question answering (VQA) [Antol et al., 2015] literature has so far largely restricted itself to answering questions about a given relevant visual context (often a single image). However, this does not necessarily suffice to satisfy our information needs since the information may be spread across multiple images and may not be present in some images. For example, consider a natural language question 'Do the rose and sunflower share the same color?', answering such a question from a pool of images as visual context (refer Figure 1), requires a model to first retrieve relevant images and then perform visio-lingual reasoning on the retrieved images to arrive at a fluent free-form natural language answer. We refer to this problem as RETVQA or retrieval-based visual question answering. The RETVQA setting has potential applications in question answering on web images, e-commerce, environmental monitoring, and health care, among others, e.g., multiple images of a particular area can be analyzed to monitor environmental changes over time; multiple MRI or CT scans of a patient's brain need to be analyzed to detect abnormalities, such as tumors.

For RETVQA, the input is a pool of images with only a few images being relevant to the question. Close to our setting, there is some exciting progress in the recent literature [Talmor et al., 2021; Bansal et al., 2020; Singh et al., 2021; Chang et al., 2022]. However, these works assume one or more of the following constraints: "without requiring explicit retrieval", "having classification-type fixed-vocabulary answers", "assuming the availability of meta-data like WikiEntities, captions", "having a homogeneous yet limited number of images in the pool", and "having only a small set of questions that need multiple images". Such constraints in the existing datasets point towards a need for a large-scale benchmark to study RETVQA. To this end, we present a derived dataset prepared from Visual Genome [Krishna et al., 2017], leveraging its questions and annotations of images. We curate questions under different categories: (i) common attributes such as color, shape, and count, (ii) other object-attributes that include non-common attributes, e.g. length, material, and (iii) subject-object relationships, e.g., 'eats', 'left of'. Further, to facilitate benchmarking capabilities of the VQA models over open-ended answers, we curate questions under binary (yes/no) and open-ended answer categories. Note that the answers are free-form fluent in both the answer categories, e.g. 'No, rose and sunflower do not share the same color' (a binary answer); 'The color of rose and sunflower is red and yellow, respectively' (an open-ended generative answer). RETVQA dataset statistics and distribution across the question-answer categories are shown in Table 1 and Table 2, respectively.

Further, to solve the RETVQA task, a model must first retrieve the relevant images for the question and then consume the retrieved images as the context to answer the question. Towards this end, we present a unified Multi Image BART that takes in the question along with the multi-image context retrieved using a relevance encoder to generate the free-form fluent natural-language answer. Our proposed framework, MI-BART, allows joint reasoning over multiple retrieved images along with the question to capture better semantics.

To summarize, our contributions are as follows: (i) We present RETVQA, a 20× larger dataset than the closest dataset [Chang et al., 2022] in this setting. RETVQA dataset is prepared by leveraging questions and image annotations from Visual Genome. It emphasizes on multi-image, metadata-independent questions over a pool of heterogenous collections of images, expecting a mix of classificationoriented and generative answers. We strongly believe that the proposed task, curated dataset and benchmarks presented in this paper will pave the way for further research. (ii) We present Multi Image BART (MI-BART) - a unified method that reasons jointly over the retrieved multi-image context along with the question to generate a free-form fluent answer for the question. (iii) We perform extensive experiments to evaluate the performance of our proposed framework on RETVQA and the image segment of WebQA. Our approach clearly outperforms baseline approaches on RETVQA dataset and achieves state-of-the-art performance on the image segment of WebQA. We make our data and implementation publicly available. 1

2 RelatedWork

Visual and Multi-modal QA. Visual Question Answering (VQA) aims at answering a natural language question in the context of a relevant image [Antol et al., 2015]. This area has seen significant progress partly due to the introduction of several challenging datasets [Malinowski et al., 2015; Ren et al., 2015a; Zhu et al., 2016; Antol et al., 2015; Goyal et al., 2017; Johnson et al., 2017; Krishna et al., 2017]. Most methods for VQA either use a multimodal fusion of language and image embeddings [Ren et al., 2015a; Gao et al., 2015; Noh et al., 2016; Kembhavi et al., 2017], attention-based multimodal fusion [Yang et al., 2016; Fukui et al., 2016; Shih et al., 2016; Lu et al., 2016; Xiong et al., 2016] or neural module networks [Andreas et al., 2016; Hu et al., 2017]. More recently, knowledge-based VQA [Shah et al., 2019; Marino et al., 2019] has gained attention where external knowledge is used for answering visual questions. Contrary to these exciting works in VQA literature, our problem setting is distinctively different as we need to mine the answer from a collection of relevant as well as irrelevant images.

Sharing a similar motivation as ours, the following tasks and accompanying datasets have been recently introduced in the literature: (i) MultimodalQA [Talmor et al., 2021], (ii) ISVQA [Bansal et al., 2020], and (iii) WebQA [Chang et al., 2022]. In MultimodalQA [Talmor et al., 2021], only a small part of the dataset (ImageListQ) is relevant to our setting; however, even on this subset, MultimodalQA assumes the availability of extra image metadata, i.e., table or WikiEntity linkage. Similarly, in ISVQA [Bansal et al., 2020], every question has a small set of homogeneous images as context. Since images are homogeneous, there is no need for explicit retrieval. Recently, WebQA [Chang et al., 2022] dataset has been proposed to target such practical QA scenarios, where a question has to be answered in the context of multimodal sources; however, the images in the dataset have associated captions. All of these settings have differences from RETVQA in either usage of additional context, constraints on images in the collection, answer schema, or classification instead of generation. In another recent work MIMOQA [Singh et al., 2021], only extractive question answering is performed. In terms of reasoning on more than one image, another related work is NLVR [Suhr et al., 2019]. However, it does not involve any retrieval and open-ended answer generation. Further, in terms of QA over multiple document images or video frames, related works are DocVQA [Tito et al., 2021] and VideoQA [Lei et al., 2018; Lei et al., 2020; Tapaswi et al., 2016]. DocVQA focuses on text-heavy document images, limiting visual reasoning, whereas the VideoQA task does not involve explicit retrieval and open-ended answer generation. Contrary to these works, our newly curated dataset is significantly larger, has no assumption of meta-data availability, and requires retrieval and reasoning over multiple images to arrive at an answer.

1 https://vl2g.github.io/projects/retvqa/

Multi-modal modeling. Recently, multi-modal transformers such as VisualBERT [Li et al., 2019], VilBERT [Lu et al., 2020], VILT [Kim et al., 2021], LXMERT [Tan and Bansal, 2019], OSCAR [Li et al., 2020], UNITER [Chen et al., 2020] have shown strong results on the downstream vision and language tasks. However, these encoder-based models are more suited for classification-style VQA settings. Multimodal transformers like VLP [Zhou et al., 2020] and VLBart [Cho et al., 2021] are pre-trained with sequence-tosequence objectives and hence are more suitable for the current setting that requires the model to generate free-form natural language answers. We follow a similar approach by devising an encoder-decoder framework to jointly reason over multiple images along with the question.

3 RETVQA Dataset

Traditionally, VQA datasets [Antol et al., 2015; Goyal et al., 2017; Singh et al., 2019; Talmor et al., 2021] assume that the context provided is always relevant to the question. Recently, [Chang et al., 2022] proposed a benchmark where given a question and a pool of multimodal sources (containing both image and text snippets as context), only a few of these sources are relevant to the question. Similar to our problem setup, it requires first retrieving the relevant context and then using it to answer the question. However, after carefully observing the WebQA dataset, we found that most questions include rare entities like 'Maracana Stadium', 'Minnetonka Rhododendron', etc. Such questions make the retrieval task of the problem non-trivial without auxiliary information about these images. Methods proposed in [Chang et al., 2022] leverage metadata like image captions, which contain information like the name of the entity in the image. Further, a rule-based retrieval using word overlap of the question with the image caption for the retrieval task has an F1 score of 37, asserting our claim that retrieval is over-dependent on image metadata and not significantly on visual data. Further, the image-based subset of WebQA has a majority of samples (55.6%) with one relevant image per question, thereby, a single image VQA method may perform equally well given the relevant image is retrieved.

To overcome such limitations, we curate a dataset RETVQA from Visual Genome, where we emphasize multiimage, metadata-independent questions over a pool of heterogeneous images, expecting a mix of classification-oriented and open-ended generative answers. We leverage questionanswer and object annotations of Visual Genome to curate the dataset. We curate truly multi-image questions spanning over five different categories, namely, color, shape, count, objectattributes, and relation-based. For each question category, we curate binary-generative and open-ended generative answers. Dataset statistics are shown in Tables 1 and 2.

The questions in RETVQA are curated as follows. We start by extracting subjects and relations of the existing questionanswer pairs from Visual Genome; for example, consider these two question-answer pairs: q1 (over image I1): "What is the cow eating in the image?" where the answer is a1: "grass"; and, q2 (over image I2): "what is the sheep eating?" where the answer is a2: "grass". Given q1 and q2, we extract subjects ("cow" and "sheep"), relations ("eating (eat/eats)"), and then we frame combined questions using templates (over images I1 and I2) as follows. q3: "what else eats the same thing as cow does?" with answer a3: "sheep eats the same thing as cow". Another question could be q4: "Does cow and sheep eat the same thing?" where the answer is a4: "Yes, cow and sheep eat the same thing". Thus, we curate binarygenerative (like q4) and open-ended generative (like q3) types of answers. We further associate negative images for each of the curated questions using their object annotations as follows. A negative image is one where both the subject and object (used to generate the question) do not exist together in the image. This enforces that the answer has to be inferred only when all the relevant images are correctly retrieved and the negatives serve as sufficiently hard negatives.

We use a random 80%-10%-10% train-validation-test split. All the questions in our dataset have at least two relevant images and 24.5 irrelevant images on average. A comparison with the other relevant datasets is shown in Table 3. Further, Figure 2 shows the distribution of unique answers across question types and question length distribution. Figure 3 shows the word cloud of top-80 frequent answers. We observe that most questions are in the 5-10 words range, and there is no noticeable bias towards the majority of answers in the dataset.

4 Retrieval QAMethodology

4.1 RETVQA ProblemFormulation

The RETVQA problem is defined as follows. Given a natural language question Q, a set of N heterogeneous images I = {I1, I2, . . . , IN}, the task is to generate an answer (A) for the question Q based on I where only a few images are relevant for the question. To answer the question Q using I, we need a method that retrieves the relevant images I r ⊆ I and then leverages the retrieved context I r . Accordingly, our labelled dataset consists of quadruplets (Q, I, I r , A).

4.2 RETVQA Framework

The proposed framework solution: (i) multi-modal relevance encoder for retrieval of relevant sources I r from I for the given question Q and (ii) a unified Multi Image BART (MIBART) to generate fluent free-form natural language answer for the question Q using the retrieved images I r as context.

Image representation. Inspired by recent vision-language pretraining literature [Li et al., 2019; Chen et al., 2020; Li et al., 2020], for every image Ii in I where i ∈ {1, 2, . . . ,N}, we first detect a fixed set of P objects using Faster R-CNN [Ren et al., 2015b] pretrained on Visual Genome [Krishna et al., 2017]. For every object p, where p ∈ {1, 2, . . . ,P}, we obtain 2048-dimensional regional feature oreg p and 4-dimensional bounding box co-ordinates obbox p . Thereby, each image Ii is represented by a set of P object proposals {(oreg , obbox )p}i. Following [Li et al., 2019], for every region p we project both 2048-dimensional regional representation and 4-dimensional bounding box coordinates into the d-dimensional space using a linear projection to obtain {op}i and then concatenate across all regions within the image to obtain image embedding oi as follows.

Question representation. We encode the textual question Q containing M words using a pretrained BERT [Devlin et al., 2019]. This results into a sequence q of M d-dimensional vectors, q = {qm} where m ∈ {1, 2, . . . ,M}. Note that if any additional metadata is available (e.g. captions in WebQA dataset), we augment it to the question.

4.3 Multimodal Relevance Encoder for Image Retrieval

Pretraining. Our multi-modal Relevance Encoder (RE) consists of three transformer encoder layers followed by a multi-layered perceptron (MLP) with a sigmoid unit over the final representation of the [CLS] token. We pretrain our relevance encoder on MS-COCO [Lin et al., 2014] using two unsupervised objectives, Image Text Matching (ITM) and Masked Language Modelling (MLM) similar to [Li et al., 2019].

Question-Image relevance learning. Each sample in our dataset contains a question Q and N images I1, I2, . . . , IN of which some have been labelled as positive and others negative. Further, for each image, we have P regions. We use each question-image pair (Q, Ii) to learn question-image relevance using our multi-modal Relevance Encoder (RE). Our pretrained multi-modal relevance encoder is fed with question-image pairs, along with two special tokens, [CLS] and [SEP]; in short, the input to our relevance encoder is [[CLS], {q1, q2, . . . , qm}, [SEP], {o1i, o2i, . . . , opi }]. Our encoder then allows the input M + P + 2 token sequence of question-image features to attend to each other and produces a sequence of contextualized embeddings. The d-dimensional contextualized embedding of [CLS] token is further fed to an MLP with a sigmoid unit to produce a relevance score (ˆ si) between 0 and 1 (Eq. 3), indicating whether the given question-image pair (Q, Ii) is relevant or not. We finetune our multi-modal relevance encoder parameters ϕ by minimizing binary cross-entropy loss LREL(ϕ) (Eq. 4).

Given a question Q and a set of N images I sampled from our dataset D, we obtain relevance scores S = {ˆ si}Ni=1 for each question-image pair (Q, {Ii}Ni=1 ) using our fine-tuned relevance encoder (Eq.5). To choose the final set of relevant images I r from the pool of images I, we rank all the images in the pool using S and choose top-K images as our relevant context I r for the given question Q (Eq. 6).

4.4 Multi Image BART for Question Answering

Given the question and the retrieved images I r , the goal of MI-BART is to generate an accurate yet fluent free-form natural language answer for the question. Towards this end, we propose an encoder-decoder architecture similar to SimVLM [Wang et al., 2022]. MI-BART encoder is a stack of six transformer layers [Vaswani et al., 2017], where each transformer layer comprises a self-attention layer, followed by a fully connected linear layer with a residual connection. Similarly, the MI-BART decoder is also a stack of six transformer layers [Vaswani et al., 2017], with an additional crossattention layer in each transformer layer. We concatenate question embedding q with image embeddings of each image Ik in I r with a special token [SEP] in between. Also, to distinguish the image features belonging to different images in the retrieved image set I r , we assign image order ids to image features from different images. Note that image order ids are not meant for assigning a sequence number to images in the retrieved set. Their sole purpose is to differentiate image features from different images. In short, the input to our MI-BART encoder is [[CLS], {q1, q2, . . . , qm}, [SEP], {o1, o2, . . . , op}1, [SEP], . . . , {o1, o2, . . . , op}k], where m ∈ {1, 2, . . . ,M}, p ∈ {1, 2, . . . , P} and k ∈ {1, 2 . . . ,K}. These inputs attend over each other through various self-attention layers of the MI-BART encoder and produce a sequence of contextualized embeddings z = {zj}, where j ∈ {1, 2, . . . ,M + (P × k) + k} (Eq. 7).

MI-BART decoder auto-regressively predicts the probability of the next token At in the answer A by attending to these encoder outputs z and previously generated answer tokens a<t through cross-attention and self-attention layers, respectively (Eq. 8). We train MI-BART decoder parameters θ by minimizing the generative loss LGEN(θ) for generating the target answer token conditioned on the question Q and retrieved image context I r (Eq. 9). During training, we leverage the ground truth relevant images as retrieved image context I r , while during inference, we obtain it from our relevance encoder.

To summarize, our proposed framework works as follows, given a question Q and a pool of N images I, we (i) obtain question-image relevance scores S for each question-image (Q, Ii) pair in (Q, I) using our multimodal relevance encoder, (ii) rank all images in the pool based on S, and choose top-K images as retrieved images context I r , and (iii) question Q along with the retrieved image context I r is fed to MI-BART which encodes the provided context and generates the free-form natural language answer A to the question Q.

4.5 Image-stitchMI-BART

Inspired by [Bansal et al., 2020], we consider stitching the retrieved images along the width into a single joint image and use single image VQA on this joint image. We use our proposed MI-BART for this baseline; however, we feed the stitched joint image instead of feeding K images. While the MI-BART combines information across images in the embedding space, the image-stitch MI-BART combines information across images in the input space.

5 Experiments and Results

We conduct our experiments on RETVQA as well as on WebQA. Since our task deals with the image set as a given context, we consider the image-only subset of the WebQA dataset. Following [Chang et al., 2022], we use accuracy (A), fluency (F), and F ×A, as metrics to evaluate the generated answers. Accuracy validates whether the correct answer is present in the generated answer, whereas fluency measures the quality of the answer paraphrase. Fluency is computed using a recently proposed natural language generation metric called BARTScore [Yuan et al., 2021]. Further, an F1 score is used for retrieving relevant images from a pool of images.

5.1 Baselines, Ablations and Implementation Details

Baselines. We compare our proposed method (MI-BART) and its variant image-stitch MI-BART with the following baselines: (i) Popularity-based Baselines: To check for prior biases associated with frequent answers globally or per question category, we use two popularity-based baselines. (a) Global popularity: the most frequent answer in the training set is always considered as the answer by the model, and (b) Per-category popularity: the most frequent answer for each question category is always considered the answer for questions in the corresponding question category. (ii) Aggregate VQA: RETVQA task involves VQA over multiple images. In the Aggregate VQA baseline, we use the traditional singleimage VQA method [Antol et al., 2015] for each image and aggregate the results. Given a question Q and its corresponding K retrieved images, I r from our relevance encoder, we feed each retrieved image along with the question Q to a single image VQA model to get a joint representation. We concatenate joint representations of all retrieved images into a single representation F and feed to a linear layer (MLP) to predict the final answer, i.e., A = MLP(F). Since traditional VQA methods follow a classification-style answer prediction approach, we use the 1000 most frequent answers as

Question: What else eats same thing as brown horse ?

Table 7: MI-BART performance on RETVQA using different retrieval strategies.

classes in the softmax layer. To generate a fluent answer, we prepend the predicted answer to the question after removing the first word from the question. This baseline is not benchmarked on the WebQA dataset, as the dataset does not provide precise answer annotations for the trainset questions. (iii) VLP: As the RETVQA task requires the model to generate the text, encoder-only multimodal transformer models like ViLBERT [Lu et al., 2020], VisualBERT [Li et al., 2019], OSCAR [Li et al., 2020], ViLT [Kim et al., 2021] and UNITER [Chen et al., 2020] are not directly suitable. Hence, we use VLP [Zhou et al., 2020] which is a unified encoderdecoder multimodal transformer as our baseline. We finetune a pretrained VLP on our datasets for evaluation.

Ablations. We perform the following ablations to better understand the various components of our proposed model. (i) Question-only: To study the role of the images in generating accurate and fluent answers, we ignore the images and use questions only as input to our model. (ii) Single-image retrieval: To study the importance of reasoning over multiple images to generate an answer to the question, we use top-1 retrieved image as our only context instead of a multi-image context. (iii) Missing captions: To study the role of image metadata in the relevant source image retrieval and, thereby, the answer generation, we conduct experiments on WebQA without leveraging the image metadata (captions). In this ablation, we augment captions (available in WebQA) as part of the textual input in both the relevance encoder and MI-BART.

Implementation details. We have implemented our framework in PyTorch [Paszke et al., 2019] and Hugging Face's transformers [Wolf et al., 2020] library. Our relevance encoder has three transformer layers, each having eight attention heads. We pretrain our relevance encoder on MS- COCO [Lin et al., 2014] with a constant learning rate of 1e-4 using Adam optimizer [Kingma and Ba, 2015]. Using the same optimiser, we finetune the relevance encoder on both datasets with a constant learning rate of 2e-5. Our MI-BART has six standard transformer encoder layers and six standard transformer decoder layers [Vaswani et al., 2017]; we initialize our MI-BART with VLBart [Cho et al., 2021] pretrained weights to leverage the strong visual-textual learning of VLBart. We further finetune MI-BART on a multi-image QA task with a learning rate of 5e-5 using Adam optimizer with a linear warm-up of 10% of the total steps. Our relevance encoder and MI-BART were trained using 3 Nvidia RTX A6000 GPUs with a batch size of 96 and 256 while training and a batch size of 360 and 480 during testing, respectively.

5.2 Results and Analysis

We conduct our experiments in two settings, namely, (i) Oracle images: Here, we use ground-truth relevant images for answer generation, and (ii) Retrieved images: Here, relevant images are retrieved using our relevance encoder. We show the results under both these settings in Table 4. We observe that the popularity-based methods perform poorly. This result is expected as popularity-based methods do not use any question or image context. Methods that involve either questions or images perform better than the popularity-based baselines. However, the question-only baseline has a F×A score of 10.4 on RETVQA, showing that image context is needed to generate accurate yet fluent answers. Transformer-based baseline VLP and image-stitch MI-BART reach a F×A score of 58.8 and 70.7 on our dataset, respectively, in the oracle setting, compared to 79.8 of our proposed MI-BART framework. Image-stitch MI-BART outperforms transformer-based VLP

by 12% on our dataset and 1.5% on the WebQA dataset, which shows that having a separate decoder in the proposed MI-BART baseline has better reasoning capabilities than a unified encoder-decoder like VLP. We further present the QA results over the retrieved images setting using our relevance encoder, which has an F1 score of 71 at the top-2. All the approaches involving image context outperform question-only baseline, emphasizing that RETVQA has a reasonable utility to develop and benchmark methods capable of jointly reasoning over multi-image context and the question.

Further, in Table 5, we show the QA results over various question categories, and in Table 6, we show the results over answer categories under the retrieved images setting. Our framework outperforms baselines, especially in questions with open-ended generative answers, which constitute nearly half of our dataset. As expected, open-ended generative questions are more challenging than binary ones. However, compared to the baselines, MI-BART provides better improvement for open-ended questions than binary ones by jointly reasoning over multi-image context. Results in Table 7 further emphasize our hypothesis of requiring multiple images to answer the given question. We show the missing caption ablation results on the image-subset of WebQA in Table 8; this result further affirms our claims that the performance of methods on the WebQA dataset depends on the image metadata like captions.

Qualitative analysis. We illustrate a selection of results using our proposed approach and one of the most competitive baselines viz. VLP in Figure 5. In both these results, our approach correctly answers the question in a large heterogeneous visual context. Further, to understand the importance of the multimodal input for question answering, we plot the multimodal attention map over the retrieved images and the question during the answer generation in Figure 6. The figure shows that our proposed MI-BART model attends to the relevant regions of both images while generating the main answer word 'sheep'. Further, we observe that it attends to brown horse regions of the first image along with the corresponding question parts while generating 'brown horse'. Thus, both images are needed and paid attention to when generating the right answer. We further conducted a detailed error analysis on 50 randomly chosen samples where our model failed to generate a correct answer. We categorize the errors into four major categories: (i) partial retrieval: images retrieved by relevance encoder are partially relevant (52%). (ii) Incorrect retrieval: images retrieved by the relevance encoder are entirely irrelevant (26%). (iii) Incorrect reasoning: model generating a partially incorrect answer despite all the retrieved images being relevant (40%).

6 Conclusion and Future Scope

In this paper, we introduced the RETVQA task. We proposed a unified Multi Image BART model to answer the question from the retrieved images using our relevance encoder. Our proposed framework shows promising improvements over the baselines. We have also performed several ablations to further understand the importance of various modules in the proposed framework. In the future, we would like to explore stronger retrieval models and QA on a large pool of images. We firmly believe RETVQA will pave the way for further research avenues in a broader theme of web image QA.

Acknowledgments

Abhirama is supported by the Prime Minister Research Fellowship (PMRF), Government of India. We thank Microsoft for supporting this work through the Microsoft Academic Partnership Grant (MAPG) 2021.