MsRAG: Knowledge Augumented Image Captioning with Object-level Multi-source RAG

1 Introduction

The rise of Large Language Models (LLMs) has significantly enhanced the utility of Artificial Intelligence Generated Content (AIGC) [Touvron et al., 2023; Bai et al., 2023a], drawing extensive attention and research interest from both academia and industry. Leveraging the robust natural language understanding capabilities of foundation LLMs, the emergence of Language-Visual Large Models (LVLMs) has notably bridged the gap between textual and visual modalities, greatly improving models' vision comprehension abilities [Liu et al., 2023; Bai et al., 2023b]. Traditional benchmarks for text-image understanding, such as NLVR2 Test [Suhr and Artzi, 2019], CommercialAdsDataset[Zhu et al., 2022], nocaps [Agrawal et al., 2019], and VQA [Goyal et al., 2017], which focus on visual reasoning, image captioning, and simple visual question answering (VQA) tasks, are no longer sufficient to comprehensively assess LVLMs visual understanding capabilities. Consequently, more complex benchmarks like knowledge-based VQA tasks [Marino et al., 2019; XENOS et al., 2023] have been proposed to test the multimodal understanding capabilities of LVLMs. However, constrained by the timeliness and knowledge breadth of pretraining data, these models often exhibit hallucination issues in real-world knowledge based VQA tasks [Bai et al., 2024]. DPO [Rafailov et al., 2024], RAG [Lewis et al., 2020], etc. are currently effective solutions for addressing model hallucinations. Among them, RAG usually provides the model with information related to user queries or images through incontext learning, namely incorporating real-world knowledge as prior information input to the model. This approach can effectively reduce model hallucinations, improving response quality of visual tasks (such as visual understanding, visual reasoning, and captioning) without post-training. Consequently, the RAG methods are widely used in LLMs/LVLMs.

∗ Corresponding author.

Similar to visual reasoning and VQA tasks, image captioning plays an important role in LVLMs based visual tasks, and high quality captioning data is critical for enhancing the performance of visual tasks. The pre-training phase of current LVLMs heavily relies on image captioning tasks to achieve effective visual and text modality alignment [McKinzie et al., 2025]. The quality of the captioning data also greatly determines the knowledge boundary of the foundation model, impacting overall performance [Yin et al., 2023]. Knowledge-augmented image captioning [Yu et al., 2024; Jiang et al., 2024] can effectively combines image descriptive language and real-world knowledge, which is crucial for practical applications or synthesizing high-quality data for Input Image

In this paper, we formally define real-world knowledge to include time-sensitive knowledge and domain-specific knowledge.

C1 (w/o RAG)

C2

(w/ mRAG)

C3

(w/ our MSRAG)

The image shows four German football players ... The players are wearing their ŶatioŶal teaŵ uŶifoƌŵs …

…shows four German football players. The goalkeeper on the far right is Manuel Neuer, who was …

From left to right, they are:1. Bastian Schweinsteiger, ... 2.Fritz Walter, … 3. Miroslav Klose, ... 4. Manuel Neuer, ...

The building features two large, white shell-shaped structures that resemble pearl oysters …

The image shows a young Asian man sitting outdoors, holding up a white smartphone or camera device …

This image shows the Zhuhai Grand Theatre … is iŶspiƌed ďy the uŶiƋue marine species and features two large "shells"

This is Zhuhai Grand Theater, … the "Sun and Moon Shells". you ĐaŶ also … Ŷaŵed Mingting Tower, foƌŵeƌly …Deyuefan Xu Zhisheng, a Chinese stand-up ĐoŵediaŶ aŶd aĐtoƌ …

… a peƌsoŶ holdiŶg aŶ OPPO Find X 8 smartphone. … is ideŶtified as Xu Zhisheng, a Chinese comedian ...

there are two stone pillars in a ďuildiŶg with a skylight …

The image shows the Temple of Dendur … ďuilt iŶ the 2nd century BC

… shows the Egyptian Temple of Dendur in the Metropolitan Museum of Art...

Figure 1: Some captioning results of our MsRAG compared to other methods. The proposed MsRAG enables LVLMs achieving high-quality knowledge augmented captioning in various domains. Text in green represents knowledge gain of MsRAG. Text in red is retrieved contents of mRAG. Text in grey is common descriptive language of the image.

improving visual comprehension capabilities.

In short, we take advantages of both RAG and knowledgeaugmented image captioning technologies to equip LVLMs with up-to-date and domain-specific knowledge, thereby enhancing the performance of image captioning tasks. Unfortunately, current mainstream RAG solutions are tailored for scenarios with user queries [Gao et al., 2023; Zhao et al., 2023], making them unsuitable for situations without such queries. Moreover, interpreting user intent and effectively utilizing retrieved content become challenging in the absence of user queries. To address this issue, we propose the MsRAG framework to guide LVLMs in effectively retrieving and utilizing real-world knowledge without user queries (see Fig. 1). By doing so, it makes AIGC technology more accessible to everyday users and unlock its full potential.

The MsRAG framework aims to solve two key research challenges. (1) Multi-source RAG information acquisition. How can abundant and related RAG content be obtained without user queries? (2) Complex RAG content utilization. How can diverse RAG information be effectively used, while noisy information can be automatically ignored and LVLMs' self-correction capabilities can be improved? To evaluate the effectiveness of MsRAG, we integrate several mainstream LVLMs, including GPT4o, Claude-sonnet-3.5, Qwen2VL [Bai et al., 2023b], and InternVL2 [Chen et al., 2024], into the framework. We also introduce a knowledgeaugmented image captioning benchmark KAC-dataset that covers multiple domains. By assessing the captioning results, we tested the MsRAG's ability to retrieve and utilize RAG information under non-query conditions. We further evaluated our MsRAG on existing benchmarks like CapFusion [Yu et al., 2024] and Kale [Awadalla et al., 2024]. Results show that MsRAG significantly improves the knowledge-based captioning capabilities of LVLMs, demonstrating its effectiveness in enhancing real-world and domain-specific knowledge integration and utilization. Additionally, MsRAG demonstrates robustness by filtering out irrelevant or noisy RAG information and correcting inaccuracies, thereby improving the overall quality of generated captions. This proves the successful fusion of RAG and knowledge-augmented image captioning techniques, leading to notable improvements in object-level knowledge based image captioning tasks.

Our motivation is inspired by the reliance of current QA systems on user language queries. With the willing of effectively applying RAG system to image captioning task, we propose MsRAG, an effective RAG framework that can address the knowledge gap of existing RAG systems in knowledge-augmented image captioning tasks. Our contributions can be summarized as follows.

• We propose a general tuning-free RAG framework to boost LVLMs' knowledge-based image captioning capability. MsRAG is plug-and-play that can be easily, flexibly, and cost-effectively integrated with any opensource or close-source LVLMs.
• We develop an object-level Parallel Visual Search module to enrich retrieved information from both online visual search engines and offline domain-specific knowledge databases.
• We propose a Visual-RAG alignment module to bridge the gap between object-level image semantics and their text-format RAG contents.

2 RelatedWork

2.1 Large Vision-Language Models (LVLMs)

The significant improvement in natural language understanding by LLMs has established a strong foundation for advanced multi-modal comprehension. Prominent mainstream LVLMs, such as Qwen2VL [Bai et al., 2023b], InternVL [Chen et al., 2024], and LLaVA [Liu et al., 2023], are typically constructed using pre-trained LLMs and visual encoders. These LVLMs demonstrate excellent performance in comprehending both user instructions and image content through visual-text modality alignment training. They deliver impressive performance in diverse visual understanding tasks. Nevertheless, due to constraints in the timeliness and breadth of pre-training corpus, LVLMs frequently exhibit inaccuracies when confronted with scenarios requiring knowledge beyond their initial training scope.

2.2 Retrieval-Augmented Generation (RAG)

RAG is widely used in LLMs to mitigate model hallucination. Typically, it takes a user query as input, retrieves relevant external information, and enhances the output with the retrieved content [Gao et al., 2023]. This technique has also been applied to LVLMs for knowledge-based VQA tasks. For example, Wiki-LLaVA [Caffagni et al., 2024] employs a two-stage retrieval process, first using the input image and then refining the content with the user query. EchoSight [Yan and Xie, 2024] introduces a Q-Former based reranker for retrieval using both query and image inputs, while SearchLVLMs [Li et al., 2024] enhance queries before retrieving and filtering web content. Entity-centric methods such as SnapNTell [Qiu et al., 2024], MAR [Zhang et al., 2024b], and MuKA [Deng et al., 2025] leverage visual grounding to retrieve knowledge about specific objects in images.

All approaches mentioned above rely on extra expert models to handle the retrieved content and ignore the inherent capacity of the LVLMs. Self-RAG [Asai et al., ] strengthens the model's self-evaluation capability by setting special reflection tokens during training. Inspired by selfRAG, mR2AG [Zhang et al., 2024a] introduces a retrievalreflection mechanism guiding the model to step-by-step judge the relevance of the retrieved content with input image and user query, reranking the retrieved content based on its own capacity. However, these RAG-based methods mainly focus on VQA scenarios that rely on user queries to retrieve domain-specific and time-sensitive knowledge. Few research addresses the retrieval and utilization of real-world knowledge under non-query conditions. This gap highlights the need for new approaches that can effectively generate image captioning without explicit user queries, expanding the applicability of such systems to a broader range of scenarios.

2.3 Knowledge Augmented Image Captioning

Yu [Yu et al., 2024] analyzed the impact of captioning data in the pre-training phase on the model's knowledge boundary, finding that LVLMs trained based on existing synthetic captioning data such as LAION-COCO [Schuhmann et al., 2022] would experience a significant degree of knowledge loss. This is attributed to the fact that the captions output by the captioning model used by LAION-COCO, such as BLIP [Li et al., 2022], often replace some specific content with more common placeholders, resulting in the final trained LLMs tends to caption image with simple language structures containing only basic semantic information rather than detailed object-level real-world knowledge.

To address this issue, CapFusion [Yu et al., 2024] generates knowledge-augmented image captioning data by prompting GPT with raw captions from Laion-2B and descriptive captions from LAION-COCO, and the experimental results prove that such captioning data can effectively broaden the model's knowledge boundary. Inspired by CapFusion, BLIP3-KALE [Jiang et al., 2024] further improves the granularity and efficiency of data annotation with a two-stage process. First, LVLM was adopted to create an initial pool of knowledge augmented captioning data. Then, captions from stage 1 were used to distill a light-weight VLM to accelerate the captioning process. Compared with previous captioning tasks, knowledge-augmented image captioning is more valuable in both real-world image caption applications and high quality captioning data generator for training stronger LVLMs. However, current knowledge-augmented image captioning task primarily focuses on extending existing datasets, and does not involve open-domain images and the application of RAG systems to achieve general knowledge-augmented image captioning. In this paper, We propose MsRAG, focusing on leveraging LVLMs to effectively retrieve and utilize fine-grained object-level multi-source RAG information under non-query conditions, so as to achieve high-quality, generalized knowledge-augmented image captioning.

3 MsRAG Framework

In this section, we introduce our MsRAG, a tuning-free framework enabling LVLMs to achieve knowledge augmented image captioning with object-level multi-source RAG contents. MsRAG framework is shown in Fig. 2, given an input image I without language query, we focus on investigating how to retrieve and utilize real-world knowledge (e.g., domain-specific knowledge) without user query, and incorporate retrieved knowledge into captioning results.

For the input image I, we propose a parallel visual search module containing a pre-process module and an RAG content summary module. In the pre-process phase, an object detection model is applied to identify objects in the image and crop object-aware image Io. Subsequently, Io is fed into the RAG content summary module to further extract and analyze detailed object-level knowledge from both online visual search engines and offline domain-specific knowledge databases. Based on the parsed information output from the RAG content summary module, we designed a set of prompt templates. This pool dynamically selects the appropriate prompt according to the object detection results and RAG contents, enhancing the LVLMs' ability to utilize complex RAG information effectively. We also propose a Visual-RAG alignment module (VRAM) to align the object-aware image Io with its corresponding text-format RAG content {Riol }ni=1 . This process generates a marked image I m o and an aligned prompt PA, allowing LVLMs to better understand the relationship between visual objects and their associated objectlevel knowledge. Finally, the marked image I m o and the Visual RAG aligned prompt PA are sent to the LVLMs to generate knowledge-augmented image captions.

3.1 Parallel Visual SearchModule

Given an input image I, undergoing preprocessing module, an object-aware image Io is obtained. We crop different objects according to their bounding boxes and then merge & filter them based on their entity attributes to obtain sub-images I io , i means the i-th object in the image. These sub-images are then inputted into online visual search engines to obtain object-level knowledge {Riol }ni=1 from the websites. However, online visual search engines perform well in retrieving time-sensitive knowledge but fail to provide accurate information in specific domains like culture relics, products, and scenes. We organized an offline domain-knowledge database to compensate for the domain-specific knowledge gap in web search. Similarly, we can also retrieve offline object-level knowledge {Riof }ni=1 from offline database. For the retrieval information of each object from both online Riol and offline Riof , we select the top three search results [Ri,1:3 ol ,Ri,1:3 of ] based on cosine similarity and summarize them using a lightweight LLM. Ultimately, we obtain rich object-level multisource knowledge from the website and offline database.

3.2 Prompt Matching

As for the RAG contents obtained from Parallel Visual Search module, we categorize them into three scenarios and dynamically adjust the corresponding prompts to enhance the model's response quality. Hence, the prompts in the Prompt Templates Pool can be mainly classified into three types.

1. Multi-object multi-source RAGprompt (AppendixA.3): In this case, the image contains multiple objects, while retrieved object-level RAG contents are from multiple RAG sources (e.g., different online search engines and different offline domain-knowledge databases). We designed a prompt suitable for multi-object scenarios and proposed a Visual-RAG alignment module to explicitly align the visual objects with the corresponding text-format object-level RAG contents, enhancing the model's ability to comprehend image information. A detailed illustration of the Visual-RAG alignment module can be found in Section 3.3 and Fig.3.
2. Single-object multi-source RAG prompt (Appendix A.2): When there is only one object in the image
3. but with multiple RAG sources, we propose a selfrectification mechanism. Instead of assuming that the RAG content is correct and using it as-is, the selfrectification mechanism first relies on the model's pretrained knowledge to judge the relevance of RAG content and the visual object (prompting LVLMs to identify whether the RAG content is related to the image objects with pre-trained knowledge).
4. General Prompt (Appendix A.1): When no RAG information is available, a general prompt is used to encourage the model to generate an objective image caption with its own knowledge.

3.3 Visual-RAGAlignment Module

In scenarios with multiple objects and multiple RAG sources, traditional RAG strategies are unable to effectively associate the text-format RAG content with the visual objects. This could hinder the effective utilization of the object-level RAG information, affecting the quality of captioning results. We propose a Visual-RAG Alignment Module (VRAM) to bridge the modality gap between textual RAG contents and visual image objects, enabling the model to better understand the image semantics and use the correct RAG content to describe corresponding object. The VRAM, as shown in Fig.3, inspired by the practices of set of marks[Yang et al., 2023], for input object aware image and object-level RAG contents, we start by marking the objects in the image via their bounding box. Then we formulate visual marks in text format and integrate them with RAG content to construct an object position-aware prompt. Our Visual-RAG alignment module effectively bridges the modality gap between visual and textual modalities. This allows the model to better understand the correspondence between object-level RAG contents and visual objects, thereby outputting high-quality, multi-object knowledge-augmented captions.

4 Experiment

4.1 Settings

Dataset Collection. We classify the open domain images into seven categories, namely Cha.(Characters), A. & P.(Animals & Plants), Sce.(Scenes), Pro.(Products), Food, Cul.(cultural relics), and Mul.(Multi-object scenario). We collect failure cases in real world applications pertinent to our business domain to form the KAC-dataset. The static results of KAC-dataset are shown in Fig.4. The pie chart on the right displays the data proportion for each domain in the KACdataset, with Multi-object domain (21%), Culture Relics domain (23%), and Plants & Animals domain (13%) taking up relatively larger shares. The Multi-object domain mainly consists of image data that includes multiple objects, thus it can effectively validate the efficiency of the VRAM within the MsRAG framework. Similarly, the offline domainknowledge databases within parallel search module aims to address the deficiencies of online visual search engines in retrieving information in areas such as Culture Relics, scenes, and products. So increasing the proportion of data in domains relevant to the offline domain-knowledge databases can verify the efficacy of the parallel visual search module.

In addition to this, we present typical examples from each domain in the KAC-dataset, as shown in the left image of Fig.4. The text in red represents the correct retrieval information. As the online visual search engine performs well in the Cha.(Character) domain, there is no need to implement offline database in Cha. domain. Therefore, data in the Cha. domain typically only contains retrieval contents searched by the online visual search engine. Pro. and P. & A. domains, however, often struggle to retrieve valuable information from the web. The Pro. domain often fails to return any useful retrieval results, while the P. & A. domain usually only retrieves broad classification information about plants or animals, such as their genus or phylum, but lacks detailed species information. For these two domains, the offline domain-knowledge databases always offer more accurate information. It is worth mentioning that since we use more than one online visual search engine, we often get multiple search results from the web, as shown in the example of the Food domain. For the Sce. and Cul. domains, the retrieval information is more complex. In addition to the information retrieved from the web, data in these two domains often simultaneously trigger Sce. and Cul. tracks in the offline database, returning multiple retrieval results. As shown in the example data for the Cul. domain, the retrieval content includes information from both the Sce. track(Henan Museum) and the Cul. track(Lotus and Crane Rectangular Hu). The Lotus and Crane Rectangular Hu is housed in the Henan Museum, which means that both pieces of information are meaningful for captioning. Finally, the data from the Mul. domain returns object-level knowledge once it has passed through the Parallel Visual Search Module. Each object's information may stem from either online or offline. As shown in the example from the Mul. domain, object 1 is Steve Jobs, whose information was retrieved online, while the iPhone 1(object 2) he holds was sourced from Pro. track in offline database. Detailed information of offline domain-knowledge database is shown in Table 1, We categorize the data in the offline database into four tracks, including the P. & A. track (Animal and Plant track), the Cul. track (Culture Relics track), the Sce. track (Scenes track), and the Pro. track(Products track). The term 'Entity' represents the number of distinct entities within each track, while 'Volume' denotes the total amount of samples in each track.

Baselines. We evaluate MsRAG on multiple LVLMs, including GPT-4o, Claude-3.5-Sonnet, Qwen2-VL, and InternVL2. For closed-source models (GPT-4o, Claude), we use their APIs; for open-source models, we deploy them with vllm[Kwon et al., 2023]. We also compare MsRAG against mRAG-integrated baselines to verify its effectiveness.

Evaluations. We evaluate MsRAG on LVLMs using three datasets: CapFusion, Kale, and KAC-dataset. CapFusion and Kale are public captioning datasets with real-world knowledge, aligning well with the knowledge-augmented captioning task, effectively testing MsRAG's retrieval and utilization of external information without queries. All experiments were run on two Nvidia A100s.

4.2 User Study

We conducted a user study on the KAC-dataset using mRAG and MsRAG methods to invoke various LVLMs for captioning the samples in KAC-dataset, with 10 caption results for each sample. Users were asked to rank these captions based on the accuracy and richness of knowledge information they provide. The scoring process was done in a fully blinded manner as the content shown to the users were randomly shuffled results of various captioning methods, ensuring fairness in the scoring process. We eventually collected 20 valid scoring results, and the summary is shown in Table 2.

MsRAG outperformed mRAG in most scenarios, with significant gains in the multi-object (2.78), cultural relics (3.24), and product domains (3.18), demonstrating the effectiveness of its offline database and Visual-RAG alignment. However, the improvement in the character domain was marginal (0.28), as pre-trained LVLMs and online visual search engines already excel in character recognition.

4.3 Quantitative Comparison

Following the evaluation metrics of current image captioning and VQA works, we adopt these metrics to evaluate the performance of our MsRAG in the CapFusion and Kale datasets.

Evaluation results are displayed in Table 3. It can be inferred that (1) GPT4o combined with our MsRAG has the best overall performance on both the CapFusion and Kale datasets. (2) Our method has continuous gains on almost all of the LVLMs (especially commercial models such as GPT and Claude, which can better exploit multi-source RAG contents). It is noteworthy that metrics like BLEU and CIDEr yielded relatively lower scores on the CapFusion dataset as opposed to the Kale. This is due to the fact that most of the ground truth data in the CapFusion are relatively short and knowledge-oriented, lacking a detailed description of the images. However, generated text often contains more detailed descriptions, which leads to text length significantly surpassing the ground truth text and hence the scores by lengthsensitive metrics such as BLEU and CIDEr are skewed.

4.4 Ablation Study

Fig.5 shows our ablation results. We first evaluated VRAM's effectiveness in multi-object scenarios by comparing full MsRAG against baselines without SoM and prompt alignment. Human evaluators scored captions (0=no real-world knowledge, 0.5=partial knowledge, 1=full external knowledge utilization). Results (Fig.5a) show VRAM significantly improves performance (w/VRAM:0.77 vs w/o:0.41), with 86% of scores in the 0.5-1 range (vs 62% without). We also selected two typical examples from the Multi-object domain as shown in Fig.5b. From the above examples, it can be seen that the captioning result with VRAM correctly understands the location of the object in the image, the content of the news and their correspondence with retrieved object-level RAG contents.(eg. Li Yuyi is the official mentioned in the news that is being publicly tried). However, the captioning result without VRAM cannot effectively use retrieval information, confuses the retrieval information and the content of news text in the image (eg. mistakenly regards Liu xiang in the content of news text as the official (Li Yuyi) under public trial). This further proves that VRAM in MsRAG can help LVLM better understand the relationship between the objects in the image and the corresponding retrieved RAG contents.

In addition, we also verified the effect of the offline domain-knowledge databases in the parallel visual search module. We removed the offline database as the baseline. The scoring results are shown in the P. & A. , Sce. and Cul. columns of Fig.5a. After adding the offline domainknowledge databases, the quality of LVLM-caption in the P. & A. , Sce. and Cul. domain significantly improved (P. & A w/ OD: 0.75, P. & A w/o OD: 0.48. Sce. w/ OD: 0.79, Sce. w/o OD: 0.44. Cul. w/ OD: 0.77, Cul. w/o OD: 0.21.), proving the necessity of using offline database to make up for the online visual search engine and verifying that our prompt design in MsRAG can make LVLM better utilize the complicated multi-source RAG information. Furthermore, We investigated the retrieval information gain of the offline database in the five domains presented in Table 1 on the KAC-dataset. Analysis results are shown in Table 4, the Gain proportion represents the proportion of data where the offline database can provide effective information while online visual search engine fails. It can be observed that the offline database has a continuous gain across all five domains, with the largest gain in the Cul. and Sce. domain.

To further demonstrate MsRAG's superiority over mRAGbased methods in query-free scenarios, we compared them under the VQA setting to highlight key differences. As illustrated in Fig.6, in the first example where the image has multiple objects and the query lacks semantic detail, MsRAG aligns visual and textual cues effectively, ensuring high-quality responses. Conversely, m-RAG struggles due to insufficient query signals. When the image has a single dominant object and the query is semantically rich (the second example), m-RAG performs better by leveraging precise knowledge retrieval. Overall, MsRAG excels with multiobject images and weak queries, while m-RAG works better with informative queries and single-subject images. Combining both approaches could further enhance VQA performance. Thus, for caption task (like VQA tasks with semantically weak queries), our MsRAG is more effective.

5 Conclusion

In this paper, we propose MsRAG, a general tuningfree framework for knowledge-augmented image captioning. MsRAG is plug-and-play that can be easily, flexibly, and costeffectively integrated with any open-source or close-source LVLMs. It includes: (1) a Parallel Visual Search Module for retrieving object-level information online and offline without language queries; (2) a Prompt Template Pool to dynamically adjust prompts based on retrieved content, enhancing the LVLMs' ability to utilize complex RAG contents; (3) a Visual RAG Alignment Module to align text-format objectlevel knowledge with visual objects, enhancing the model's understanding of the correlation between image contents and RAG contents. The aligned image and prompt are fed into the LVLM to generate knowledge-centric captions. Our method is evaluated on the proposed KAC-dataset, CapFusion, and Kale, demonstrating its effectiveness.