Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models

1 Introduction

Recently, the development of large language models (LLMs) [Chung et al., 2022; Touvron et al., 2023a; Touvron et al., 2023b] has notably propelled advancements in artificial general intelligence. Various domains within artificial intelligence have actively embraced LLMs to enhance their performances across different tasks [Liu et al., 2023c; Chen et al., 2023; Hong et al., 2023]. The multimodal dialogue field is no exception, witnessing a surge in the development of multimodal large language models (MLLMs) in recent months [Liu et al., 2023c; Zhu et al., 2023; Dai et al., 2023; Ye et al., 2023b; Bai et al., 2023]. These works commonly insert visual encoders into LLMs, followed by finetuning a light-weight projection network to project extracted visual information into the language latent space.

† Corresponding author

While recent advancements have notably enhanced the performances of downstream multimodal dialogue tasks [Goyal et al., 2017; Singh et al., 2019; Mishra et al., 2019; Masry et al., 2022; Shah et al., 2019], these improvements primarily stem from the collection of instruction data in various formats [Liu et al., 2023c; Liu et al., 2023a; Zhu et al., 2023; Ye et al., 2023a; Chen et al., 2023]. Pioneering works such as MiniGPT-4 [Zhu et al., 2023] and LLaVA [Liu et al., 2023c] introduced an automatic mechanism for generating general multimodal instruction data, leveraging the capabilities of ChatGPT [OpenAI, 2023]. By subsequently fine-tuning MLLMs with the generated data, these approaches have achieved substantial enhancements in general multimodal tasks. Additionally, mPLUG-DocOwl [Ye et al., 2023a] targets to amass instruction data related to documents, specifically enhancing the performance of MLLMs in document understanding tasks [Masry et al., 2022; Mishra et al., 2019;

Mathew et al., 2021; Berant et al., 2019]. Shikra [Chen et al., 2023], on the other hand, proposed to collect referring expression pairs and fine-tune MLLMs on these pairs, thereby strengthening the models' ability to handle the referential dialogue tasks [Kazemzadeh et al., 2014]. Furthermore, Instruct-BLIP [Dai et al., 2023] and other related works [Bai et al., 2023; Wang et al., 2023] have proposed to assemble various multimodal datasets with distinct instruction templates. Subsequent fine-tuning of MLLMs on these consolidated datasets has proven instrumental in significantly improving their performances across different multimodal tasks.

As outlined above, while prior works have demonstrated advantages across various multimodal dialogue scenarios, they predominantly focus on collecting different types of instruction data, sharing a similar learning framework. Specifically, these works consistently employ a light-weight projection module (e.g., Q-Former in BLIP-2 [Li et al., 2023]) to project visual information, extracted by CLIP-like encoders (e.g., EVA-CLIP [Sun et al., 2023]), into the language latent space. Given that the CLIP-like encoders cannot comprehensively describe the entirety of visual inputs (for them pre-trained with short textual captions, as shown in Fig.1(a)), MLLMs grapple with the visual information loss dilemma, which further affects the final performances. Moreover, though the detailed instruction data generated in LLaVA [Liu et al., 2023c] or other works [Zhu et al., 2023; Chen et al., 2023] can alleviate the above problem to some extent, there are still lots of details that cannot be fully described (as shown in Fig. 1(b)). To address this challenge, there is a need for novel strategies that transcend the existing learning frameworks, enabling a more nuanced and accurate representation of visual inputs in MLLMs.

Inspired by the above, this paper explores MLLMs from the perspective of visual perception ability enhancement. Consequently, we introduce a simple but effective visual information learning framework, referred to as Incorporating Visual Experts (IVE), designed to augment the visual perception capabilities of MLLMs through aggregating available visual information extracted by specific experts. Specifically, IVE mainly involves two additional modules, i.e., multi-task encoders and structural knowledge enhancement, for comprehensively describing the visual inputs. The multi-task encoders module integrates three auxiliary encoders, namely the low-level information encoder and the document-related information encoder, alongside with a CLIP-like encoder for semantics extraction. The above fusion aims to provide a more comprehensive description of visual inputs within the latent embedding space. The synergistic combination of these encoders facilitates a more nuanced understanding of the visual inputs. The structural knowledge enhancement module mainly utilizes specific visual tools to extract structural data (e.g., the categories and locations of instances or textual information inside images). These structural data will serve as prior knowledge and then be cooperated with the extracted latent embeddings fed into LLMs.

The introduced IVE is easy to implement, and its effectiveness has been substantiated through comprehensive experiments across various multimodal tasks. In general multimodal dialogue scenarios [Goyal et al., 2017; Marino et al.,

2019], IVE excels in recognizing the intrinsic content of input images, thereby generating more accurate responses to each query in comparison to recent works [Liu et al., 2023c; Zhu et al., 2023]. Furthermore, when applied to specific multimodal dialogue tasks such as DocVQA [Mathew et al., 2021], IVE shows competitive results compared with mPLUG-DocOwl [Ye et al., 2023a], the recent method designed specifically for document analysis tasks. The above experiments further demonstrate the excellent visual perception ability achieved through the proposed integration strategy of visual experts.

2 RelatedWork

2.1 Vision-and-Language Pre-training

Recent multimodal large language models (MLLMs) [Liu et al., 2023c; Zhu et al., 2023; Dai et al., 2023; Ye et al., 2023a; Bai et al., 2023; Ye et al., 2023b] are commonly built on vision-and-language pre-training models (VLPs) [Chen et al., 2020; Radford et al., 2021; Li et al., 2022], therefore this paper first revisits the development of VLPs before introducing MLLMs. The predominant VLP approaches can be broadly categorized into two frameworks: the one-stream framework [Chen et al., 2020] and the two-stream framework [Radford et al., 2021; Li et al., 2022]. Methods within the one-stream framework [Chen et al., 2020] typically employ a single transformer architecture to process both text and image data, incorporating various designs of loss functions. In contrast, the two-stream framework involves the independent extraction of modality information using distinct backbones. For efficiency, current MLLMs [Zhu et al., 2023; Dai et al., 2023; Ye et al., 2023b] predominantly leverage the visual module of two-stream methods to encode the latent embeddings of visual inputs.

2.2 Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) have garnered considerable attention from both academia and industries, with a surge in novel works emerging recently [Liu et al., 2023c; Dai et al., 2023; Ye et al., 2023b; Bai et al., 2023; Huang et al., 2024; Zong et al., 2024; Xuan et al., 2024]. A common framework underpins most of these works, featuring CLIP-like encoders responsible for extracting information from visual inputs, an abstractor summarizing the extracted information with few tokens, a light-weight layer further projecting the summarized information into the language latent space and a pre-trained large language model handling user questions in the context of the above extracted visual information. Despite their similar architectures, these works demonstrate versatility in addressing various multimodal dialogue tasks through training on distinct types of instruction data. For instance, LLaVA [Liu et al., 2023c] excels in generating detailed answers for generic images with training on comprehensive instruction data. On the other hand, mPLUGDocOwl [Ye et al., 2023a] achieves significant improvements in the performance of MLLMs on document analysis tasks by training on document-related instruction data. Shikra [Chen et al., 2023] enhances the model's capability in handling referring questions by training on referring expression pairs.

Although these works yield remarkable results, they remain constrained by the limited perception ability of CLIP-like encoders. In contrast to previous approaches, this work takes a novel perspective by focusing on enhancing the visual perception ability of MLLMs. The proposed approach involves aggregating available visual experts to provide a more comprehensive description of visual inputs, aiming to overcome the constraints imposed by the existing limitations in visual perception ability.

3 Our Approach

3.1 Preliminaries

Generally, the multimodal large language models (MLLMs) [Liu et al., 2023c; Dai et al., 2023; Ye et al., 2023b; Bai et al., 2023] are usually composed of three modules, i.e., the visual perception module, the light-weight projection module, and the large language model, respectively. Specifically, the visual perception module extracts the inside contents from visual inputs and then the light-weight projection module projects the above visual information into the language latent space. The large language model receives the projected visual information and generates textual responses for each user query. Therefore, given the visual input as xi, the query as qi, the visual perception module as Fvis(·), the light-weight projection module as Fproj(·) and the large language model as LLM(·), the process of generating response in MLLMs can be formulated as:

where Responseqi:xi denotes the generated response for the query qi based on the visual input xi.

Limited by the computing and data resources, most current MLLMs directly utilize well-trained large language models, such as Flan-T5 [Chung et al., 2022] and LLaMA [Touvron et al., 2023a], as the encyclopedia to answer the given question. Therefore, the key for MLLMs lies in how to properly summarize the information of visual inputs into language space. Currently, most MLLMs [Bai et al., 2023; Zhu et al., 2023; Liu et al., 2023c; Dai et al., 2023; Ye et al., 2023b; Ye et al., 2023a] usually utilize CLIP-like encoders to extract the visual information, and then fine-tune a light-weight projection network with the collected instruction-following data to project extracted visual information into language latent space. Though extensive experiments have validated its effectiveness, the descriptions of visual inputs extracted by CLIP-like encoders are still not enough. As said "a picture is worth a thousand words", the CLIP-like encoders can only extract coarse semantic features inside each image in spite of their training on the billions of image-text pairs. To facilitate the above information loss dilemma, this paper proposes to incorporate visual experts in MLLMs, for comprehensively summarizing the visual contents of inputs. Details of the proposed approach will be described carefully in the next.

3.2 Incorporating Visual Experts intoMLLMs

Different from previous works [Zhu et al., 2023; Liu et al., 2023c; Ye et al., 2023b; Bai et al., 2023; Chen et al., 2023], this paper improves the visual perception ability of MLLMs from the perceptive of knowledge enhancement, and thus proposes a simple but effective framework with primarily Incorporating different types of Visual Experts into the current MLLMs, referred as IVE. As shown in Fig. 2, the visual perception within IVE relies on two pivotal modules: the multi-task encoders and structural knowledge enhancement module. The multi-task encoders is designed to amalgamating various types of latent visual information extracted by multiple visual encoders. This integration improves its comprehensiveness in the view of latent embedding of visual inputs. Additionally, the structural knowledge enhancement module is crafted to leverage visual tools, such as OCR tools [JaidedAI, 2020] and object detectors [Zhang et al., 2023; Liu et al., 2023d], to extract prior knowledge from images. This extracted knowledge is then treated as visual prompts and incorporated into the large language model alongside the previously fused latent embeddings. Through the above cooperative modules, IVE can comprehensively encode the contents of visual inputs from diverse perspectives, thereby enhancing the visual perception ability of MLLMs.

Multi-task Encoders. The majority of current MLLMs commonly rely on CLIP-like encoders for extracting semantic information from visual inputs. However, the limited perception ability associated with this strategy restricts their performances across various multimodal dialogue scenes. In contrast, IVE seeks to resolve this limitation by integrating multiple types of complementary encoders to collaboratively capture the latent information within visual inputs. As depicted in Fig. 2, three types of encoders are employed: the semantic information encoder, the low-level information encoder, and the document-related information encoder, each contributing distinct perspectives to the overall understanding of visual content.

The semantic information encoder is designed to extract the semantics from visual inputs and subsequently project them into the language embedding space. Consistent with prevalent methodologies [Chen et al., 2023; Liu et al., 2023c; Li et al., 2023; Zhu et al., 2023; Ye et al., 2023b], IVE adopts the CLIP-like encoder proposed in BLIP-2 [Li et al., 2023], where EVA-CLIPg [Sun et al., 2023] is initially employed to extract visual information, followed by the Q-former [Li et al., 2023] designed to condense this information into a concise representation using a few tokens. Leveraging extensive training with abundant image-text pairs, this encoder generates embeddings adept at capturing the global semantic information of each visual input. The process of semantic feature extraction can thus be presented as follows:

where Enceva denotes the visual encoder of EVA-CLIPg [Sun et al., 2023], CrossAttQ represents the operations in QFormer [Li et al., 2023], {T0, T1, ..., Tm} denotes the query tokens and m is the sum of query tokens, respectively.

Given the brevity of captions that only provide a coarse description of the global semantics within each image, the semantic information extracted by Eq. (2) is apparently insufficient. To enhance the richness of detailed information within the extracted latent embedding, a low-level information extractor is introduced as the supplement. This paper adopts the encoder from VQGAN [Esser et al., 2021] as the corresponding low-level information extractor, which can encode images into latent embedding and then reconstruct them into original images with the corresponding decoder. However, directly integrating the extracted embedding into MLLMs is costly because of its high dimensionality. Following Flamingo [Alayrac et al., 2022], we also utilize several query tokens (32 tokens) to summarize this latent embedding with a 3-layer Perceiver Resampler [Alayrac et al., 2022], and the resultant tokens are then considered as low-level latent embedding. Consequently, the process of low-level information extraction can be formulated as:

where Encvqgan denotes the pre-trained encoder of VQGAN [Esser et al., 2021], CrossAttPR represents the operations in Perceiver Resampler [Alayrac et al., 2022] and n represents the sum of query tokens for low-level information, respectively.

While the aforementioned low-level information extractor contributes additional details upon the semantic embedding, it's noteworthy that both are trained on general images and may lack specificity for certain types, such as the document image. To address this, a document-related information encoder is incorporated into the latent embedding learning framework. In our framework, Pix2Struct-Large [Lee et al., 2023], a recent state-of-the-art approach in document analysis tasks, is employed for this purpose. Similar to the lowlevel information encoder, 64 query tokens are employed to succinctly summarize the extracted document-related information using a 6-layer Perceiver Resampler [Alayrac et al., 2022]. Generally, the process of document-related information extraction can be formulated as:

where Encpix denotes the pre-trained encoder of Pix2StructLarge [Lee et al., 2023] and k represents the sum of query tokens for document-related information.

Consequently, the final fused latent embeddings of each image in IVE can be formulated:

where Fsproj , Flproj and Fdproj represents the linear projection layer for projecting the semantic information, low-level informaion and documented-related information into a common language feature space, respectively.

Structural Knowledge Enhancement. In view of that query tokens for each extractor undergo end-to-end training, ensuring that the summarized embeddings encompass the entirety of visual input remains a challenge. Thereby, this paper further introduces a structural knowledge enhancement module to explicitly extract structural data within each image using specific visual tools. Finally, these data are subsequently treated as prior knowledge and fed into the large language model alongside the fused latent embeddings.

Typically, human observation of an image involves first identifying the objects (their categories and locations) or textual information within this image. Drawing inspiration from this human cognitive process, the structural knowledge enhancement module is designed to extract three types of information: the category and localization of instances, together with textual content, respectively. We first utilize two specific visual tools (i.e., RAM [Zhang et al., 2023] and Grounding DINO [Liu et al., 2023d]) to recognize and localize the objects inside each image. Furthermore, we utilize EasyOCR [JaidedAI, 2020] to detect the contained textual information of each visual input. Therefore, thanks to the above visual tools, most instances [(c0, x00 , y0 0 , x10 , y1 0 ), ..., (cq, x0q , y0 q , x1q , y1 q )] and textual information [t0, t1, ..., to] inside each image can be detected, where ci denotes the category of the detected i-th instance, (x0i , y0 i , x1i , y1 i ) represents the corresponding bounding boxes, tj means the detected j-th visual text segment, q and o are the sum of detected instances or textual segments, respectively. Thereby, the final extracted structural knowledge can be formulated as:

To better align with LLM, we design the template in which inserting the extracted structural knowledge. The details of this template are shown in Appendix A of the supplemental materials.

Difference with previous works. While extant works [Gao et al., 2023; Shen et al., 2023], exemplified by LLaMA-Adapter v2 [Gao et al., 2023], have explored the integration of visual tools to extract structural knowledge with the aim of augmenting the visual perceptual capabilities of MLLMs, it is notable that these approaches have predominantly restricted the deployment of visual tools solely into the inference stage. In contrast, the proposed IVE is meticulously crafted to harness structural knowledge throughout both the training and inference phases of MLLMs. This strategic design of IVE serves the dual purpose of mitigating the inherent noise introduced by the visual tools and comprehensively capitalizing on the helpful cues they provide.

3.3 Training Pipeline

Once the latent embeddings and structural knowledge are available, we feed them into a well-trained large-scale language model (LLaMA2-chat (7B) [Touvron et al., 2023b], utilized in this work) and conduct the overall training, which makes LLM better handle these prompts while ignoring the inevitable noises. Following previous works [Bai et al., 2023; Dai et al., 2023], we reorganize several public multimodal datasets [Marino et al., 2019; Masry et al., 2022; Mathew et al., 2021; Liu et al., 2023c], and conduct supervised finetuning on them. Overall, the training process goes through three stages: pretraining, multi-task tuning, and specific finetuning. In the pretraining stage, we primarily utilize weakly labeled image-text pairs to train the alignment module in the semantic information encoder. The multi-task tuning stage involves training on various multimodal instruction datasets. Subsequently, in the specific fine-tuning stage, we fine-tune the model on the training set of specific datasets to better adapt to their unique characteristics. Details of each training process are provided in Appendix B of the supplemental materials.

4 Experiments

4.1 Datasets

Training Dataset. The entire training pipeline comprises three stages. In the pre-training stage, about 300M image-text pairs crawled from the Internet [Li et al., 2022] are initially utilized to train Q-Former. Subsequently, the LLaVA-CC3MPretrain-595K from LLaVA [Liu et al., 2023c] is employed to further train Q-Former and the projection layer. In the multitask tuning stage, following previous work [Bai et al., 2023], multi-task datasets are combined to jointly guide the further training of IVE, including several general VQA datasets (e.g., VQAv2 [Goyal et al., 2017]), OCR-related VQA datasets (e.g., OCRVQA [Mishra et al., 2019]), document-related VQA datasets (e.g., DocVQA [Mathew et al., 2021]), grounding datasets (e.g., RefCOCO [Kazemzadeh et al., 2014]), image captioning datasets (e.g., COCO Caption [Chen et al., 2015]), and multimodal instruction datasets (e.g., LLaVAInstruct-150K [Liu et al., 2023c]). The statistics of the used training data in the mutl-task tuning stage are presented in Appendix C of the supplemental materials. In the fine-tuning stage, further fine-tuning is conducted on the training set of specific datasets individually to fit their unique characteristics, thus achieving better performances on specific tasks.

Evaluation Dataset. The evaluations cover general scene recognition, character recognition, chart and document analysis, as well as other multimodal dialogue tasks. To this end, the VQAv2 test set [Goyal et al., 2017], OKVQA test set [Marino et al., 2019], TextVQA validation set [Singh et al., 2019], OCRVQA test set [Mishra et al., 2019], DocVQA validation set [Mathew et al., 2021], ChartQA test set [Masry et al., 2022], WTQ test set [Berant et al., 2019], and MME Benchmark [Fu et al., 2023] are chosen for the evaluations.

4.2 Implementation Details

The overall training process of IVE includes three stages. In the pre-training stage, only the Q-Former and the projection layer of the semantic information encoder are trainable, while the other two encoders are temporarily removed. Moreover, the parameters of other modules remain frozen. The input resolution for the semantic information encoder is set as 224×224. When training with the 300M image-text pairs [Li et al., 2022], the training encompasses only 1 epoch, and a global batch size of 2048. While training with the LLaVACC3M-Pretrain-595K [Liu et al., 2023c], the training encompasses 5 epochs. The learning rate in this stage employs a cosine warm-up strategy (2000 steps), with a maximum learning rate of 1e-4, and a minimum learning rate of 1e-6.

In the multi-task tuning and specific fine-tuning stage, the language model undergoes tuning using LoRA [Hu et al., 2021] with the hyper-parameters of rank=64. The Q-Former, Perceiver Resampler, and their corresponding projection layers are trainable, while the parameters of other modules remain frozen. The input resolution for the semantic information encoder is increased to 448×448, while the low-level information encoder is configured with the input resolution of 256×256. The input resolution of the document-related information encoder is set as 1024×1024. As for the learning rate, we employ a cosine warm-up strategy (500 steps), with a minimum learning rate of 1e-6 and a maximum learning rate of 3e-5 for the multi-task tuning stage, 1e-5 for the specific fine-tuning stage. AdamW serves as the optimizer for all three training stages, with β1 = 0.9, β2 = 0.98, and the weight decay of 0.05, respectively.

4.3 Direct-transfer performances on VQA Datasets

The VQA task entails multimodal large language models (MLLMs) answering questions based on both the visual inputs and user query. In this section, we conduct direct-transfer evaluations on multiple VQA benchmarks using the IVE model trained after multi-task tuning stage. We compare the proposed method with several state-of-thearts, including Qwen-VL-Chat [Bai et al., 2023], mPLUGDocOwl [Ye et al., 2023a], mPLUG-Owl2 [Ye et al., 2023b], and LLaVA-1.5 [Liu et al., 2023b]. The evaluation encompasses seven benchmarks: VQAv2 [Goyal et al., 2017] and OKVQA [Marino et al., 2019] for the general VQA task, TextVQA [Singh et al., 2019] and OCRVQA [Mishra et al., 2019] for the OCR VQA task, and ChartQA [Masry et al., 2022], DocVQA [Mathew et al., 2021], and WTQ [Berant et al., 2019] for the document or chart VQA task, respectively. We employ the following prompt template for all evaluations on these datasets: "<Img>{latent embedding}</Img>{structural knowledge}{question}. Answer the question using a single word or phrase." In addition, as the object detection results of the chart and document images are usually useless, we design an automatic filtering mechanism to filter out the detection results of these images.

As presented in Tab. 1, our proposed method shows competitive performances when compared to recent approaches. Specifically, IVE achieves an accuracy of 60.3% on OKVQA [Marino et al., 2019], which significantly surpasses the performance of recent state-of-the-art method (mPLUG-Owl2 [Ye et al., 2023b], achieved with 57.7%). In TextVQA [Singh et al., 2019] and OCRVQA [Mishra et al., 2019] datasets, IVE achieves accuracies of 62.0% and 71.1%, outperforming Qwen-VL-Chat [Bai et al., 2023] with 0.5% and 0.6%, respectively. As for the DocVQA [Mathew et al., 2021] and WTQ [Berant et al., 2019], IVE still achieves consistent improvements compared with recent approaches. More visualized examples have been shown in Appendix D of the supplemental materials.

Additionally, to demonstrate the robustness of our method, more validation about other widely-used VQA benchmarks with unseen images are shown in Appendix E of the supplemental materials.

4.4 Fine-tuning Performances on VQADatasets

To compare our approach with different specific VQA methods, we assess the performance of IVE further fine-tuning on the VQAv2 [Goyal et al., 2017], OKVQA [Marino et al., 2019], OCRVQA [Mishra et al., 2019], and ChartQA [Masry et al., 2022]. We still employ the prompt template: "<Img>{latent embedding}</Img>{structural knowledge}{question}. Answer the question using a single word or phrase." during evaluation. The further fine-tuning results of IVE on these VQA datasets are shown in Tab. 2.

The experimental results demonstrate that IVE, following additional fine-tuning on specific datasets, achieves favorable improvements. Specifically, there are 5.2% and 4.9% improvements compared with the direct-transfer results on VQAv2 [Goyal et al., 2017] and OKVQA [Marino et al., 2019], respectively. Notably, in the tasks related to character and chart, IVE significantly outperforms the Pix2StructLarge [Lee et al., 2023] in OCRVQA [Mishra et al., 2019] and ChartVQA [Masry et al., 2022], with 3.6% and 9.7% improvements, respectively. Additionally, when compared to the recent state-of-the-art (CogVLM [Wang et al., 2023]), IVE still shows competitive results.

Given that the MME Benchmark [Fu et al., 2023] focuses on answering "yes/no" formats, we conduct further fine-tuning of our multi-task tuning model using a mixed dataset composed of VQAv2 [Goyal et al., 2017] and LRVInstruction [Liu et al., 2023a]. Subsequently, we evaluate the model on the MME Benchmark. As demonstrated in Tab. 3, our method achieves the scores of 1455.6 and 384.1 in the perception and cognition task of MME Benchmark [Fu et al., 2023], respectively. Compared with recent state-of-the-arts (mPLUG-Owl2 [Ye et al., 2023b] and LLaVA-1.5 [Liu et al.,

2023b]), our IVE demonstrates superior stability.

4.5 Ablation Study

To better evaluate the effectiveness of each proposed module, we further conduct ablation studies with utilizing different combinations of proposed modules. All the ablations are conducted with training on multi-task tuning datasets and performing direct-transfer evaluations on different VQA test sets.

Effectiveness of Multi-task Encoders. To evaluate the individual contributions of each encoder within our multi-task encoders, three distinct experiments have been conducted. The initial experiment exclusively employs the semantic information encoder. Subsequently, the low-level information encoder and document-related information encoder are utilized progressively. As shown in Tab. 4, combing the semantic information encoder and the low-level information encoder leads to improvements across various datasets compared to only using the semantic information encoder. Further fusion with the document-related information encoder results in a significant improvement on document-related analysis tasks, with its performance on ChartVQA [Masry et al., 2022] rising from 59.1% to 61.8% and DocVQA [Mathew et al., 2021] rising from 60.8% to 62.3%, respectively. More visualized analysis have been present in Appendix D of the supplemental materials.

Effectiveness of Structural Knowledge Enhancement. To validate the effect of structural knowledge enhancement and compare the different impacts of integrating structural knowledge only in the inference phase or in both the training and inference phases, we further conduct two additional experiments built upon the multi-task encoders.

As shown in Tab. 4, while evaluating VQAv2 [Goyal et al., 2017] and OKVQA [Marino et al., 2019], the performances will decrease with incorporating structural knowledge solely during the inference phase. Conversely, integrating this structural knowledge during both the training and inference phases yields improved results across a spectrum of datasets. The above phenomenon demonstrates that the structural knowledge introduces inherent noises, negatively impacting the capability of MLLMs while it is directly utilized. However, when introducing these extracted knowledge during the training phase, the LLM is guided to autonomously discern pertinent information, thereby mitigating the adverse effects of noise. More visualized analysis and effectiveness validation about structural knowledge enhancement module are shown in Appendix D and Appendix F of the supplemental materials, respectively.

Moreover, to demonstrate that integrating structural knowledge during both training and inference phases can mitigate the disturbation of noises in knowledge rather than simply aligning prompt formats, we conduct additional experiments with fine-tuning on the sampled VQAv2 [Goyal et al., 2017] dataset. Specifically, we replace the automatically detected results with the ground truth as finally utilized structural knowledge. Then, we conduct the comparisons with integrating structural knowledge only in the inference phase or in both training and inference phases. As shown in Tab. 5, utilizing the ground truth as structural knowledge and integrating it during both training and inference phases only achieves slight gains (0.5%) compared to the mechanism of integrating ground truth during the inference phase (1.6% gains). This observation demonstrates that our proposed method goes beyond simple prompt format alignment. Instead, it focuses on autonomously discerning and extracting pertinent information, thereby mitigating the adverse effects of noise.

5 Conclusion

This paper firstly reevaluates the existing limitations within current multimodal large language models(MLLMs), and points out that they always grapple with the information loss dilemma. To enhance the corresponding visual perception ability of MLLMs, this paper presents Incorporating Visual Experts(IVE), the first work to aggregate available visual information through a mixture-of-experts mechanism in both training and inference stages. Extensive experiments on a wide range of multimodal dialogue datasets have evaluated the effectiveness of IVE. Though the significant improvements achieved by IVE, the types of visual experts utilized in current pipeline are still limited. In addition, compared to the methods which use only a semantic encoder branch, IVE incurs an additional 979M parameters, which is far less than the total parameters of MLLMs. In the future, we aim to explore more efficient multi-encoder fusion strategies and develop a more effective information compression projector.

Contribution Statement

Xin He and Longhui Wei are co-first authors.