CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

1 Introduction

Currently, multi-modal large language models (MLLM) like GPT-4[OpenAI, 2023], Gemini[Team et al., 2023], LLaVA[Liu et al., 2023a] and Qwen-VL [Bai et al., 2023] have showed powerful abilities in this field of multi-model. At the same time, the ability to evaluate MLLMs more rationally and comprehensively is receiving increasing attention. Researchers have made many efforts to address this problem. Datasets like MMBench [Fu et al., 2023], MME [Fu et al., 2023], and SEED-Bench [Li et al., 2023b; Li et al., 2023a] evaluate models through a diverse range of questions, ranging from perception to reasoning abilities. However, these datasets primarily access common-scene knowledge more than domain-specific knowledge. The recently introduced GAIA benchmark [Mialon et al., 2023] focuses on testing fundamental abilities like reasoning, multi-modal processing, and general tool use. However, GAIA also presents certain limitations. It primarily tests tasks that are conceptually simple for humans, which may not fully capture the complex problem-solving capabilities required in some specialized domains.

In addition to the above benchmarks, alternative evaluation datasets containing questions from textbooks and other educational materials are proposed to evaluate domain-specific knowledge, which are inspired by human exams for measuring knowledge levels and selecting talents. For instance, ScienceQA[Lu et al., 2022] is a dataset that evaluates the scientific knowledge of models, while MMMU[Yue et al., 2023] assesses university-level knowledge. These two datasets only contain English questions, while some datasets, such as M3Exam [Zhang et al., 2023], turn attention to the multilingual setting. However, the above benchmarks mainly focus on multiple-choice questions, which limits the comprehensiveness of evaluation. Multiple-choice questions cannot evaluate the text generation abilities of the models, as the models only need to choose the correct answer from a few existing options. Meanwhile, the models may obtain correct answers through guessing, which could impact the accuracy of the evaluation. Therefore, there is a need for a diversified and comprehensive benchmark to evaluate the understanding and reasoning abilities of MLLMs.

To bridge the dataset gap, we introduce a novel benchmark, CMMU, for multi-modal and muli-type question understanding and reasoning in Chinese. CMMU encompasses multimodal content across 7 subjects. Every question requires the model to combine image and text content to generate a comprehensive response. While CMMU shares similarities with datasets like ScienceQA and M3Exam [Zhang et al., 2023], it offers a broader range of question types. Previous datasets only have multiple-choice questions, while CMMU offers a wider variety of question types, including multiple-choice, multiple-response, and fill-in-the-blank questions, as shown in Table 1, which poses a more significant challenge to the comprehension abilities of MLLM. In addition, to mitigate the position bias[Zheng et al., 2023] in LLM and ensure genuine correctness rather than guessing, inspired by CircularEval [Liu et al., 2023c], we adopt a Positional Error Variance approach to measure the position bias for multiple-choice question. Specifically, we cycle through the position of options to ensure that the answer can appear at any position with equal probability, which is the same as CircularEval, aim at reducing position bias, minimizing the influence of randomness on correctness. Then we produce Positional Error Variance, a quantitative analysis to measure position bias. We evaluate 11 models using the CMMU benchmark, and the results indicate that CMMU presents a significant challenge to current MLLMs.

To sum up, our contributions are as follows:

• We present a novel benchmark of multi-modal and multitype questions in Chinese, featuring a wider variety of question types, including multiple-choice, multipleresponse, and fill-in-the-blank questions.
• We evaluate 10 models and analyze their performances in Chinese language proficiency and multi-modal comprehension.
• We propose Positional Error Variance, which is designed to conduct a quantitative analysis of position bias in MLLMs.

2 RelatedWork

2.1 Multi-modal Benchmarks

With the development of large language models (LLMs), there is a growing emphasis in research communities on assessing the capabilities of LLM such as HELM[Liang et al., 2022], CLEVA [Li et al., 2023d], especially their multimodal understanding abilities. Datasets like VQAv2 [Goyal et al., 2017], TDIUC [Kafle and Kanan, 2017], TextVQA [Singh et al., 2019] and GQA [Hudson and Manning, 2019] are used in visual question answering tasks, while COCO [Lin et al., 2014], NoCaps [Agrawal et al., 2019], and Flickr30K [Plummer et al., 2015] are employed in image captioning tasks. Additionally, Visual7w [Zhu et al., 2016] and RefCOCO [Kazemzadeh et al., 2014] are commonly utilized for visual grounding purposes. With the rapid development of multi-modal large language models, researchers have achieved good results on these datasets. We require more extensive data to evaluate MLLMs, and there have been recent studies evaluating models from various perspectives. LVLM-eHub [Xu et al., 2023] collects 47 existing benchmarks and evaluates 6 types of capabilities of MLLMs, however, it does not create any new benchmarks. MME [Fu et al., 2023] comprehensively measures the perception and cognition abilities of models. However, its question types are simplistic, merely requiring yes or no responses. MMBench [Liu et al., 2023c] and SEED-Bench [Li et al., 2023b; Li et al., 2023a] contain many multiple-choice questions covering various ability dimensions, but these datasets mainly consist of common-sense questions and do not require lots of domain-specific knowledge and complex reasoning. To enhance the evaluation of domain-specific knowledge, ScienceQA [Lu et al., 2022] was introduced. This dataset encompasses a wide range of science topics from elementary and high school curricula. MMMU [Yue et al., 2023] is designed to evaluate college-level subject knowledge, questions of CMMU are collected from college exams and textbooks, and many of them require expert-level skills. BenchLMM [Cai et al., 2023] assesses MLMMs in three distinct styles: artistic style, sensor style, and application style. M3Exam [Zhang et al., 2023] is a multilingual and multi-modal benchmark designed to evaluate domain knowledge and problem-solving skills, and it spans seven languages. However, less than one-third of the questions include images.

2.2 Multi-modal Large Language Models

Benefiting from the development of large language models(LLM) like GPT [Macfarlane, 2023], LLaMA [Touvron et al., 2023] and Vicuna [Chiang et al., 2023], MLLMs have made significant improvements. Many works integrate additional modal inputs on LLM and bridge the modality gap between vision and language, and the majority of MLLMs possess multilingual capabilities. BLIP-2 [Li et al., 2023c] propose Q-Former to align image representation and text representation, InstructBLIP [Dai et al., 2024] based on BLIP2 and propose an instruction tuning framework to improve the instruction following capability. CogVLM [Wang et al., 2023] propose a visual expert module to enable deep alignment of the vision-language features. LLaVA [Liu et al., 2023b; Liu et al., 2023a], Emu2 [Sun et al., 2023], and MiniGPT4 [Zhu et al., 2023] adopt a simple but effective projection scheme to connect image feature into the language space. A modality-adaptive module is introduced by mPLUG-Owl2 [Ye et al., 2023], aiming to enhance modality collaboration by projecting visual and linguistic features into a shared space. In this paper, we will provide a comprehensive evaluation of some of these models using the CMMU benchmark and assess their abilities in domain-specific knowledge.

3 The CMMU Benchmark

CMMU is a novel multi-modal benchmark designed to evaluate domain-specific knowledge across seven foundational subjects: math, biology, physics, chemistry, geography, politics, and history. We collect questions encompassing both text and images sourced from diverse exams. CMMU covers questions from primary school to high school, providing a

✩

✪

comprehensive evaluation of the abilities of models across various grades.

Previous benchmarks, such as ScienceQA and MMMU, only have multiple-choice questions. In contrast, our CMMU benchmark contains 3 types of questions:

• Multiple-choice question (MCQ): Each question presents 3 or 4 options, with only one correct answer.
• Multiple-response question (MRQ): Each question includes 4 options, and the number of correct answers can range from 1 to 4.
• Fill-in-the-blank question (FBQ): The question is to fill in the blanks with the correct answers to complete the sentence or passage.

In addition to providing the correct answer, CMMU also provides the explanations of the answers about MCQ and MRQ.

3.1 Data Pre-process

Data Collection and Processing. We extract text and images from the original PDF and convert them into JSON format. In addition, we transform all formulas, including mathematical and chemical ones, into LaTeX format. For fill-in-theblank questions, if there are many sub-questions within one question, we will split them into a sub-question list, attempting to have only one blank to fill in each sub-question, except for some indivisible questions. In the end, we decompose 639 fill-in-the-blank questions into 1,632 sub-questions, with 83% of them requiring only one blank to be filled.

Data Cleaning. We manually review the questions, filtering out images that are blurry, low-quality, or have a resolution less than 50×50 dpi, eliminating questions that are incorrectly parsed, and correcting mistakes made during the automatic conversion to LaTeX. Furthermore, experienced teachers consider the depth of knowledge and the complexity of questionsolving methods to categorize each question into two levels: normal and hard based on their experience and subjective judgment.

3.2 Data Distribution

CMMU has a total of 3603 questions, divided into validation set and test set, with 1,800 and 1,803 questions respectively. The validation set is open source to the community. As shown in Figure 1, the benchmark contains multi-modal content from middle and high school across 7 types of subjects, while primary school only contains math. The ratio of normal and hard questions are 8:2. Over 70% of the questions have detailed answer explanations, with an average length of 311 characters for each analysis. The detailed statistics are shown in Table 2. We translate the original questions into English and analyze the distribution of their first three words. As shown in Figure 2, the questions have diverse formats and are relevant to images or diagrams.

3.3 Comparisons with Existing Datasets

Table 3 shows the comparisons with existing datasets. We compare the benchmarks from five dimensions: quantity, with or without images, with or without explanation, question type and language. It can be seen that CMMU is the first benchmark for multi-modal and muli-type question understanding and reasoning in Chinese.

4 Evaluation

The evaluation of multiple-choice questions confronts two challenges: First, considering the particularity of the formats of multiple-choice questions, when the model correctly answers a question there is an uncertainty about whether the model has truly mastered the relevant knowledge or it just guesses the correct answer. When a model chooses answers through guessing, there will be positional bias, which means a LLMwill prefer the answer in a certain position. Position bias is an issue that appears in many LLMs and MLLMs, however, existing methods have not quantitatively measured the extent of the position bias.

To address the above problems, we propose Positional Error Variance. Firstly, following the CircularEval [Liu et al., 2023c], we cyclically change the positions of the options and let the model answer questions. Subsequently, we calculate metrics to quantify position bias. We will describe the whole process in detail.

4.1 CircularEval

For a multiple-choice question with k options, we perform a right circular shift on the options. For example, if the original order of the options is ABCD, then after one shift, the order will change to DABC. A detailed example is provided in Figure 3. Given a question Q with k possible options, we generate k distinct shifted-option questions, denoted as Qi, i ∈ [0, k]. Each Qi is then evaluated by the MLLMs to generate the corresponding answers Ai. We consider the model to have sufficient knowledge to answer the question Q only if all of Ai are correct, in which case the accuracy score of Q is 1, otherwise it is 0.

4.2 Positional Error Variance

Conceptually, an unbiased model assigns equal probability to each option. Under the shifted-option setting, if the probability of each option is not equal, it indicates that the model has a bias towards a certain option. Considering this, we define the BiasRate as follows:

Questions that are completely answered correctly in CircularEval do not reflect position bias, so we just focus on the incorrectly answered questions. If there are M incorrectly answered questions with n options for each, there will be a total of m ∗ n answers combination. We count the occurrences So of each option o and then calculate the probability Po = So m∗n , o ∈ {A, B, C, ...}. And then we define the BiasRate as the variance of P , the formula is BiasRate = σ2 (P). The larger the BiasRate, the greater the positional bias of the model.

4.3 Evaluations on Different Question Types

To avoid the impact of the analysis process of model outputs on the evaluation, we filter the answers by retaining only the last line of the answer. For multiple-choice and multiple-response questions, we extract option letters from the responses of the models. After that, we apply different strategies to evaluate the three types of questions.

Evaluation on Multiple-choice Question. We adopt CircularEval and Positional Error Variance in section 4, which allows us to analyze both the accuracy and the BiasRate.

Evaluation onMultiple-response Question. This question type may have more than one correct option. We consider the correctness only when all the chosen options are correct, excluding any incorrect choices.

Evaluation on Fill-in-the-blank Question. The answers to fill-in-the-blank questions may not be unique and responses with similar meanings to the groundtruth can also be considered correct. Hence, we utilize GPT-4 to judge the answer, providing a binary score of 0 or 1 to determine correctness. Further details about the evaluation prompts are in Table 5.

5 Experiments

5.1 Models

We evaluate the performance of various MLLMs, including both closed-source and open-source models. The closedsource models are evaluated by using their official API, while open-source models are evaluated by running inferences on NVIDIA A100 GPUs. For the closed-source models, we select state-of-the-art models like GPT-4V, Gemini-Pro. We also choose Qwen-VL-Plus, which performs well on Chinese datasets. For the open-source models, model sizes vary from 7b to 13b, including LLava-1.5-13b, CogVLM-7b, InstructBLIP-13b, Qwen-VL-Chat-7b, Intern-XComposer-7b, mPLUG-Owl2-7b and ShareGPT4V-7b.

5.2 Prompts and Settings

All models are tested in zero-shot settings as we only specify the output format in prompts. Each type of question has its own prompt template, and we utilize the same prompt template for all models. The prompt 1 of MCQ is "Answer with the option's letter from the given choices directly", the prompt of MRQ is "Please directly provide the letters of the correct options. There may be more than one correct option." the prompt of FBQ is "Complete each blank with a single word or phrase. If there is more than one blank, split answers with a semicolon (;)".

Parameters are configured with distinct settings for each question type. For MCQ and MRQ, the temperature is set to 0, and the max new token is set to 10. For FBQ, the temperature is set to 0.2, and the max new token is set to 128.

5.3 Results

Results on Different Subjects and Grades

The overall results are shown in Table 4. GPT-4V achieves an accuracy of 30.19% and 30.91% on the validation and test sets, respectively, reaching the highest level among all models. Moreover, all three closed-source models perform better than all open-source models. From the perspective of question types, most models show lower accuracy in FBQ and MRQ compared to MCQ. Specifically, 7 models achieve an accuracy of less than 10% on the Val-hard set of MRQ, and 5 models achieve an accuracy of less than 10% on the Val-hard set of FBQ, highlighting the difficulty and challenge of FBQ and MRQ.

Results on Different Question Types

Figure 4 shows the overall results of different models in subjects and grades, and the detailed statistics are shown in Table 6. We can see that there is a significant subject bias in all models. Subjects such as politics and history rely on the knowledge reservoir have higher accuracy than subjects such as physics, math which require computation and reasoning. When comparing different grades, the results show that MLLMs generally perform better on middle school questions than high school ones, suggesting that more complex knowledge presents a more significant challenge for these models.

1 In the experiments, we use the Chinese version and we translate it into English for reading.

Position Bias Analysis

We employ the Positional Error Variance for quantitative analysis of position bias. As shown in Figure 5, most models have a positional preference for one or two specific options. An interesting finding is that, although these positional preferences are inconsistent across models, none of them choose Option D as their most preferred choice. By analyzing the BiasRate, we find that superior models, such as GPT-4V, tend to have a relatively lower BiasRate.

Case Study with CoT Prompts

To further analyze the performance of models using Chain of Thought(CoT), we change the prompt of MCQ to "Please analyze the question step by step and eventually provide a single correct option letter. (This is a multiple-choice question.)" Then, we choose GPT-4V, which has a strong ability in instruction-following, to answer 500 randomly selected MCQs. We identify three common mistake types in the model outputs: image misunderstanding, misleading reasoning, and question misunderstanding, with proportions of 27.48%, 35.41%, and 13.03%, respectively. Cases in Table 7 show the above common mistakes respectively: In Question 1, the model fails to identify the route A in the image correctly. In Question 2, the model thinks that the kinetic energy during free fall is transformed into gravitational potential energy, leading to an incorrect reasoning result. In Question 3, the model misunderstands the question and produces a hallucination that the overlap area is 1. Although it guesses the answer correctly, it cannot pass the CircularEval. All bad cases demonstrate that even one of the most advanced MLLMs cannot perfectly solve questions of CMMU, highlighting both the potential and challenges of the benchmark.

6 Conclusion and FutureWork

In conclusion, our work introduces a novel benchmark named CMMU to evaluate the multi-modal and multi-type question understanding and reasoning abilities of MLLMs in Chinese. Unlike existing benchmarks focusing on multiple-choice questions, CMMU offers a more comprehensive evaluation by incorporating a broader question type, including MCQ, MRQ, and FBQ. We also propose Positional Error Variance to quantify the position bias of MLLMs. The evaluation results contribute to a deeper understanding of current MLLMs in the context of diverse and complex question formats. In future work, we will consider enriching the problem types and increasing the quantity to increase the challenge of the benchmark further.

Acknowledgments

This work is supported by National Key R&D Program of China (2022ZD0116306). Thanks to Duo Zheng and Ning Su for their valuable suggestions and contributions to this paper.

Contribution Statement

Zheqi He and Xinya Wu contributed equally to this work. This research was conducted during Pengfei Zhou's internship at the Beijing Academy of Artificial Intelligence. Guang Liu and Xi Yang serve as corresponding authors for this study.