M4Bench: A Benchmark of Multi-domain Multi-granularity Multi-image Understanding for Multi-modal Large Language Models

1 Introduction

Multi-modal large language models (MLLMs) [Liu et al., 2024a] have demonstrated outstanding performance in various vision-language tasks. Early attempts mainly focus on bridging the gap between text and image modalities in single-image scenarios, ranging from image captioning [Yuan et al., 2024] and visual grounding [Xuan et al., 2024] to VQA [Schwenk et al., 2022]. Recently, increasing demands in analyzing complex associated scenes pose necessities to researching multi-image understanding abilities. For example, in automatic driving, models use multiple images to compare environmental changes [Naranjo et al., 2005]. In video monitoring, models compare multiple temporal images to analyze human behavior [Ye et al., 2024; Ye et al., 2025].

∗Corresponding author.

Compared with understanding individual images, the key to understanding multi-image lies in both the alignment and distinction between images [Wang et al., 2024a]. However, existing benchmarks [Liu et al., 2024c; Liu et al., 2024b; Wu et al., 2025] face difficulties in addressing both of these aspects simultaneously. On the one hand, some efforts [Wu et al., 2025; Fu et al., 2023; Wu et al., 2024] have been directed towards enhancing the ability to identify differences between images, though the majority focus on a coarsegrained level. In practice, fine-grained differences, such as continuous action understanding [Lu et al., 2024], are very common, which presents significant challenges for deploying existing MLLMs in such a scenario. On the other hand, some efforts [Fu et al., 2025] have been made to enhance the ability to align parts between images. This typically requires a fine-grained alignment, yet the images in current benchmarks tend to originate from the same domain [Liu et al., 2024b; Kil et al., 2024]. In addtion, the input given to the model often consists of multiple images across different scenarios. For example, when comparing different design products, the model is required to overcome scenario difference and compare the detailed differences between products. Consequently, the challenge of simultaneously modeling relationships and differences across various granularities and domains has become critical, but remains largely underexplored.

To bridge this research gap, in this paper, we introduce M 4Bench, a benchmark dedicated to simultaneously evaluating the capability of aligning and distinguishing multi-images with multi-domain multi-granularity comparison for multimodal language models. Humans naturally excel at synthesizing information from multiple image sources [Wang et al., 2024a]. Following human-like thinking mode, M 4Bench is structured along three dimensions, encompassing a total of five multi-image subtasks. Our benchmark concatenates two images horizontally as the visual input, employing multigranularity visual prompts as contextual cues and response options to minimize the model's dependence on linguistic prior knowledge. In comparison tasks, humans typically first identify the main objects in each image, which tests Multidomain Coarse-grained Comparison ability. Once these high-level objects are determined, they employ a top-down processing approach [Noudoost et al., 2010] to guide their attention towards more detailed features, which assesses multidomain Fine-grained Comparison ability. Single-domain Fine-grained Comparison ability is a prerequisite for this assessment. Correspondingly, as illustrated in Figure 1, State Invariance (SI) first examines the models' ability to correctly recognize objects in different states. After correct recognition, State Comparison (SC) requires the model to distinguish between different physical states of the same category object. Detailed Difference (DD) evaluates the models' fine-grained perception ability. Finally, multi-domain tasks Spatial Perception (SP) and Instance Comparison (IC) test whether models can overcome scenario differences and focus on key finegrained features. With the above design, M 4Bench evaluates the comparison capabilities of models at different stages.

We evaluated 13 MLLMs with various sizes (i.e., 2B, 4B, 7B) on M 4Bench. Our results reveal that existing MLLMs struggle in fine-grained comparison tasks, especially in multidomain scenario. Besides, we also designed the caption test experiment to prove that our M 4Bench previously overcomes the conflation of multi-image comparison with language reasoning ability in previous benchmarks. We perform further analysis of error cases, providing insight for future MLLM improvements. Our contributions are summarized as follows:

• We present M 4Bench for evaluating the multigranularity, multi-domain comparison capability of MLLMs. Following human thinking mode, M 4Bench provides multi-grained visual prompts as contextual cues and response options, which can reliably evaluates the comparison capabilities of models at different stages.
• We designed four automatic pipelines and employed manual annotation to construct M 4Bench, which addresses a gap in the community for multi-domain finegrained comparison datasets.
• The evaluation on M 4Bench reveals that current MLLMs perform poorly in multi-granularity, multi-

domain comparison tasks. We also provide further analyses and insights for future improvement.

2 RelatedWork

2.1 Multi-Image mid-scale Multimodal Models

Multimodal Large Language Models (MLLMs) refer to models capable of processing multiple forms of input (e.g., text, images, and videos) and performing joint reasoning [Yin et al., 2023]. Most open-source autoregressive vision-language models [Liu et al., 2023; Zhang et al., 2023; Li et al., 2023c] are primarily focused on processing a single image, and exhibit limitations when handling more complex tasks that require comparison across multiple images. As research progresses, an increasing number of models [Awadalla et al., 2023; Li et al., 2023a; Sun et al., 2024] have begun to support multi-image input. OpenFlamingo [Awadalla et al., 2023], for instance, leverages the Perceiver Resampler and cross-attention layers to achieve in-context learning capabilities. However, in low-data scenarios, this approach struggles with performance efficiency and computational cost. InternVL 1.5 [Chen et al., 2024] and Qwen2VL [Wang et al., 2024b] introduce dynamic resolution support, which dynamically transforms multi-image input into a variable number of visual tokens, optimizing MLLM computational efficiency in multi-image scenarios. Qwen2-VL also employs a Multimodal Rotary Positional Embedding (M-RoPE) technique to enhance cross-image contextual reasoning. MiniCPM [Hu et al., 2024] further improves efficiency by optimizing the density of visual tokens, allowing the model to handle multiple images without sacrificing performance. The LLaVA-NEXT series [Li et al., 2024b; Li et al., 2024a], while inheriting LLaVA's minimalist design, enhances multi-image task processing through a highquality multi-image instruction dataset. Despite these advancements, current multi-image models still exhibit shortcomings in multi-domain comparison tasks.

2.2 Multi-Image Multimodal Benchmarks

Most benchmarks [Liu et al., 2025; Yu et al., 2023] primarily focus on single-image evaluation, and often overlook multiimage perception and reasoning abilities, which hold even greater practical value. Some recent studies exploring multiimage scenarios, but they mainly focus on coarse-grained multi-image perception and lack object-centric comparison. Q-Bench [Wu et al., 2023] is proposed to evaluate the lowlevel perception of MLLMs by comparing multiple image inputs. Memontos [Wang et al., 2024c] evaluates the temporal understanding of the image sequences. The DEMON benchmark [Li et al., 2023b] also present several subset of questions requiring multi-image reasoning, where the main focus is on evaluating the demonstrative instruction following abilities of MLLMs. MIBench [Liu et al., 2024b] and CompBench [Kil et al., 2024] has expanded the evaluation dimensions and sample numbers for multi-modal tasks, but each subtask is limited to either coarse-grained or fine-grained capabilities separately. BLINK [Fu et al., 2025] evaluates the core visual perception ability based on classic computer vision tasks, but by directly drawing the options on the images, it indirectly compromises the reliability of evaluating models' multi-image comparison ability. Additionally, BLINK's images are entirely sourced from existing datasets, failing to address the community's lack of object-centric comparison datasets. In this paper, we propose a brand-new "visualcentric" multi-image comparison benchmark that provides visual object-level and part-level visual prompts, evaluating multi-granularity perception ability.

3 M 4Bench

3.1 Overview of M 4Bench

The M 4Bench includes three dimension of single and multi domain scenarios along with five vision comparison tasks as illustrated in Figure 2:

• Multi-domain Coarse-grained scenario aims to recognize objects in different structural forms and distinguish the physical states of objects.
• Single-domain Fine-grained scenario aims to identify fine-grained difference by detecting subtle changes between images.
• Multi-domain Fine-grained scenario aims to compare fine-grained features of the interest object under diverse environments.

When given two images, coarse-grained scenarios require distinguishing features at the image or object level, while fine-grained scenarios focus on the comparison of subtle local parts of objects. Furthermore, multi-domain scenarios have diverse prior environments of two images, such as image attribute settings, background complexity, arrangement of objects, lighting, and shadows, which are challenges in distinguishing the invariance of objects in two images.

The subtasks consist of intra-class image pairs and multigrained visual prompts collected via four automatic pipelines and manual annotation. Table 1 shows detailed settings.

In addition, we compare our M 4Bench with previous benchmarks from four aspects: with or without visual prompts, visual features granularity, scenario, and benchmark construction methodology. Table 2 shows the details.

3.2 Dataset Collection Process

Multi-domain Coarse-grained

State Invariance (SI). The ability to correctly detect objects in different states is a prerequisite for the multi-image comparison task. However, when the state of an object changes, the object embeddings generated by current MLLMs also change. This task helps us evaluate the robustness of the model in generating discriminative features at the object level. We design the task with each sample containing two images and a yes-no multiple-choice question to judge the difference. We collect positive samples from the ObjectsWithStateChange dataset [Sarkar and Kak, 2024] to test whether the model can correctly classify objects with different structural forms as the same object. Negative samples are constructed from the Grocery Store dataset [Klasson et al., 2019] to test the model's ability to distinguish between similar but different objects.

State Comparison (SC). Understanding object states is essential for common sense physical reasoning. We design a subtask to evaluate MLLM's ability to encode the objects' physical states, focusing on the comparison of diverse states of objects. Our data is sourced from the ChangeIt dataset [Souˇcek et al., 2022], which features videos of various objects undergoing state changes in different scenarios, along with annotations for actions and state localization. Our pipeline automatically capture the initial and end state keyframes from each video, and then pair these state keyframes of objects in the same category into image pairs with corresponding state pairs as answers.

Single-domain Fine-grained

Detailed Difference (DD). Given that objects can be futher subdivided into diverse parts, reasoning at a coarse-grained level is insufficient for subtle comparison tasks, and partlevel features should be considered to support the detailed difference. Therefore, we consider datasets that provide image pairs with similar layouts but subtle changes. To generalize diverse vision scenarios, we provide both synthetic (DD-SI) and natural (DD-NI) multi-image pairs from MagicBrush [Zhang et al., 2024] and Spot-the-diff [Jhamtani and Berg-Kirkpatrick, 2018] respectively. The former consists of triplets of source image, mask images and target image, while the latter provides image pairs in several outdoor scenes.

We also provide an automatic pipeline for constructing the detailed difference task with multiple-choice questions.

Cross-domain Coarse-grained

Single-domain Fine-grained

Cross-domain Fine-grained

Figure 2: Overview of the M 4Bench benchmark. M 4Bench offers diverse triplets comprising two images from single and multi scenario, comparision question, and and answer based multi-grained key visual feature.

Our options consist of three parts: (1) Generating bounding boxes of the difference region. In synthetic images, the bounding boxes are derived from the masked area provided by mask image. In natural images, they are calculated by the surrounding bounding box of the difference cluster. (2) Designing distractors. We design an automatic pipeline to randomly generate non-overlapping bounding boxes in nondifferent areas of the image. (3) Collecting image pairs with the same images. We collect the same images and label the answer with "no difference" as negative samples.

Multi-domain Fine-grained

Spatial Perception (SP). Other than capturing coarsegrained global features and fine-grained local features, MLLMs should establish connections between them to learn the hierarchical layout correspondences in the image pairs to gain spatial perception. Therefore, we derive our samples from the PACO-LVIS dataset [Ramanathan et al., 2023], which provides object and part maskon images in the COCO dataset [Lin et al., 2014]. We select the images in which objects contain relative spatial parts for the spatial perception task, such as symmetrical components like a mouse with left and right buttons and subassembly parts like bottles with necks, shoulders and bottoms. We retrieve from the PACOLVIS based on the object category and filter out image pairs containing four or more partial components with multiple choice questions designed. In the option design, we provide bounding boxes of the potential interest regions, with the correct answer setting the corresponding part and the other options randomly matching unaligned parts as distractors.

Instance Comparison (IC). We hope that the models will overcome the semantic ambiguity caused by changing vision scenarios when comparing multiple images. Thus, they are suggested to not only focus on the corresponding parts of the interest instances (in SP) but also distinguish the part-level differences. Therefore, we propose a challenging VQA task of Instance Comparison to address the lack of comparable image pairs of different designs for the same instances. Furthermore, we annotate a test set that contains images of the same product in different designs across diverse scenarios, along with multi-grained visual prompts, on the AIData platform.

Instance Comparison in deed is a two-stage task: objectlevel positioning and part-level comparision, so we provide two settings: with or without visual prompts. Visual prompts tell the models the bounding boxes on the interest objects, which facilitates the completion of the initial localization phase, subsequently allowing for the assessment of models' comparative capabilities in the second phase.

3.3 Avoiding Data Flaws

We construct test data of M 4Bench by utilizing the validation or test sets from existing datasets. Furthermore, we combine automated filtering and manual verification to ensure the quality and reliability of the test data. We discard samples that can still be answered correctly without visual input. This avoids the overestimation of model performance due to the textual bias of the questions and options. To improve the quality of the generated samples, we apply manual verification after automated filtering. The process is conducted by three human annotators recruited within the team. They delete inferior image pairs such as the images are blurry and the main subject is relatively small.

4 Experiments

4.1 Models

We evaluate 13 MLLMs which can be categorized into two groups: (1) closed-source API models, including GPT-4o, Gemini-Pro and Qwen-VL-Max; (2) open-source models, including DeepSeek-VL2 (tiny and small), Qwen2VL (2B and 7B), MiniCPM-V-2.6 (8B), InternVL (v2, v2.5, model size 4B and 8B) and LLaVA-OneVision (7B). All these models inherently support multiple image inputs.

4.2 Evaluation Setup

To ensure reproducibility, we follow the approach used in VLMEvalKit [Contributors, 2023], extracting answers from the free-form outputs of MLLMs. Specifically, for multiplechoice questions, we obtain the options through predefined rules and the assistance of GPT-3.5-turbo, and finally report the results using accuracy (ACC). In addition, to solve the position bias [Liu et al., 2025], we shuffle the options randomly to balance the distribution of correct answers among A, B, C, and D to mitigate the position bias.

We conducted Multi-domain Coarse-grained testing on all models. However, it is noteworthy that some of the models were not pre-trained on input data in the visual prompts format. They exhibited a negative effect in the Single-domain Fine-grained and Multi-domain Fine-grained tests, i.e., their performance decreased as the number of demos increased. Therefore, we only present the evaluation results of models that support visual prompt input in these two dimension tests.

4.3 Main Results

Overall performance. Table 3 shows the evaluation results on our M 4Bench, where "Random Choice" implies that the correct answer probability is 50% for yes-no questions and 25% for four-choice questions. From the table, the average accuracy across all tasks for MLLMs is 39.6%, which is 8.3% higher than random choice (31.3%). Even the top-performing models in our benchmark, the open-source DeepSeek-VL2-small and the closed-source Qwen-VL-Max, achieved only 51.3% and 46.3% accuracy, respectively. In general, the multi-domain coarse-grained tasks (68.3%) are significantly simpler compared to single-domain fine-grained tasks (31.9%) and multi-domain fine-grained tasks (29.0%), with the latter being particularly more challenging.

Performance limitations of models on M 4Bench. Figure 3 shows that MLLMs generally perform better on multidomain coarse-grained tasks (SI and SC) in the upper right half, while struggling with multi-domain fine-grained tasks (IC and SP) in the left half. This performance disparity reveals the varying levels of challenge in multi-image understanding tasks. Although Qwen-VL-Max achieves the best performance on the DD-SI task, surpassing DeepSeekVL2-small and demonstrating the potential of closed-source MLLMs on synthetic images, there are still notable limitations in current MLLMs. These limitations can be attributed to two main factors. First, current MLLMs primarily adopt data-driven optimization approaches that process multiple images independently and concatenate them at the token level, potentially constraining their ability to perform multidomain and multi-granularity visual comparisons, which are crucial for M 4Bench tasks. Second, most MLLMs rely on LLMs for reasoning, projecting visual information into textual space before analysis, which may result in the loss of essential fine-grained visual details.

4.4 Single-domain vs. Multi-domain

In single-domain scenarios, the input images have highly consistent visual scenes, such as similar backgrounds, lighting conditions, object sizes, and visual angles. Models with finegrained perception capabilities can extract reliable features for difference comparison. In contrast, multi-domain scenarios introduce irrelevant parts(i.e. noise and biases), so we design SP and IC to examine the model's ability to overcome irrelevant factors and compare the corresponding and opposite details between two images. Experimental results show that the average value of SOTA in fine-grained multi-domain scenarios decreases by 37.88% compared to the average value of SOTA in fine-grained single-domain scenarios. The challenge of multi-domain tasks lies in the fact that multi-domain can divide the semantic image regions into positively correlated (corresponding parts), negatively correlated ("aligned but different" parts) and irrelevant parts. The irrelevant regions of interest come from prior knowledge of the options. As illustrated in Figure 4, IC requires the model to overcome the confusion caused by biases and noise, such as differences in background environment, and identify the differences of details in the interest regions. Changes in the visual angle and specific coordinates of the bounding box are irrelevant parts with instances of interest, but these changes may affect the model's judgment. From the above comparisons, it can be concluded that current MLLMs may have lost their sensitivity to fine-grained differences in the transition between different scenarios due to a lack of effective multi-domain adaptation strategies, resulting in a decrease in accuracy.

4.5 Coarse-grained vs. Fine-grained

In coarse-grained tasks, encoding global features is sufficient to answer questions on object categories and attributes, while there still exists flaws in answering questions on diverse states of objects, excluding the close-source model Gemini-Pro. In the construction of fine-grained scenarios, more part-level relations between the images can be excavated such as texturefeature and attribute-association relationships. As shown in the case of IC in Figure 2, to answer the question, the models need to pay attention to the hot kid's emoticon on the candy package, requiring part-level representation. As shown in Table 3, the average accuracy of multi-domain fine-grained scenarios compared to multi-domain Coarse-grained scenarios decreases by 39.3%, which indicates that current MLLMs are inadequate in fine-grained perception tasks.

4.6 Effect of Model Scaling on Performance

In the analysis of the performance of open-source MLLMs on the M 4Bench benchmark, we observe that increasing the model parameter scale does not uniformly lead to performance improvements across all tasks. Fine-grained tasks require modeling more complex features, and thus increasing the parameter scale can enhance the model's performance on single-domain tasks. Notably, model scaling has negligible impact on the performance of InternVL2 series on DD-SI, InternVL2.5 on SI, SC, and IC-vp, Qwen2VL on IC-vp, and DeepSeek series on SP-none. This indicates that further improvements may require architectural modifications or different strategies beyond merely improving the parameters scale. Increasing the parameter scale for test tasks with significantly different data distributions from the training set can lead to performance degradation. As shown in Table 3. InternVL2.5 on the DD-SI task drops from 8.1% to 4.7%, and Qwen2VL on the DD-SI task drops from 21.3% to 15.7%. The above analysis indicates that achieving an optimal balance across multiple subtasks when scaling up model parameters represents a promising direction for future research.

4.7 Visual Prompts Analysis

Some studies show that humans tend to adopt the topdown processing approach when comparing multiple images [Gilbert and Li, 2013; Oliva et al., 2003]. This means that the human visual system first locates the main objects in images, and then compares the specific details of those objects. To mimic human cognition, we provided object bounding boxes as visual prompts (vp) and without visual prompts (none) settings for the SP and IC tasks. We found that SP-vp had a 4.6% increase in average accuracy compared to SPnone, which claims that the SP task requires the model to extract geometry layouts for building pixel-level semantic correspondences. Notably, providing visual prompts did not improve the average accuracy for the IC task. DeepSeek-VL2small is the model most significantly affected, with an accuracy drop of 15.2%. Meanwhile, we analyzed four models, DeepSeek-VL2-small, Qwen2VL-7B, Qwen-VL-Max, and GPT-4o, where the accuracy decreased after incorporating vp. When analyzing error cases, we found that the drop in accuracy mainly stemmed from a decrease in the probability of the models answering D. Figure 4 depicts the details. Without vp, the models considered A, B, and C as semantically misaligned parts and therefore chose D. However, the provided bounding boxes label the regions of interest and help establish spatial perception, testing the models' ability to distinguish irrelevant factors in multi-domain tasks( § 4.4).

4.8 M 4Bench is more robust against "single-image" solutions

Popular multi-image benchmarks like MIBench [Liu et al., 2024b] inadvertently encourage "single-image" models that exploit language biases. We demonstrate this by conducting native test and caption test on the previous benchmark and our M 4Bench's subtasks, respectively. The caption test involves using image captions instead of the actual images as input. We use GPT-4 to generate dense captions for each image. Figure 6 presents a comparative analysis of the impact of the caption test on the performance of the previous multiimage benchmarks and our M 4Bench. On MIBench, the caption test led to accuracy declines of only 16.5% and 12.8% for Qwen-VL-Max and DeepSeek-VL2-small, respectively. On BLINK, Qwen-VL-Max and DeepSeek-VL2-small achieved accuracy increases of 13.7% and 5.0% respectively. However, on our M 4Bench DD-SI task, compared to native test, the caption test led to a 40.2% drop in accuracy for QwenVL-Max and a 49.5% drop for DeepSeek-VL2-small. On the M 4Bench SP task, the caption test even caused QwenVL-Max's accuracy to be 9.4% lower than random choice. Figure 5 confirms that M 4Bench's design prevents models from solving multi-image comparison problems through a "single-image" approach, establishing it as a more visioncentric benchmark for reliable evaluation of MLLMs.

5 Conclusion

In this paper, we introduce M 4Bench, a benchmark dedicated to evaluate the capability of aligning and distinguishing multi-images with multi-domain multi-granularity comparison for MLLMs. M 4Bench constructs a hierarchical evaluation that includes five subtasks from coarse-grained to finegrained, encompassing both single and multi domain scenarios. We evaluate 13 popular MLLMs on M 4Bench and measure the effect of converting images to dense captions. The experimental results shows multi-domain multi-grained comparison tasks pose significant challenges for current MLLMs. Even the SOTA models only achieve around 50% accuracy on M 4Bench. We offers detailed analyses and insights for future advancements in the multi-image tasks and the annotated data is publicly available to facilitate further research.

Acknowledgments

This work is supported in part by the National Natural Science Foundation of China (Grant No.62372408).