Modality-Fair Preference Optimization for Trustworthy MLLM Alignment

1 Introduction

Recent advances in multimodal large language models (MLLMs) [Liu et al., 2024b; Wang et al., 2023b; Jiang et al., 2024b] have achieved remarkable performance on diverse multimodal tasks. However, separate training of visual and textual encoders often causes a misalignment of the modality [Cui et al., 2023; Guan et al., 2024] of MLLMs, which can cause the model to generate content that does not exist in the visual input-a phenomenon known as hallucination [Bai et al., 2024]. This issue compromises the reliability of MLLMs in real-world applications [Shah et al., 2019].

† Corresponding author: zuozhuliu@intl.zju.edu.cn

To address this issue, recent work has applied preference optimization to improve the alignment of modalities in MLLM [Rafailov et al., 2024; Naveed et al., 2023]. These methods typically train models to favor accurate responses over those that contain erroneous image descriptions. They achieve this by using inaccurate outputs as rejected options to enhance learning from correct responses, thus improving the alignment of modality [Wang et al., 2024; Zhang and Rong, 2024].

Although current preference optimization methods show success on trustworthiness benchmarks, they mainly rely on text preference optimization. This raises a critical question: What does the model learn during preference optimization when only text preferences are used? Is it merely memorizing text preferences or is it learning to align image and text data? Our experiments shown in Figure 1 reveal that even with severe noise-degrading image content [Leng et al., 2024], the model consistently provides accurate answers. Across a sample of 100 images, the disparity in performance between the original and degraded images is only 9%. Furthermore, the visualization of visual token attention shows that the model mainly focuses on the surrounding contexts rather than the key object mentioned in the question. These findings suggest that the model seems to lean more towards memorizing text preferences rather than achieving modality alignment after preference optimization.

We explore implementing more modality balanced preference optimization to encourage MLLMs to focus more on modality alignment for improved trustworthiness. As shown in Figure 2, we evaluated adding image-related rewards with a simple image preference dataset and a more fine-grained image preference dataset. As we progressively update the approach, the image and text rewards become higher and also more balanced, and the model's trustworthy performance consistently improves. The limited impact of simple image degradation may be due to the model quickly learning to distinguish these variations after only a few training steps, which leads it to continue prioritizing text preference optimization. These findings illustrate the effectiveness of more balanced optimization in achieving trustworthiness. Motivated by our investigation, we propose Modality-Fair Preference Optimization (MFPO) for trustworthy MLLM alignment. To circumvent mere memorization of text responses, we propose to jointly optimize over both text and image preference in a modality-fair manner. In particular, we first construct a multimodal preference dataset encompassing both text and image preference data. Beginning with a text preference dataset, we characterize the relationship between words using a multipartite graph [Jin and Zhang, 2014; Boudin, 2018] to identify the top-K keywords. These keywords are then linked to relevant image regions using a modified Segment Anything Model (SAM) [Kirillov et al., 2023]. By introducing diffusion noise, we generate dispreferred image data that remains largely faithful to the originals while introducing meaningful differences [Jin and Zhang, 2014; Dawande et al., 2001], which are critical for preference optimization. Afterwards, to encourage fair alignment over both modalities, we devise a novel loss function consisting of text preference alignment, image preference alignment and a margin loss for stablized training. Furthermore, to tackle the instability in joint modality optimization, we propose an iterative alignment approach [Paas et al., 2003]. By categorizing training data into varying complexities using semantic entropy, we initially train the model on simpler data and gradually advance to more challenging samples.

Extensive experiments demonstrate the effectiveness of MFPO in improving model trustworthiness. Remarkably, MFPO empowers 7B LLaVA-v1.5 to achieve trustworthiness levels that are comparable to, or even exceed, those of significantly larger 13B, 34B, and larger-scale models. Specifically, when applied to models such as LLaVA-v1.5-13B, MFPO outperforms GPT-4V, achieving a substantial 40% improvement on Object HalBench and setting new state-of-the-art results on both Object HalBench and AMBER benchmarks. Furthermore, LLaVA-v1.5-7B+MFPO and LLaVA-v1.5-13B+MFPO demonstrate superior performance over GPT-4V across five out of eight evaluation metrics. Comprehensive ablation studies further confirm the efficacy of the three core components in MFPO. Our work underscores the importance of balanced preference optimization over different modalities for MLLMs.

2 Preliminaries

Reinforcement Learning from Human Feedback (RLHF) aligns models with human preferences using a reward model rϕ [Schulman et al., 2017], trained on pairwise preference data [Christiano et al., 2017; Ouyang et al., 2022; Casper et al., 2023]. The model assigns higher rewards to preferred outputs, with the cross-entropy loss:

where σ(·) is the logistic sigmoid, and rϕ(x, yw) and rϕ(x, yl) are the rewards for the preferred and less preferred outputs.

After training, the policy πθ is optimized by maximizing the expected reward while regularizing it to prevent divergence from the reference policy πref, using a KL divergence penalty:

where β balances reward maximization and regularization. The KL divergence ensures policy stability [Peng et al., 2023]. Direct Preference Optimization (DPO) directly optimizes a policy πθ using preference data D [Rafailov et al., 2024], without relying on a reward model. Concretely, DPO derives a mapping between the reward r(x, y) and the policy πθ:

where Z(x) is the partition function, defined as Z(x) = ∑y πref(y|x) exp ( 1β r(x, y)). Substitute Eq. 3 into Eq. 1, we can derive the following loss:

where yw and yl denote the preferred and less preferred outputs, respectively. This approach bypasses the complexity of RLHF by allowing the policy gradient to be computed analytically, ensuring efficient alignment with human preferences [Xu et al., 2024; Feng et al., 2024].

The common approach in DPO for MLLMs [Yu et al., 2024a; Yu et al., 2024b; Sun et al., 2023; Zhou et al., 2024] involves concatenating the image m and question t into a single input x, followed by optimization using Eq. 4. Chosen responses yw typically provide accurate, hallucination-free information, while rejected responses yl often contain hallucinated details, making them less preferred.

3 Modality-Fair Preference Optimization

3.1 Image Preference Data Generation

We first introduce the construction of our multimodal preference data as shown in Figure 4. A key question is: what defines effective image preference data in DPO for MLLMs? Previous works on text preference data [Chen et al., 2024] suggest that optimal preference data consists of chosen and rejected responses with moderate differences, as this balance enhances gradient effectiveness during training and helps the model capture subtle content distinctions. Inspired by this, our approach focuses on generating image preference data with fine-grained variation, which involves introducing perturbations only to the key regions of the original image most relevant to the chosen responses. The details of the hyperparameters in this section can be found in Supplementary Section 2.

Keyword Selection. To achieve balanced optimization of image and text preferences, we refine the construction of image preference data at the region level to better capture image preferences. We first construct a multipartite graph [Boudin, 2018], G = (V, E), where nodes V represent words and edges E encode both positional and semantic relationships. The positional influence between two words ki and kj is defined as θij = ∑li∈L(ki) ∑lj∈L(kj) 1 1+|li-lj |φ , where φ controls the decay with distance. Semantic similarity is calculated using cosine similarity: S(ki, kj) = v(ki)·v(kj) ∥v(ki)∥∥v(kj)∥ , and the edge weight is a combination of positional and semantic factors: µij = θij + σij, with σij = γ · S(ki, kj). We further refine these relationships by adjusting edge weights contextually using related words and apply positional decay κij = e-λli to emphasize proximity. The final edge weight ωij = τij · κij integrates positional, semantic, and contextual factors.

Once the graph is constructed, we compute a ranking score for each word based on its centrality using a weighted PageRank-inspired algorithm [Bianchini et al., 2005; Mihalcea and Tarau, 2004]. The initial score for each word ki is r0(ki) = (1 - αpage), where αpage is a damping factor. The score of each word is influenced by its neighbors, with the contribution from a neighboring word kj calculated as rc(ki, kj) = ωij·r(kj) ∑km∈N(kj) ωjm , where ωij is the edge weight, r(kj) is the neighbor's score, and N(kj) represents the neighbors of kj. The score for each word ki is updated iteratively using r(ki) = (1-αpage)+αpage · ∑kj∈N(ki) rc(ki, kj), with αpage controlling the influence of neighbors. This process continues until convergence, where all words are ranked by their final scores r(ki). The top-K keywords, corresponding to the most significant regions in the image, are then selected.

Region Mapping and Perturbation. The top-ranked keywords guide a modified Segment Anything Model (SAM) [Kirillov et al., 2023], mapping each keyword ki to its corresponding visual regions. We then apply diffusion noise to perturb these regions [Ho et al., 2020]. For each region Ri of the image m, the perturbation follows the diffusion process m′ = √αdiff,t · Ri + √1-αdiff,t · ϵ, where αdiff,t = ∏tj=0 βdiff,j (with βdiff,t ∈ (0, 1) as a hyperparameter controlling the noise schedule), and ϵ is random noise sampled from a standard normal distribution. This perturbation selectively distorts key regions, encouraging the model to focus on critical areas. For images whose keywords fail to locate any visual region, we add diffusion noise across all image areas to generate rejected image data. These perturbed images m′ serve as rejected preference images in DPO, refining the model's alignment with fine-grained visual details.

3.2 MFPOTraining Process

We leverage the generated image preference data for a unified optimization of the policy πθ across both text and image.

Text Preference Alignment. The DPO policy objective for text preference optimization is:

where t and m represents the original text and image inputs. yw and yl are the preferred and less preferred outputs. πref is the reference policy.

Image Preference Alignment. For image preference optimization, we use the original image m and the perturbed image m′ generated in earlier steps. The image preference loss is defined as:

Note that we only use the preferred response yw for image preference alignment.

Margin Loss for Stability. To address potential instability during the joint optimization of text and image preferences, we introduce a margin loss to penalize situations where both chosen and rejected responses experience a reduction in reward, drawing inspiration from previous research [Meng et al., 2024; Chen et al., 2024; Wang et al., 2024]. The margin loss is defined as:

where η is the margin parameter that enforces a greater separation between positive (chosen) and negative (rejected) responses. The total loss for the training phase is defined as Ltotal = Ltext +Limage +Lmargin, ensuring that both text and image preferences are jointly optimized while maintaining stability in their alignment.

3.3 Easy-to-Hard Iterative Alignment

We introduce an Easy-to-Hard Iterative Alignment algorithm to stabilize the training of MFPO.

Entropy Calculation. Leveraging semantic entropy [Venhuizen et al., 2019; Farquhar et al., 2024], we estimate the uncertainty of the model's responses. Given a probability distribution of predicted answers P = {p1, p2, . . . , pn}, where pi represents the probability of the i-th predicted token, the entropy H is calculated as H(P) = - ∑ni=1 pi log(pi). This measure quantifies the uncertainty inherent in the model.

Sorting by Difficulty. After calculating entropy for all training samples, we rank the training dataset according to their entropy scores, where higher values denoting more challenging inputs. We then divide the dataset into three distinct difficulty levels: "easy", "medium", and "hard".

Iterative Alignment. By progressively moving from easy to hard examples, the feedback distribution is iteratively updated. This ensures that the model first learns simpler patterns to build a foundation before tackling more complex cases. As the training progresses, the alignment becomes more refined, allowing the model to effectively handle harder examples.

4 Experiements

4.1 Experimental Setup

Implementation Details. We use LLaVA-v1.5 as the backbone for all experiments and include the latest top-performing LLaVA-v1.6 to validate the effectiveness of our method. The training consists of three stages: the first two stages follow standard LLaVA training, while MFPO is introduced in the third stage. Here, we construct image preference data based on Section 3.1, using text preference data from RLHF-V [Yu et al., 2024a], and apply MFPO optimization. Details are in Supplementary Section 4.

Evaluation. We evaluate the trustworthiness and general model capabilities with comprehensive experiments. For trustworthiness, we employ three widely used benchmarks to evaluate trustworthiness reflecting the degree of hallucination. Object HalBench [Rohrbach et al., 2018] was used to assess common object hallucinations in image descriptions, with CHAIR scores for both response-level (CHAIRs) and objectlevel (CHAIRi) hallucination rates. MMHal-Bench [Sun et al., 2023] measured response quality and hallucination rates by comparing model outputs with human responses and object labels. AMBER [Wang et al., 2023a] evaluated generative tasks, providing metrics on CHAIR variants, object coverage, and cognitive hallucination rates. For general capabilities, we employ the LLaVA-Bench [Liu et al., 2024b] for systematic comprehension, encompassing two categories: conversation, detailed description. See details in Supplementary Section 5. Baselines. We compare our model with state-of-the-art baselines under different settings. For general MLLMs, we include LLaVA1.5 [Liu et al., 2024a], Qwen-VL-Chat [Bai et al., 2023], LLaVA1.6, and MiniGemini [Li et al., 2024]. We also compare with preference-feedback models such as POVID [Zhou et al., 2024], RLHF-V [Yu et al., 2024a], Silkie [Sun et al., 2023] and DPO [Rafailov et al., 2024], as well as feedback-independent methods such as VCD [Leng et al., 2024]. We also benchmark against GPT-4V [Achiam et al., 2023]. See details in Supplementary Section 6.

4.2 Main Results

Comparison to Existing Methods. The main results are presented in Table 1. With MFPO integration, both LLaVA-v1.5 and LLaVA-v1.6 show consistent improvements across all three trustworthiness evaluation datasets. On MMHalBench and Object HalBench, our approach achieves state-of-the-art performance among all 7B-parameter models. As shown in Table 2, MFPO enhances performance across all metrics, demonstrating universal improvements in MLLM trustworthiness. These results highlight MFPO's ability to balance preference optimization across text and image modalities, leading to superior trustworthiness.

Notably, LLaVA-v1.5-7B+MFPO achieves trustworthiness levels comparable to or exceeding those of larger models, such as LLaVA-v1.5-13B and MiniGemini-34B. Furthermore, while baseline LLaVA-v1.5-7B and LLaVA-v1.5-13B underperform GPT-4V across all eight metrics, MFPO integration enables both models to outperform GPT-4V on five out of eight metrics. This underscores MFPO's capability to significantly enhance the trustworthiness of smaller MLLMs, bridging the performance gap with much larger models.

General Perception Evaluation. To verify whether MFPO can enhance general perception capability of MLLM, we analyze different preference collection strategies on LLaVABench benchmark. We focus on the performance related to conversation and detailed descriptions, which represent general perception abilities. As shown in Table 3, our method outperforms both LLaVA-v1.5 and other alignment methods, indicating that our approach not only enhances trustworthiness but also improves overall perception capability of the model.

4.3 Ablation Study and Analysis

Ablation on Joint Text-Visual Reward Loss. We conduct experiments to validate the loss composition in MFPO. As shown in Table 4, optimizing only the text modality (Ltext) improves performance but overfits to text preferences, while optimizing only the image modality (Limage) addresses the text-image preference imbalance but underperforms due to lack of text optimization. This highlights the necessity of joint text-visual optimization in multimodal tasks.

Ablation on Margin Loss. Adding margin loss (Lmargin) further improves performance. Figure 6 shows that without Lmargin, rewards for both chosen and rejected responses decrease simultaneously, while Lmargin stabilizes rewards by penalizing reductions for chosen responses. Table 5 demonstrates consistent performance gains across margin values (0, 0.2, 0.4), with the best results at 0.

Ablation on Loss Ratio. In Table 6, we present an ablation study on the weighting of different loss components. A balanced ratio of 1:1:1 yields the best results.

Ablation of Easy-to-Hard Alignment Scheme. We evaluate the effectiveness of the easy-to-hard alignment scheme against simultaneous training on all data. Table 7 shows that our approach consistently outperforms the traditional method. Figure 7 further reveals that the easy-to-hard strategy achieves better convergence, with losses steadily decreasing across easy, medium, and hard phases, indicating more efficient alignment learning. The loss reduction is fastest in the easy phase and gradually slows in later phases. Moreover, the easy-to-hard training curve exhibits greater stability compared to full training, demonstrating its stability and efficiency.

How Does Noise Proportion Affect Preference Optimization Balance? We study how noise granularity and proportion affect preference optimization balance by comparing three methods: applying global diffusion noise to the entire image, adding noise to 20% of random image regions, and our finegrained noise addition strategy. Table 8 demonstrates that image preference data construction enhances image-based optimization during alignment. However, as noise granularity becomes coarser, optimization increasingly favors text preferences, leading to performance degradation. Coarse-grained methods oversimplify image feature optimization, impairing image-text alignment. In contrast, fine-grained construction shifts optimization toward the image modality, achieving a more balanced multimodal alignment.

How Does Keyword Selection Perform? To evaluate the accuracy of extracting key visual keyphrases from the original text preference data using a multipartite graph, we randomly sample 100 examples and manually verify their correctness. We find that 86% of the extracted keyphrases correspond to important objects in the associated preference data. In the remaining 14% of cases, the selected keywords are not the most critical objects; however, using SAM to add diffusion noise to these regions still generates finely dispreferred images. We attribute the effectiveness of the multipartite graph to the relatively short text length and simple object relationships in the preference datasets, which enable robust keyword selection.

How Does SAM Perform on Region Mapping? To validate SAM's accuracy in region mapping, we randomly sample 100 images for manual evaluation. SAM achieves over 90% accuracy, with the remaining 10% of failures occurring when keywords refer to objects requiring contextual understanding beyond SAM's scope. In such cases, we apply global diffusion noise to the entire image. While this approach is suboptimal, the resulting dispreferred images still improve model performance. Furthermore, we experiment with more advanced segmentation models, such as LISA [Lai et al., 2024], which achieves over 95% accuracy in region mapping. Models with reasoning-based segmentation capabilities offer greater robustness to diverse keywords, enabling the generation of more refined dispreferred images.

How Do Different Noise Levels Affect Performance? We investigate the impact of varying noise levels on preference optimization. As shown in Figure 8, adding diffusion noise to the entire image improves performance, with the best results achieved at 500 steps. At this level, the image undergoes an appropriate level of corruption: it avoids excessive noise that would erase critical details and make dispreferred images too easily distinguishable, while also preventing insufficient noise that would result in minimal image changes, failing to establish meaningful rewards for image optimization.

How to Achieve Modality-Fair Optimization? We investigate modality-fair optimization from both data and loss perspectives. On the data side, we analyze the impact of noisy region proportion and noise intensity in images. Our results show that region-level noise is more effective than global noise in guiding preference toward the visual modality. Moreover, moderate noise levels yield a better optimization balance than extremely high or low noise. On the loss side, jointly optimizing text and image rewards facilitates balanced modality alignment while preserving overall performance.

More Comparisions to mDPO. While mDPO demonstrated its effectiveness by only training on Silkie, Table 9 reports MFPO's performance when trained on Silkie and mDPO's performance when trained on RLHF-V. This ensures a fair comparison under equivalent dataset conditions. The results show that dataset differences have minimal impact on MFPO's performance, and MFPO consistently outperforms mDPO across both datasets, further validating MFPO's effectiveness.

5 Conclusion

In this paper, we investigate the limitation of text preference optimization and propose Modality-Fair Preference Optimization (MFPO) for trustworhty aligntment in MLLMs. We constructed a new dataset with fine-grained image preference, and implemented balanced preference optimization between text and image using a multi-stage alignment strategy alongside novel loss functions. Extensive experiments and analysis demonstrate that MFPO notebly diminishes hallucinations and attains state-of-the-art performance across trustworthiness and general capability benchmarks.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant No. 12326612, 62476241), the Natural Science Foundation of Zhejiang Province, China (Grant No. LZ23F020008), and the Zhejiang University-Angelalign Inc. R&D Center for Intelligent Healthcare.