TextMEF: Text-guided Prompt Learning for Multi-exposure Image Fusion

1 Introduction

Natural scenes exhibit a vast range of light intensities, ranging from intense sunlight to faint starlight, differing by up to 100 million orders of magnitude [McCann and Rizzi, 2011]. However, conventional photography equipment, such as mobile phones and SLR cameras, captures only a fraction of this

(LDR) images that often exhibit over- or under-exposed areas, thereby failing to faithfully reproduce details visible to the human eyes under extreme lighting conditions. High dynamic range (HDR) imaging has emerged as a solution to this challenge, offering well-exposed images that enhance visual perception and support various computer vision tasks including super-resolution [Park et al., 2003], panoramic photography [Inanici, 2006], object detection [Biswas and Milanfar, 2017], and remote sensing [Palsson et al., 2017].

While specific hardware devices can directly produce HDR images [Nayar and Mitsunaga, 2000], their high production costs restrict widespread adoption in civil applications. Consequently, MEF methods have garnered significant attention. MEF aims to integrate multiple LDR images captured at different exposures into a unified HDR image. Depending on the number of input LDR images used, MEF methods are broadly categorized into non-extreme [Li et al., 2013; Shen et al., 2011; Li and Kang, 2012] and extreme exposure fusion [Ma et al., 2017; Li et al., 2020] techniques. Non-extreme methods rely on a larger set of input images to achieve optimal fusion performance, albeit at the cost of increased storage requirements and computational complexity. In contrast, our work focuses on extreme fusion techniques, which streamline the process by utilizing only a pair of extreme exposure images to achieve efficient fusion without compromising quality.

Recently, drawing on non-liner fitting ability of deep neural networks, a large numbers of learning-based MEF methods [Xu et al., 2020b; Xu et al., 2020a; Zhang and Ma, 2021; Xu et al., 2023; Wu et al., 2024] have been proposed and applied well. Despite great achievement, there are still come across several challenges that hinder further improvement of producing high-quality fused results. First, most of existing approaches cannot precisely perceive the bright and dark regions, causing over-enhancement in well-exposed regions or under-enhancement in low-light ones. Second, when designing neural networks, existing approaches often overlook subtle details. As a result, the integrated outputs frequently fail to accurately reproduce realistic textures, especially in excessively dark or bright regions.

To address these challenges, we propose TextMEF, a prompt guided multi-exposure fusion framework enhanced by learning-based prompts, which effectively merges multiple LDR images into an HDR result. Specifically, to mitigate color discrepancies caused by uneven exposures, we develop a new pipeline that adapts the Contrastive Language-Image Pre-Training (CLIP) [Radford et al., 2021] model to our task. This approach refines the prompts by learning distinct feature distributions among over, under, and properly exposed images. The optimized prompts then direct the training of our multi-exposure fusion network through text-image similarity constraints in the CLIP embedding space. Given that network interference often results in the loss of critical information, we incorporate the advanced Mamba [Gu and Dao, 2023; Zhu et al., 2024] block along with newly introduced attention Mamba and mutual guided Mamba, achieving enhanced contrast and detail in the resultant fused image.

Thanks to the meticulously designed modules and the effective prompt learning strategy outlined above, our TextMEF framework demonstrates superior performance across several widely used datasets. A straightforward example is given in Figure 1, where the results of our method not only enhance visual appeal but also deliver fused images with distinct targets and realistic details. The main contributions of this paper are summarized as follows:

• We leverage a pretrained CLIP model to provide crossmodal contrastive constraints from text to image, benefiting from its robust visual-language priors. Our method achieves a trade-off between metric-oriented and perceptual-oriented objectives.
• To fully unleash the potential of the CLIP priors, we employ prompt learning to discover more precise cues, thus providing richer and more detailed semantic guidance.
• We design the attention Mamba module and the mutualguided Mamba module. The former helps capture prominent features and protect them under varying exposure levels, the latter promotes efficient feature fusion, enabling the model to autonomously learn the complementary information between different exposure inputs.
• Evaluations on three prevailing benchmark datasets and against seven state-of-the-art multi-exposure fusion approaches demonstrate that our proposed TextMEF can significantly enhance color representation while accurately recovering texture details.

2 RelatedWork

2.1 Learning-basedMEF Approaches

In recent years, deep learning [Liu et al., 2024a] has made significant advancements in the field of multi-exposure fusion. DeepFuse [Ram Prabhakar et al., 2017], a pioneering approach, set a precedent by applying the feature processing capabilities of neural networks to MEF. Subsequent methods have predominantly focused on innovations in network architecture and improvements in loss functions. Architecturally, MEF-GAN [Xu et al., 2020b] and AGAL [Liu et al., 2022] introduced Generative Adversarial Networks (GANs), with the former leveraging GANs to effectively extract latent representations from reference images, while the latter employed global-local hierarchical constraints to enhance network learning. More recently, TransMEF [Qu et al., 2022] combined CNN and Transformer modules to extract richer features through self-supervised multi-task learning, and SwinFusion [Ma et al., 2022] utilized the advanced SwinTransformer to achieve global information integration and detail preservation. Differently, CRMEF [Liu et al., 2024c] employed neural architecture search to adaptively derive a supernetwork. In terms of loss functions, following the pioneering MEF-SSIM [Ma et al., 2015], MEF-CL [Xu et al., 2023] and HoLoCo [Liu et al., 2023] introduced contrastive learningbased loss constraints, guiding the fusion process through distinct positive and negative sample settings. HSDS [Wu et al., 2024], on the other hand, used an automated search method to construct loss functions without human intervention. Unfortunately, most current structural improvements tend to focus on network stacking rather than simplification [Liu et al., 2024b], and the potential of cross-modal constraints in loss function improvements remains largely untapped.

2.2 Prompt Learning in Vision

Contrastive Language-Image Pre-Training (CLIP)[Radford et al., 2021] achieved significant success in zero-shot prediction by leveraging large-scale image-text pairs. It has since been widely adapted for a range of visual tasks. By incorporating learnable prompt tokens, CLIP can internalize datasetspecific biases, enhancing its recognition ability for tasks such as object detection[Vidit et al., 2023], style transformation [Kwon and Ye, 2022], and image enhancement [Liang et al., 2023]. CLIP-based prompt learning typically adopts templates from natural language processing, which are fed into the text encoder, while image features are aligned with the textual prompts through the image encoder. Techniques such as StyleCLIP [Patashnik et al., 2021] and StyleGAN [Gal et al., 2022] combine the generative power of GANs with CLIP's semantic guidance, allowing images to be optimized toward target directions while being steered away from undesired features. However, existing prompt learning approaches have rarely addressed low-level vision tasks. In this work, we explore the potential of prompt learning for extracting more accurate exposure representations, enabling more effective multi-exposure image fusion.

where the cosine similarity is used to measure the alignment between image features Φimage(Ii) and prompt features Φtext(Tc).

The labels yi,c are one-hot encoded, indicating which class the image belongs to. Specifically, [1, 0, 0] corresponds to over-exposed images Io, [0, 1, 0] for under-exposed images Iu, and [0, 0, 1] for well-exposed images Iw.

3.2 Loss Function

Once the prompt learning phase is complete, the learned prompts can be used to guide the fusion network. The loss function for the multi-exposure image fusion task consists of two major components: Mean Squared Error (MSE) Loss and CLIP Perceptual Loss. These losses are designed to ensure that the fused image both retains pixel-level accuracy and aligns well with the learned semantic cues from the prompts. CLIP Perceptual Loss. The purpose of the CLIP perceptual loss is to constrain the fused image to be semantically closer to the positive prompt Tp and farther from the negative prompts Tn1 and Tn2 in the CLIP feature space. The formula for this loss is as follows:

This function uses cosine similarity to measure the alignment between the image features Φimage(Ii) and the text features of the prompts. The numerator calculates the distance between the image and the negative prompts, and the denominator normalizes the result by considering all prompts, including the positive one.

Mean Squared Error Loss. To minimize pixel-wise differences between the ground truth and the fused image, we employ the Mean Squared Error (MSE) Loss, calculated as:

where Igt is the ground truth image, and If is the fused image. The MSE loss ensures that the final fused image retains similar pixel intensities to the ground truth image, improving the quality of the fusion process.

3 The ProposedMethod

Our TextMEF is developed in two distinct phases. In the first phase, using CLIP priors, we initialize a set of image prompts derived from under-/over-exposed, and correctly exposed images. These prompts are then refined and fine-tuned to enable better semantic alignment. In the second phase, the learned prompts are fixed and used in conjunction with a contrastive CLIP perceptual loss, which guides the multi-exposure image fusion network based on Mamba architecture.

3.1 Prompt Learning

We start with a set of images, {Io, Iu, Iw}Mi=1 , where Io represents over-exposed images, Iu represents under-exposed images, and Iw represents well-exposed images. The variable M denotes the total number of images in the dataset.

For these images, we initialize one positive prompt Tp ∈ RN×512 and two negative prompts Tn1 and Tn2 ∈ RN×512 , where N refers to the number of embedded tokens in each prompt. These prompts are intended to represent the different lighting conditions of the images. The initialization is done by feeding the images into the image encoder of a pretrained CLIP model to extract image features Φimage, and the prompts are processed through the text encoder to generate their respective text features Φtext.

For effective prompt learning, we leverage a multi-class cross-entropy loss function to fine-tune these prompts so that they properly differentiate between under-exposed, overexposed, and well-exposed images inspired by [Liang et al., 2023]. The multi-class cross-entropy loss function is formulated as:

where ˆ yi,c is the predicted probability of the i-th sample belonging to class c, calculated as:

U2Fusion SDNet DeFusion MEFLUT HSDS-MEF DeepM2CDL FILM

Ours GT

Figure 3: Visual comparison of our method with seven state-of-the-art methods on SICE dataset. The signal maps demonstrate accurate pixel intensity between the ground truth and our method.

Total Loss. The total loss function combines the CLIP Perceptual Loss and the MSE Loss, weighted by a hyperparameter λ:

where λ is a hyperparameter that controls the trade-off between the pixel-level similarity and the semantic alignment.

3.3 Network Architecture

To fully leverage the advantages of the State-Space Model [Gu and Dao, 2023], we have integrated the Mamba into our network. Specifically, our network architecture includes the Attention Mamba, Mutual-guided Mamba, and Adaptive Deep Fuse modules. The source image is first processed through a CNN to convert to feature space, then the Attention Mamba module is used to extract important features. Next, the Mutual-guided Mamba module performs preliminary feature fusion, and then, the Adaptive Deep Fuse module achieves adaptive information fusion. Finally, the fused features are passed through a convolutional layer and a skip connection from the source images to produce the final fused image. Each module will be introduced below.

Attention Mamba Block. Given the feature maps F Iu i and F Io i ∈ RB×C×H×W for under-exposed and over-exposed images, respectively, the Attention Mamba module contributes to compute attention weight maps to guide the extraction of more prominent features, thereby laying a foundation for more complementary and complete information fusion. The feature extraction process can be defined as follows:

where Au(·) and Ao(·) are the Attention Mamba Modules, and ⊙ indicates point-wise multiplication.

Mutual-Guided Mamba Block. The mutual-guided Mamba block achieves lightweight information complementarity through a simple channel exchange method. Given the image features F Iu i ∈ RB×N×C and F Io i ∈ RB×N×C , the Mutual-Guided Mamba aims to achieve lightweight information complementarity and interaction through channel swapping. Specifically, Mutual-Guided Mamba splits the features along the channel dimension into two halves. The first half of the channels from F Io i is concatenated with the second half of the channels from F Iu i to obtain F Iu i+1 and vice versa.

Adaptive Deep Fuse Block. The common element-wise addition fusion method overlooks the weights of the fused objects, which is detrimental to preserving prominent features. Inspired by [Chen et al., 2024], we adopt a Content Guided Attention (CGA)-based Mixup Fusion scheme to form our adaptive deep fusion module. The CGA module provides self-learned channel and spatial attention weights to adaptively modulate the feature fusion. The detailed feature fusion process is defined as follows:

This final fused feature map Ffuse integrates the adaptive fusion process, where the weights ω control the contributions of the under-exposed and over-exposed images in the fusion process. The convolutional layer C1×1 is used to process the weighted fusion output.

In this formulation, we ensure that the network learns optimal fusion strategies by adapting the weight allocation dynamically, considering both channel and spatial attention.

4 Experiments

4.1 Implementation Details

We conduct experiments on the SICE [Cai et al., 2018] dataset. The training process consists of prompt learning and During the training process, all images are cropped to a size of 224×224, and data augmentations including flip, rotate, and zoom are used. The batch size and epochs for the prompt learning and image fusion are set to 16/8 and 160/200, respectively. We employ the Adam optimizer to guide parameter optimization. The learning rate for the prompt learning is set to 5e-5 , while the initial learning rate for the image fusion is set to 2e-4 , with a learning rate decay of 0.1 at epochs 100 and 135. The overall framework is implemented on Pytorch with an NVIDIA Tesla V100 GPU.

4.2 ComparisonMethods &EvaluationMetrics

We compare our method with seven state-of-the-art deep learning-based competitors, i.e., U2Fusion [Xu et al., 2020a], SDNet [Zhang and Ma, 2021], DeFusion [Liang et al., 2022], MEFLUT [Jiang et al., 2023], HSDS-MEF [Wu et al., 2024], DeepM2CDL [Deng et al., 2023], and FILM [Zhao et al., 2024]. All methods are tested with their official codes.

For quantitative analysis, a total of nine metrics are adopted, including three reference-based metrics, i.e., PSNR [Huynh-Thu and Ghanbari, 2008], SSIM [WANGZ et al., 2004], and TMQI [Yeganeh and Wang, 2012], and six no-reference metrics, i.e., EN [Roberts et al., 2008], AG [Es-

4.3 Qualitative Comparisons

We present a visual comparison on the SICE [Cai et al., 2018] dataset in Figure 3. Certain methods, e.g., U2Fusion and HSDS-MEF, suffer from significant detail loss in extremely under-exposed and over-exposed areas (see the paper in the red region of the first image sequence and the building in the blue region of the second image sequence). Other methods, e.g., DeFusion, MEFLUT, DeepM2CDL, and FILM, exhibit noticeable artifacts (particularly evident in the unnatural colors of the clouds in the second image sequence). Thanks to the CLIP Perceptual Loss, our method excellently balances details and luminance, producing fused results that realistically simulate natural illumination. Moreover, in the RGB curves shown below, our method exhibits a distribution closest to the ground truth, demonstrating the outstanding color balance capability of our method.

Furthermore, we evaluate our method on the MEF [Ma et al., 2017] and Mobile [Jiang et al., 2023] datasets, with the comparison results shown in Figures 4 and 5. As illustrated in Figure 4, most methods suffer from blurred artifacts, leading to the loss of high-frequency information. Additionally, as seen in the second image sequence, many competing meth-

4.5 Ablation Studies

Study on Prompt Learning. We investigated the impact of prompt learning on the fusion stage using three variants for

Figure 6: Quantitative comparison with seven state-of-the-art methods on the MEF dataset.

ods display dull colors. Empowered by the specially designed Mamba modules, our method excels at preserving edge information while maintaining good contrast. Figure 5 presents a similar situation to Figure 4, where our method demonstrates superior ability to restore texture details and vibrant colors compared to the competitors.

4.4 Quantitative Comparisons

Table 1 presents a qualitative comparison on the SICE and Mobile datasets. Our method achieved significantly higher performance on reference-based metrics compared to the competitors, demonstrating that the proposed method delivers image content structure and visual perception closest to the ground truth. Moreover, on the learning-based no-reference metrics, MUSIQ and PaQ-2-PiQ, our method achieves the best performance on the SICE dataset and came in second only to SDNet on the Mobile dataset, indicating that our method strongly aligns with the priors of high perceptual quality. Additionally, the quantitative comparison on the MEF dataset is shown in Figure 6. It is notable that our method significantly outperforms all other competitors across all metrics, demonstrating its ability to produce images with rich details, high edge strength and contrast, and alignment with human visual perception.

experiments on the weight of the CLIP perceptual loss. As shown in Figure 8, λ = 0 (i.e., w/o Lclip) maintains overall proximity to the reference but struggles with varying brightness, leading to local detail loss (prominently shown on the building in the red rigion of the third image sequence). Higher weights (i.e., λ = 5e-1 ) enhance texture details but introduce unnatural colors and severe noise due to weakened pixel constraints (see the first and third sequences). At a weight of 1e-2 , the loss function strikes an optimal balance between pixel-level and semantic constraints, resulting in visually appealing outcomes. The quantitative results in Table 2 further validate this fact.