Interactive All-in-One Image Restoration and Fusion

1 Introduction

Multi-modal image fusion integrates complementary cues from different sensors into a unified representation [Li and Wu, 2024; Cao et al., 2024]. A prime example is InfraredVisible Image Fusion (IVIF) [Zhang and Demiris, 2023], where infrared sensors capture thermal radiation to highlight salient targets [Ibarra-Castanedo et al., 2004], while visible sensors provide rich textural and chromatic details. However, the modality gap makes it challenging to balance target prominence with fine-grained scene detail. Ideally, fusion should preserve thermal saliency and visible texture without artifacts, benefiting downstream perception such as object detection and surveillance [Takumi et al., 2017].

∗Corresponding author.

Despite impressive progress, practical deployment faces a critical hurdle: supervised IVIF models are often trained on limited fusion datasets and can overfit the training distribution, resulting in a pronounced generalization gap under open-world degradations. In real scenarios, inputs are inevitably corrupted by complex degradations (e.g., rain, haze, noise, and blur) [Bulat et al., 2018; Tang et al., 2023]. Conventional fusion pipelines [Fang et al., 2021; Zhao et al., 2020] typically treat inputs as clean signals, inadvertently propagating and even amplifying degradations in the fused results. Such corruption distorts modality-specific structures and salient cues, leading to inconsistent textures and weakened target boundaries. Consequently, downstream perception may degrade in safety-critical scenarios. Thus, robust restoration-aware fusion is urgently needed to restore signal fidelity while integrating complementary information.

To address this, recent unified frameworks like Text-IF [Yi et al., 2024b] attempt to couple restoration with fusion. However, most rely on training task-specific networks from scratch on limited fusion datasets. Compared with largescale generative corpora, these datasets lack diversity, causing scratch-trained models to struggle with open-world degradations and varying imaging conditions. As illustrated in Fig. 1, pre-trained generative models possess robust priors for restoration but lack inherent multi-modal fusion capability, whereas unified methods trained from scratch lack strong visual priors for restoration. Bridging this gap requires leveraging the restoration priors of foundation models for generalizable multimodal fusion under complex degradations.

Motivated by these priors, we propose AIR-Fusion, a framework that adapts a frozen restoration-capable LDM backbone for robust fusion under degradations. Instead of training a fusion network from scratch, we reformulate degraded IVIF as aligning multi-modal conditions with this frozen generative substrate, so that restoration priors learned from large-scale data can generalize beyond the limited fusion training distribution. We term this reformulation generative capability elicitation. Yet fusion imposes stricter sourcefidelity requirements than ordinary restoration: the model should remove degradations without hallucinating structures or weakening thermal targets. However, directly repurposing such a backbone for high-fidelity fusion is non-trivial due to three key gaps: (i) the backbone is trained for single-modal conditional generation and thus lacks a native multi-modal conditioning interface to jointly condition on degraded visible and infrared inputs; (ii) full adaptation risks degrading the pre-trained restoration priors that underpin open-world robustness; and (iii) stochastic sampling together with VAE latent compression can suppress high-frequency structures and induce structural drift, conflicting with the source fidelity requirements of fusion.

To overcome these hurdles, we employ a parameterefficient fine-tuning (PEFT) strategy. We freeze this backbone and introduce a lightweight Cross-Modal Bridging Adapter (CMBA). CMBA maps infrared cues and text instructions into the frozen conditioning space and injects them into multi-scale denoising features, enabling instructionguided restoration-aware fusion without expensive full-model training. Furthermore, to counteract the loss of highfrequency details caused by VAE-based latent compression in LDMs [Chen et al., 2023] and regularize the stochastic denoising trajectory, we propose a Trajectory-Constrained Rectifier (TCR). TCR explicitly references infrared and visible raw source inputs for modality-aware rectification in a pixellatent closed loop, constraining each denoising step toward source-consistent structures while recovering fine-grained details. The main contributions of this work are summarized as follows:

• We propose AIR-Fusion, an interactive all-in-one image restoration and fusion framework that performs generative capability elicitation for degraded IVIF by aligning multi-modal conditions with a frozen generative manifold to preserve restoration priors while eliciting fusion capability under complex degradations.
• We propose CMBA for cross-modal conditioning and TCR for trajectory regularization via a pixel-latent closed loop, jointly improving restoration-aware fusion and source fidelity.
• Extensive experiments on multiple datasets and realworld degradations show consistent gains in restoration quality, fusion fidelity, and cross-dataset robustness.

2 RelatedWork

Multi-modality Image Fusion. Deep learning has substantially advanced image fusion beyond traditional optimizationbased pipelines [Shi et al., 2005; Pradnya and Sachin, 2013], primarily due to its powerful representation learning capabilities. Early data-driven methods were dominated by CNN and auto-encoder architectures [Liu et al., 2017; Zhang et al., 2020; Jian et al., 2020], followed by adversarial learning [Zhou et al., 2023; Ma et al., 2020] and Transformerbased mechanisms (e.g., SwinFusion [Ma et al., 2022]) to enhance perceptual fidelity and capture long-range dependencies. However, a critical limitation of these networks is their implicit assumption of clean, high-quality inputs. Under degradations (e.g., rain, haze, noise, low-light and blur), such assumptions often break, leading to degraded fused outputs and weakened saliency/texture cues. To address degradations, recent studies have shifted towards unified restorationand-fusion pipelines. Text-IF [Yi et al., 2024b] pioneered the integration of textual prompts to guide interactive fusion, while ControlFusion [Tang et al., 2025] and MMAIF [Cao et al., 2025] further incorporated degradation-aware prompts. Despite their progress, these approaches predominantly adhere to a training-from-scratch paradigm; constrained by limited fusion data, they often exhibit a generalization gap under complex, open-world degradations.

Diffusion Models and Foundation Model Adaptation. Denoising Diffusion Probabilistic Models (DDPMs) [Ho et al., 2020; Nichol and Dhariwal, 2021] and Latent Diffusion Models (LDMs) [Rombach et al., 2022] provide strong generative priors for solving inverse problems. (i) Generative Priors in Restoration. Leveraging large-scale pre-training, LDMs encode rich statistics of natural images. Recent works like AutoDIR [Jiang et al., 2024] and Spire [Qi et al., 2024] show that robust blind restoration can be elicited from frozen generative priors. Nevertheless, these frameworks are tailored for single-modality restoration and lack mechanisms to integrate and align complementary information from multi-modal sensors when the backbone is kept frozen. (ii) Diffusion-based Fusion. Diffusion architectures have also been applied to image fusion, such as DDFM [Zhao et al., 2023b], Diff-IF [Yi et al., 2024a], and Text-DiFuse [Zhang et al., 2024]. However, most existing approaches treat diffusion models as trainable backbones and adapt extensively on limited fusion data, which can underutilize a frozen restorationcapable manifold and weaken robustness under severe degradations. In contrast, leveraging a frozen restoration manifold for fusion calls for dedicated multi-modal condition alignment rather than full-model training. (iii) Foundation Model Adaptation. Parameter-efficient fine-tuning (PEFT) techniques like ControlNet [Zhang et al., 2023] and T2IAdapter [Mou et al., 2024] inject conditional signals via lightweight modules while freezing heavy backbones. While effective for controlled generation, robust degraded fusion further requires preserving restoration priors, aligning multimodal conditions, and enforcing source fidelity during sampling. AIR-Fusion is designed to bridge these gaps via generative capability elicitation with condition alignment (CMBA) and trajectory-level fidelity constraints (TCR).

(c) Process

3 Methodology

The architecture of AIR-Fusion, illustrated in Fig. 2, is founded on the principle of generative capability elicitation. Diverging from traditional paradigms that attempt to reconstruct fusion mappings from limited data, our framework adopts a transfer learning strategy. Concretely, in this work we instantiate this substrate with AutoDIR [Jiang et al., 2024], a restoration-capable latent diffusion backbone rather than a vanilla LDM, and keep it frozen to preserve its pretrained restoration priors. Upon this foundation, we introduce and exclusively optimize lightweight modules to align multimodal conditions and constrain the denoising trajectory for robust fusion under degradations.

The overall pipeline comprises two synergistic stages. First, the Condition Alignment stage employs a Cross-Modal Bridging Adapter (CMBA) to align multi-modal conditions with the frozen backbone, steering the iterative denoising process via textual prompts. Second, the Trajectory-Constrained Rectification stage utilizes a Trajectory-Constrained Rectifier (TCR) module to enforce source-referenced fidelity, mitigating the spectral bias introduced by VAE latent compression and significantly enhancing fine-grained visual quality.

3.1 CMBAAdapter for Fusion

We formulate the problem using Visible-Infrared Fusion as a representative case study. Let Iraw ∈ RH×W denote the raw infrared image and V ∈ RH×W×3 denote the clean visible image. In practice, for compatibility with the backbone, we replicate the infrared modality channel-wise to obtain I ∈ RH×W×3 (I = Replicate(Iraw)), and we use I to denote this 3-channel representation throughout the paper. In real-world scenarios, the visible input is often corrupted, yielding a degraded observation Vd, where the degradation type d ∈ {low-light, rain, haze, blur, noise}.

We employ AutoDIR [Jiang et al., 2024] as the frozen restoration-capable latent-diffusion backbone. The degraded image Vd is projected into a latent representation ZVd ∈ Rh×w×c via the pre-trained VAE encoder:

To inject complementary infrared information without disrupting the pre-trained generative manifold, we introduce the Cross-Modal Bridging Adapter (CMBA). CMBA is inspired by ControlNet-style conditional branches [Zhang et al., 2023]: it is a trainable conditional branch whose condition input is the infrared modality. Specifically, CMBA encodes the infrared input I into a latent condition ZI that lies in the same latent space as ZVd and is spatially aligned with it, while using twice the channel capacity:

where ZI ∈ Rh×w×2c . During denoising, CMBA produces multi-scale residual features that are injected back into the frozen backbone through zero-initialized 1 × 1 convolutions and are added to the corresponding decoder blocks.

Training Strategy: Manifold Alignment. Image fusion is inherently unsupervised due to the lack of ground-truth fused labels. Following [Yi et al., 2024a], we construct a curated repository of reference priors, denoted as Fg, comprising high-quality fused images generated by representative algorithms. At training time, we sample a proxy fused prior ˜ F ∼ Fg and encode it as Z ˜ F = EVAE( ˜ F). We then sample a timestep t ∼ Uniform(1, T ) and add standard Gaussian noise ϵ ∼ N(0, I) to obtain the noisy latent:

where Zt ∈ Rh×w×c . Meanwhile, user text prompts (e.g., "A clear photo, no haze") are encoded into embeddings c via a frozen CLIP text encoder to provide semantic guidance.

The feature extraction process follows two parallel pathways. Let [·; ·] denote channel-wise concatenation and ⊕ denote element-wise addition. The Frozen Restoration Path employs the frozen backbone U-Net encoder Er to encode the concatenation of the current noisy latent Zt and the degradedvisible latent ZVd , producing restoration-prior features f r t :

In parallel, the Trainable Fusion Path uses the CMBA encoder Ef , which further incorporates the infrared latent ZI by adding it to the concatenated visible-conditioned latent, yielding fusion-oriented features f f t :

Within the decoder of the frozen backbone U-Net (Dr), the fusion features are projected by a zero-initialized convolution (denoted as C0(·)) and injected at multiple scales via element-wise addition, yielding a steered representation for noise prediction:ˆ ϵt = Dr (f r t ⊕C0(f f t ), t, c). (6)

Synergistic Inference Process. During inference, the Unified Core Architecture FUCA (the denoising core that couples the frozen backbone with the CMBA) starts from pure Gaussian noise ZT ∼ N(0, I) and performs iterative denoising, unlike training where Zt is constructed by noising a proxyprior latent. At each timestep, FUCA predicts ˆ ϵt to update the latent according to the diffusion scheduler.

This design fosters a mutual reinforcement mechanism: the fusion cues from CMBA provide modality-complementary guidance, while the frozen restoration-capable LDM backbone contributes strong degradation removal priors. The process yields a preliminary fused result that is semantically consistent but may exhibit high-frequency loss due to latent compression, motivating the subsequent trajectory-constrained rectification stage.

3.2 Trajectory-Constrained Rectification

A known limitation of LDMs [Chen et al., 2023; Huang et al., 2023] is that the VAE-induced spatial compression may suppress high-frequency textures, which is particularly harmful for image fusion where source fidelity is critical. To mitigate this issue, we propose a Trajectory-Constrained Rectifier (TCR), denoted as R, which explicitly references raw source inputs to enforce a structure-preserving constraint in pixel space. Architecturally, TCR adapts the refinement design from the NAFNet block [Chen et al., 2022]; crucially, we extend the input to enable multi-modal awareness.

We decode the clean-latent estimate ˆ Z0|t into a coarse fused image:

and feed it to TCR together with the degraded visible image Vd and infrared image I via channel-wise concatenation:

Since Ft already contains the main clean fusion content, TCR uses Vd mainly as a structural reference under final fusedimage supervision rather than copying its degradation artifacts.

To maximally leverage TCR, we embed it directly into the sampling loop, forming a pixel-latent closed loop that regularizes the denoising trajectory. At each timestep t, we first estimate the clean latent ˆ Z0|t and decode it to pixel space. TCR then rectifies this estimate, which is subsequently reencoded to guide the next sampling step:

By iteratively enforcing this constraint, AIR-Fusion anchors the generative trajectory to the physical structures of the source inputs, yielding high-fidelity fusion results under severe degradations.

3.3 Loss Function

Our training process is divided into two sequential stages, and our loss function is designed accordingly.

In the first stage, we train CMBA and frozen backbone. The primary objective is to learn the denoising process fundamental to the diffusion model. This is formulated as a noise prediction loss, Lnoise:

To guide the model towards better fusion results and enhance visual quality, we introduce a set of auxiliary fusion losses: a Structural Similarity loss (Lssim), an intensity loss (Lint), a gradient loss (Lgrad), and a color loss (Lcolor). The total loss for this stage is a weighted sum of the primary and auxiliary objectives:

In the second stage, we freeze CMBA and optimize TCR. The training objective, LTCR, leverages the same set of auxiliary fusion losses. Crucially, these losses are now computed on the final output image to directly refine its detail and visual quality.

4 Experiments

Datasets. Our model was trained exclusively on the LLVIP dataset [Jia et al., 2021]. To comprehensively evaluate its effectiveness and generalization capability, we utilized the MSRS [Tang et al., 2022] and RoadScene [Xu et al., 2020] datasets as unseen test sets. For the training data, we leveraged both the inherent degradations found in LLVIP's visible images (e.g., low light, blur) and further applied a suite of five artificially simulated degradations: low light, rain, haze, blur, and noise. We construct a template-based prompt set (hundreds of variants) covering the five degradation types to enable instruction-guided interaction. In addition, we use AWMM-100k [Li et al., 2024] for qualitative evaluation of real-world generalization under adverse weather.

Implementation Details. Our method utilizes the pretrained restoration-capable AutoDIR latent-diffusion backbone [Jiang et al., 2024] as its frozen backbone. During training, we optimize only the parameters of the CMBA and the TCR. Parameter updates are performed using the Adam optimizer with a learning rate set to 1e-5. All experiments were conducted on an NVIDIA RTX A6000 GPU. With 100 denoising steps, AIR-Fusion has only 0.361B trainable parameters and takes 11.397s for inference, requiring far fewer trainable parameters than full AutoDIR tuning while running much faster than Diff-IF and DDFM.

EvaluationMetrics. To quantitatively assess the quality of the fused images, we employed a variety of evaluation metrics: SD [Ma et al., 2019], EN [Ma et al., 2019], VIF [Han et al., 2013], QAB/F [Ma et al., 2019], PSNR [Wang et al., 2019], SSIM [Wang et al., 2004], and LPIPS [Zhang et al., 2018]. We compared our proposed method with state-ofthe-art approaches, including: U2Fusion [Xu et al., 2020], TarDAL [Liu et al., 2022], SwinFusion [Ma et al., 2022], CDDFuse [Zhao et al., 2023a], DDFM [Zhao et al., 2023b], EMMA [Zhao et al., 2024], Text-IF [Yi et al., 2024b] and ControlFusion [Tang et al., 2025].

4.1 Comparisons with other Methods

Existing fusion models are typically developed for clean inputs and thus degrade noticeably when the visible modality suffers from severe corruptions, while most of them also lack a mechanism for instruction-guided control. We therefore benchmark AIR-Fusion under five representative visible degradations (blur, haze, low-light, rain, and noise) and a clean case, covering both in-domain and cross-dataset evaluation. Notably, AIR-Fusion uses one set of parameters to handle all conditions without per-degradation retraining.

AIR-Fusion is trained only on LLVIP and tested on LLVIP as well as two unseen datasets (MSRS and RoadScene). For methods without an explicit restoration component, we prepend an off-the-shelf AutoDIR module to restore the degraded visible input before fusion; such pipelines are marked with ∗ in Table 1. For interactive baselines (Text-IF and ControlFusion) and AIR-Fusion, we use degradation-matched prompts, while other methods are run with their default inference settings.

Quantitative Comparisons. Table 1 reports results under degraded conditions. AIR-Fusion achieves the best performance on blur, low-light, and rain across PSNR, SSIM, VIF, and LPIPS, demonstrating strong joint restoration-and-fusion capability with improved perceptual fidelity. For haze, AIRFusion attains the highest PSNR and SSIM, the lowest LPIPS, and remains competitive on VIF, indicating better structure recovery and fidelity under scattering. Under noise on the unseen RoadScene dataset, AIR-Fusion achieves the top PSNR and strong VIF, while maintaining competitive LPIPS, suggesting a favorable trade-off between denoising, detail preservation, and perceptual quality. Overall, the consistent performance across datasets supports adapting a frozen restorationcapable diffusion backbone via generative capability elicitation, rather than training a specialized fusion network from scratch.

Qualitative Comparisons. Fig. 3 presents visual comparisons under both degraded inputs and a clean case. Across degradations, AIR-Fusion better suppresses corruption while preserving complementary information from infrared and visible modalities. In blur/rain/haze cases, it restores clearer edges and scene textures with fewer artifacts. Under lowlight, it improves visibility with more natural colors and avoids the over-darkening or color shifts observed in competing methods. For noise, AIR-Fusion reduces granular noise while maintaining fine structures. In the clean case (last row), AIR-Fusion highlights thermal targets while retaining visible details such as clothing textures and ground markings, yielding more faithful and visually consistent fusion results.

4.2 Generalization to Real-world Scenarios

Models trained and evaluated on synthetic degradations may still struggle in real-world scenes, where weather patterns and sensor characteristics differ from simulated settings. To assess robustness in the wild, we conduct qualitative evaluation on the public AWMM-100k benchmark [Li et al., 2024], which contains large-scale aligned infrared-visible pairs captured under adverse weather. Specifically, we select realworld haze cases to test transferability beyond the degradations seen in LLVIP training. We compare AIR-Fusion with two instruction-guided baselines, Text-IF [Yi et al., 2024b] and ControlFusion [Tang et al., 2025].

As shown in Fig. 4, other fusion methods trained from scratch exhibit limited transfer to real haze scenes. Text-IF often leaves substantial haze and artifacts, while ControlFusion alleviates haze to some extent but may introduce color shifts and residual veiling. In contrast, AIR-Fusion produces clearer structures and more consistent colors while preserving salient thermal targets from the infrared modality. We attribute this improved transferability to adapting a frozen restoration-capable diffusion backbone with constrained generative adaptation, which provides strong restoration priors beyond the training distribution.

4.3 Performance on High-level Task

To examine whether the fused images preserve task-relevant semantics, we evaluate AIR-Fusion on downstream object detection. Following common practice on LLVIP, we adopt YOLOv5 as the detector and fine-tune it on the LLVIP training split. At test time, the detector takes the fused images produced by each fusion method as input. Fig. 5 shows qualitative detection examples. Compared with competing fusion methods, AIR-Fusion yields fused images that lead to more accurate and complete detections, especially for the car and pedestrian under adverse conditions. These results suggest that AIR-Fusion better preserves discriminative structures and salient thermal cues that are beneficial for downstream perception.

4.4 Ablation Study

We ablate key components and report quantitative results in Table 2. Using only diffusion noise loss performs worst because latent denoising alone does not enforce cross-modal complementarity. Removing the color loss can retain competitive reconstruction scores but causes chromatic inconsistency. Disabling TCR hurts structural/perceptual quality, confirming that pixel-space rectification regularizes latent sampling and recovers details. The improved PSNR and structure-related metrics over the No TCR variant further indicate that TCR does not reintroduce degradation artifacts. Overall, the full model performs best by combining fusionaware supervision with structure-aware rectification.

4.5 Discussion of Emergent Capabilities

Beyond the supervised degraded-IVIF objective, we observe several qualitative behaviors that are not directly targeted by our fusion training data (Fig. 6):

Low-resolution recovery. Although low-resolution degradation is not included in our fusion training set, AIR-Fusion still enhances perceptual sharpness and alleviates blocky pixelation when presented with LR inputs (top). We attribute this behavior to the restoration priors already encoded in the frozen restoration-capable diffusion backbone, which AIRFusion can effectively elicit and reuse during fusion through our conditional modulation. This suggests that AIR-Fusion is able to transfer general-purpose restoration capabilities to unseen degradations without task-specific retraining.

Text-guided robustness. This experiment (middle) is designed to compare the language controllability of different methods using a natural human instruction such as "remove fog from the image." Under this prompt, AIR-Fusion responds more faithfully and consistently. In contrast, TextIF exhibits weaker prompt-following robustness. These observations suggest that AIR-Fusion better aligns the naturallanguage intent with the restoration behavior during fusion, enabling more reliable instruction-driven editing without drifting away from the fusion objective.

Selective restoration under compound degradations. As shown in the bottom row (e.g., "remove blur/haze from the image"), when prompted with a haze-specific command, AIR-Fusion selectively lifts the global haze veil and restores scene visibility, yet keeps the blur characteristics largely unchanged. Conversely, when using a blur-specific instruction, AIR-Fusion primarily sharpens local structures and recovers fine details, while leaving the atmospheric veil mostly intact instead of over-dehazing. This behavior indicates that AIRFusion can disentangle restoration actions via language, enabling fine-grained, instruction-faithful control.

5 Conclusion

We presented AIR-Fusion, a generative capability elicitation framework for degraded infrared-visible image fusion that adapts a frozen restoration-capable latent diffusion backbone via parameter-efficient modules. CMBA aligns infrared cues and textual instructions with the frozen conditioning space for instruction-guided restoration-aware fusion, while TCR constrains denoising with a pixel-latent closed loop to preserve structures and recover fine details. Experiments across datasets and degradations verify consistent gains in fusion fidelity and robustness. Our current evaluation mainly targets visible-side degradations; degraded infrared inputs, dualmodality degradation, and misregistration remain important future directions, together with more efficient inference and broader real-world evaluation.

Acknowledgments

This work was sponsored by the National Natural Science Foundation of China (Nos. 62222608, 62436002, 62476198), the Tianjin Natural Science Funds for Distinguished Young Scholar (No. 23JCJQJC00270), the Zhejiang Provincial Natural Science Foundation of China (No. LD24F020004), and the Natural Science Foundation of Tianjin (No. 25JCYBJC00950).