Temporal-Synergistic Policy Optimization for Unsupervised Low-Light Image Enhancement

1 Introduction

Real-world low-light scenarios suffer from coupled degradation factors, including low signal-to-noise ratio, uneven illumination, and significant color deviation, rendering low-light image enhancement a highly ill-posed inverse problem. Beyond merely improving global brightness, the enhancement process must simultaneously suppress noise while preserving structural consistency and color naturalness. Traditional methods predominantly rely on fixed hand-crafted priors, which struggle to adapt to the complex and varying conditions of real-world scenes [Wu et al., 2022]. While supervised learning approaches have achieved remarkable progress [Wu et al., 2023], their application is severely constrained by the high cost of acquiring paired labeled data. Furthermore, synthetic degradation fails to accurately model the complex imaging characteristics of real-world environments. The resulting significant domain gap severely impairs the model's generalization capability.

∗Corresponding author.

In recent years, diffusion models have demonstrated significant potential in low-light image enhancement tasks. Existing methods typically rely on heuristic physical properties for conditional guidance or directly incorporate no-reference perceptual metrics into timestep-wise training objectives [Yi et al., 2023]. Fundamentally, these methods constitute proxy perceptual constraints: perceptual evaluation is performed in the final state, whereas the optimization process is distributed across the entire denoising trajectory. In unsupervised settings, this misalignment not only tends to amplify metric noise, leading to training instability, but also induces the model to generate visual artifacts in an attempt to exploit these metrics. Consequently, how to stably and effectively integrate human perceptual preferences into the diffusion generation process, without relying on paired data, remains a critical problem to be addressed.

In tasks such as text-to-image generation, where human subjective evaluation is difficult to explicitly model, reinforcement learning (RL) methods based on preference modeling have recently achieved significant success [Xu et al., 2023]. By formulating quality assessment as a terminal preference reward, RL can directly guide the model at the generative distribution level to align with human perception, bypassing the need to explicitly model the perceptual process. However, traditional RLHF methods, such as Proximal Policy Optimization (PPO) [Schulman et al., 2017], rely on an additional value network for advantage estimation, incurring significant computational and memory overheads. Group Relative Policy Optimization (GRPO) [Shao et al., 2024] mitigates this by leveraging multiple samples from the same input to estimate advantages based on intra-group relative performance, thereby eliminating the dependency on an explicit value network. This characteristic renders GRPO particularly suitable for integration with diffusion models to directly optimize terminal generation quality.

However, directly applying GRPO to low-light enhancement entails two critical challenges. First, the denoising process exhibits distinct temporal characteristics: global structures are established in the early stages, while details and textures are refined in the later stages. Consequently, fulltrajectory stochastic exploration proves inefficient. Furthermore, the trajectory-level advantage, derived solely from terminal feedback, is approximately uniformly distributed across all time steps. This ambiguity obscures the causal relationship between the terminal reward and specific denoising behaviors, leading to erroneous credit assignment [Luo et al., 2025]. Second, relying solely on a single human-preference reward makes the policy prone to reward hacking, where the model caters to perceptual metrics by deviating from original structures or introducing unnatural artifacts. Moreover, directly aggregating multi-dimensional rewards within the GRPO framework presents inherent limitations. Naive aggregation tends to blur subtle discrepancies among different reward components, resulting in reward signal collapse [Liu et al., 2026b]. Simultaneously, it allows high-variance rewards to dominate the optimization direction, thereby suppressing gradient signals from other critical objectives [Lyu et al., 2025].

In this paper, we propose an unsupervised low-light image enhancement framework based on Group Relative Policy Optimization (GRPO), as shown in Fig. 1, with the advantage computation detailed in Eqs. (14) and (17). We introduce online reinforcement learning into the low-light enhancement task, modeling human preferences as terminal rewards while requiring only low-light image inputs. By directly optimizing the generative distribution, the model generates highquality results that are aesthetically pleasing, well-exposed, and vividly colored. To mitigate the issues of phasic exploration and credit assignment during diffusion denoising, and inspired by its success in generation tasks [Li et al., 2025], we introduce the Sliding Window Hybrid ODE-SDE Sampling strategy. This strategy defines a dynamic time window that slides along the denoising direction, restricting stochastic exploration to the sub-interval covered by the window while maintaining deterministic sampling in the remaining stages. Through this mechanism, we achieve an ordered, coarse-tofine exploration and enable the precise attribution of terminal reward differences to the exploratory noise within the window, thereby significantly alleviating the credit assignment problem.

We construct a Synergistic Perception-Fidelity Reward that encompasses human perceptual preferences, content fidelity, illumination uniformity, and color naturalness, to effectively constrain the model's exploration space while enhancing visual quality. To achieve stable and balanced multi-reward optimization, we adopt Independent Advantage Estimation. This strategy calculates normalized relative advantages separately within each sampling group, mapping distinct reward components to a consistent magnitude. This ensures that policy updates rely on relative performance rather than raw numerical discrepancies, thereby effectively mitigating the gradient suppression problem in multi-objective optimization. Furthermore, based on our experimental setup, we perform a Trajectory Branching Analysis (Fig. 3) to quantify the terminal reward variance induced by independent stochastic noise injection at specific timesteps. Experimental results indicate that this variance exhibits a distinct non-uniform distribution along the time axis. Building on this observation, we propose a Temporal-Aware Dynamic Weighting Mechanism. By assigning higher weights to these critical stages during optimization, we guide the model to prioritize the denoising intervals that most significantly impact final perceptual quality, thereby efficiently achieving synergistic optimization of multiple rewards. Our contributions can be summarized as follows:

1. We pioneer the application of GRPO to unsupervised low-light image enhancement and, by integrating the Sliding Window Hybrid ODE-SDE Sampling strategy, achieve efficient coarse-to-fine exploration and precise advantage attribution.
2. We construct a Synergistic Perception-Fidelity Reward and, coupled with Independent Advantage Estimation and the Temporal-Aware Dynamic Weighting Mechanism, achieve stable and precise synergistic optimization of multiple rewards.
3. Extensive experiments on multiple public real-world low-light datasets demonstrate that the proposed framework significantly improves the perceptual performance of existing generative models, yielding enhanced results that better align with human aesthetics.

2 RelatedWork

2.1 Low-light Image Enhancement

Traditional low-light image enhancement (LLIE) methods, such as histogram equalization [Wu et al., 2024], gamma correction [Wang et al., 2023], and Retinex models [Ren et al., 2020], rely on hand-crafted priors or fixed transformations. When applied to perceptually challenging realworld scenes [Ji et al., 2024], these approaches often suffer from over-enhancement or color distortion [Wu et al., 2022]. While deep learning-based LLIE methods improve accuracy and robustness [Li et al., 2024], supervised paradigms depend on large-scale paired datasets, which are costly to collect and often limit generalization under real-world distribution shifts [Mumuni et al., 2024; Lu et al., 2025]. In contrast, unsupervised methods [Ma et al., 2022] eliminate the need for paired training data by introducing illumination [Shi et al., 2024] or physical priors [Wang et al., 2024] as constraints, or by adopting generative architectures [Jiang et al., 2021]. Consequently, unsupervised approaches often generalize better in complex real-world environments.

2.2 Preference Alignment

Preference-driven reinforcement learning has been effective in image generation [Xu et al., 2023]. DanceGRPO [Xue et al., 2025] and FlowGRPO [Liu et al., 2026a] apply GRPO to visual generation by reformulating the sampling process via SDEs. Recent studies have explored optimization strategies for the advantage attribution mechanism within the GRPO framework. Chunk-GRPO [Luo et al., 2025] partitions denoising sequences into independent sub-chunks for localized estimation, effectively mitigating high variance and credit assignment ambiguity caused by global attribution. TEMPFLOW-GRPO [He et al., 2025] implements time-sensitive advantage assignment by utilizing noise-aware weighting and trajectory branching to account for the varying contributions of different timesteps to generation quality. Dynamic-TreeRPO [Fu et al., 2025] uses structured treebased sampling in the Markov Decision Process, providing accurate reference baselines through shared nodes and breaking the attribution bottleneck of independent trajectories.

3 Method

The proposed online reinforcement learning enhancement framework initializes its policy network with LightenDiffusion, a pre-trained unsupervised diffusion-based model. The overall pipeline is illustrated in Fig. 2.

3.1 Diffusion Policy Optimization

RL Formulation. Diffusion models define a forward process that gradually adds noise to perturb original data into a simple distribution. The model is then trained to approximate the reverse generation process, which reconstructs the original data through progressive denoising. DDIMs [Song et al., 2020] are defined as a class of Non-Markovian diffusion processes.

Specifically, given a noisy sample xt and a trained noise prediction network ϵθ(xt, t), the reverse denoising process of DDIM from t to t - 1 can be expressed as:

Here, αt denotes the cumulative product of the noise schedule, and ϵt ∼ N(0, I) represents standard Gaussian noise. The parameter σt controls the stochasticity of the sampling and is defined as:

where η ∈ [0, 1] is a hyperparameter. When η = 0, the sampling process is deterministic. Conversely, when η > 0, a random noise term is introduced to add stochasticity to the generation trajectory. Such controlled stochasticity is pivotal for the integration of reinforcement learning, offering the requisite diversity for policy exploration.

Following current diffusion-based reinforcement learning methods [Black et al., 2024], we utilize the denoising network of the diffusion model as the policy π(at|st). We reframe the diffusion process as a multi-step Markov Decision Process (MDP) defined by the tuple (S, A, ρ0, π, P, R):

Here, state st consists of the condition c, current diffusion timestep t, and latent variable xt. The action at is defined as the denoised latent variable at the next reverse diffusion step, xt-1. The initial MDP state s0 is sampled from the condition distribution p(c), the initial diffusion timestep δT , and standard Gaussian noise N(0, I), corresponding to the starting point of the denoising process.

The state transition P is deterministic. Once the policy takes action at by selecting the next reverse denoising state, the system deterministically transitions from st to st-1 = (c, t - 1, at). The reward function R(st, at) is defined as a sparse reward. It returns the quality score r(x0, c) of the generated image only at the end of the denoising process (t = 0) and remains zero at all other time steps.

We optimize the diffusion model to learn a policy πθ that maximizes the expected reward of the generated samples at the final step. This is typically formulated as maximizing the following objective function:

where p(c) represents the distribution of input conditions, and pθ(x0|c) denotes the sample distribution generated by the model.

Standard GRPO. To efficiently maximize the objective function defined in Eq. (4), we employ Group Relative Policy Optimization (GRPO). GRPO eliminates the need for a parametric value function by leveraging group-based statistics to estimate the baseline and advantage directly from sampled trajectories.

Specifically, for each input condition c, we generate a group of M independent trajectories {τi}Mi=1 using the current policy πθ, resulting in a set of enhanced images {xi0 }Mi=1 . We then compute the reward R(xi0 , c) for each sample. The advantage Ai for the i-th sample is estimated by normalizing its reward against the group statistics:

where µ(R) and σ(R) denote the mean and standard deviation of the rewards within the group, and δ is a small constant for numerical stability.

To update the policy, we optimize the following surrogate objective, which combines the clipped advantage for stability and a KL-divergence penalty to ensure the model stays close to the reference pre-trained distribution πref:

Here, ρit (θ) = πθ(ait |sit ) πθold (ait |sit ) is the probability ratio, T denotes the total number of denoising timesteps, ϵ is the clipping threshold, and β controls the strength of the KL regularization.

3.2 SlidingWindowHybrid ODE-SDE Sampling

The standard diffusion-based GRPO method injects randomness at every timestep t ∈ {T, T -1, . . . , 0} of the denoising process. The trajectory generated by this sampling method is defined as τ = (sT , aT , sT-1, aT-1, . . . , s0, a0). The sparse terminal reward r(x0, c), calculated at the end of the trajectory, is uniformly distributed to the actions of all timesteps. This time-uniform attribution method overlooks the inherent temporal characteristics of the diffusion denoising process. To achieve precise credit assignment, inspired by previous work [Li et al., 2025], we adopt a sliding window mixed ODE-SDE sampling strategy.

We concentrate the injection of stochasticity within a dynamic sub-interval of the denoising timesteps. We define a sliding window W(k) that covers a portion of the timesteps in the denoising process:

where k is the starting timestep index of the sliding window, and L is the window span.

Based on this window, we construct a hybrid ODE-SDE sampling method. Starting from the initial noise xT , the trajectory evolves deterministically until reaching the specified window. To implement the dynamic switching of sampling, we define the hyperparameter η in Eq. (2) in the following time-step dependent form:

Specifically, upon entering the window (t ∈ W(k)), we set ηt to a preset constant ηset > 0. According to Eq. (2), this yields σt > 0. This also activates the random noise term σtϵt in Eq. (1). The model shifts to performing SDE sampling to explore the policy space within the current timestep window. After leaving the window, deterministic sampling is resumed until the final image x0 is generated. This strategy shortens the effective sequence length of the Markov Decision Process (MDP) to the subset τk = (sk, ak, sk-1, ak-1, . . . , sk-L, ak-L).

The window position k slides dynamically from T to L to guide an ordered coarse-to-fine exploration. Governed by the sliding interval I and step size ∆k, the update rule is defined as:

For brevity, we omit the subscript iter and use k to represent the current window position in subsequent discussions. By concentrating randomness solely within W(k), any variation in the final image (and resulting reward) is directly attributable to the noise in this specific window, thereby achieving precise credit assignment.

3.3 Synergistic Perception-Fidelity Reward

Human Preference Reward (RP). We employ MUSIQ, a no-reference image quality assessment metric, as the perceptual reward to estimate the visual quality of generated images without requiring paired ground truth. This reward encourages visually plausible and perceptually preferred enhancement results.

Fidelity Reward (RF). We employ LPIPS as a fidelity constraint to better capture human-perceptual structural differences. Due to the lack of ground truth, we compute the negative LPIPS distance between the generated image x0 and a reference image xref produced by a pre-trained model.

Exposure Reward (RE). To achieve adaptive dynamic range adjustment, following [Lin et al., 2025], we utilize a spatially variant exposure map E to constrain the local exposure of the generated image ˆx0. Unlike global adjustments, E assigns non-uniform target values based on the input luminance Y . This balances underexposed and overexposed regions. Based on the statistics of normal-light images, E is calculated as:

We define the exposure reward as the negative squared difference between the mean exposure of the output image and the target map E:

Color Reward (RC). To prevent color distortion, we rely on the Retinex theory. We utilize the property that reflectance R

is independent of illumination changes as a physical consistency prior. We employ a pre-trained decomposition network F(·) (RNet) to extract reflectance features. We enforce consistency between the reflectance of the generated image x0 and the input condition c using the following reward:

3.4 Temporal-Aware Independent Advantage Aggregation

Independent Advantage Estimation. Traditional multireward reinforcement learning methods typically calculate the total reward by performing a weighted summation of all reward components:

Subsequently, group-wise normalization is applied to this total reward using Eq. (5) to calculate the relative advantage. However, significant discrepancies exist in the numerical scales and variance distributions of different reward functions. Direct summation causes reward components with high variance to dominate the optimization direction, suppressing gradient signals from other critical objectives.

To address this issue, we compute the advantage values for each reward independently. Specifically, for each input condition c, we sample a group of independent trajectories {τi}Mi=1 using the current policy πθ. Through this sampling process, we obtain a set of enhanced images {xi0 }Mi=1 . For the j-th reward objective, we first calculate its raw reward value Rj(xi0 , c) within the group, and then compute its normalized relative advantage Aij :

where µj and σj denote the mean and standard deviation of that reward within the group, respectively, and δ is a small constant for numerical stability. In this manner, each reward component is rescaled to a consistent magnitude.

Temporal-Aware Dynamic Weighting. After obtaining the independent normalized advantages, direct linear weighted summation fails to reflect the non-uniform contribution of different denoising stages to the final generation quality. To precisely quantify the contribution of different time steps to the final perceptual reward, we employ singlevariable perturbation analysis. Specifically, we employ deterministic ODE sampling for the denoising trajectory, introducing stochastic noise exclusively at a targeted single timestep. We then assess the significance of this timestep by measuring the variance of the terminal perceptual rewards. As shown in Fig. 3, the reward variance exhibits a non-uniform distribution along the time axis and reaches a peak during the earlyto-intermediate stage. This result reveals the dominant role of this stage in optimizing perceptual quality. To this end, we design a time-step-based dynamic weighting mechanism:

Here, t denotes the index of the elapsed sampling step starting from the initial noise. Eq. (15) defines a temporal importance distribution that peaks at the early-to-intermediate stage. Considering that our sampling strategy is based on a sliding window W(k) of width L, the weights of individual time steps need to be aggregated into window-level weights. Therefore, we define the perceptual dynamic weighting coefficient wP(k) for the current sliding window W(k) as the arithmetic mean of the weights of all discrete time steps within the window:

Finally, the aggregated advantage Aik for the i-th trajectory under sliding window k consists of the dynamically adjusted human preference advantage and other constraint terms with fixed weights:

3.5 Optimization Objective

We perform exploration within the dynamically sliding denoising sub-interval W(k) and guide policy updates using the advantage Aik . This advantage effectively integrates the Synergistic Perception-Fidelity Reward via independent normalization and time-aware dynamic weighting mechanisms. Following DanceGRPO [Xue et al., 2025], we omit the explicit KL regularization term to simplify the optimization objective. The final optimization objective is defined as:

The detailed optimization algorithm is provided in the Supplementary Material.

4 Experiments

4.1 Training and Testing Data

The proposed enhancement framework further optimizes the LightenDiffusion backbone using the publicly available EnlightenGAN dataset [Jiang et al., 2021]. The dataset contains over 1,000 unpaired images from low-light and normal-light domains, while only the low-light images are used as training inputs. To comprehensively evaluate the performance of the proposed framework, extensive experiments were conducted on a variety of benchmark datasets. The test suites include paired datasets such as LOL-v1 [Wei et al., 2018] and LOLv2 [Yang et al., 2021], as well as real-world unpaired datasets, including DICM [Lee et al., 2013], LIME [Guo et al., 2016], MEF [Ma et al., 2015], NPE [Wang et al., 2013], and VV [Vonikakis et al., 2018]. Due to space constraints, detailed evaluation results on some datasets are provided in the supplementary materials.

4.2 ComparisonMethods and Analysis

To validate the effectiveness and generalization of the proposed framework, we designed comparative experiments from two perspectives. To demonstrate the optimization efficacy in improving perceptual quality, we conducted a direct comparison between the optimized model and the initial policy. To evaluate the overall competitiveness of our algorithm, we selected eight advanced unsupervised low-light image enhancement methods as baselines for performance benchmarking, specifically including: Zero-DCE [Guo et al., 2020], Zero-DCE++ [Li et al., 2021], RUAS [Liu et al., 2021], SCI [Ma et al., 2022], PairLIE [Fu et al., 2023], CUE [Zheng et al., 2023], QuadPrior [Wang et al., 2024], and IUJDE [Li and Wang, 2025].

Table 1 shows that the optimized model improves nearly all perceptual metrics across five unpaired datasets. Visual comparisons in Fig. 4 and Fig. 5 demonstrate that our method generates images that are visually more natural and richer in detail. Furthermore, quantitative results on the LOL-v1 dataset in Table 2 highlight our advantages in perceptual metrics. Although slightly outperformed by other methods in certain metrics, our approach achieves overall improvements over the baseline, especially in perceptual quality. This indicates that the proposed strategy effectively exploits the model's potential, maximizing perceptual quality under constrained base performance.

4.3 Ablation Study

All experiments were conducted on the LOL-v1 test set. As shown in Table 3, relying solely on the perceptual reward RP yields the highest MUSIQ but extremely low PSNR, confirming the reward hacking phenomenon. Incorporating the fidelity reward RF significantly improves PSNR, effectively constraining structural consistency. Consequently, the complete synergistic reward mechanism achieves the optimal perception-fidelity balance, validating the effectiveness of Independent Advantage Estimation in mitigating multi-objective signal collapse. Regarding the sampling and weighting strategies in Table 4, our full model improves all three metrics compared with both the w/o SWHS and w/o TADW variants. These results indicate that sliding-window stochastic exploration helps alleviate credit-assignment ambiguity, while Temporal-Aware Dynamic Weighting further strengthens optimization by assigning higher weights to critical denoising stages.

5 Conclusion

We proposed an unsupervised framework based on GRPO to directly optimize diffusion policies using terminal rewards. To mitigate credit assignment in trajectory-wise optimization, we introduced Sliding Window Hybrid ODE-SDE Sampling, confining stochasticity to dynamic sub-intervals for efficient exploration. Furthermore, to enforce fidelity and prevent reward hacking, we constructed a Synergistic PerceptionFidelity Reward with Independent Advantage Estimation and Temporal-Aware Dynamic Weighting. This approach mitigates signal collapse in multi-objective optimization. Experiments demonstrate that our method generates high-quality results aligning with human preferences without paired data.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 62422609, and in part by the Fundamental Research Funds for the Central Universities under Grant WK2102026001.