A Case for Validation Buffer in Pessimistic Actor-Critic

1 Introduction

Approximation errors, although ubiquitous in machine learning, are particularly exaggerated in the context of value-based Reinforcement Learning (RL). Such exaggeration stems from Temporal Difference (TD) in which the critic is supervised via value estimate calculated at a different state [Silver et al., 2014], [Mnih et al., 2015].

Inaccuracies in this estimate lead to propagated errors in state-action updates, and the use of maximization in value estimation inherently promotes overestimation. Addressing such overestimation has proven to be an effective strategy in discrete and continuous action environments [Hasselt, 2010], [Van Hasselt et al., 2016], [Hessel et al., 2018], [Haarnoja et al., 2018]. Clipped Double Q-Learning (CDQL), a common solution to overestimation in continuous action actorcritic algorithms aims to mitigate overestimation by balancing errors against a pessimistic lower bound value approximation [Fujimoto et al., 2018]. However, challenges arise if the lower bound is insufficiently pessimistic, leading to continued overestimation, or overly pessimistic, causing underestimation [Cetin and Celiktutan, 2023]. The latter, though less recognized, can significantly reduce sample efficiency and degrade actor-critic agents' performance in both low and high replay ratio settings which we show in Figure 1.

In this paper, we investigate the relationship between pessimism in Q-value approximation and error accumulation in critic networks. We start by characterization of existing strategies for pessimism adjustment. Furthermore, we analyze the pessimistic critic approximation error and show that such error can be represented recursively forming a fixedpoint model, akin to values. This recursive representation helps us highlight the bias inherent in pessimistic actor-critic algorithms, examine their convergence dynamics, and identify the conditions under which pessimistic critics can achieve zero error. Building on these insights, we propose the Validation Pessimism Learning (VPL) algorithm. VPL employs a small validation replay buffer to adjust the pessimism levels online, aiming to minimize the approximation error of critic targets while preventing overfitting to accumulated experience. We evaluate VPL against existing pessimism adjustment methods on DeepMind control [Tassa et al., 2018] and MetaWorld [Yu et al., 2020]. Our findings demonstrate that VPL not only achieves performance improvements but also exhibits less sensitivity to hyperparameters compared to the baseline algorithms. We list our contributions below:

• We show that critic error can be defined recursively through a fixed-point model. We demonstrate that pessimistic TD learning, a method often used in RL, converges to the true value under strict conditions.
• We present an empirical analysis showing that the performance loss associated with not including every transition in the replay buffer diminishes as training progresses. This observation challenges the traditional belief that every transition must be used in value learning for sample-efficient RL and builds a case for employing

a validation buffer in an online RL setting.

• We propose VPL, an algorithm that uses a small validation buffer for online adjustment of pessimism associated with lower bound Q-value approximation. We test the effectiveness of VPL and other pessimism adjustment strategies in low and high replay regimes. We show that VPL offers performance improvements across a variety of locomotion and manipulation tasks.

2 Background

2.1 MaximumEntropy Reinforcement Learning

We consider an infinite-horizon Markov Decision Process (MDP) [Puterman, 2014], represented by the tuple (S↪ A↪ r↪ p0↪ ω), where S and A are continuous state and action spaces, rs↪a is the reward, p0(s) is the initial state distribution, and ω → (0↪ 1] is the discount factor. The policy ε(a♣s) is a distribution of actions given states. The goal of Maximum Entropy Reinforcement Learning (MaxEnt RL) [Haarnoja et al., 2017] is to maximize the expected cumulative discounted return augmented with an entropy term:

where ϑ controls the balance between reward and entropy [Haarnoja et al., 2018]. The soft Q-value is defined as:

and the soft value is given by:

MaxEnt RL algorithms, such as Soft Actor-Critic (SAC), parameterize the policy (actor) εε and the Q-value (critic) Qϑ, which are optimized iteratively using objectives derived from policy iteration [Haarnoja et al., 2018]. The critic ensemble is often used to address Q-value overestimation via Clipped Double Q-Learning (CDQL) [Fujimoto et al., 2018]. In CDQL, the value lower bound is approximated as:

2.2 PessimismAdjustment

Building upon CDQL, recent works propose parameterizing the lower bound of Q-values as:

where Qθϑ and Qϖϑ represent the mean and standard deviation of the critic ensemble, respectively [Ciosek et al., 2019], [Nauman and Cygan, 2023]. This formulation allows the adjustment of the pessimism level ϖ, controlling the influence of critic disagreement. Algorithms such as Generalized Pessimism Learning (GPL) [Cetin and Celiktutan, 2023] and OnPolicy Pessimism Learning (OPL) [Kuznetsov et al., 2021] optimize ϖ online by aligning pessimism with approximation error. For example, GPL treats this adjustment as a dual optimization problem:

where V lb ϑ (s↔ ) ↓ Qθϑ (s↪ a) ↑ ϖQϖϑ (s↪ a) ↑ ϑ log εε(a♣s) and sg denotes the stop-gradient operator. This approach risks overfitting, as the adjustment heavily relies on critic outputs. Alternative methods, such as Tactical Optimism and Pessimism (TOP) [Moskovitz et al., 2021], use external bandit controllers for ϖ, but they can be less effective with continuous pessimism adjustments. We include additional discussion of critic disagreement in the appendix, as well as key comparisons of pessimism adjustment methods in Table 2.

3 Approximation Error and Pessimism

In this section, we focus on the analysis of critic approximation errors within the framework of pessimistic updates. For simplicity, we consider a fixed policy εε and use V (s) and Q(s↪ a) to represent the value and Q-value under this policy. We define the mean and lower bound approximation errors denoted as U θ ϑ and U lb ϑ respectively:

Here, Q(s↪ a) denotes the true Q-value, the term Qθϑ (s↪ a) represents the mean Q-value estimated by an ensemble of k critics, calculated as Qθϑ (s↪ a) = 1k ∑k Qiϑ (s↪ a), and Qlb ϑ (s↪ a) is the lower bound Q-value as defined in Equation 5. Additionally, we introduce the mean and lower bound temporal critic errors, denoted as uθϑ and ulb ϑ , respectively:

These temporal critic errors quantify the deviation between the Q-values Qθϑ (s↪ a) and the mean or lower bound Temporal Difference (TD) targets. The value V lb ϑ (s) is equal to the expected value of Qlb ϑ (s↪ a) over all state-action pairs under policy ε, such that V lb ϑ (s) = EωQlb ϑ (s↪ a) ↑ log εε(a♣s).

Lemma 3.1 (Approximation error operator). Given policy ε, k on-policy q-value approximations Q1ϑ ↪Q2ϑ ↪ ▷▷▷↪Qkϑ , sample mean Qθϑ and standard deviation Qϖϑ , the mean and lower bound approximation errors follow a recursive formula:

We expand on Lemma 3.1 in Appendix A. The lemma reveals that approximation errors exhibit a recurrent pattern analogous to Q-values. Specifically, the temporal errors function as an immediate signal, akin to rewards, while the future approximation errors serve as the bootstrap signal. Furthermore, this observation formalizes the intuitive concept that minimizing the lower-bound error necessitates a precise calibration of the pessimistic correction against the temporal error and the approximation errors of subsequent states.

Key Insight

Lemma 3.1 shows that approximation errors propagate through recursive updates, just as Q-values propagate through Bellman equations. The temporal errors, uθϑ and ulb ϑ , act as immediate signals, while future approximation errors (weighted by ω) bootstrap over time. This highlights the need to balance immediate errors and long-term disagreement for effective learning.

It can be shown that similarly to the Bellman operator, both mean and lower bound error approximation operators are monotonic contractions:

Theorem 3.2 (Approximation error contraction). Let F be the space of functions on domain S ↗ A. We define the mean error and lower bound error operators U θ ↪ U lb : F ↘ F as:

Above, f(s↪ a) : S ↗ A ↘ R represents an estimate of the approximation error, and we assume that Qϖϑ (s↪ a) ↘ 0 as training progresses. Then it follows that both U θ and U lb are monotonic contractions for any f1 and f2:

We provide the relevant derivations in Appendix A.

Key Insight

Theorem 3.2 formalizes that both mean and lowerbound approximation error operators are contractions. This ensures that repeated applications of these operators converge to a fixed point. Intuitively, this means that as training progresses, the approximation errors stabilize, allowing the model to converge to a consistent value function.

As follows from Theorem 3.2, repeated application of the approximation error operator yields a Cauchy sequence, and therefore leads to a fixed point:

Corollary 3.3 (Approximation error fixed point). We denote repeated k applications of either approximation error operator to function f as Uk(f). Then, due to Banach fixed point theorem:

The corollary shows that the approximation error of values can be effectively modeled using a fixed-point approach, analogous to treating values themselves. The potential ramifications and applications of this concept are further explored in Appendix A. Principally, the convergence of a pessimistic value model signifies that the approximation errors converge to zero, implying U θ ϑ = U lb ϑ = 0. The convergence proof of CDQL indicates that the value model should align with the true on-policy values under the conventional Q-learning convergence assumptions [Watkins and Dayan, 1992], [Fujimoto et al., 2018]. Lemma 3.1 explicitly shows that for all s, a and s↔ , both approximation errors equate to zero iff the following conditions are satisfied:

Figure 2: High-level overview of the proposed approach. After each environment step, the transition is stored in either the training buffer (used for updating actor-critic modules) or the validation buffer (used for updating the pessimism module). The pessimism is updated via a "reverse" TD loss, optimisation of which on the training buffer would be prone to overfitting.

The convergence of a pessimistic model necessitates either the absence of critic ensemble disagreement (i.e., Qϖϑ (s↪ a) = 0 for all state-action pairs) or an algorithmic ability to diminish the level of pessimism over time, culminating in ϖ = 0 asymptotically. We emphasize that this pertains specifically to convergence toward the zero-error fixed point defined in Lemma 3.1; an algorithm may still converge to a biased solution even if these conditions are not met, but exact fixed-point convergence with zero approximation error is unattainable otherwise. Figure 10 shows that the critic disagreement does not completely diminish on popular DeepMind Control and MetaWorld benchmarks. Given the improbability of achieving zero critic disagreement in overparameterized deep RL contexts, the adjustment of ϖ emerges as a compelling strategy. Additionally, it can be demonstrated that under the scenario of critic underestimation, the lower-bound error exceeds the mean approximation error:

The inequality follows from the third formula from Lemma 3.1, where ϖ ⇑ 0 and Qϖ (s↪ a) ⇑ 0 as the standard deviation of the critic ensemble. We refer the reader to Appendix B for detailed proof.

As follows, pessimistic learning is advantageous only in overestimation, whereas it becomes detrimental in cases of underestimation. To this end, the pessimism levels should be adjusted in tandem with changes in the approximation errors. In practical terms, achieving a zero approximation error for either mean or lower bound is unrealistic. Given that Uϑ(s↪ a) → R, one might be interested in optimization of norm of U θ ϑ (s↪ a) or U lb ϑ (s↪ a). This leads to the possibility of defining an "optimal" level of pessimism, where optimality is considered in relation to minimizing the respective approximation error norm. We note that our analysis yields a different approach to updating pessimism as compared to the method derived from dual optimization [Cetin and Celiktutan, 2023], which we discuss in Section 2.2.

Key Insight

Equation 10 highlights that pessimistic updates are beneficial in situations of overestimation, but they may harm learning when underestimation occurs. Therefore, adjusting the level of pessimism dynamically is critical to balance the benefits of avoiding overestimation with the risks of underestimation.

4 Validation PessimismLearning Algorithm

Building on the analysis conducted in the previous Section, we propose the Validation Pessimism Learning module (VPL). The goal of the VPL module is to adjust the pessimism parameter such that the critic targets (lower bound Qvalue approximation) has the least approximation error. As such, VPL can be used as an alternative to CDQL or GPL in conjuction with any off-policy actor-critic algorithm. For our analysis, we utilize the Soft Actor-Critic (SAC) [Haarnoja et al., 2018] as the backbone algorithm. VPL is based on a simple premise of adjusting pessimism via a TD loss. Given that the critic concurrently optimizes this loss function, such setup is especially prone to overfitting. To mitigate this, the optimization of the pessimism parameter is conducted on a distinct set of validation data, which remains unseen by the actor-critic modules. From a theoretical standpoint, VPL can be interpreted as a strategy for pessimism model selection, with the selection process aimed at minimizing the lower bound approximation error delineated in the previous section. A critical aspect of VPL involves conducting the pessimism model selection on validation data. The model selection is achieved through gradient-based optimization of the proposed pessimism loss. The utilization of validation data in this process reduces the probability of overfitting to bootstrapped supervision signals used by TD learning. We summarize VPL approach in Figure 2 and share pseudo-code in Section B.1, where we colour changes wrt. regular SAC.

4.1 Validation Buffer

The employment of validation data is a well-established practice in supervised learning frameworks [Bishop and Nasrabadi, 2006]. It serves a dual purpose: providing an unbiased assessment of model performance trained on the training dataset, and facilitating regularization techniques such as early stopping [Prechelt, 2002] or hyperparameter tuning [Bergstra and Bengio, 2012]. However, the integration of validation data entails a trade-off, notably the reduction of the training set size. In supervised learning, the regret associated with decreasing the training set can be quantitatively evaluated through the lens of neural scaling laws [Rosenfeld et al., 2019]. Such regret is, to the best of our knowledge, a relatively understudied area in the context of online RL. In online RL, the notion of a validation buffer is not popular, primarily due to the requisite sacrifice of actor-critic learning on the validation transitions. Given inherent sample inefficiency of RL, this cost is often deemed as overly burdening. Contrary to supervised learning setup, RL is characterized by a high correlation between successive samples, thereby diminishing the marginal utility of processing additional samples from the same trajectory. Consequently, we posit that in online RL, the cost associated with the use of validation data can be counterbalanced, provided the validation data is leveraged to enhance the learning process. In the case of the VPL, we allocate the validation transitions exclusively for the adjustment of the pessimism parameter. This approach presents a novel utilization of validation data in online RL in a manner that is RL-specific, diverging from supervised methodologies.

4.2 PessimismUpdate Rule

The persistence of critic disagreement throughout training implies that the standard convergence guarantees of the pessimistic temporal difference update towards on-policy values are not upheld when ϖ ⇓= 0. Moreover, in cases where minimizing the mean approximation error is not achievable, particularly in scenarios characterized by strong overestimation, the presence of non-zero critic disagreement can be leveraged to decrease the lower bound approximation error by increasing ϖ. This observation forms the basis for our proposed method of adjusting ϖ. The aim is to minimize the expected lower bound approximation error U lb ϑ (s↪ a), formulated as follows:

Unfortunately, obtaining U lb ϑ (s↪ a) is challenging as it necessitates an estimate of the true on-policy Q-value. Typically, such estimates are derived through methods like MonteCarlo (MC) rollouts, TD(n), or TD(ϱ), with MC being the only unbiased method. However, in the context of off-policy learning or non-terminating environments, employing MC rollouts is impractical. Consequently, we leverage the simple approach proposed by [Cetin and Celiktutan, 2023] in which it is assumed that the critic output for prerecorded off-policy actions is unbiased. Therefore, we assume that Qω (s↪ a) = Qθϑ (s↪ a) for actions that do not maximize the output of the policy. Additionally, akin to the approach in offpolicy actor-critic algorithms, the policy-induced distribution is approximated using an off-policy replay buffer. This approach leads to the formulation of the following:

In this formulation, DV represents the validation replay buffer, with s↪ a↪ s↔ denoting transitions sampled from this buffer. In line with other stochastic policy algorithms, we further approximate V lb ϑ (s↔ ) with the critic output for a single action a↔ ↔ εε(a↔ ♣s↔ ). As follows, VPL adjusts the pessimism under the assumption that Qθϑ (s↪ a) is a good representation of Qω (s↪ a). Since the actions at which Qθϑ (s↪ a) is evaluated are sampled from the validation buffer and are off-policy, these actions are likely to produce less overestimation than the adversarial actions sampled from a value-maximizing policy.

The inclusion of a square in the loss function ensures that the optimization remains non-negative and convex with respect to ϖ, focusing the updates on reducing significant errors, which are particularly impactful in reinforcement learning scenarios. Additionally, removing the stop-gradient operator allows ϖ to be updated dynamically based on both the approximation error and the disagreement among critics (Qϖ ), tightly coupling the level of pessimism to the degree of uncertainty in the critic estimates. This ensures that the level of pessimism is adjusted adaptively during training, aligning with the intuition that higher disagreement indicates greater uncertainty, necessitating increased pessimism.

While we acknowledge that providing a strict mathematical justification for all aspects of the proposed rule is challenging, our empirical results (Figure 3) demonstrate its practical effectiveness in reducing overfitting and improving performance across diverse environments. VPL mitigates the risk of overfitting by computing the pessimism loss exclusively on validation samples that are not used by the actorcritic modules. Unlike GPL, VPL fully incorporates critics' disagreement into the optimization process and dynamically adjusts ϖ, enabling more precise control of the level of pessimism based on the observed uncertainty.

5 Experiments

Our experiments are based on the JaxRL codebase [Kostrikov, 2021]. Since all considered algorithms use SR-SAC [D'Oro et al., 2022] as their backbone, we align the common hyperparameters with those recommended for Scaled-By-Resetting SAC (SR-SAC) as per [D'Oro et al., 2022]. This includes using the same network architectures and a two-critic ensemble in accordance with established practices [Fujimoto et al., 2018], [Haarnoja et al., 2018],

[Ciosek et al., 2019], [Moskovitz et al., 2021], [Cetin and Celiktutan, 2023]. We conduct our experiments in two environments: the DeepMind Control (DMC) suite [Tassa et al., 2018] and the single-task MetaWorld [Yu et al., 2020]. Our study encompasses two replay regimes: a compute-efficient setup with 2 gradient steps per environment step without resets, and a sample-efficient setup with 16 gradient steps per environment step, including full-parameter resets every 160k steps, as suggested by [D'Oro et al., 2022]. We provide robust analysis using the RLiable package [Agarwal et al., 2021] and detail the experimental setting in Appendix E.

5.1 Performance and Sample Efficiency

Firstly, we test the performance and sample efficiency of the proposed approach. To this end, we compare SR-SAC [D'Oro et al., 2022] (DMC state of the art) to four algorithms that extend SR-SAC with online pessimism adjustment: GPL [Cetin and Celiktutan, 2023]; OPL [Kuznetsov et al., 2021]; TOP [Moskovitz et al., 2021]; and VPL (the proposed approach). We run the tested algorithms in both replay regimes for 1mln environment steps on 20 medium to hard tasks (10 from DMC and 10 from MetaWorld). We discuss the chosen baselines in Sections 2.2 & C. We discuss hyperparameter selection in Appendix G and the tested tasks in F. We report the results of this experiment in Figures 3, 4 & 5. We find that the proposed approach surpasses baseline algorithms, demonstrating 48% and 27% higher performance than the baseline SR-SAC in low and high replay regimes, respectively. As depicted in Figure 4, VPL exhibits particular effectiveness in MetaWorld manipulation tasks, developing robust policies in environments where other approaches fail, such as the assembly task. To provide further practical context, we evaluate the computational overhead introduced by each of the tested pessimism adjustment methods. Table 1 shows that VPL maintains a low wall-clock overhead compared to SR-SAC, with only a 3▷5% increase in the low-replay regime and 3▷8% in the high-replay regime. This minimal computational overhead highlights VPL's suitability for large-scale and real-time reinforcement learning applications.

5.2 Validation Buffer Regret

To understand the impact of a validation buffer on online RL training, we analyze three distinct agent setups: baseline SRSAC, which operates without a validation buffer, thus updating actor-critic modules with all experienced transitions; regret SR-SAC, which maintains a validation buffer but does not employ validation transitions for pessimism adjustment; and SR-SAC-VPL, which not only maintains a validation buffer but also utilizes validation transitions for pessimism adjustment (we present these results in Figure 6).

This comparative analysis aims to isolate the performance loss attributable to the presence of a validation buffer and the efficiency gains derived from employing VPL for updating pessimism. We evaluate these agents in high-replay regime on 4 tasks (listed in Table 4) over 1mln environment steps, using varying ratios of validation to training samples, specifically at proportions of 1 128 , 1 32 , 18 , and 12 . The results for this experiment are presented in Figure 6. We observe that the regret associated with maintaining a validation buffer, and thus not utilizing it for actor-critic updates, diminishes over the course of training. Specifically, the regret SR-SAC reaches parity with the SR-SAC in performance for all validation proportions except at 12 . We note that the rate of regret reduction correlates with the size of the validation proportion, with smaller proportions converging to baseline performance more rapidly. When examining the effectiveness of pessimism adjustment, we observe its most pronounced impact during the early stages of training. This trend aligns with the expectation of reducing critic disagreement over time. Additionally, the extent of performance gain appears to be influenced by the size of the validation buffer, where larger proportions yield greater improvements. This effect is likely due to the increased diversity of environment transitions available for pessimism adjustment in larger buffers. When considering the combined effects on performance, our findings indicate that, except for the 12 proportion, all validation proportions successfully compensate for the performance loss due to validation buffer maintenance. This result is in line with the broader experimental results presented in Figures 3 & 5.

5.3 Other Experiments

We investigate the sensitivity of VPL to varying pessimism learning rates as compared other pessimism adjustment algorithms. Given the dependency of such learning rate on reward scales and environmental dynamics, determining an optimal rate a priori is challenging, which is a significant restriction for practical applications. To address this, we test the performance of VPL, GPL, and OPL across four environments detailed in Table 4 in the high-replay regime. We evaluate agents after 500k environments steps for learning rates of [5e↑5↪ 5e↑4↪ 5e↑3↪ 5e↑2]. The results, presented in Figure 7, indicate that VPL exhibits less sensitivity to changes in the pessimism learning rate than the other considered algorithms. Furthermore, we investigate the importance of the two proposed design elements: the use of a validation buffer and the VPL pessimism loss as formulated in Equation 12. To this end, we compare the performance of six agents, each employing different combinations of pessimism loss - either the dual optimization pessimism loss or the VPL pessimism loss - along with varying sources for pessimism updates. These sources include samples from the replay buffer, the validation buffer, and the most recent transitions. The results of this analysis are presented in Figure 9. In our final analysis, we focus on validating the premise of VPL: its effectiveness in reducing approximation error and mitigating overfitting compared to baseline algorithms. Our methodology for quantifying approximation error and overfitting are described in Appendix E. We conducted these measurements across both low and high replay regimes, using a selection of 20 tasks from the DMC and MetaWorld as listed in Table 3. The findings, depicted in Figure 3 and Appendix H, confirm that VPL achieves the lowest levels of critic overfitting and approximation error in both replay scenarios.

6 Limitations

The primary challenge of VPL lies in estimating the lowerbound approximation error necessary for the pessimism adjustment mechanism. This estimation currently relies on a simplified assumption, inherited from GPL and discussed in Section 4.2, that Qθ provides a reasonable approximation of Qω . While this assumption works well in many standard benchmarks, it may limit the applicability of VPL in environments with high policy dynamics or entropy, where offpolicy actions are less representative of on-policy behavior. Future research could explore alternative estimation methods for the lower-bound approximation error that do not rely on this assumption, potentially leading to more robust algorithms. Surprisingly, our experiments (see Figure 6) reveal that using a validation buffer does not detrimentally impact agent performance in high-replay scenarios, except in extremely sample-scarce environments (e.g., fewer than 250k environment steps). Addressing the limitations of the current error estimation framework may also mitigate this samplescarcity sensitivity.

7 Conclusions

This paper examined the approximation error in critic networks optimized via temporal difference variants. We introduced a fixed-point model for estimating mean and lower bound errors and used this model to analyze the convergence of pessimistic actor-critic algorithms. We proposed the VPL algorithm, which dynamically adjusts pessimism levels to minimize approximation errors of critic supervision in validation samples. We tested VPL against baseline algorithms in various locomotion and manipulation tasks, showing performance and sample efficiency improvements. We explored the impact of VPL components and their sensitivity to hyperparameter selection. Our results confirm VPLs effectiveness in complex continuous action tasks.

Acknowledgements

Marek Cygan was partially supported by an NCBiR grant POIR.01.01.01-00-0433/20. Mateusz Ostaszewski was partially supported by the National Science Centre, Poland, under a grant 2023/51/D/ST6/01609. We gratefully acknowledge Polish high-performance computing infrastructure PLGrid (ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PLG/2025/018377