Situational-Constrained Sequential Resources Allocation via Reinforcement Learning

1 Introduction

Sequential resource allocation (SRA) involves distributing limited resources across locations over time, where an agent allocates resources at a sequence of demand nodes while satisfying upper and lower bound constraints. The objective is to allocate resources efficiently while adhering to these constraints. SRA arises in critical domains such as healthcare, public safety, energy, and agriculture, where dynamic demands and societal priorities play a key role. For example, healthcare resource distribution during pandemics must balance immediate needs with future demand [Malenica et al., 2024], while pesticide distribution must adapt to regional crop health and sustainability requirements [Qin et al., 2021].

Beyond efficiency, resource allocation algorithms must consider societal constraints, such as equity [Pu, 2021], sustainability [Heffron and McCauley, 2014], and justice [Zhao et al., 2020]. Moreover, these constraints often depend on context, such as prioritizing equity when regions' demands conflict. For example, during the COVID-19 pandemic, New Zealand proposed a Traffic Light system [Taylor et al.,

(b) Agricultural Scenario

Figure 1: Left: Simulated medical resource demand in Beijing, where darker colors represent higher demand levels. The experiment focuses on a district in the southwest. Right: Farmland in Saskatchewan, Canada, used for pesticide allocation. Numbers indicate regions with varying pesticide requirements.

2023] to adjust policies according to the level of emergency. E.g. one-meter distancing measures were only enforced when public medical resources faced high pressure. Similarly, the U.S. clean-energy supply-chain strategy [Igogo, 2022] highlighted adaptive systems to address bottlenecks during disruptions. They underscore the need for context-aware allocation strategies.

Traditional solutions to SRA, such as dynamic programming [Lien et al., 2014] and multi-armed bandits [Kaufmann, 2018], are effective for small-scale problems with explicit models but struggle with scalability. Reinforcement learning (RL) offers a promising alternative by learning optimal policies through interaction with the environment, without requiring prior system knowledge [Bhatia et al., 2019]. RL has been successfully applied to diverse SRA tasks, such as pesticide spraying [Qin et al., 2021], healthcare resource allocation [Li et al., 2023], and dynamic electricity distribution [Bahrami et al., 2020]. Constrained reinforcement learning (CRL) extends standard RL by incorporating constraints into the learning objective, typically through Lagrangian methods or constrained policy updates. Recent advancements, such as density-constrained reinforcement learning (DCRL) [Qin et al., 2021], have extended RL by incorporating constraints on state distributions. However, existing algorithms rely on static constraints, limiting their ability to adapt to evolving demands and situational requirements. Addressing this gap calls for an advanced RL framework capable of incorporating conditional constraints to enable adaptive decision-making. This leads to the question: How can we design a density-constrained RL framework that ensures situational fairness and adapts to dynamic, context-dependent constraints in resource allocation? To address this question, one needs to (1) develop a formal framework for SRA under such constraints. (2) Propose a new density-constrained RL algorithm that handles "if-then" logic for such constraints.

In this paper, we initiate the study of situational constraints for SRA tasks. We formulate the problem as a conditional DCRL problem, where the constraints are implications. To address this problem, we propose a new algorithm, Situational-Constrained Reinforcement Learning (SCRL). The algorithm extends the conventional CRL framework by introducing a violation degree-based punitive term function that quantifies the extent of constraint violations and adjusts policy updates accordingly. Unlike previous approaches [Tessler et al., 2018; Ray et al., 2019; Qin et al., 2021], SCRL incorporates an adaptive aggregation mechanism that handles disjunctive constraints by selectively prioritizing one constraint within the disjunction. This design allows SCRL to dynamically balance reward optimization with constraint satisfaction in context-sensitive environments. To the best of our knowledge, this is the first work to address situational, disjunctive constraints within the CRL paradigm.

We evaluate SCRL in two real-world-inspired scenarios: medical resource allocation during the COVID-19 pandemic in Beijing, China [Hao et al., 2021] and agricultural resource distribution in Saskatchewan, Canada [Qin et al., 2021]. In both cases, the constraints are designed to balance equity and adequacy, such as ensuring fairness when resources are insufficient and maintaining sufficient coverage when resources are ample. Experimental results demonstrate that SCRL significantly improves the satisfaction of situational constraints compared to baseline methods and effectively adapts resource distributions to meet context-specific requirements. Additionally, we present a case study to illustrate how SCRL adjusts resource allocation across regions in response to shifting situational demands, further highlighting the algorithm's ability to provide adaptive and equitable decision-making in complex, real-world environments.

The following is a summary of key contributions:

• Formulation of sequential resource allocation with situational, disjunctive constraints.
• Development of the SCRL algorithm with a violation degree-based punitive term for dynamic policy updates.
• Introduction of an aggregation mechanism to handle disjunctive constraints in context-sensitive environments.

2 RelatedWork

Sequential Resource Allocation (SRA). SRA focuses on distributing resources in systems where demands arrive sequentially, making it distinct from traditional resource allocation due to its dynamic nature and uncertainty. Its relevance spans socially impactful applications, such as allocating medical testing resources during pandemics [Malenica et al., 2024] and optimizing industrial gas deliveries to minimize costs and prevent shortages [Berman and Larson, 2001]. Ethical considerations, such as equity in resource distribution, have also been explored in government and community planning [Johnson and Smilowitz, 2007].

Early approaches to SRA used dynamic programming to optimize costs under uncertainty, including supply chain management for sequential customers [Bassok and Ernst, 1995]. Bayesian methods were later introduced to handle stochastic dynamics, with Bayes-UCB demonstrating asymptotic optimality [Kaufmann, 2018]. To address fairness, heuristic algorithms were proposed for equitable and sustainable allocation [Lien et al., 2014]. Reinforcement learning (RL) has recently become a predominant approach for SRA, offering scalable solutions for complex environments. Deep RL has been applied to supply chain management [Peng et al., 2019] and network slicing [Liu et al., 2021], enabling efficient resource allocation under constraints. Resourceconstrained RL frameworks have further improved performance over conventional policies [Bhatia et al., 2019].

Despite these advancements, existing methods often focus on fixed constraints or single-objective optimization, leaving a gap in addressing situational and context-sensitive constraints, which are critical for real-world applications.

Constrained reinforcement learning (CRL). CRL extends traditional RL by incorporating constraints to ensure policies satisfy predefined requirements while maximizing rewards [Gu et al., 2022; Garcıa and Fern´andez, 2015]. Rooted in Constrained Markov Decision Processes (CMDPs)[Altman, 1993], methods like Reward Constrained Policy Optimization (RCPO)[Tessler et al., 2018] and SACLag [Ha et al., 2020] use Lagrange multipliers to balance rewards and constraints. Constrained Policy Optimization (CPO)[Achiam et al., 2017] introduced trust region methods for maintaining feasibility during updates, while Projectionbased CPO (PCPO)[Yang et al., 2020] extended it to further avoid infeasible policy during optimization.

Density-based CRL imposes constraints directly on state density functions, offering clear physical interpretations suitable for resource and safety-critical applications [Rantzer, 2001]. Qin et al.[Qin et al., 2021] applied this approach to a pesticide spraying scenario by constraining pesticide density, and Zhang et al.[Zhang et al., 2023] extended it to multi-agent settings with ethical constraints. These methods demonstrated the efficacy of density constraints for resource allocation but do not address situational constraints, which require dynamic adaptation across scenarios.

Other RL paradigms are less applicable to SRA. Logicbased RL [Hasanbeig et al., 2018; Hasanbeig et al., 2020] relies on qualitative specifications, which are unsuitable for quantitative resource allocation and introduce significant computational complexity. Fuzzy-logic-based RL, e.g., FQL [Glorennec and Jouffe, 1997], applies fuzzy rules to represent value functions and actions, and has been used in tasks like robot navigation [Fathinezhad et al., 2016] and resource management [Prasath et al., 2024]. However, it typically yields soft rule satisfaction, which is unsuitable for strict constraints like fairness and safety. Shielding [Waga et al., 2022;

Alshiekh et al., 2017] focuses on safe exploration, which is unnecessary for SRA with reliable simulations.

Our approach builds on density-based CRL, extending it to handle situational constraints that cannot be addressed by traditional Lagrangian methods. We propose a novel algorithm tailored to this problem, ensuring dynamic and contextsensitive resource allocation.

3 ProblemFormulation

3.1 Situational Constraints

We study resource allocation to demand nodes represented as finite regions within a 2D spatial domain, reflecting applications where resources are distributed geographically. Formally, a supplier agent distributes resources over a bounded region D ⊆ R2 with m ∈ N demand nodes, indexed as M = {1, . . . ,m}. Each demand node i ∈ M covers a subregion Si ⊆ D, and an allocation function f : M → R maps each node i to a non-negative amount of resources f(i) ≥ 0. These sub-regions may overlap. Applications include drones spraying pesticides [Qin et al., 2021], mobile immunization vehicles distributing vaccines, and policing resource allocation [Maslen and Paine, 2024], as illustrated in Figure 1.

In many situations, resource allocation must satisfy interval and equity constraints. Interval constraints ensure that each demand node i receives resources within a specified range f(i) ∈ [ai, bi], where 0 ≤ ai < bi ≤ ∞ [Qin et al., 2021]. Equity constraints ensure fairness, expressed as |f(i) - f(j)| ≤ b, where b ≥ 0 [Lien et al., 2014; Zhang et al., 2023]. These constraints address sufficiency and fairness but lack flexibility for context-aware allocation. We therefore introduce situational constraints, which enable conditional relationships between constraints. For example, a situational constraint may state: "If the resources allocated to region A exceed a certain threshold, then region B must receive a minimum amount." Let ⃗ f = [f(1), . . . , f(m)] represent the allocation vector.

Definition 1. An atomic constraint is of the form ⃗a · ⃗ f ≤ b where vector ⃗a ∈ Rm and b ∈ R are parameters. A situational constraint is of the form φ1(⃗ f) → φ2( ⃗ f), where φ1(⃗ f) and φ2( ⃗ f) are atomic.

Situational constraints generalize existing formulations. For example, an interval constraint f(i) ∈ [ai, bi] can be expressed as ⊤ → [-f(i) ≤ -ai] and ⊤ → [f(i) ≤ bi], where ⊤ denotes an always-true atomic constraint. Similarly, an equity constraint can be rewritten as a conjunction of two interval constraints. While interval and equity constraints have been studied [Qin et al., 2021; Lien et al., 2014; Zhang et al., 2023], they fail to capture context-sensitive requirements. Situational constraints address this gap by capturing conditional demands.

3.2 SRAwith Situational Constraints

Formally, the SRA problem is represented by the MDP M = ⟨S, A, r, P, γ, η, L⟩, where:

• S: The state space, which captures the agent's position in the 2D region and its movement dynamics, such as velocity.
• A: The action space, which represents the set of actions available to the agent.
• P : S × A → ∆(S): The transition function, which defines the probability distribution over next states s′ given the current state s and action a.
• r : S ×A → R: The reward function, which quantifies the efficiency of resource allocation.
• γ ∈ (0, 1): The discount factor.
• η ∈ ∆(S): The initial state distribution, which specifies the probability distribution over initial states s0 ∈ S.
• L: S → 2M : The labeling function maps s ∈ S to the set of demand nodes L(s) receiving resources at that state.

To formalize the reward function r, we follow [Qin et al., 2021] and assume that the agent distributes resources at a constant rate as it moves through the region. This means that the amount of resources f(i) allocated to a demand node i ∈ M is captured by the time the agent spends within the corresponding sub-region Si. This simplification links resource allocation directly to the agent's trajectory, allowing the agent to control resource distribution through its movement across the space. Formally, a trajectory is a potentially infinite sequence τ = (s0, s1, s2, . . .) where each st ∈ S represents the agent's state at time t. The following definition follows the definition in [Qin et al., 2021; Rantzer, 2001; Syed et al., 2008; Chen and Ames, 2019].

Definition 2. Given a trajectory τ, the density of resources allocated to demand node i ∈ M is defined as: ρτ (i) := ∑∞t=0 γ t · 1L(st)(i), where 1 is the indicator function.

The density ρτ (i) quantifies the cumulative, discounted amount of resources allocated to demand node i by the agent while following the trajectory τ. The discount factor ensures the density function does not diverge under infinite-horizon settings. A policy π: S → ∆(A) defines a probability distribution over the agent's actions at each state. The sequence of states τ = (s0, s1, . . .) conforms to a policy π, written τ ∼ π, if: st+1 ∼ P(st, at), at ∼ π(st) for all t ≥ 0, where P(st, at) is the transition function.

Definition 3. Given a policy π: S → ∆(A), the expected density of resources allocated to demand node i ∈ M is:

where Pr(i ∈ L(st) | π, η) denotes the probability that demand node i is receiving resources at time t, given the policy π and initial state distribution η.

The expected density ρπ (i) captures the average amount of resources allocated to demand node i when the agent follows policy π. Thus any atomic constraint φ( ⃗ f) of the form⃗a· ⃗ f ≤ b can be rephrased as the following constraint over the policy π of the agent: a1ρπ (1) + · · · + amρπ (m) ≤ b. We now formalize our main problem:

Problem 1 (SRA with situational constraints).

where Vπ(s) is the expected cumulative reward under policy π, defined as: Vπ(s) = Eτ∼π,s0=s [∑∞t=0 γ t r(st, at)] , and Ψ(π) is a conjunction of situational constraints over π.

4 Method

4.1 AlgorithmOverview

To solve Problem 1, we propose a general algorithm framework (Alg. 1) designed for constrained reinforcement learning (CRL) tasks. The framework dynamically addresses the dual objectives of maximizing cumulative rewards and satisfying constraints by incorporating a punitive mechanism through a punitive term function, denoted as σ(Ψ, s). This term penalizes constraint violations directly within the reward structure, effectively transforming the constrained RL problem into an unconstrained optimization task. The objective then becomes maximizing the cumulative punished reward, allowing the agent to learn constraint satisfaction implicitly while pursuing reward optimization.

The framework operates iteratively through three core steps: (1) Trajectory generation: The current policy π is used to generate a collection of trajectories Dπ. (2) Constraint evaluation and punitive term update: Using Dπ, the violation degree V ioπ (Ψ) is computed to evaluate how well the policy satisfies the constraints, and σ(Ψ, s) is updated accordingly. (3) Policy optimization: The policy π is updated by maximizing the cumulative punished reward. This iterative process continues until convergence, ensuring that both reward maximization and constraint satisfaction objectives are met. This framework generalizes existing CRL methods, including RCPO [Tessler et al., 2018], PPOLag [Ray et al., 2019], and DCRL [Qin et al., 2021], which utilize scalar Lagrangian multipliers as punitive terms.

We extend the punitive mechanism to handle situational constraints. This involves: 1. Defining the punitive term σ(φ, s) for atomic constraints, ensuring it accurately reflects the violation degree. 2. Extending σ to situational constraints, such as σ(φ1 → φ2, s).

4.2 Punitive Mechanism: Atomic Constraints

For an atomic constraint φ of the form⃗a· ⃗ ρπ ≤ b, the violation degree is defined as: V ioπ (φ) := ⃗a· ⃗ ρπ -b. This quantifies the extent to which the constraint φ is violated under the current

Algorithm 1 Algorithm Scheme with Punitive Term

• 1: Input: An MDPMwith a constraint Ψ
• 2: Initialize: An initial policy π; A punitive term σ
• 3: while Not converged do
• 4: Generate trajectories Dπ ← {τ1, τ2, · · · | η, π, P}
• 5: Evaluate violation degree V ioπ (Ψ) using Dπ
• 6: Update punitive term function σ(Ψ, s)
• 7: for each transition (s, a, r, s′ ) ∈ τi where τi ∈ Dπ do
• 8: Apply punitive term on reward r′ ← r -σ(Ψ, s)
• 9: end for
• 10: Update policy π ← argmaxπ EDπ [∑t γ t r′ (st, at)]
• 11: end while
• 12: Return π

policy π. A positive V ioπ (φ) indicates a violation, while a value of zero or less signifies satisfaction of the constraint.

To design a punitive mechanism for atomic constraints, we decompose it into two complementary components: the penalty factor, which measures the overall severity of a constraint violation, and the weighting factor, which determines the state-level impact of the violation. These components collectively define the punitive term σ(φ, s).

1. Penalty factor: The penalty factor κ(φ) is updated iteratively to reflect the accumulated violation of φ over time:

where β is the learning rate. This approach aligns with existing CRL methods, such as DCRL [Qin et al., 2021], where the penalty factor is dynamically adjusted to enforce density constraints. For instance, DCRL enforces an upper bound ρmax on state density ρπ (s′ ) by updating κ as κ′ = max(0, κ + β · (ρπ (s′ ) - ρmax)). Our mechanism generalizes this idea to arbitrary atomic constraints.

1. Weighting Factor: The weighting factor w(φ, s) accounts for the fact that visiting different states may contribute unequally to the violation of φ by setting

where 1 is the indicator function, and L(s) identifies the demand nodes affected by state s. Thus the weighting factor w(φ, s) reflects how allocating resources to state s impacts the violation degree V ioπ (φ).

1. Punitive termfunction: The punitive term for an atomic constraint φ is defined as: σ(φ, s) = w(φ, s) · κ(φ). The penalized reward is then computed as:

This formulation ensures that the agent is guided toward satisfying the constraint by dynamically adjusting the reward based on the impact of each state on the violation degree.

Example 1. Consider atomic constraint φ that specifies ρπ (i) - ρπ (j) ≤ 0, which enforces that the resources allocated to demand node i should not exceed those allocated to demand node j. Suppose that the penalty factor κ(φ) > 0, i.e., the constraint is violated under the current π. For s ∈ S, when i ∈ L(s) and j /∈ L(s), by definition, we have w(φ, s) = 1/2. In this case, σ(φ, s) = 1/2κ(φ) > 0. This leads to a decrease on the reward, which discourages the agent to allocate resource at s. On the other hand, when i /∈ L(s) and j ∈ L(s), w(φ, s) = -1/2, which means σ(φ, s) < 0, leading to an increase on the reward, which encourages the agent to allocate resources at s. In any other case, w(φ, s) = 0, which means that any penalty at state s will not affect the satisfaction of φ.

With the definition of the punitive term σ(φ, s) provided above, we can directly instantiate Algorithm 1 for addressing an SRA problem with an atomic constraint φ.

Proposition 1. Consider an SRA problem with an atomic constraint φ of the form ⃗a · ⃗ρπ ≤ b. With a sufficiently small learning rate β, the algorithm framework incorporating the punitive term σ(φ, s) converges to a feasible solution.

Proof. The proof builds on principles from canonical Lagrangian-based CRL frameworks, such as [Tessler et al., 2018].

4.3 Punitive Mechanism: Situational Constraints

Situational constraints ψ := φ1 → φ2 can be reformulated as disjunctive constraints ¬φ1 ∨ φ2. This requires handling disjunctions of atomic constraints φ1 ∨ φ2, which pose unique challenges. Traditional CRL methods work well for conjunctive constraints as they enable gradient-based optimization within connected feasible regions. However, disjunctive constraints create disconnected feasible regions that make gradient-based methods ineffective.

Conventional approaches usually reformulate disjunctive constraints into mixed-integer linear programming (MILP) [Trespalacios and Grossmann, 2015; Kronqvist et al., 2021] and solve them using techniques such as branchand-bound [T¨urkay and Grossmann, 1996]. While effective for problems with explicit system models (e.g., linear programming), these methods struggle with the complexity and dynamic nature of realistic tasks like the SRA problem, where constraints are context-dependent and the environment evolves sequentially. Moreover, branch-and-bound approaches scale poorly in large or high-dimensional problems due to their exhaustive exploration of disjuncts. Importantly, recent machine learning methods have tackled disjunctive constraints by interpreting them as a min-operator over loss functions or as unions of feasible sets [Ren et al., 2020; Huang et al., 2022; Li and Srikumar, 2019; Nandwani et al., 2019]. Building on these ideas, our approach extends the use of min-operators to RL by designing a punitive mechanism for disjunctive constraints.

To define the punitive term σ(ψ, s) on a state s, we adopt the principle that prioritizes the "leastviolated" disjunct in the constraint, i.e., φj where j = argminj{σ(φ1, s), σ(φ2, s), . . . , σ(φj, s)}. The min operator aligns with the logical semantics of disjunctions and has been applied in prior works [Ren et al., 2020; Huang et al., 2022]. This design encourages the policy to satisfy the most attainable disjunct in a disjunction.

While it is intuitive to select the least-violated atomic constraint during the optimization, greedily applying the minoperator may suffer from sub-optimality in cases where it focuses on an infeasible disjunct that appears easier to satisfy. Specifically, consider a disjunction ψ := φ1 ∨ φ2, where φ1 is infeasible for the policy set Π, meaning ∀π ∈ Π, V ioπ (φ1) > 0, while φ2 is feasible. In this scenario, for a given policy π′ ∈ Π at state s, if the punitive term for the infeasible disjunct φ1, σ(φ1, s), is smaller than that for the feasible disjunct φ2, σ(φ2, s), the algorithm would encourage the policy to prioritize φ1. Consequently, the policy is misguided to focus on an unattainable constraint.

To address this issue, we propose a probabilistic mechanism for selecting disjuncts within a disjunctive constraint.

Algorithm 2 Situational-Constrained RL

• 1: Input: An MDP (S,A, P, r, η, γ,L), situational constraints Ψ := ∧i∈I ∨j∈{1,2} φj
• 2: Initialisation: Let π be a random policy, κ(φ) = 0 be penalty factor for each φ, β be learning rate for κ
• 3: repeat
• 4: Generate trajectories Dπ = {τ1, τ2, · · · | η, π, P}
• 5: Empirically compute density ρπ according to Dπ
• 6: for all atomic constraint φ do
• 7: Compute the violation degree V ioπ (φ) for φ
• 8: Update penalty factor as:
• κ(φ) ← max(0, κ(φ) + βV ioπ (φ))
• 9: end for
• 10: for each τi ∈ Dπ, each transition (s, a, r, s′ ) ∈ τi do
• 11: Calculate σ(Ψ, s).
• 12: Apply punitive term on reward r′ ← r -σ(Ψ, s)
• 13: end for
• 14: Solve π that maximizes the expected punished return based on Dπ
• 15: until Convergence
• 16: Output: A policy π, with density values ρπ .

For a disjunction ψ = ∨j∈[J] φj, where φj are atomic constraints, probabilistic mechanism defines a random variable Φ over the set of atomic constraints. The punitive term σ(ψ, s) is then defined as σ(Φ, s), where Φ follows a probability distribution that assigns a probability pj to each φj . These probabilities are defined as pj := κ(φj)-1 ∑j∈J κ(φj)-1 , where κ(φj) is the penalty factor associated with φj. This formulation ensures that constraints closer to satisfaction are prioritized, while constraints with larger penalty factors are still occasionally explored due to their nonzero probabilities.

Finally, for a conjunction of I situational constraints, Ψ := ∧i∈Iφi, we define σ(Ψ, s) = ∑i∈I σ(ψi, s).

4.4 SCRL Algorithm

Alg.2 presents the Situational-Constrained Reinforcement Learning (SCRL) algorithm, an instance of the general framework (Alg.1) tailored for Problem 1. SCRL incorporates the punitive mechanism defined above to handle situational constraints. The algorithm initializes the penalty factor κ(φ) = 0 for each atomic constraint φ, with a learning rate β to encourage exploration during the early stages of training. Trajectory data is used to empirically estimate the state density ρπ (s), leveraging either discrete state counts or kernelbased methods for continuous spaces [Qin et al., 2021; Chen, 2017], ensuring computational efficiency for largescale problems. The algorithm iteratively alternates between generating trajectories, updating penalty factors based on constraint violations, applying punitive terms to the rewards, and optimizing the policy. This iterative process ensures both constraint satisfaction and reward maximization.

5 Experiment

We aim to validate SCRL algorithm's performance through empirical evaluations over two real-world scenarios.

(a) Med-Regions

(b) Agri-Regions

Figure 2: The sub-regions in two scenarios. For both scenarios, the map is divided into 50×50 grids, and 5 regions (regions with label 0 are ignored in our settings).

5.1 Experiment Scenarios and Tasks

Medical Resource Allocation. This scenario models the allocation of medical resources in Beijing during the COVID19 pandemic using a simulation from [Hao et al., 2021]. The city is divided into modules, grouped into five sub-regions based on demand levels (Figure 2a). The challenge is to prioritize high-demand regions during resource shortages while maintaining fairness, reflecting real-world public health requirements for dynamic, context-sensitive allocation policies.

Agricultural Spraying Drone. [Qin et al., 2021] This scenario involves pesticide allocation in farmland in Saskatchewan, Canada, divided into five sub-regions based on crop types (Figure 2b). The agent must optimize pesticide usage by responding to pest outbreaks while avoiding overuse, balancing sufficiency and fairness across regions. This mirrors real-world agricultural challenges with economic and environmental implications.

For both scenarios, the agent's goal is to maximize resource allocation efficiency while satisfying constraints. Three tasks of increasing complexity evaluate the agent's performance: 1. Situational Task: A single situational constraint requires, e.g., "If resources allocated to certain regions exceed a threshold, others must receive a minimum allocation." This tests the agent's ability to adapt dynamically to conditional requirements. 2. Priority Task: Involves equity constraints (equal resource allocation across specific regions) and adequacy constraints (minimum resources for specific regions). The situational requirement states, "If adequacy cannot be met, ensure equity." This evaluates the agent's ability to prioritize fairness under resource limitations. 3. Joint Task: Combines adequacy and equity constraints simultaneously without prioritization, requiring the agent to balance potentially conflicting requirements. In some cases, satisfying both constraints may be infeasible.

5.2 Baselines

We compare our approach against four baseline methods:

1. Deep Deterministic Policy Gradient (DDPG) [Gu et al., 2017] serves as an unconstrained RL baseline, optimizing solely for reward without considering constraints.
2. Reward Constrained Policy Optimization (RCPO) [Tessler et al., 2018] is adapted to our setting by defining cost functions over state-action pairs

instead of using density-based constraints. RCPO cannot natively support density-based situational constraints, requiring approximations for implementation.

1. Conservative Augmented Lagrangian (CAL) [Wu et al., 2024] is a recent primal-dual CRL method. We adapt CAL to our setting in the same manner as RCPO.
2. Density Constrained Reinforcement Learning (DCRL) [Qin et al., 2021] is included as a baseline but lacks native support for situational constraints. To adapt, we decompose each situational constraint φ1 → φ2 into: (DCRL1): The premise ¬φ1, treated as an interval constraint ρπ (s) ∈ [a, b]. (DCRL2): The conclusion φ2, addressed independently as an interval constraint.

5.3 Performance Metrics

The primary evaluation metric, constraint violation (Cons.Vio.), measures violation of constraints. A lower Cons.Vio. indicates better compliance with these critical constraints and is preferred, as we prioritize the safety and fairness in real-world applications. The density function in experiments takes an undiscounted sum due to the finitehorizon setting. The secondary metric, reward, assesses resource efficiency, with higher rewards indicating less resource allocation amount. A negative reward is assigned when one unit of resources is allocated.

(b) Agri-Situational

Figure 3: SCRL's resource allocation as heatmap. Higher temprerature indicates more resources allocated. In joint task, agent tries to satisfy the equity constraint so allocating resources to different regions in different amounts; In situational task, agent prioritizes adequacy constraint so allocating each region with sufficient resources.

5.4 Experiment Result

The result in Table 1 shows that DDPG attains high rewards at the cost of severe constraint violations. Both RCPO and CAL rely on a surrogate cost function, so they suffer high violations (with the lone exception of CAL on the Med. joint task). Few DCRL instances satisfy constraints due to the separate implementation. However, DCRL lacks scalability against situational constraints. SCRL consistently offers near-zero cost on priority and situational tasks. For joint task, SCRL also offers low-level costs, demonstrating its advantage in satisfying context-sensitive requirements. Besides, even when a few DCRL instances satisfy constraints, they perform worse on rewards (e.g., -31 vs. -9). In contrast, SCRL successfully guides the agent to higher-reward feasible solutions.

5.5 Case Study: Priority and Joint Tasks

We investigate SCRL agent's behavior by comparing it with DDPG agent's behavior, as an unconstrained baseline.

Constraint Violation. As shown in Table 2, SCRL effectively prioritizes adequacy constraints in priority tasks, satisfying them at the expense of higher equity violations. In joint tasks, SCRL balances both adequacy and equity constraints, minimizing total violation degrees. This demonstrates SCRL's adaptability to diverse constraint structures.

Resources Allocation. Figure 3 visualizes SCRL's allocation strategies on Agri. scenario, with warmer colors indicating higher allocations. In joint tasks, SCRL balances equity constraints, reflecting region size variations. In priority tasks, it emphasizes adequacy, ensuring critical regions meet minimum demands. These results demonstrate SCRL's adaptability to dynamically meet task-specific constraints.

Resource Efficiency. In the Agri. scenario, DDPG minimizes resource usage, completing its trajectory in 800 time steps, achieving higher rewards at the cost of constraint violations. In contrast, SCRL meets adequacy constraints, requiring at least 900 time steps to satisfy minimum demands across regions, leading to lower rewards. A similar trade-off is observed in the Med. scenario. This underscores the inherent trade-off between reward maximization and constraint satisfaction, as CRL algorithms prioritize constraint compliance over unconstrained efficiency.

5.6 Case Study: More disjunctions

Case study in Table. 3 shows algorithms performance under multiple disjunctions. The result is consistent with earlier analysis: RCPO and CAL fail due to the surrogate constraint; DCRL occasionally satisfies the constraint with a loss of reward (DCRL 3 and 4); SCRL still shows its advantage in capturing situational constraints.

5.7 Ablation Study: Probabilistic Mechanism

We compare the proposed probabilistic mechanism (SCRL) with a variant using the min operator (SCRL-min). As shown in Table 4, SCRL-min performs similarly in reward but incurs slightly higher constraint violations. This aligns with our earlier claim that the min operator may mislead the agent to satisfy an infeasible constraint with a lower violation.

6 Conclusion and Limitations

This paper tackles sequential resource allocation (SRA) under situational constraints, formulating the problem as an MDP with density functions to quantify regional resource allocation. We propose the SCRL algorithm, which trains agents under context-sensitive constraints.

Limitations. This study focuses on a single resource allocator and single-resource type. Extending the framework to multi-agent systems and multi-resource scenarios offers promising directions for future work.