PID-Controlled Constrained RL for Hub-based Joint Pricing, Dispatching, and Routing with Service Guarantees

1 Introduction

Hub-based mobility-on-demand (MoD) systems are integral to modern urban transit and paratransit services [Pavia et al., 2024], concentrating demand and supply within tight spatiotemporal windows [Gao et al., 2021]. Efficiently managing these systems requires the synergistic optimization of three coupled decisions: destination-level pricing, fleet dispatching, and vehicle routing [Sun et al., 2022; Yuan et al., 2023; Sun et al., 2024; Zhou et al., 2019]. As illustrated in Fig. 1, these decisions are intrinsically linked in a closed loop. Pricing reshapes the spatial distribution of demand by leveraging user sensitivity [Xin et al., 2025], dispatching allocates scarce capacity to specific regions using advanced control policies [Han et al., 2025], and routing determines the feasibility and operational cost of executing these commitments.

Decoupling these components or employing myopic optimization strategies often yields suboptimal or executioninfeasible solutions. For instance, seemingly optimal dispatching decisions may become infeasible when grounded in actual road network constraints and travel times, a gap that recent neural routing solvers aim to bridge [Meng et al., 2025; Xiao et al., 2025]. Beyond profit maximization, platforms must strictly adhere to service-level requirements to ensure user retention and regulatory compliance [Pujari and Panda, 2025]. A critical metric in this domain is ORR, which measures the proportion of requests successfully served within a time slot. Neglecting this constraint in favor of pure profit maximization incentivizes reinforcement learning (RL) agents to "cherry-pick" high-value demand hotspots while systematically underserving remote or low-profit areas, violating fairness principles [Jiang et al., 2025]. This behavior not only degrades user experience but also violates long-term operational mandates. Consequently, the problem evolves into a Constrained Markov Decision Process (CMDP) characterized by a hierarchical action space: continuous pricing and dispatching that dictate the inputs for combinatorial routing.

Solving this CMDP poses significant algorithmic challenges. Standard approaches, such as Lagrangian relaxation, often struggle with stability in this context [Wachi et al., 2024; Gao et al., 2023]. The interplay between profit maximization and service constraints often induces dual-variable oscillations, as widely reported in constrained RL [Tessler et al., 2019]. The system alternates between aggressively overserving difficult regions to reduce accumulated penalties and reverting to greedy, profit-seeking behavior when penalties subside. Furthermore, standard primal-dual updates are sensitive to the high variance typical of MoD environments, preventing convergence to a stable policy.

To address these challenges, this paper proposes PIDSACA, a unified framework for the Constrained Joint Dispatching, Routing, and Pricing (C-JDRP) problem. We model the system as a CMDP and employ an entropy-regularized actor-critic policy to jointly generate pricing and dispatching actions. To bridge the gap between high-level decisions and physical execution, we embed an attention-based routing solver into the learning loop to construct feasible routes and provide feedback on realized costs, enabling routingaware policies. The core of our approach is the application of PID control theory to stabilize the Lagrangian multiplier updates, inspired by control-theoretic adaptations in deep RL [Stooke et al., 2020]. By incorporating proportional, integral, and derivative feedback, this mechanism effectively dampens dual variable oscillations. Specifically, the derivative term helps anticipate error trends, smoothing the learning dynamics against stochastic noise and enabling the system to satisfy long-term ORR guarantees.

Our main contributions are:

• We formalize a hub-based joint pricing, dispatching, and routing problem with an explicit long-term ORR service requirement, capturing realistic operational coupling between learning decisions and routing feasibility.
• We demonstrate that adapting PID control to the Lagrangian dual update significantly improves training stability in high-variance MoD environments compared to standard primal-dual methods.
• We validate our approach on public large-scale realworld mobility benchmarks, showing consistent gains in revenue while meeting ORR targets, supported by ablations that isolate the benefits of the routing integration and the PID mechanism.

2 RelatedWork

This work lies at the intersection of joint decision-making in MoD systems and constrained reinforcement learning. We review prior efforts on coupled optimization of pricing, dispatching, and routing, and then highlight the missing serviceaware perspective that motivates our CMDP formulation.

2.1 Joint Decision-Making for MoD

Due to the complexity of jointly optimizing pricing, dispatching, and routing, most studies focus on coupled subsets of decisions. In pricing-dispatching, [Zheng et al., 2018] studied price-aware ridesharing dispatch with profit objectives and detour constraints, while [Tong et al., 2018a] proposed matching-based dynamic pricing for spatial crowdsourcing to balance supply and demand. These methods provide economic control but often simplify route feasibility and network-level execution. In dispatching-routing, [Tong et al., 2018b] developed a unified route-planning framework with flexible insertion operators, and [Jin et al., 2019] proposed CoRide, a hierarchical multi-agent RL approach integrating dispatching with fleet repositioning. However, executionfocused methods typically assume exogenous demand and thus cannot shape demand through pricing.

Recent progress moves toward fully unified JDRP. [Gao et al., 2021] introduced SACA, an end-to-end framework that uses a Soft Actor-Critic agent for pricing and dispatching together with an attention-based route planner, allowing highlevel decisions to leverage low-level routing feedback. Despite strong performance, existing unified approaches largely pursue unconstrained revenue maximization and do not explicitly encode service-level requirements such as ORR. This can induce systematic preference for dense, high-value regions and chronic underservice of remote areas, conflicting with long-term service mandates. Our work addresses this gap by formulating JDRP as a CMDP, enabling stabilityaware optimization under explicit service guarantees.

2.2 Constrained Reinforcement Learning for MoD

Enforcing strict service guarantees transforms the problem into a CMDP. While standard Lagrangian relaxation is the de facto method for solving CMDPs [Wachi et al., 2024], it faces severe stability issues in MoD environments characterized by high variance and sparsity. Standard primal-dual updates often suffer from "overshoot," where the dual variable oscillates violently, causing the policy to alternate between ignoring constraints and over-conserving resources.

Recent works have attempted to mitigate this. [Hazra et al., 2025] proposed IP3O, which uses adaptive incentives as soft barriers, and [Zhang et al., 2025] introduced SCRL to handle situational constraints via probabilistic selection. However, these methods often introduce sensitive hyperparameters or fail to dampen the momentum of error accumulation in longhorizon tasks. Distinct from these approaches, we adapt PID control theory to the Lagrangian update [Stooke et al., 2020]. By incorporating Integral (accumulated error) and Derivative (rate of change) feedback, our method explicitly dampens the oscillation of the multiplier, ensuring robust convergence to the target service level without manual reward shaping.

3 ProblemFormulation

3.1 Preliminaries

We consider a hub-based bus service scenario where all buses depart from a central depot to deliver passengers to various destinations across the city and then return. The system operates in a round-based online manner.

Definition 1 (Time Slot). The operation time horizon is divided into discrete time slots t = 1, 2, . . . , T with a fixed interval ∆t. In each slot t, the platform collects orders, updates vehicle states, and executes dispatching decisions.

Definition 2 (Order Request). Let D = {1, . . . , K} be the set of discrete destination regions. An order request oti arriving at slot t is denoted as a tuple:

where lj represents the geographical location of destination j, and pi ∈ [pmin, pmax] denotes the trajectory-based price charged to the passenger. The aggregate demand for destination k at slot t is collected in the set Otk , and the total order set is Ot = ⋃k∈D Otk .

Definition 3 (Bus Service). Consider a fleet of buses B = {b1, . . . , bM}. At the beginning of slot t, the state of bus bi is defined as a tuple:

where Ci is the maximum passenger capacity, vi is the average travel speed, ζ t i represents the planned route, and ψt i ∈ R≥0 denotes the remaining service time. A bus is available if and only if ψt i = 0. The transition of the remaining service time follows:

where Time(·) estimates the travel time for a newly assigned route, and the indicator function 1{·} ensures the service time is updated only when a valid route is assigned (ζ t+1 i ̸= ∅).

3.2 Supply and DemandModeling

Demand Modeling. Let Ot be the set of order requests in slot t. We denote the location of destination region k as lk. The potential demand N tk is the count of users wishing to travel to lk, i.e., N tk = |{oti ∈ Ot | dest(oti ) = lk}|.

We define a unified pricing strategy ptk for destination k. The realized demand Dtk is modeled by a conversion rate function F d k (·), which captures the user's sensitivity to both price and travel distance. Let dk denote the distance from the hub to location lk. The demand function is formulated as:

where dmin and dmax represent the minimum and maximum distances in the service area, respectively. This formulation reflects that users traveling longer distances are less sensitive to price increases compared to those with shorter trips.

SupplyModeling. The platform controls the fleet and allocates capacity explicitly. The total available capacity at slot t is St total = ∑bi∈B Ci · 1{ψt i = 0}. Let atk ∈ [0, 1] be the dispatching ratio for region k, subject to ∑k atk ≤ 1. The effective supply for destination k is Stk = St total · atk . Consequently, the number of fulfilled orders Otk is the minimum of demand and supply:Otk = min{Dtk , Stk }.

3.3 Objective and Constraints

Definition 4 (Profit). The platform's profit Rt in slot t is the profit from fulfilled orders minus operational costs:

The routing process is formulated to find the optimal trajectory ζ t i that minimizes the total travel distance given the assigned destination set. Consequently, the operational cost is defined as cost(ζ t i ) = βd dist(ζ t i ), where βd is a distance coefficient and dist(ζ t i ) is the length of this distance-optimized route. The indicator terms charge this cost only when a bus is newly dispatched.

The routing process is formulated as finding the optimal trajectory ζ t i that minimizes travel distance/time given the assigned destination set, serving as the physical constraint for the dispatching layer.

Definition 5 (Order Response Rate). The ORR quantifies the service level at slot t:

Definition 6 (C-JDRP). The C-JDRP problem aims to sequentially determine pricing and dispatching decisions over a horizon T, while explicitly enforcing a minimum servicelevel requirement measured by ORR. Formally, we solve:

where δ denotes the required minimum long-term average ORR over the time horizon.

4 Methodology

As illustrated in Fig. 2, our framework employs an alternating optimization scheme: the primal policy jointly generates pricing and dispatching, while the dual variable is regulated by a PID controller to stabilize constraint satisfaction.

4.1 CMDP Formulation

We first map the JDRP elements to a standard MDP tuple ⟨S, A, γ⟩.

State Space S. The state st captures the global demand distribution and fleet status. We define:

where Nt = {N tk }k∈D represents the potential demand across all regions, Btidle = {bti ∈ B | ψt i = 0} represents the set of available buses, and G t encapsulates global context features such as time of day and traffic conditions.

Action Space A. The action at = ⟨Pt ,At ⟩ corresponds to the decision variables in the JDRP formulation:

• Pricing Pt : A continuous vector determining the price ptk ∈ [pmin, pmax] for each destination k ∈ D.
• Dispatching At : A continuous vector representing the dispatching ratio atk ∈ [0, 1] for each region, normalized such that ∑ atk ≤ 1.

Reward Function. We consider Rt in Definition 4 as the immediate reward function.

With the state, action, and reward defined above, we now cast C-JDRP (Definition 6) as a CMDP. Specifically, we maximize the discounted cumulative profit while enforcing the long-term ORR constraint, which will be handled via Lagrangian relaxation in the next subsection.

4.2 Execution Layer: Attention-Based Routing

To ensure high-level dispatching decisions are physically feasible and service-oriented, we integrate a differentiable routing solver into the learning loop. Unlike heuristic estimates, this solver explicitly constructs routes to minimize the total travel distance, providing accurate feedback on operational costs and improving real-world execution reliability.

We formulate the routing sub-problem via the pointernetwork paradigm [Vinyals et al., 2015]. For a bus bi assigned a set of destinations, the decoder sequentially selects the next station to visit based on the encoder embeddings of all pending stations. The probability of visiting station km at step m is given by:

where hmdec is the decoder state and {hk} are station embeddings. This routing module effectively acts as the environment's transition function component, determining the realization terms Time(τ t+1 i ) and cost(τ t i ) defined in Eq. (1) and Definition 4. By closing the loop, the upper-level pricing and dispatching policy πθ learns to avoid decisions that would incur excessive routing latency or costs.

4.3 PID Lagrangian Relaxation and Alternating Optimization

Since the direct optimization of the constrained objective in a high-dimensional continuous action space is intractable, we employ the Lagrangian multiplier method to convert the CMDP into an unconstrained dual problem.

We construct the Lagrangian function L(θ, ν) by relaxing the constraint into the objective:

where ν ≥ 0 is the learnable Lagrange multiplier acting as a penalty coefficient for constraint violation. This formulation establishes a min-max game (primal-dual problem):

To solve this minimax problem within the SAC framework, we perform alternating updates during the training phase.

Primal Update (Policy Optimization). Fixing ν, we update the policy parameters θ via the standard SAC gradient estimator to maximize [Haarnoja et al., 2018] the Lagrangian reward rt lag :

This formulation dynamically shifts the agent's focus between profit maximization and service quality (ORRt ) based on the magnitude of ν.

Dual Update (PID Multiplier Optimization). We update ν while fixing πθ. To strictly enforce non-negativity, we parameterize ν = exp(x). Since standard gradient descent acts as pure integral control and is prone to oscillation in dynamic environments, we employ a Proportional-Integral-Derivative (PID) controller on x to ensure robust convergence.

We define the constraint violation error at step t as:

Based on this error, the PID update magnitude ∆xt is computed as:

where KP , KI , KD ≥ 0 are the hyperparameters for the proportional, integral, and derivative terms, respectively. The log-multiplier x is then updated via gradient ascent with a learning rate η:

This mechanism offers superior robustness over standard Lagrangian methods by integrating three terms: the Proportional (KP ) addresses immediate violations (et); the Integral (KI ) eliminates steady-state bias by accumulating past errors over long horizons; and the Derivative (KD) dampens oscillations by penalizing the error rate (et - et-1), collectively ensuring training stability and strict constraint satisfaction.

4.4 Theoretical Analysis

We establish the convergence and long-term constraint satisfaction of our PID-based Lagrangian relaxation using Lyapunov optimization theory. By constructing a Lyapunov function based on Kullback-Leibler divergence-tailored to the exponential updates of log-parameterization-we prove that the dual multiplier remains bounded and the service constraint is satisfied over the long run.

Definition 7 (Lyapunov Function). Let ν ∗ be the optimal Lagrange multiplier. We define the potential function V (νt) as:

V (νt) is strictly convex and non-negative, with V (νt) = 0 if and only if νt = ν ∗ .

Theorem 1 (Boundedness). Define the violation et ≜ δ - ORRt and integral It ≜ ∑tτ=0 eτ . Assume: (1) Slater's condition: ∃¯π, ϵ > 0 s.t. E[et|¯π] ≤ -ϵ; (2) Boundedness: et ∈ [-1, 1] and rewards are bounded; (3) PID parameters KP ,KD ≥ 0,KI > 0; (4) Update rule νt+1 = νt exp(η∆xt) with effective step ∆xt = KPet + KIIt + KD(et - et-1) satisfying |η∆xt| ≤ 1 (via clipping). Then, the multiplier sequence {νt} is bounded.

Proof. Decomposition. Rewrite the PID signal as ∆xt = aet + bt, where a ≜ KP + KI + KD aggregates current feedback, and bt ≜ KIIt-1-KDet-1 captures history. Note that the derivative component in bt is bounded since et ∈ [-1, 1].

Drift. Using ez ≤ 1 + z + z2 for |z| ≤ 1, the drift ∆Vt ≜ V (νt+1) - V (νt) satisfies:

The actor minimizes the Lagrangian-like term aνtet - Rt.

Boundedness. By Slater's condition, for sufficiently large νt ≥ νth, the optimal policy forces E[et|Ft] ≤ -ϵ′ < 0. Consequently, the integral state It acquires a negative drift: E[It|Ft] ≤ It-1 - ϵ′ . If νt were to diverge to +∞, it would persistently exceed νth, causing It-1 (and thus bt) to decrease indefinitely. Since KI > 0, a sufficiently negative It-1 dominates the bounded terms aet and -KDet-1, forcing ∆xt < 0 and νt+1 < νt. Therefore, νt cannot diverge.

Theorem 2 (Long-Term Constraint Satisfaction). Under the assumptions of Theorem 1, the time-averaged constraint violation is non-positive:

Proof. Let et = δ -ORRt and It = ∑tτ=0 eτ as in Theorem 1. By Theorem 1, νt is bounded above, hence xt = log νt is bounded above as well.

We prove by contradiction. Suppose there exists ¯ ϵ > 0 such that

Then for infinitely many T large enough, implying

From the PID law and the bound |et - et-1| ≤ 2Emax,

Combining with (15) shows that ∆xt becomes unboundedly positive along an infinite subsequence, and thus

contradicting the boundedness of {νt} established in Theorem 1. Hence the assumed positive long-run average violation cannot occur, and we must have

i.e., the service constraint is satisfied in the long run.

5 Experiments

In this section, we present a comprehensive evaluation of our proposed framework. We aim to answer the following research questions:

• RQ1: Can our method maximize the platform's profit while strictly satisfying ORR constraint compared to state-of-the-art baselines?
• RQ2: How does the proposed PID-based Lagrangian mechanism contribute to training stability and convergence speed?
• RQ3: What is the impact of individual modules and hyperparameters on the final performance?

5.1 Experimental Settings

Dataset. We evaluate our method on a large-scale realworld ride-hailing order dataset released by the Didi Chuxing GAIA Initiative [DiDi Chuxing GAIA Initiative, 2016]. The dataset contains orders from Chengdu, China, spanning November 1-30, 2016 (30 days) and includes 7,065,937 records. Each record provides fields such as a unique ID, billing time, and GPS coordinates.

Baselines. We compare our method against the following representative state-of-the-art baselines:

• JDRL-POMO: A robust baseline that combines the JDRL dispatching policy [Sun et al., 2022] with POMO [Kwon et al., 2020], a routing solver leveraging instance augmentation to improve routing quality.
• GRC-ELG: A hybrid baseline integrating goalconditioned RL for order dispatching [Lei and Ukkusuri, 2023] and ELG attention for route planning [Yoon et al., 2024]. It jointly optimizes pricing and routing to improve long-term fleet revenue and operational efficiency.
• SACA: A two-module baseline with a SAC-based policy for joint pricing and dispatching, coupled with an attention-based routing policy [Kool et al., 2019] to construct routes for assigned demand [Gao et al., 2021].
• ST-SACA: An enhanced SACA variant that injects order coordinates during encoding, strengthening stationlevel spatio-temporal demand representation.

5.2 Destination Selection

Data Denoising. We apply DBSCAN [Ester et al., 1996] to filter raw GPS noise and outliers, retaining a high-density destination set that characterizes core travel patterns.

• (a) Profit Convergence

(b) Lagrange Multiplier

Figure 4: Analysis of training stability and convergence.

DestinationConstruction. The filtered points are clustered using Partitioning Around Medoids (PAM) [Kaufman and Rousseeuw, 1990] to ensure robustness, yielding K = 30 destination medoids D = {1, . . . , 30}.

IntegrationwithDispatching. Given pricing and dispatching decisions, this module constructs feasible routes ζ t i to determine realized travel times and operational costs.

5.3 Results and Analysis

Comparison. We evaluate the performance of our proposed PID-SACA against four baselines: JDRL-POMO, GRCELG, ST-SACA, and the vanilla SACA. The training curves for Profit, Operational Cost, and ORR are illustrated in Fig. 3, and the quantitative metrics are summarized in Table 1.

As shown in Fig. 3 and Table 1, while ST-SACA and GRC-ELG maximize profit via "cherry-picking," they suffer from poor coverage (ORR ≈ 0.47). In contrast, PID-SACA achieves a dominant ORR of 0.70 (∼30% improvement) with only a marginal profit trade-off, successfully balancing global service guarantees. Regarding cost (Fig. 3(b)), PID-SACA maintains consistently high operational efficiency comparable to ST-SACA. In contrast, JDRL-POMO's lower cost is merely an artifact of fleet idleness.

Ablation Study And Training Stability. To investigate the impact of the proposed mechanism on learning dynamics, we analyze the convergence behavior and stability of the Lagrange multiplier. The unconstrained baseline w/o Lagrange yields the lowest profit of 77.18, indicating that without service constraints acting as regularization, the agent adopts a myopic policy failing to sustain long-term revenue. Among constrained methods, the w/o PID variant exhibits instability characterized by a higher multiplier standard deviation of 241.68. This oscillation implies the dual variable overshoots and disrupts policy updates, resulting in suboptimal convergence with a profit of 90.18. In contrast, PID-SACA effectively utilizes derivative feedback to dampen fluctuations and reduces the multiplier standard deviation to 208.11. This stabilized dual update process enables the agent to reach a superior equilibrium, achieving the highest profit of 100.14 while satisfying the ORR requirement of 0.70.

Sensitivity Analysis. Table 4 and Table 2 investigate the impact of the constraint threshold δ and vehicle capacity C, respectively. (1) Impact of Threshold δ: Table 4 illustrates a non-trivial trade-off between revenue and service quality (ORR). Interestingly, at moderate levels (δ = 0.70), the system achieves a synergy where improved coverage boosts revenue to a peak of 100.46. However, as we tighten the constraint (δ ≥ 0.80), the PID mechanism effectively prioritizes strict service guarantees (ORR rises to 0.85) over peak profitability. This confirms PID-SACA functions as a flexible tuner, navigating the Pareto frontier between operational efficiency and social responsibility. (2) Scalability on Capacity C: Table 2 further validates the robustness of our method against varying capacities. While baselines like ST-SACA aggressively pursue profit at the cost of service fairness (low ORR), PID-SACA consistently maintains the highest ORR across all capacities (e.g., 0.70 vs. 0.47 at C = 80) while delivering competitive revenue growth. This demonstrates that our approach ensures reliable service coverage regardless of resource abundance or scarcity.

Case Study. Fig. 5 visualizes the pricing policy during a peak snapshot. As demand concentrates in the CBD and Transport Hub (Fig. 5(a)), PID-SACA (Fig. 5(b)) demonstrates topology-aware regulation: it applies surge pricing in hotspots while lowering prices in sparse western regions. This confirms that the PID controller guides the agent to stimulate conversion in underserved areas to satisfy global ORR constraints without sacrificing core profit. In contrast, the Vanilla Lagrangian baseline (Fig. 5(c)) counter-productively maintains high prices in low-demand zones, suppressing order volume. This highlights the necessity of the PID mechanism in stabilizing the trade-off between local profit maximization and global service mandates.

6 Conclusion

This paper investigates the constrained joint pricing, dispatching, and routing problem (C-JDRP) for mobility-ondemand systems, emphasizing the critical trade-off between profit maximization and service level guarantees. We propose PID-SACA, a unified framework that couples an entropyregularized actor-critic policy with an attention-based routing module to ensure operational feasibility. To address the instability of standard Lagrangian methods in mixed continuous-combinatorial action spaces, we introduce a PIDcontrolled dual update mechanism. Theoretical analysis confirms that this approach ensures bounded dual variables and long-term constraint satisfaction. Extensive experiments on real-world datasets demonstrate that PID-SACA significantly outperforms competitive baselines, achieving high profitability while robustly maintaining the target Order Response Rate through stable learning dynamics. This work provides a scalable and theoretically grounded paradigm for sustainable urban fleet management.

Acknowledgments

The authors would like to thank the anonymous reviewers for their helpful comments and suggestions. This work was partially supported by the National Key Research and Development Program of China (No. 2024YFF0617702); the National Natural Science Foundation of China (Nos. U22A2025, 62232007, and U23A20309); the 111 Project (No. B16009); the Ant Group Research Program (No. 2025021900003); the Fundamental Research Funds for the Central Universities (No. N2417007); and the CCF-DiDi GAIA Collaborative Research Funds for Young Scholars.