SplitScaling: Adaptive Scaling for Disaggregated LLM Serving Against Traffic Bursts via DRL
SplitScaling: Adaptive Scaling for Disaggregated LLM Serving Against Traffic Bursts via DRL
Wei Xiao, Xuefeng Huang, Weijia Shi, Baokang Zhao
Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence
Main Track. Pages 5101-5109.
https://doi.org/10.24963/ijcai.2026/568
The disaggregated Prefill-Decode (PD) architecture has emerged as a prominent paradigm for efficient Large Language Model inference serving. However, resource management remains a critical challenge, particularly under the dual burstiness of real-world scenarios—characterized by volatile fluctuations in both request arrival rates and Prompt-to-Response ratios. Existing rule-based heuristics often fail to accurately identify system bottlenecks, leading to severe resource misallocation and Service Level Objective (SLO) violations. To address this, we propose a Deep Reinforcement Learning-based auto-scaling framework tailored for the PD architecture. By modeling the resource allocation problem as a Markov Decision Process, our framework enables the agent to capture non-linear load dynamics, thereby achieving decoupled and precise scaling for prefill and decode pools. Furthermore, to mitigate Head-of-Line blocking caused by scaling latency, we design an immediate rescheduling mechanism that migrates queued tasks to newly ready nodes in real-time. Experimental results driven by Azure bursty load traces demonstrate that our framework significantly reduces computational costs by 25.2% and 28.3% compared to the Static configuration and HeteroScale, respectively, while strictly adhering to SLOs.
Keywords:
Machine Learning: Reinforcement learning
Planning and Scheduling: Learning in planning and scheduling
