GMM-TDQN:Two-Stage Multi-Objective Reinforcement Learning for Large-Scale Edge Server Deployment

1 Introduction

Edge computing has emerged as a key paradigm for supporting latency-sensitive and resource-intensive applications in large-scale Internet of Things (IoT) systems, such as intelligent transportation, smart cities, and industrial automation [He et al., 2025]. By offloading computation from centralized cloud servers to geographically distributed edge servers (ESs), edge computing can significantly reduce service latency, alleviate backbone network congestion, and improve system reliability. However, the effectiveness of edge computing critically depends on the deployment strategy of ESs, distributed and dynamically utilized.

∗Corresponding author.

Designing an effective ES deployment strategy is challenging for several reasons. First, the deployment problem is inherently large-scale, involving numerous base stations (BSs) with heterogeneous spatial distributions and workload characteristics. Second, the deployment aim is multi-objective, requiring the simultaneous optimization of conflicting objectives, such as service latency, energy consumption, load balance, and system reliability. Improving one objective often degrades another, making the problem difficult to solve using traditional single-objective optimization techniques. Third, the deployment environment is frequently dynamic and uncertain, with time-varying user demands and network conditions, which limit the applicability of static or heuristic-based methods.

Existing approaches to ESs deployment can be broadly categorized into meta-heuristic optimization approaches, clustering and graph-based methods, and learning-based solutions [Wu et al., 2024]. Heuristic and meta-heuristic approaches, while more scalable, usually lack adaptability and offer no guarantees on solution quality. Clustering-based methods typically group BSs or user nodes based on spatial proximity, traffic demand, or service similarity, and deploy ESs at the cluster centers. Recently, reinforcement learning (RL) has gained increasing attention for edge computing problems due to its ability to learn adaptive policies through interaction with the environment. Nevertheless, directly applying RL to large-scale ES deployments remains challenging, as high-dimensional state and action spaces can lead to slow convergence and unstable learning, especially in multiobjective settings.

To address these challenges, we propose GMM-TDQN, a two-stage multi-objective reinforcement learning framework for large-scale ES deployment. The core idea is to decompose the complex deployment problem into two complementary stages that jointly improve scalability and learning efficiency. In the first stage, a Gaussian Mixture Model (GMM) is employed to capture the spatial and workload heterogeneity of base stations, providing a structured initialization that significantly reduces the deployment search space. In the second stage, the deployment problem is formulated as a sequential decision-making process, and a Transformerenhanced Deep Q-Network (TDQN) is developed to learn adaptive deployment policies that balance multiple objectives. The Transformer-based attention mechanism enables the model to capture long-range dependencies and complex interactions among deployment decisions, leading to more effective multi-objective optimization.

Extensive experiments conducted on real-world datasets demonstrate that the proposed framework consistently outperforms state-of-the-art methods across multiple performance metrics. The results show significant improvements in latency reduction, energy efficiency, load balancing, and system reliability, validating the effectiveness and scalability of GMM-TDQN in realistic large-scale edge computing environments.

The main contributions of this paper are summarized as follows:

• We formulate large-scale ES deployment as a multiobjective reinforcement learning problem, capturing multiple conflicting performance objectives within a unified learning framework;
• We propose a two-stage deployment strategy: a GMMbased clustering stage that reduces decision complexity and enhances scalability, followed by a Transformerenhanced DQN (TDQN) that learns adaptive deployment policies;
• We design a TDQN architecture tailored for multiobjective deployment optimization, enabling effective modeling of complex dependencies among deployment decisions;
• We conduct extensive experiments on real-world datasets, demonstrating that the proposed framework significantly outperforms existing approaches across multiple objectives.

2 RelatedWork

Edge server placement (ESP) has emerged as a critical issue in mobile edge computing (MEC) and smart city networks, aiming to provide low-latency services while optimizing resource utilization and system stability. Prior research in ESP can be broadly categorized into three classes: meta-heuristic optimization approaches, clustering and graph-based methods, and machine learning and reinforcement learning-based approaches.

2.1 Meta-Heuristic Optimization Approaches

Meta-heuristic algorithms have been widely applied to ESP due to its NP-hard nature. Genetic Algorithm (GA), Particle Swarm Optimization (PSO), and their hybrid variants are commonly used to search for near-optimal server placements under complex constraints. Han et al. [Han et al., 2024] proposed GP4ESP, a hybrid GA-PSO algorithm that combines evolutionary exploration with fast convergence, achieving a reduction in overall response time compared with baseline approaches. Pandey et al. [Pandey et al., 2023] introduced a traffic-aware meta-heuristic ESP scheme to improve load balancing and energy efficiency. Similarly, Sajadinia and Yari

[Sajadinia and Yari, 2023] applied multi-objective genetic algorithms combined with clustering for virtual machine placement in cloud environments, demonstrating reduced energy consumption and improved resource utilization.

Recent studies have also focused on robustness and costawareness. In Ref. [Liu et al., 2024], an improved snake optimization-based dynamic ESP scheme was introduced , which minimizes service delay and operational cost through an interior-point-assisted optimization strategy. In addition, multi-objective evolutionary algorithms have also been extensively explored for ESP. For instance, an NSGA-II-based edge server placement and allocation framework was proposed in [Ghasemzadeh et al., 2024], where latency and server workload variance were jointly optimized to obtain balanced Pareto-optimal solutions. Although effective in handling conflicting objectives, such approaches rely on population-based search and lack learning capability, which limits their adaptability to highly dynamic MEC environments.

2.2 Clustering and Graph-BasedMethods

Clustering and graph-based techniques exploit the spatial and topological characteristics of BSs and ESs to improve placement efficiency. Vali et al. [Vali et al., 2024] proposed RESP, a recursive clustering-based ESP approach that places ESs according to cluster medians, achieving balanced workloads and reduced network traffic. Yucel [Yucel, 2021] extended spectral clustering with hierarchical and density-aware enhancements to support collaborative virtual services. In addition, Zhang et al. [Zhang et al., 2024] introduced a graph convolutional network (GCN)-based framework combined with differentiable K-Means, transforming ESP into an end-to-end optimization task that jointly minimizes latency and server workload. For application-specific scenarios, Doostmohammadi et al. [Doostmohammadi et al., 2023] proposed DyMECC, a dynamic clustering strategy for MEC video streaming based on queuing theory, reducing delivery delay under fluctuating traffic. Qin et al. [Qin et al., 2024] studied joint optimization of base station clustering and service caching, demonstrating that spatial clustering plays a critical role in latency and cache cost reduction. However, such methods typically rely on static assumptions and lack adaptability to time-varying workloads.

2.3 Machine Learning and Reinforcement Learning-Based Approaches

Recent studies increasingly leverage machine learning (ML) and RL to capture dynamic network conditions and optimize ESP adaptively. Chen and Guo [Chen and Guo, 2023] proposed an RL-based framework (ESDR) for dynamic MEC environments, achieving notable energy savings through learned deployment policies. Ling et al. [Ling et al., 2023] integrated graph convolutional networks to predict user traffic and guide ESP decisions, improving both latency and energy efficiency. Deep reinforcement learning has also been applied to reliability and mobility-aware scenarios. Zhang and Jabbari [Zhang and Jabbari, 2024] employed deep RL for joint content placement and IoT device allocation. Toufga et al. [Shinde and Tarchi, 2024] proposed a multi-time-scale reinforcement learning approach for joint service placement and task offloading, utilizing aerial platforms and advanced deep Q-learning to minimize latency and energy consumption.

3 SystemArchitecture and Problem Formulation

This section explores the system architecture and problem formulation of ESs placement.

3.1 SystemArchitecture

Fig. 1 illustrates a three-layer architecture for ESs deployment, consisting of the user layer, edge layer, and cloud layer . This work focuses on the edge layer, which includes n base stations (BSs) and m ESs, denoted by B = {b1, b2, . . . , bn} and S = {s1, s2, . . . , sm}, respectively.

The user layer comprises heterogeneous intelligent devices (e.g., mobile terminals, sensors, and cameras) that continuously generate computation requests [Nguyen et al., 2024]. These requests are first transmitted to the nearest BS and then forwarded to the edge layer for processing.

The edge layer hosts ESs with limited computing resources to serve latency-sensitive tasks locally. When edge resources are insufficient, tasks are offloaded to the cloud layer, which provides abundant computation and storage at the cost of higher transmission delay and energy consumption. Therefore, efficient edge server deployment strategies are critical to achieving low latency, energy efficiency, and reliable service delivery.

3.2 Related Definition and the Optimization Problem

In the ES placement algorithm, four critical performance metrics are identified: delay, energy consumption, load balancing, and network reliability. Each of them is described below.

Delay Metrics

The delay of an ES sj is defined as the total geographical distance to all BSs it serves:

where Bj is the set of BSs assigned to sj, and d(bi, sj) is the great-circle distance calculated using the Haversine formula:

Where R is the Earth's radius, and ∆lat, ∆lon are the latitude/longitude differences (in radians) between bi and sj . The overall system delay is the sum over all ESs:

Load Balancing Metrics

The load on ES sj is defined as the sum of its associated BSs' workloads:

where Bj is the set of BSs served by sj, and wi is the workload of bi.

The load balancing performance is measured by the coefficient of variation of ES loads:

with σL and µL being the standard deviation and mean of the ES loads, respectively.

Energy Consumption Metrics

The energy consumption of an ES sj is determined by its power profile and CPU utilization:

where Pidle and Pmax denote the idle and maximum power, and UP (t) is the instantaneous CPU utilization.

CPU utilization is modeled as a piecewise function based on the server load W(s) and a capacity threshold WTH:

The total energy consumption of the system is:

Network Reliability

Network reliability quantifies the system's ability to maintain service under failures. Redundancy is enhanced when BSs are covered by multiple ESs.

Define a binary indicator Cji for whether ES sj can serve base station bi within its coverage radius Re:

The normalized reliability R(S) is the average coverage per BS:

A value R(S) ∈ [0, 1] close to one indicates high redundancy and fault tolerance.

3.3 ProblemFormulation

Given the definitions in Section 3.2, the ESs deployment problem aims to determine the optimal placement of m ESs over n BSs. Each BS is associated with its nearest ES based on the Haversine distance, and each ES serves the traffic demands of its assigned BSs.

The problem is formulated as a multi-objective optimization by jointly considering system delay, energy consumption, load balancing, and network reliability:

The optimization is subject to the following constraints:

These constraints ensure exclusive BS-ES association, bounded resource utilization, balanced workload distribution, limited communication distance, energy feasibility, and minimum network reliability. Due to its combinatorial and nonconvex nature, the formulated ES deployment problem is NPhard.

To address this challenge, a hybrid GMM-TDQN framework is adopted, where GMM provides spatial initialization and a TDQN refines ES placement through reinforcement learning.

4 GMM-TDQNAlgorithm

In this section, we introduce the ESs placement algorithm GMM-TDQN.

4.1 The Overall Process of the GMM-TDQN

To address the ESs placement problem under complex spatial distributions and service demands, we propose a two-stage optimization framework termed GMM-TDQN, as illustrated in Fig. 2. The framework integrates GMM clustering for deployment initialization with a TDQN for adaptive refinement [K et al., 2024].

In the first stage, a GMM is employed to initialize ES locations by clustering BSs according to their spatial distribution and traffic load. After convergence of the EM algorithm, each BS is assigned to its most probable Gaussian component, and the centroid of each cluster is selected as the initial ES position. This procedure places ESs in dense and high-demand regions, providing a favorable starting point for optimization.

In the second stage, the TDQN iteratively refines ES positions through reinforcement learning. Agents interact with the environment under an ε-greedy policy, and transition samples are stored in a replay buffer. The Transformer-based Q-network is updated using mini-batch training to capture global spatial dependencies among ESs and improve deployment decisions.

By combining probabilistic clustering, global attention modeling, and value-based reinforcement learning, the proposed GMM-TDQN framework enables efficient initialization and adaptive optimization of ES deployment. Detailed descriptions of the GMM and TDQN components are provided in Sections 4.2 and 4.3, respectively.

4.2 Initialization Using GMM

To provide a principled initialization for ES deployment, a GMM is adopted to capture the spatial distribution of BSs. Each BS bi ∈ B is characterized by its geographical coordinates and traffic load, and the overall BS distribution is modeled as a mixture of m Gaussian components, where m corresponds to the number of ESs.

The probability density of BS bi is given by

where πk, µk, and Σk denote the mixture weight, mean, and covariance of the k-th component, respectively, and Θ represents the model parameters. In this work, the feature dimension is set to d = 3, incorporating spatial coordinates and traffic load.

The parameters are estimated via the ExpectationMaximization (EM) algorithm by maximizing the loglikelihood

After convergence, each BS is assigned to the Gaussian component with the highest posterior responsibility.

Server Initialization

Based on the clustering results, the initial position of ES sk is determined as the centroid of BSs belonging to cluster Bk:

Algorithm 1 Initialization of ES deployment based on GMM

Input: Base station set B

Output: Initial ES positions S(0)

• 1: Randomly initialize µk, Σk, πk;
• 2: Set iteration counter t = 0 and log-likelihood L(0) = -∞;
• 3: while not converged do
• 4: t ← t + 1;
• 5: Compute the responsibility of component k for each base station bi;
• 6: Update model parameters based on current responsibilities µk, Σk, πk;
• 7: Compute the updated log-likelihood L(t) ;
• 8: Stop if ∣∣L(t) -L(t-1) ∣∣ < ξ;
• 9: end while
• 10: Assign each base station bi to themost probableGaussian component Bk;
• 11: Compute the initial position of each ES as the centroid of its assigned cluster s(0) k ;
• 12: return S(0) = {s(0) 1 , s(0) 2 , . . . , s(0) m }.

where loni and lati denote the spatial coordinates of ES i, Gmax is the maximum grid size of the deployment area, N near i and Ntotal represent the number of nearby BSs associ-

This initialization places ESs close to dense and highdemand regions, providing a favorable starting point for reducing latency and energy consumption while improving load balance and network reliability. The subsequent reinforcement learning module further refines the deployment to optimize the overall objective.

Algorithm 1 summarizes the GMM-based ES initialization procedure.

4.3 Optimal ES Deployment Based on the TDQN Algorithm

To further enhance the ES deployment optimization, we propose a TDQN algorithm that leverages the self-attention mechanism to capture complex dependencies among ES. This section introduces the details of the ES environment modeling and the TDQN algorithm for ES deployment.

Environment Modeling for ES Deployment

The ES deployment environment is modeled as a grid-based system with the following components:

State Space: The state representation for each ES includes its normalized coordinates, nearby BS statistics, and load information. Formally, the state for ES i is represented as:

Algorithm 2 TDQN for ES deployment

Input: Initial ES positions from Algorithm 1

Output: Final ES deployment positions S∗

• 1: Initialize TDQN parameters θ and target network θ-;
• 2: Initialize replay buffer D and environment;
• 3: for episode e = 1 to E do
• 4: Reset environment, obtain initial global state o0 , construct embedding X0;
• 5: for time step t = 0 to Tmax - 1 do
• 6: Encode Xt to Z L t via Transformer;
• 7: For each ES i, compute Q-values Qi(oi, ai; θ) for ai ∈ A;
• 8: Select actions for all ESs using ϵ-greedy;
• 9: Execute joint action at = (at1 , . . . , atm), update ES positions;
• 10: Observe reward r and next state o′ ;
• 11: Store transition (o, a, r, o′ ) in buffer D;
• 12: if buffer D contains at least B samples then
• 13: Sample mini-batch {(o, a, r, o′ )} of size B;
• 14: Compute current Q-values Qi(oi, ai; θ) for all ESs i;
• 15: Compute TD targets according to Eq.(24);
• 16: Update θ by minimizing DQN loss according to (23);
• 17: Every C steps update target parameters θ- ← θ;
• 18: end if
• 19: end for
• 20: end for
• 21: Apply the learned optimal policy π∗ (o) to determine the final ES positions.
• 22: return Final ES deployment positions S∗

ated with ES i and the total number of BSs in the system, ¯ di is the average Euclidean distance between ES i and its nearby BSs, Dmax denotes the maximum possible distance within the considered region, ¯W nearby is the mean traffic load of the nearby BSs, and Wtotal denotes the aggregated load of all BSs.

Action Space: Each ES can perform one of five possible actions:

Reward Function: The reward function incorporates multiple optimization objectives, including latency, energy consumption, load balancing, and network reliability:

where λ1 to λ4 are weighting coefficients that balance the importance of each objective.

ES Deployment Based on TDQN

To enhance the representation capability of conventional DQN, a Transformer encoder is integrated to model spatial correlations and interaction dependencies among multiple ESs, as illustrated in Fig. 3. Unlike CNN- or MLP-based architectures, the proposed TDQN explicitly captures global deployment patterns through self-attention.

Each ES i is represented by an embedding vector xi ∈ Rd , constructed from normalized spatial coordinates, local BS statistics, and load indicators. The embedding sequence is denoted as

which is projected to the model dimension and combined with positional encodings:H0 = XWE + P.

The Transformer encoder consists of L stacked self-attention layers, enabling each ES to aggregate information from all other ESs and generate a context-aware representation ZL ∈ Rm×dmodel . This representation encodes global spatial dependencies and workload interactions among ESs.

The encoded features are fed into a linear Q-head to produce per-server Q-values over a discrete action set

Let Qi(oi, ai) denote the Q-value of ES i. The TDQN is trained using the standard DQN loss:

where the temporal-difference target is defined as

During training, each ES selects actions following an ϵgreedy policy, and all ESs execute their actions simultaneously to update deployment positions. A replay buffer is used to store transition tuples (o, a, r, o′ ), and a target network is periodically synchronized to stabilize training.

By continuously interacting with the environment, the TDQN learns an optimal deployment policy π∗ (o) that adaptively adjusts ES locations to jointly reduce latency, energy consumption, and load imbalance while maintaining network reliability. Benefiting from the self-attention mechanism, the

(c) Energy consumption

(d) Network reliability

Figure 4: Performance comparison of ES deployment strategies with different numbers

proposed method enables coordinated server movements and achieves faster convergence and more balanced deployment compared with conventional DQN-based approaches.

Algorithm 2 summarizes the pseudocode of the TDQN algorithm.

5 Experimental Results and Analysis

To comprehensively evaluate the performance of the GMMTDQN approach, we compare the GMM-TDQN algorithm against several representative baseline methods spanning diverse optimization paradigms. These baseline algorithms include K-Means [Huang et al., 2025], Top-K [Chen and Xiao, 2025], Simulated Annealing (SA) [Lan et al., 2025], and TDQN (without GMM Initialization).

(a) Service delay

(b) Energy consumption

• (c) Load balancing

(d) Network reliability

Figure 5: Performance comparison under different conditions

5.1 Experimental Parameters and Configuration

This study utilizes a real-world network dataset provided by the Shanghai Telecommunications Bureau. The processed dataset comprises records for 2,753 base stations, including their unique identifiers, latitude and longitude coordinates, and access durations over a continuous 15-day period, as shown in Table 1 and Table 2.

5.2 Experimental Comparison and Analysis

Performance Comparison under Different Numbers of ESs

In this experiment, the number of base stations is fixed at 2753, while the number of ESs increases from 20 to 100. The five deployment schemes are evaluated in terms of system latency, load balancing, energy consumption, and network reliability, as illustrated in Fig. 4(a)-4(d).

The experimental results consistently demonstrate that GMM-TDQN significantly outperforms the four baseline strategies across all evaluation metrics. Its superiority becomes increasingly pronounced as system scale grows. This performance advantage stems from GMM-TDQN's multiobjective decision mechanism, which jointly optimizes latency, energy usage, load distribution, and reliability within a unified learning framework. In contrast, clustering-based K-Means and heuristic SA methods are limited by their static objectives, while the TDQN and Top-K strategies lack the capacity to reason about spatial-temporal patterns or global resource coordination. These findings underscore the necessity of intelligent and adaptive optimization methods for largescale edge server deployment in complex and dynamic environments.

Performance Comparison under Different Numbers of BSs

To comprehensively evaluate the performance of the proposed GMM-TDQN algorithm, we compare it with four baseline methods, including K-Means, Top-K, SA, and TDQN deployment, under different numbers of base stations (300-1900). The evaluation metrics include average latency, energy consumption, load balancing, and system reliability.

As shown in Fig. 5(a)-5(d), experimental results confirm that the GMM-TDQN algorithm exhibits better scalability and robustness compared with traditional clusteringand heuristic-based methods. It dynamically adapts to the increase in network density while optimizing multiple objectives, demonstrating its potential for practical deployment in large-scale edge computing environments.

6 Conclusion

We investigated the problem of large-scale ESs deployment under multi-objective optimization requirements. To address the challenges posed by scalability, conflicting objectives, and dynamic environments, we proposed GMM-TDQN, a two-stage multi-objective reinforcement learning framework. Experimental results on real-world datasets demonstrate that the proposed approach consistently outperforms state-of-theart methods, highlighting its effectiveness and scalability in realistic edge computing scenarios.

Future work will explore extending the proposed framework to more dynamic settings, such as online deployment with time-varying workloads and mobile edge nodes.

Ethical Statement

There are no ethical issues.

Acknowledgments

This paper is supported by the National Natural Science Foundation of China (grant 62402065), Humanities and Social Sciences Research Planning Fund of the Ministry of Education of China (grant 25YJA710076 and 24YJC710107), the Hunan Provincial Education Science Research Project during the 14th Five-Year Plan period (grant XJK23CXX004), the Hunan Provincial Degree and Postgraduate Teaching Reform Research Project (grant 2025JGYB360), the Natural Science Foundation of Hunan Province (grant 2025JJ50344), the Natural Science Foundation of Changsha city (grant kq2502340).