Modular Deep Reinforcement Learning for Multi-Workload Offloading in Edge Networks

1 Introduction

In recent years, the proliferation of wireless devices (WDs) has led to an unprecedented increase in the generation of realtime workloads [Zhang and Debroy, 2023]. These WDs frequently encounter considerable challenges due to their limited computational power and battery life, rendering them incapable of processing all workloads locally [Chen et al., 2024]. Mobile edge computing (MEC) has emerged as a promising solution, allowing for the offloading of workloads from WDs to nearby MEC servers [Yu et al., 2024]. This policy improves computational efficiency, minimizes latency, and enhances the overall user experience [Qu et al., 2024].

∗The corresponding author

In dynamic MEC environment, the key problem lies in determining the offloading decision-making for workloads [Feng et al., 2022] or resource allocation policy [Zhang et al., 2023], which must take into account the prevailing computation and communication conditions of the MEC servers (such as channel state and energy availability) alongside timevarying workloads (like workload size and deadlines), with the objective of minimizing the cost of finishing workloads [Luo et al., 2021]. The traditional heuristic [Almadhor et al., 2022] or convex optimization algorithm [Tan et al., 2022] can implement the workload offloading or resource allocation decision in the reliable MEC model environment. To this end, when faced with the complex environment of multiple workloads and multiple MEC servers, the aforementioned methods struggle to make near-optimal offloading or allocating decisions to minimize the system cost. Fortunately, deep reinforcement learning (DRL) emerges as an effective solution to these challenges, owing to its robust decision-making capabilities [Yang et al., 2024].

Currently, there are many efforts dedicated to developing DRL-based computation offloading strategies, e.g., vehicular edge computing [Shi et al., 2022; Hazarika et al., 2022], collaborative edge computing [Ren et al., 2024; Zhang et al., 2024]. However, most related works focus on scenarios involving a single MEC server or address situations where the workloads exhibit low complexity. In addition, research has been conducted on workload offloading strategies involving multiple edge servers, as illustrated in studies [Xu et al., 2023]. Nevertheless, these studies universally assume that there is no interaction among MEC servers and a static environment for MEC, making it challenging to address the demands posed by complex multi-workloads. Consequently, it is the general trend to study workload offloading in environments with numerous time-varying workloads and multiple edge servers, characterized by fluctuating channel states. Complex multi-workload offloading within a collaborative framework of multiple MEC servers presents several significant challenges. Figure 1 illustrates the primary challenges. The first is the heterogeneity of workloads. Workloads generated by WDs exhibit diverse categories, sizes, required CPU cycles, deadlines, etc. This diversity poses significant challenges in tackling workloads as different characteristics demand varying computation resources and processing methods, making it difficult to manage and schedule them in a unified manner. The second is the homogeneity of agents. Neural network (NN) structures employed in DRL for agents tend to be similar, without considering the reuse of different networks for different workloads. This implies that agents may not perform optimally when confronted with diverse complex workloads, as they lack the flexibility to adequately address the specific requirements associated with different workload types. The third is the inefficiency of action exploration. In a dynamic MEC environment, random sampling or uniform sampling is not applicable and leads to high deviations. such inefficient sampling strategies may hinder the agents' ability to identify optimal action plans, as they do not accurately represent the true value of various actions within a complex environment, thereby impacting the system's overall performance and efficiency.

In terms of the above issues, we conceive and design the solution, summarized as follows: State representation. We perform state representation for the state features and workload features of the workloads, and delineate each workload individually. This approach allows for a more comprehensive and accurate depiction of workloads and their states, yielding valuable insights for subsequent decision-making and facilitating improved management of workload heterogeneity. Multi-module reuse in policy networks. We implement multi-module reuse in the policy networks of DRL. This enables the agent to flexibly select and combine appropriate modules according to the characteristics of different workloads, thereby enhancing the agent's ability to process different types of workloads and addressing the issues caused by the homogeneity of agents. Prioritized experience replay with weighted importance sampling. Agents can utilize experience samples more effectively during the learning process. Through weighted sampling of experiences and preferentially replaying those experiences that are more valuable for learning, it helps to improve the efficiency of action exploration, reduce sampling deviations, and enable the agent to quickly learn the optimal action strategy in complex environments.

To the best of our knowledge, we are the first to propose DRL-based state representation, policy network modularization and prioritized experience replay with importance sampling. A flexible module weighted fusion DRL with state representation for multi-workload offloading policy (DRLMWF) in edge network is presented to learn the optimal offloading decision-making policy for minimizing the weighted average cost for finishing workloads. The main contributions of our work are summed up as follows:

• We propose a representation learning-based state representation method for multi-workload and multi-MEC environments. Different from the conventional imagebased representation policy, we characterize and normalize the features of computation and communication resources for MEC server, as well as workload features of WDs respectively to adapt to the proposed DRL-MWF's policy network.
• We formulate DRL-MWF, and redesign its policy network on the basis of soft actior-critic (SAC) to execute offloading decision-making actions with adaptive module fusion. The learning efficiency of DRL-MWF can be enhanced through the reuse and composition of its modules.
• We develop a twin critic network that facilitates module composition to address the overestimation challenges associated with the policy network, arising from modular reuse and composition, thereby enhancing the stability of DRL-MWF. In addition, we propose a prioritized experience replay with weighted importance sampling for DRL-MWF, aimed at improving the tradeoff between exploration and exploitation, and adapting to multi-workload offloading decision-making contexts in diverse MEC server environments.
• We conduct comprehensive experiments to validate the convergence and advantage of DRL-MWF compared to five other benchmarks in terms of the average cumulative reward, the ratio of unfinished workloads.

2 RelatedWork

DRL focused on how the agent took a series of actions in an environment to maximize the expected cumulative reward by learning and approximating of intricate state representations as well as value functions from deep neural network [Franc¸ois-Lavet et al., 2018]. DRL consisted of value-based scheme (deep Q-Network (DQN) [Mnih et al., 2015], double deep Q-Network (DDQN) [Van Hasselt et al., 2016], dueling deep Q-Network [Wang et al., 2016]) and policy-based scheme (Actor-Critic [Peters and Schaal, 2008], SAC [Haarnoja et al., 2018], proximal policy optimization(PPO)[Schulman et al., 2017]). DRL had been proved to be useful for controlling the discrete and continuous actions, such as: [Xiang et al., 2023], [Yang et al., 2020]. Because of its powerful decision-making ability, DRL was extensively applied in the field of MEC for workload offloading and resource allocation.

Currently, several studies had explored DRL-based workload offloading in the MEC environment. [Aghapour et al., 2023] formulated a DRL-based policy to optimize computation offload and resource allocation decision-making selection. [Huang et al., 2024] combined directed acyclic graphs (DAGs) and DRL to optimize the offloading scheme in the MEC framework. [Consul et al., 2024] proposed federated reinforcement learning (FRL)-based offloading strategy to tackle the tasks fromWDs. [Ling et al., 2024] implemented PPO-based optimization method to address the computation resource allocation issue. However, the MEC environments set by the aforementioned works were relatively simple, and the multi time-varying workloads or channel states were not considered.

To sum up, we proposed DRL-MWF offloading scheme tailored for intricate multi-workload scenarios in the MEC environment. First, we considered the state representation for complex features of multi-workload and MEC environment. Different frommost existing representation learning for complex image-based workloads, we could characterize the state of workloads and environments, thereby enabling DRL to accommodate multi-workload learning. Second, Multitasking was heterogeneous, but the WDs were similar. Assigning each WD or MEC server as an independent learning agent resulted in reduced learning efficiency. We designed network modularization and weighted combination for specific modules to improve the efficiency of DRL's learning. Third, we employed prioritized experience replay coupled with importance sampling to facilitate the exploration of actions taken by the scheduling agent, thereby enhancing the learning effect.

3 SystemModel and ProblemFormulation

3.1 SystemModel

As illustrated in Fig.2, our focus is on the offloading of multiple workloads in an MEC environment that comprises various WDs and multiple MEC servers. There are W WDs and M MEC servers connected to M base stations (BSs) 1 , meeting W = {1, 2, · · · ,W}, M= {1, 2, · · · ,M}. WD w generates different K sub-workloads Kw = {1, 2, · · · ,Kw}, that must be processed within the stringent deadline. However, given the constraints of computing capacity and battery life, WDs cannot tackle all generated workload sets, a significant portion of the sub-workloads must be offloaded to the MEC servers for processing. MEC server m contains several core components: a computing unit,a communication unit and a workload scheduling unit. The computing unit is responsible for processing the received workloads in the buffer queue from WDs. The communication unit facilitates data transmission between the WDs and the MEC

1 Note that each MEC server is associated with a BS; in this context, we refer to them as MEC servers instead of BSs.

servers. The workload scheduling unit manages the scheduling of various workloads, taking into account the available computation resources of the current MEC server. Without loss of generality, WD w may select an appropriate offloading strategy to optimize system performance. A conventional method involves the complete offloading of all workloads to one MEC server; however, due to the intricate nature of these workload sets, we explore the option of utilizing edge collaborative processing. The sub-workloads in the workload set from each WD are assigned to several MEC servers, our attention shifts to their processing on multiple MEC servers, which encompasses resource distribution of MEC servers for finishing workloads. To this end, xkw,m is introduced to denote the offloading decision variable, and ykw,m signifies the ratio of computational resources allocated for the execution of sub-workload kw generated by WD w. The decision variable xkw,m ∈ {0, 1, 2, · · · ,M} means whether the workload generated by WD m can be executed locally (when xkw,m = 0) or offloaded to MEC server m (when xkw,m = m). The decision variable ykw,m ∈ [0, 1] is the ratio of allocated computational resources at MEC server m.

Communication Model

Due to the limitation of the computation capacity of the local WD, workloads can be transmitted (offloaded) to the MEC server via the wireless network for execution. Initially, we discretize a continuous time interval T into discrete time slots t, meeting t ∈ T. Similar to [Wu et al., 2019], we consider that each MEC server operates on one of the bandwidth subcarriers that are accessible. As a result, the communication method is established as orthogonal frequency division multiple access (OFDMA). The total bandwidth available between the WDs and MEC server m is designated as Bm. According to Shannon's theorem, the transmission rate between WD w and MEC server m in time slot t can be given by

where pw,m(t) represents the transmission power for offloading workloads. hw,m stands for the channel states between WD w and MEC server m, meeting the Gaussian Markov autoregressive model [Ke et al., 2020]:

where ρm means the path-loss coefficient, describing signal attenuation as it propagates between WD w and MEC m, which is close to 1. em(t) is an error term that follows a complex Gaussian distribution, independent of the channel state.

Computation Model

We redefine the sub-workload generated by WD w in the time slot t as the tuple kw(t) = ⟨ksw (t), kcw (t), kD w (t)⟩, where kw(t) ∈ Kw. ksw (t) denotes the size of workload kw(t), ksw (t) represents the number of required CPU cycles for kw(t), and kD w (t) means the deadline for finishing kw(t).

Local Computation. As described in section 3.1, while xw,m = 0, the processor of WD w can accomplish the subworkload kw(t) it generates. To this end, the latency llc kw (t) consists of the local execution latency llc,exe kw (t) and the queue waiting latency llc,queue kw (t), which can be calculated by

where Qkw (t) is the size of workloads queue at WD w in the time slot t, and F w is the available computation frequency of WD w.

The energy consumption elc kw (t) can be given by

where κw is the effective switching capacitance who depends on the computation ability of local processor. Since the queue waiting latency are far less the execution latency, which can be ignored [Van Huynh et al., 2022].

Offloading to MEC server. While xkw,m ̸= 0, it is necessary to offload the sub-workload kw(t) from WD w to MEC server for execution. Therefore, the latency loff kw,m(t) consists of the transmission latency loff,tran kw,m (t), the execution latency loff,exe kw,m (t) and the queuewaiting latency loff,queue kw,m (t), which can be calculated by

To this end, the energy consumption eoff m (t) includes the transmission energy consumption eoff,tran m (t) and the execution energy consumption eoff,exe kw,m (t) can be derived by

3.2 ProblemFormulation

In this study, we define the optimization objective as finding the optimal workload offload selection xkw,m and the ratio of allocated computation resource ykw,m for the MEC server throughout the whole time period T, with the aim of minimizing the weighted average latency and energy consumption costs of the MEC system. Hence, the optimization objective function can be derived as

where l(t) means the average latency of all W WDs, meeting l(t) = [∑w∈W ∑kw∈Kw (1 - xkw,m)llc kw (t) + (xkw,m/m)lof f kw,m(t)]/W, and e(t) is the average energy consumption. β1, β2 are the weighted parameters for latency cost and energy consumption cost, meeting β1 + β2 = 1.

Furthermore, the optimization problem can be described as

4 Multi-workload Offloading Policy with DRL-MWF

As outlined in Section 3.2, P1 is an NP-hard problem, posing significant challenges for conventional convex optimization techniques or heuristic methods. The complexity arises from the dynamic nature of the MEC environment, characterized by factors such as the time-varying arrival workloads from WDs and varying channel states between the WDs and MEC servers. Consequently, we transform the optimization problem P1 into a Markov decision process (MDP) and define the tuple ⟨S,A, P,R, γ⟩. The state s(t) in state space S in the time slot t can be characterized as

Hence, The action a(t) in action space A in the time slot t can be defined as

Furthermore, the reward r(t) ∈ R can be defined as r(t) = -C(t) + I(.)p, where I(.) is the indicator function which means whether the workload is completed within the deadline. The term p represents the penalty factor associated with any incomplete workload. In this way, the problem of minimizing the weighted average cost is transformed into obtaining the maximum expected cumulative reward; P1 can be transformed into P2:

Due to the change of environment, we cannot obtain the state transition probability matrix P. Therefore, we convert the MDP to DRL, and propose the DRL-MWF according to the multi-workload inputs and dynamic channel states.

Figure 3 illustrates the DRL-MWF framework, highlighting state representation and module fusion strategies. Building upon SAC, we initially modularize the actor network for multi-task complex MEC scenarios, and design a twin critic network to solve the overestimation issue while enhancing the stability of the evaluation strategy for the multi-module of actor. In terms of policy network, there are two networks: a state representation network and a workload schedule network. In each time slot, the state s(t) is divided into two parts, the state s1(t) related to the environment and the state s2(t) related to the workload. We represent s1(t) and s2(t) as V s 1 and V s 2 with two-layer multilayer perceptron (MLP) and one-layer MLP, one fully connected layer, respectively. For the represented state vector V s 1 , we implement AvgL1Norm to prevent representation collapse and unify the vector scale by V s 1 ||V s 1 ||1 /D , where D is the dimension. To this end, the schedule network obtains the represented and normalized states V s 1 and V s 2 with D-dimension vector and further output the probability vector by ReLU function. The probability vectors for the first layer and the l + 1-layer can be computed as p1 = FC1d (ReLU(V s 1 , V s 2 )) and pl+1 = FCld (ReLU(plV s 1 , V s 2 )), respectively. In terms of policy network, there are L layers and K modules in each layer and the state representation for module k in layer l can be written as V l j = ∑k∈K S(pl-1 (j, k))ReLU(W l-1 k V l-1 k ), where S is the Softmax function and W l-1 k is the weighted matrix. This means that we can select modules of each layer based on probability, realize module reuse, and improve the efficiency of network training. Finally, the last layer outputs the action by mean and variance.

To ensure the reuse efficiency of the proposed DRLMWF's policy module, we propose the following two improvement strategies to enhance the network training effectiveness and algorithm performance.

First, leveraging the twin critic networks of SAC, we modify the maximum entropy objective function and incorporate stability constraints to minimize the subsequent loss function aimed at preventing overestimation:

where y = r(t) + γ · mini=1,2 Qj (s′ (t), µ(s′ (t)) + N ).

Therefore, the objective function of policy network is updated by J(π) = Es∼ρπ ,a∼π[Q(s, a) - α logπ(a|s) - λd∆(s, a)], where λd means the weighted adjusting coefficient. ∆(s, a) is the deviation correction term which can be estimated by multiple sampling, satisfying ∆(s, a) = 1N ∑i∈N (max(Q+1i ,Q+2i ) -Q(st, at). Q+1i ,Q+2i stand for the estimations of the Q function. This approach allows for the mitigation of Q-function overestimation, thereby enhancing the stability of policy optimization.

Second, beyond regulating the balance between exploration and exploitation in the policy through the temperature parameter α of the traditional SAC, we also develop a priority experience replay mechanism utilizing weighted importance sampling to further enhance policy exploration by:

where Ld means the loss of sample d. pd signifies the priority of sample which can be obtained through TD Error as pd = |∆d| + ϵ. ∆d stands for the TD error of sample d, ϵ is a small constant used to ensure that the priority is not zero.

We employ the sampled experiences to update the learning process; however, it is necessary to adjust the learning rate. Consequently, we introduce weighted importance sampling to maintain the unbiased nature of the learning process. the actual importance weight associated with the sampled experience is denoted as ωd, which can be expressed as,

where β is another hyperparameter used to control the scaling of the importance weight. During the actual training process, the loss function is adjusted by incorporating the weighted TD error and loss, updated by L(Q) = ∑d∈D ωd · Ld.

5 Experiments

In this section, we first introduce the datasets for multiworkload, multi-edge server in dynamic MEC environment, then describe the experiment setups, metrics and benchmarks. Finally, we analysis evaluation results from the convergence and performance of DRL-MWF.

5.1 Datasets

We utilize a universal EUA dataset [Lai et al., 2018] comprising 125 MEC Servers/BSs and 816 WDs located in Melbourne street district. We initialize 5 BSs equipped with MEC servers and 50 WDs from EUA dataset. For the size ksw (t) and required CPU cycles kcw (t) of workloads, we employ the UE application data [Rojas, 2023] and the SPEC CPU 95 benchmark [Phansalkar et al., 2005] to simulate multiple workloads in the MEC environment.

5.2 Environment Settings andMetrics

Regarding the workload distribution, the sizes of three subworkloads are set as 1 ∼ 4Mb, 3 ∼ 6Mb, 5 ∼ 8Mb from UE application dataset. Additionally, the required CPU cycles per bit for three sub-workloads are set as 800, 1000, 1200 from SPEC CPU 95 benchmark dataset. Table 1 lists additional key parameters in the environment. For DRL-MWF, the parameters include the number of training episodes, the learning rates for both the actor and critic, the discount factor, the size of experience memory, the number of modules, and the number of layers in the actor's network, which are set to 1000, 0.0001, 0.001, 0.9, 10000, 3, 2, respectively.

To assess the convergence of DRL-MWF, we utilize the average cumulative reward [Ke et al., 2022], represented as RACR = 1 T ∑t∈T r(t). In terms of the performance comparison of DRL-MWF against benchmarks, we evaluate not only RACR, but also the average latency cost C ALC = 1 T ∑t∈T l(t), the average energy consumption cost C AEC = 1 T ∑t∈T e(t), and the ratio of unfinished workloads U RUW calculated by dividing the number of unfinished workloads by the total workloads [Tang and Wong, 2020].

5.3 Benchmarks

We introduce five benchmarks for comparative analysis: (1) DQN-LSTM [Tang and Wong, 2020], where Dueling DQN is employed for MEC selection decisions, and LSTM is used to forecast the MEC server's task queue information. (2) MCODRL [Ke et al., 2022], which combines DDQN and gated recurrent unit (GRU) to facilitate workload offloading to the MEC server while estimating both the workload size queue. (3) SAC [Shi et al., 2020], which proposes a computation offloading policy based on SAC aimed at reducing the average latency of tasks. (4) PPO [Shang et al., 2024], which implements a computation offloading policy utilizing PPO to minimize the weighted costs. (5) TD7 [Fujimoto et al., 2023], an emerging DRL strategy designed for multi-task management, is applied for the first time in MEC environment.

5.4 Evaluation Results

Convergence analysis of DRL-MWF. Firstly, we analyze the convergence of DRL-MWF. RACR represents the average cumulative reward with penalty terms p for unfinished workloads, enabling the evaluation of DRL-MWF's convergence.

Figure 4 reveals the convergence of DRL-MWF under different parameters. As shown in Fig.4(a), DRL-MWF can converge under four different deadline settings which reflects the effectiveness of the proposed modular DRL. When kD w (t) = 6, a significant number of workloads fail to be completed within the stringent deadline, leading to latency penalties and a decrease in RACR. As kD w (t) gradually increases from 8 to 12, MEC servers can fully schedule their own resources, which in turn enhances the workload completion ratio and leads to an increase in RACR. kD w (t) is set as 10 in this work. As revealed in Fig.4(b), we compare DRL-MWF with the weighted policy correction factor λd and importance sampling factor ωd for prioritized experience replay and without λd and ωd. Although DRL-MWF can converge without λd and ωd due to the modular structure, the performance of DRL-MWF without λd and ωd is worse than that of DRLMWF with λd and ωd in terms of RACR, which verifies the key of introducing two important mechanisms 2 .

Performance comparison with benchmarks. Table 2 illustrates the performance of benchmarks and our proposed DRL-MWF in terms of average cumulative rewards RACR ± 95% confidence interval (CI). The performance of each algorithm varies across different combinations of MEC servers and WDs. For instance, in dataset with MEC2-WD10, the proposed DRL-MWF performs best with RACR of - 8.58±0.013 because of the advantages of state representation and module grouping. As the configuration of datasets changes, the performance rankings of the algorithms also change. However, the performance of DRL-MWF with weighted modules is relatively stable across various configurations, and its RACR remains at a relatively high level compared to other benchmarks. Priority experience replay with importance sampling and maximum entropy revision mechanism ensure the effectiveness and stability of DRL-MWF.

Figure 5 illustrates the comparison on six algorithms with different probability of arriving workloads. From Fig.4(a), DRL-MWF has a greater RACR in any cases. With an increase in the probability of incoming workloads, RACR of DRL-MWF does not show a sharply decreasing trend due to the state representation and module fusion. The relationship between C ALC and the probability of arriving workloads is depicted in Fig.5(b). Compared with other benchmarks, DRL-MWF shows relatively high RACR at increased probability of arriving workloads; however, this advantage diminishes progressively as the probability declines. Fig.5(c) presents C AEC of six algorithms. DRL-MWF retains lower C AEC throughout the process, indicating its advantage in energy conservation. Fig.5(d) illustrates that the probability of arriving workloads influences U RUW. DRL-MWF shows superior performance in controlling the ratio of unfinished workloads under different probability of arriving workloads.

2 DRL-MWF focuses on exploration at the beginning of training, setting λd to 0.2. The over-estimation of Q-value is noticeable. λd gradually increases from 0.2 to 0.7. For the importance factor ωd, β in Eq.14 is initially set as 0.5 to enhance exploration, then gradually increased to prioritize experiences, finally reaching 1.

6 Conclusion

In this work, to address the state representation of heterogeneous workloads, multi-agent training difficulties, and efficiency of action exploration of DRL-based offloading methods in multi-workload MEC environment, we proposed a flexible modular DRL with state representation for multiworkload offloading policy (DRL-MWF). DRL-MWF can implement state representation and normalization, and modularize the policy network, then utilize policy function weighted correct and priority experience replay with importance sampling to improve the efficiency and stability of training. Experiment results show that DRL-MWF can converge in a certain number of episodes and whose performance is better than that of DRL-based workload offloading schemes.

Acknowledgments

This work was supported by the Outstanding Young Science and Technology Talents Project of Jilin Province Science and Technology Development Plan(20240602004RC), the Natural Science Foundation of Jilin Province Science and Technology Development Plan (20250102246JC), the Key R & D Project of Jilin Province Science and Technology Development Plan (20240302080GX) and the Key Scientific Research Project of Jilin Provincial Department of Education (JJKH20251146KJ).