Model-Free Real-Time Autonomous Energy Management for a Residential Multi-Carrier Energy System: A Deep Reinforcement Learning Approach

1 Introduction

1.1 Background andMotivation

Energy systems worldwide are facing major flexibility challenges driven by the limited system ability to provide secure and economical supply-demand balance in face of the largescale integration of renewable energy sources and their inherent variability. In this respect, it is increasingly recognized that integrated optimization and management of multi-carrier energy systems (MCES) represents a key opportunity to provide the required flexibility to support cost-effective evolution to smart low carbon future. This is enabled by their significant yet untapped potential to shift supply and demand across energy carriers by exploiting different forms of energy storage, compared to traditional energy systems whose carriers are operated independently [Mancarella, 2014].

∗Contact Author

At the residential level, with the increasing prevalence of distributed energy resources and advanced metering and communication infrastructure, end-users rely on autonomous energy management systems (AEMS) to actively control their generation, conversion, consumption and storage of energy in real-time, in order to reduce their demand placed onto the grid and their costs. Therein, substantial socio-environmental benefits emerge as a result of the reduced carbon emission.

An integrated solar photovoltaic (PV) and electricity energy storage (EES) system serves to compensate the mismatch between the time-varying and uncertain PV generation and electricity demand (ED) as well as making some revenue by injecting surplus PV to the electrical grid (EG). Meanwhile, the increasing adoption of electric vehicles (EV) by end-users is observed, driven by their ability to store electrical energy in their batteries, enabling the temporal decoupling between the absorption of energy from EG and its actual consumption for travelling purposes. Furthermore, coupling the operation of an electric heat pump (EHP), a gas boiler (GB) and a thermal energy storage (TES) introduces opportunities for an energy-shifting arbitrage between electricity and gas to supply electricity and heat to the end-users, which is particularly useful in face of intermittent RES. Furthermore, TES enables redistribution of heat demand (HD) and production across time which contributes to greater energyshifting flexibility. In view of such complex interactions and inter-dependencies between different energy carriers as well as the multi-source system uncertainties, developing an effective AEMS plays an crucial role in uncovering the flexibility potential and harvesting the real-world benefits of the MCES.

1.2 RelatedWork

Energy management of the MCES has been traditionally addressed with model-based control approaches. A centralized operation cost minimization problem is solved to determine the optimal schedule of various kinds of controllable energy sources, loads and storage devices [Bozchalui et al., 2012; Rastegar and Fotuhi-Firuzabad, 2015; Moghaddam et al., 2016; Basit et al., 2017]. This optimization problem generally requires full knowledge of the operational model and parameters of the MCES, and the energy usage schedules are determined using forecasted parameters, such as the energy demand, price patterns, EV user's commuting behavior and weather-dependent PV production. However, such deterministic optimization models fail to account for the inherent uncertainties associated with these parameters. To this end, stochastic programming approaches are employed to schedule different energy devices [Pazouki et al., 2014; Vahid-Pakdel et al., 2017; Sedighizadeh et al., 2018]. A scenario-based method is used to model the uncertainties. However, it poses significant challenges to identify appropriate probability distributions and construct a representative set of scenarios for all the involved uncertain parameters since they are influenced by various exogenous factors which are independent to the energy management strategy. Furthermore, the scale of the optimization problem increases drastically with the number of scenarios, resulting in significant computational burden [Conejo et al., 2010]. An alternative approach consists in applying distributed control techniques. The alternating direction method of multipliers algorithm is adopted for energy usage scheduling of smart buildings and energy internet [Zhang et al., 2017]. A consensus-based algorithm is employed in [Li et al., 2016] for energy management of a combined heat and power system. However, the convergence to the optimal solution for both algorithms is only guaranteed assuming convex operating characteristics of the system and a very large number of iterations is generally necessary before a good measure of convergence can be obtained.

In contrast, reinforcement learning (RL) is a model-free control approach that consists of an agent (AEMS) gradually learning the optimal control policy by utilizing experiences acquired from its repeated interactions with the environment (MCES). RL also makes no assumption regarding the convexity of the operating characteristics of the MCES. In other words, the AEMS does not require prior information on the model dynamics of the MCES and considers the latter as a black box. In the big data era, RL can utilize the increasing volume of data collected from smart meters and perform successive interpretation of data to learn optimal management strategies and thereby coping with the uncertainties that are encapsulated in the data. The trained model can be deployed to deliver real-time control on timescales of milliseconds. Furthermore, RL enables a representation of the control actions to be constructed that generalizes to previously unseen situations. In this context, previous works have employed Q-learning (QL) for optimal demand response [Liang et al., 2013; Wen et al., 2015] and control of an integrated PV and EES system [Berlink et al., 2015; Kim and Lim, 2018]. However, QL suffers severely from the curse of dimensionality. The discretization of both state and action spaces may distort the feedback that the agent receives regarding the influence of its actions on the environment and adversely affect the feasible action space, resulting in suboptimal policies. This challenge is aggravated in the setting of the examined problem, since both state of the environment (e.g. PV output, the energy content of the EES) and agent's actions (e.g. charging / discharging schedule of EES) are not only continuous but also multi-dimensional.

In view of these limitations, more recently, the deep Q network (DQN) method is applied to determine home energy management decisions considering a heating, ventilation and air conditioning system [Wei et al., 2017], shiftable loads [Mocanu et al., 2019], EV [Wu et al., 2018; Wan et al., 2019] and an integrated renewable energy and EES system [Chen and Su, 2018]. DQN employs a deep neural network (DNN) to approximate the Q-value function. Despite the generalization capability of DQN to multi-dimensional continuous state space, its performs sub-optimally in problems with continuous action spaces, because the employed DNN is trained to produce discrete Q-value estimates rather than continuous actions [Lillicrap and et al., 2016]. For instance, only seven discrete power levels are assumed for the EV charging in [Wan et al., 2019] and the action of the EES is limited to three options: fully charging, fully discharging or idle in [Chen and Su, 2018]. This substantially limits the flexibility potential of energy storage devices and thus hinders the effectiveness of DQN in addressing the examined problem, since the energy management decisions are multidimensional and continuous. To this end, the deep policy gradient (DPG) method is adopted in [Mocanu et al., 2019; Ye et al., 2019]. DPG also employs a DNN, but instead of estimating the Q-value function, it directly estimates the probability of taking an action at a specific state. However, the underlying energy management actions considered in [Mocanu et al., 2019] are limited to discrete on / off status of different loads while their actual energy consumption schedules are determined by solving a cost minimization problem, which implies that the employed management strategy is not model-free and discrete in nature. Furthermore, DPG generally suffers from high variance in its gradient estimates which results in slow convergence [Silver et al., 2014].

1.3 Contributions

This paper attempts to fill the knowledge gap and address the fundamental limitations of previous approaches through the following novel contributions:

- A novel autonomous energy management strategy is proposed for a residential MCES by applying the prioritized deep deterministic policy gradient (PDDPG) method. This approach is model-free and requires neither the knowledge of the system modeling of the MCES nor any forecasted exogenous information, as opposed to traditional model-based approaches. To the best of the authors' knowledge, and according to the up-to-date literature review conducted in [Mason and Grijalva, 2019], this is the first time that this approach has been used to address the optimal energy management problem of an MCES.

• In contrast with earlier works which all target a single and isolated energy carrier and only sought to control the power output of a single energy device, the energy management of a MCES constitutes a more challenging task as it necessitates monitoring and managing the activity of each device in the MCES so as to adequately capture the synergistic effects of the couplings between different energy carriers.
• Furthermore, as opposed to previous works which largely simplify the energy management problem employing discrete control RL method, the proposed approach conforms to the nature of the energy management problem by setting it up in multi-dimensional continuous state and action spaces and properly accounting for the effect of non-convex operating characteristics of the MCES.
• The value of the proposed real-time energy manage-

ment strategy is validated through numerous numerical experiments in a real-world scenario accounting for uncertainties stemmed from the supply / demand sides of the MCES.

2 EnergyManagement as anMDP

A finite Markov decision process (MDP) with discrete time step is applied to formulate the energy management problem. The time interval between two adjacent steps is one hour. The AEMS constitutes the agent, while the residential MCES represents the environment, featuring an integrated electricity, heat and gas systems, as depicted in Figure 1 (a). The electricity system is composed of the EG, a PV, an EES and an EV. The heat system consists of an EHP, a GB and a TES. The MCES is also connected to the gas grid (GG) for the operation of GB. The objective of AEMS is to optimally manage the energy usage schedules of all devices in each carrier of the MCES so as to minimize the total cost for the end-user.

State. The state st at time step t is defined as a 12dimensional vector st = [Et, t, P ev t , Etr t , P d t , λt, P pv t ], where Et = [Ees t ,Eev t ,Ets t ] represent the energy contents of the EES, EV and TES, these are endogenous state features which are affected by agent's actions; t denotes the step identifier; P ev t indicates the maximum charging / discharging limit 1 of EV; Etr t represents the electrical energy requirements 2 of EV for travelling purposes; P d t = [P ed t , P hd t ] represent ED and HD; λt = [λet , λe+ t , λgt ] represent electricity buy / sell prices and gas price; and P pv t denotes the PV production. These are exogenous state features which are decoupled from agent's actions and are characterized by inherent variability and uncertainty. Note that, our approach takes into account the consumer preference by incorporating the EV user's commuting behavior as exogenous state features.

1P ev t is equal to a fixed charging / discharging limit when EV is parked at home (assuming home-charging) and 0 otherwise.

2Etr t is equal to a fixed energy consumption level (dependent to the distance travelled) when EV is traveling and 0 otherwise.

Action. Given the state st, the actions at of the AEMS at time step t is defined as a 11-dimensional vector at = [aesc/d t , aevc/d t , atsc/d t , aes t , aev t , ats t , agg,gb t , agb,ts t , apv,hp t , apv,ed t , aes,ed t ]. As illustrated in Figure 1 (b), the blue dashed arrows indicate the action that manages the respective power input / output for each device of the MCES.

Specifically, aesc/d t , aevc/d t and atsc/d t ∈ {0, 1} represent the charging (1) / discharging (0) status of the EES, EV and TES, these actions represent the inherent non-convex operating characteristics of energy storage devices [Ye et al., 2014], ensuring that they operate exclusively either in the charging or discharging mode; aes t , aev t and ats t ∈ [0, 1] represent the charging / discharging power of EES, EV and TES as a percentage of their maximum limits P es , P ev t and P ts ; agg,gb t ∈ [0, 1] represents the gas input of GB as a percentage of its maximum limit P gb ; agb,ts t ∈ [0, 1] represents the charging power of TES from GB as a percentage of the total charging power of TES P tsc t ; apv,hp t and apv,ed t ∈ [0, 1] represent the power flow from PV to EHP and to ED both as a percentage of P pv t ; and aes,ed t ∈ [0, 1] represent the discharging power of EES to supply ED as a percentage of its total discharging power P esd t .

State transition. The state transition fromst to st+1 is governed by a function: st+1 = F(st, at, ωt). The transition may be not only affected by the action at but also influenced by the randomness ωt existed in some state features. In the examined problem, the transitions for P ev t , Etr t , P d t , λt and P pv t are subject to variability and uncertainties. Identifying probability distributions to accurately capture such randomness can be very challenging since they are affected by many factors, such as EV user's commuting behaviour, pricing process of the utility, and the weather conditions. To resolve this issue, a model-free approach is proposed to learn the transition for such features from real-world data-set using machine learning techniques. On the other hand, the transitions for Ees t , Eev t and Ets t are directly affected by energy management actions. After executing actions at, the state transitions of these features as well as the resultant power flows indicated in the MCES (Figure 1 (b)) can be derived according to the operational constraints that characterize the MCES. For space limitation reasons, these derivations are not presented here.

Reward. The reward rt resultant from the energy management decisions at is set to be equal to the negative operation cost Ct of the MCES as given by:

where the three terms represent, respectively, the cost of purchasing electricity from EG, purchasing gas from GG, and the revenue from from selling excess PV production to EG.

Performance and value functions. At each time step, the agent employs a policy π to interact with the MDP and emit a trajectory of states, actions and rewards: s1, a1, r1, s2, a2, r2, ... over S × A × R. The return Rt = ∑Tl=t γ (l-t) rl is the discounted reward where γ ∈ [0, 1] is the discount factor. The agents' goal through RL is to construct a policy that maximizes the cumulative discounted reward, denoted by the performance function J(π) = E[R1|π] = Es∼ρπ ,a∼π[r], where ρπ denotes the discounted state distribution. The Q-value function Qπ(s, a) = E[R1|s1 = s, a1 = a; π] forms an estimation of the discounted reward.

3 Proposed EnergyManagement Strategy

As discussed in Section 1.2, despite the wide popularity of adopting QL and DQN for energy management problems in the most recent smart grid literature, these approaches suffer from the curse of dimensionality to some extent, since they necessitate discretization of the state and / or the action spaces. However, the discretization of the state space may distort the feedback that the agent receives regarding the impact of its actions on the environment while the discretization of the action space may adversely affect the feasible action space, resulting in sub-optimal policies. Furthermore, although DPG can be used for continuous control, the gradient estimator of DPG generally suffers from low sampling efficiency and high variance in its gradient estimates which results in slow convergence [Konda and Tsitsiklis, 2000].

Aiming at addressing these limitations, the PDDPG method [Ye et al., 2020] features an actor-critic architecture and employs two DNNs for different purposes [Lillicrap and et al., 2016]. The critic network Qθ takes as input a state st and action at and outputs a scalar estimate of the Q-value function Qθ(st, at). The actor network µφ takes as input a state st and implements the policy improvement task which updates the policy with respect to the estimated Q-value function and outputs a continuous action µφ(st). Learning the Q-value function in addition to the policy serves to significantly reduce the variance in the gradient estimates compared to DPG and consequently promises better convergence properties.

Concerning policy improvement, the common approach adopted in QL and DQN is a greedy maximization of the Qvalue function. However, greedy policy improvement tends to be intractable in multi-dimensional continuous action spaces as it necessitates maximizing the Q-value function globally at every time step. Instead, the proposed method employs the actor network µ to generate an action µφ(st+1) for the next state. The critic network then implements the policy evaluation task, appraising the policy by producing an estimate of the Q-value function with TD learning. Rather than globally maximizing Qθ(st, at), the critic calculates gradients ∇aQθ(st, at) which indicate directions of change of action resulting in higher estimated Q-values. These gradients are computed via back-propagation through the critic, which is more computational efficient than solving an optimization problem in continuous action space.

Analogous to DQN, PDDPG incorporates target network and the experience replay [Mnih and et al., 2015] as mechanisms to stabilize the training process. In the former, an online and a target network are used to separate the Q-value update and the target Q-value evaluation. The weights of the target networks are updated by having them slowly tracking the online networks to constrain the target values to change slowly so as to improve the stability of the learning process. In the latter, the sequentially generated training experiences are stored in a replay buffer and sampled to train the DNNs, diminishing the temporal correlations existed in the replayed experiences. To further enhance the sampling efficiency, the prioritized experience replay [Schaul et al., 2016] method is employed which prioritizes learning from experiences corresponding to higher absolute TD error which promises both improved policy and faster learning speed. When prioritized sampling a minibatch of N transitions {(sn, an, rn, sn+1)}Nn=1 , the actor is updated by applying the policy gradient theorem [Silver et al., 2014]:

Then, by defining the absolute TD error |δn|, the critic is updated by minimizing the loss function Lθ,

where Wn represents the weighting factors related to the probability of sampling experience n based on |δn|.

4 Case Studies

4.1 Experiment Setup and Implementation

A real-world scenario developed by the UK government [Pudjianto et al., 2013] is used to train and evaluate the proposed energy management strategy. The data is provided over a yearly horizon and hourly resolution. The electricity buy price follows the time-of-use tariff structure provided in [Hydro, 2019], partitioned into summer and winter periods (Figure 2). The sell price is set as the UK feed-in tariff [Ofgem, 2019]. The gas price is provided by a major UK gas supplier [E.ON, 2019]. The average driving patterns of the EV is taken from [UK Department for Transport, 2018]. Based on which, the EV is assumed to make two journeys per day, each is defined by a start time, end time and electrical energy requirement. The EV is assumed connected to EG during the period between the end of their second and the start of their first journey. The technical parameters of EES, TES, EHP and GB are taken from [Pudjianto et al., 2013].

The one-year data-set is split into training, validation and test sets. In each month, the first 20 days are used for training, and the remaining days are selected for performance evaluation. The values of most of the hyperparameters associated with the design and training of the neural networks and the design of the PER have been gathered from [Lillicrap and et al., 2016] and [Schaul et al., 2016], respectively. We found that the discount rate and the mini-batch size have the greatest impact on the algorithm performance. To tune these hyperparameters, we split the training data and used the first 15 days for training and the last 5 days for validation.

To demonstrate the value of PDDPG, we compare it with DQN and DPG. Their implementations are briefly discussed as follows. In order to apply DQN, we discretize all the action dimensions, apart from the first three, in three integer values representing 0%, 50%, and 100% energy usage. It should be noted that although the action space can be discretized with a higher granularity, it leads to an exponential growth of the number of actions, and the resultant DQN is impractical to train. In order to model continuous control, we represent the probability distribution of agent's action with a normal distribution, and predict the mean and variance of it with a DNN.

4.2 Performance Evaluation

In this section, the proposed PDDPG approach is evaluated and compared with several benchmark solutions, including DQN, DPG, as well as three model-based approaches. The first one employs a "theoretical" optimal controller which minimizes the daily energy cost, assuming full knowledge of the model and parameters of the MCES and perfect forecast of the uncertain parameters. This controller formalizes the problem as a mixed-integer linear program (MILP), the optimal solution of which can be regarded as a lower bound on the cost, indicating how far from the optimum the model-free DRL controllers are. The second one is an hourly ("myopic") MILP which solves an hourly energy cost minimization problem that neglects the time-coupling operating characteristics of all energy storage devices. The third one resorts to model predictive control (MPC) [Kou et al., 2015]. At each time step, the MPC computes the control action by solving a cost minimization over a rolling horizon of 8 hours, and only the first element of the obtained control sequence is implemented. An LSTM network is used to forecast the exogenous system parameters. We train each DRL method for 2 × 104 episodes for 10 different random seeds. Each episode represents a random day from the training set and consists of 24 time steps. We assess the quality of the energy management strategy every 200 episodes during training by evaluating it on the test set. Figure 3 illustrates the average daily cost over the 125 test days for the examined three DRL methods with 10 seeds. The mean and the standard deviation of the average daily cost over the 10 seeds are illustrated through the solid lines and the shaded areas, respectively, in Figure 3. The cumulative daily energy costs of the test 125 days under PDDPG and all examined baseline methods are presented in Figure 4.

As illustrated in Figure 3, among the model-free DRL approaches, PDDPG improves its policy in a stable fashion and eventually only PDDPG manages to converge to a near-optimum solution with a decreasing standard deviation.

PDDPG significantly outperforms benchmark DRL methods, achieving the lowest average daily cost 520.95 pence and the smallest standard deviation 3.48 pence at convergence. In relative terms, PDDPG achieves 16.75% / 5.20% lower average daily cost and 25.32% / 44.10% lower standard deviation over DQN / DPG, respectively. PDDPG and DPG both outperform DQN towards the objective of cost saving. This is attributed to their ability to model multi-dimensional continuous action space, in contrast to the naïve discretization approach employed in DQN. DPG and PDDPG allow the AEMS to represent more accurate information from the entire action space and thus discover more cost-effective management strategy by exploiting it. Furthermore, PDDPG exhibits advantageous convergence properties compared to DPG in terms of the obtained average daily cost and learning stability. Its superior performance could be explained by i) PDDPG implements a critic, which estimates the Q-value function and appraises each action that the agent takes. In contrast, DPG lacks the policy evaluation step, resulting in high variance in its gradient estimates and ii) PDDPG incorporates the PER mechanism which more frequently replays experiences corresponding to higher TD-error, improving the learning performance.

As depicted in Figure 4, the costs obtained by the four benchmark approaches, DPG (blue solid line), MPC-LSTM (magenta dashed line), Hourly-MILP (cyan dashed line), and DQN (green solid line) are 7.24%, 10.08%, 15.83%, and 21.71% higher than the theoretical optimum, respectively. Among the model-based baselines, neglecting the time-coupling operating characteristics of EES, TES and EV, hourly MILP corresponds to the highest average daily cost.

The cost under MPC-LSTM is second highest as a result of accumulated forecasting errors of the exogenous parameters. In comparison, PDDPG (red solid line) achieves an average cost that is only 1.84% higher than the theoretical optimum (black dashed line), outperforming significantly both modelfree DRL and model-based baselines.

To further investigate the performance of PDDPG in coping with the MCES uncertainties, we deploy the trained model and analyse the obtained energy usage schedules of the MCES for a typical summer and winter day selected in the 125 test days, as shown in Figures 5 and 6, respectively. The summer day (Figure 5) is characterized by abundant PV generation and small HD. PDDPG learns to supply the majority of ED from PV and EES discharge; as such, the end-user only imports energy from EG during a few off-peak hours, leading to a significant cost saving. The majority of the peak PV generation is stored in EES (which discharges at night to supply ED when PV production is low or none) instead of selling to EG because electricity buy price at night is still higher than the sell price. EV is charged at lowest-priced early morning hours to fulfill its energy requirement to travel. Furthermore, PDDPG learns to exploit the flexibility offered by multiple supply sources and storage devices so as to supply HD at zero energy cost. Specifically, EHP is controlled to take advantage of the abundant PV generation and the EES discharge to supply HD; additionally, the operation of TES is coupled with EHP which enables a redistribution of HD across time, diminishing the need of EHP to source any electricity from EG.

The winter day (Figure 6), on the other hand, is characterized by scarce PV generation and large HD. As a result, the overall energy import from EG and GG are both increased. Nevertheless, PDDPG can adaptively adjust to the winter condition towards the cost-saving objective by making very efficient use of the limited PV generation during the mid-day periods to supply ED and HD (through EHP and TES), and also charge a significant amount of PV to EES, which later discharges at peak ED and HD periods where the electricity buy price is also at its highest. Additionally, apart from fulfilling EV's traveling energy requirement, more energy is charged to the EV battery during off-peak periods and discharged to supply ED before the first and after the second EV journey when the buy price adopts its mid-peak value and the PV generation is absent, significantly shaving the ED peak. Furthermore, during the periods where the electricity buy price is relatively low, and driven by the considerably higher conversion efficiency of EHP (compared to GB), PDDPG learns to source the supply of HD from EHP. During the periods where electricity buy price is at its peak, it makes efficient use of i) GB to convert gas to heat and ii) TES to charge from EHP during the off-peak periods and discharge to supply HD at these periods.

4.3 Value of Continuous EnergyManagement

This section analyzes in more depth the physical significance of PDDPG to model continuous actions in the context of the examined problem by comparing to the DQN method. Figure 7 illustrates the energy scheduling decisions of the MCES obtained using DQN for the examined summer day.

Due to the discretization of actions aes t and apv,hp t , the enduser imports more energy from EG, leading to a higher cost of 238.86 pence (PDDPG: 26.00 pence) for supplying ED. DQN also makes inefficient use of the bountiful PV production compared to PDDPG with only a small amount of PV supplied to EHP and charged to EES, but a significant amount sold to EG. Consequently, to meet the majority of HD, EHP is required to purchase a significant amount of electricity from EG, leading to a sharp cost increase of 80.50 pence (PDDGP: 0) for supplying HD. Furthermore, TES can only charge / discharge at peak HD periods resultant from the discretisation of action ats t , limiting its flexibility in redistributing HD in time. Lastly, due to the discretisation of action aev t , EV needs to charge for 3 hours at the maximum limit (3kW) to fulfill its daily traveling energy requirement (6.6kWh), as indicated in Figure 5(d), contributing to a higher cost of 58.50 pence (PDDGP: 41.75 pence) for EV charging.

Overall, although the MCES obtains a higher revenue from selling PV surplus to EG, the daily energy cost of DQN (239.96 pence) is approximately 3.68 times as high as the one of PDDPG (65.22 pence). Furthermore, the end-user's total energy demand placed on the grids of DQN (42.19kWh) is around 4.05 times as high as the one of PDDPG (10.42kWh). It can be concluded that learning in discrete action space may result in sub-optimal management strategies where the nonconvex operating characteristics of different storage devices and complex interactions between different energy carriers must be taken into account. On the other hand, PDDPG preserves all the relevant information concerning the entire action space, and thus is capable of learning more cost-effective management strategies.

5 Conclusions

This paper showcases the first application of PDDPG to realtime autonomous energy management for a MCES. The problem is formalised as a MDP which takes into account the uncertainties stemming from both the supply and demand sides of the MCES. The proposed approach is model-free and datadriven, which does not rely on knowledge of the system modeling of the MCES and any forecasted exogenous information. The proposed approach respects the nature of the examined problem by posing it multi-dimensional continuous state and action spaces, enabling the AEMS to receive accurate feedback regarding the impact of its energy management strategy on the status of the MCES, and devise more costeffective strategies by exploiting the entire action domain, also accounting for the effect of non-convex operating characteristics of the MCES.

Case studies on a real-world scenario have provided numerous new and valuable insights around the value of the proposed energy management strategy. Quantitative results have demonstrated that PDDPG manages to converge to a nearoptimum solution and achieves a significantly higher cost savings for the MCES compared with existing model-free DRL as well as model-based baselines. Furthermore, PDDPG enables representation of real-time energy management policies to be constructed that can adaptively cope with system uncertainties towards the cost-saving objective and generalize well to previously unseen situations.