Exploring the Vulnerability of Deep Reinforcement Learning-based Emergency Control for Low Carbon Power Systems

1 Introduction

To facilitate the transition to a secure, smart, and low carbon power system, the increasing penetration of renewable energy sources, the widespread deployment of power electronic devices, and the integration of cyber and physical spaces bring about unprecedented challenges for Transmission System Operators to maintain the reliability and resilience of power systems. For example, the UK suffered its most significant power outage in over a decade in 2019 because of the generator faults, lasting more than 1.5 hours and causing widespread disruption to the traffic signal network, affecting about 1 million people. As a last resort, power system emergency control is of great importance for real-time operation to diminish the occurrence frequency and severe impact of power outages or blackouts. In particular, emergency control includes Under Voltage Load Shedding (UVLS), Under Frequency Load Shedding (UFLS), generation redispatch or tripping, dynamic braking, and controlled system separation [Kundur et al., 2000], which are highly dependent on power system physical models and their abilities are limited to deal with complex and rapidly changing dynamic conditions [Zhang et al., 2018]. For example, although there are some adaptive schemes of UFLS [Banijamali and Amraee, 2018] based on system behavior that have been investigated, these approaches exhibit limited efficiency and effectiveness in guaranteeing the stability for the unknown failures as the parameters for calculating integral are almost off-line designed based on the specific failure set. To this end, it is imperative to develop a new paradigm of adaptive and realtime emergency control schemes by employing cutting-edge Deep Reinforcement Learning (DRL) technologies.

∗Corresponding Author

In the literature, the DRL algorithm, which makes use of the advantages of Reinforcement Learning (RL) in sequential decision-making problems and combines with the idea of Deep Learning (DL) to improve the limitations of slow convergence of RL [Mnih et al., 2015], has shown to be well suitable for power system emergency control, which is a dynamic, sequential decision-making problem underuncertainty. For example, the Deep Q Network (DQN) algorithm [Huang et al., 2019] and the Deep Deterministic Policy Gradient (DDPG) algorithm [Chen et al., 2020] have been exploited to enhance the adaptiveness and timeliness of generator dynamic braking under-voltage load shedding, and emergency frequency control. Moreover, multi-agent DDPG algorithms are used to conduct load frequency control of the multi-area power system in the continuous action domain, and the experimental results demonstrate the superior performance of the multi-agent framework [Yan and Xu, 2020]. In addition, a meta-learning method combined with DRL algorithms is applied to solving emergency control problems in the context of extremely limited data availability [Nikoloska and Simeone, 2021].

Although the DRL-based emergency control schemes have shown dominant performance compared to conventional model-based approaches, with the integration of information and communication technologies, modern power systems are facing potential threats (e.g., cyber-attacks), and thus, it puts forward higher requirements for the security and reliability of the DRL model itself. For the machine learning community, the vulnerability of DRL algorithms has been preliminarily studied via developing various attack methods. Huang et al. [Huang et al., 2017] altered the observation by the Fast Gradient Sign Method (FGSM), which resulted in a significant performance decline of the DRL algorithms. Moreover, Lin et al. [Lin et al., 2017] implemented Carlini &Wagner (CW) attack to generate adversarial samples, which disturbed observations only at 25% of time steps but produced the same result as FGSM. On the other hand, machine learning algorithms for the regression and classification tasks in power systems have been recognized. Chen et al. [Chen et al., 2018] first showed the impacts of adversarial attacks against ML-based power system tasks, including classification of power quality disturbances and forecast of building loads. In addition, the adversarial attack methods proposed in [Li et al., 2021] successfully decrease the accuracy of Deep Neural Networks-based false data detection approaches and thus may cause significant security issues.

However, there is no research about the vulnerabilities of DRL-based emergency control systems in the literature, which is of great importance to be investigated before using the system in practice. Therefore, designing adversarial attacks against the DRL-based emergency control system and analyzing their system impacts is essential to assess its vulnerability. After that, the corresponding defense mechanisms against adversarial attacks for DRL-based emergency control schemes are also critical problems to ensure the stability of cost-effective operation in the power system.

To this end, this paper aims to fill the knowledge gap and investigate the fundamental limitations of existing approaches through the following novel contributions:

-To the best of the authors' knowledge, this is the first work that comprehensively investigates the vulnerability of the DRL-based power system emergency control schemes. In particular, this paper focuses on the potential threats in the test phase. Five recovery-targeted (RT) adversarial attack methods are designed for gradient-based approaches that aim to prevent the system from restoring to a stable state effectively.

-Based on the explored vulnerabilities, two distance-based robust defense mechanisms are proposed to enhance the robustness of the DRL-based emergency control model under various adversarial attacks. Furthermore, four evaluation indicators are designed to assess the effectiveness, stealthiness, instantaneity, and transferability of the tested attacks as well as the performance of the defense methods.

Experiments are conducted based on the world's first opensource platform, Reinforcement Learning for Grid Control (RLGC), which was designed to develop and benchmark DRL algorithms for power system control. The proposed attacks and defense mechanisms against SAC-, PPO- and DQN-based emergency control models are tested with both white-box and black-box settings.

The remainder of this paper is organized as follows: Section 2 introduces power system emergency control and formulates the problem as an MDP model; Section 3 explains the proposed RT adversarial attacks and robust defense (RD) mechanisms in detail; Section 4 demonstrates the performance of the proposed approaches based on the platform of RLGC; Concluding remarks are provided in Section 5.

2 Background and ProblemDefinition

2.1 Power SystemEmergency Control

Power system security control, including the two main stages of preventive and emergency control before and after contingencies occur, plays a key role in maintaining the safety and stability of power system operation. The target of emergency control is to restore the system state from an unstable to a stable one. For large-scale power systems, the emergency control problem is a highly non-linear and non-convex optimal decision-making problem [Huang et al., 2019], which can be expressed as:

where F(·) is the cost function of the power grid emergency control; g(·) and h(·) represent the constraint relationships;xt represents state vector of the power system, consisting of voltage magnitudes and angles; at represents the vector of controls available to the operator, such generator speed; dt represents disturbance vector that could occur in the grid. In this problem, the main objective function is to minimize cost during the time horizon Tc - T0 by at.

For this nonlinear time-varying system, traditional emergency control methods can not achieve real-time, and are difficult to obtain the optimal decision, especially in the context of a large-scale power system with high renewable energy penetration. To this end, the DRL algorithms are considered one of the most effective solutions in the literature to address these issues.

2.2 Emergency Control as anMDP

In this work, we focus on one of the emergency control technologies, UVLS, which has been implemented in the opensource platform RLGC. For the ULVS, load shedding is the ultimate countermeasure to prevent widespread outages and recover the power system from an insecure state to a normal one. The UVLS scheme reduces the load percentage of the high load area on the bus by a set of distributed controllers. In this study, the UVLS problem can be formulated as a Markov Decision Process (MDP) consisting of four parts < S, A, R, P >, where S, A, R, P represent the state space, the action space, the reward function, and the state transition probability, respectively. In anMDP, the agent takes observations of environment states and executes actions generated by its policy π to the environment; in turn, the environment provides feedback rewards to the agent. During the interaction with the environment, the agent constantly adjusts its policy to achieve the best decision according to rewards. Given that the state transition probability is unknown, the state, reward, and actions can be defined as:

Observation and State. The voltage amplitude on the observation bus, represented by V (t), and the remaining load percentage on the control bus, represented by L(t), are selected as the observation ot; and the last T simulation steps' observations are regarded as the states, that is, the input of DRL agent's network:

Action. The action is selected as the percentage of load shedding on the control bus, which quantity decides the dimension of at. The action can be discrete or continuous:

where i represents the ith control bus, N is the number of control buses and ait is the control action of the ith bus. In the discrete action space, ait is set as a value at a fixed interval, and in the continuous space, ait is a arbitrary value of a reasonable range.

Reward. Given the state st and action at, the reward function is a feedback from the environment that can be represented as:

where C is a large penalty given to the reward in this step if the limit of Transient Voltage Recovery Criteria (TVRC) (i.e., the voltage recovery to 0.95 within 4s) can not be satisfied; Tpf refers to the instant of fault clearing; Vi(t) is the bus voltage amplitude of bus i in the power grid; ∆Pj(t) refers to the total load shedding of load bus j at time t; uivld represents a penalty when the load on a specific bus has been reduced to zero at the previous moment, but the DRL agent still provides load shedding action. Note that the reward is always negative and will get a great penalty when the action violates constraints. The specific definitions are illustrated in [Huang et al., 2019].

Overall, the DRL-based UVLS problem can be formulated as an optimization problem to find the optimal control policy π, which gets maximum cumulative expected rewards during time horizon Tc - T0, as shown below:

2.3 Adversarial Attack Against the UVLS

Adversarial attacks against the DRL-based UVLS model aim to influence the effectiveness of the control action via manipulating the states in the test phase. Let πθ denote the policy. The target of an adversarial attack at every time step is to find an adversarial example sadv t in the ϵ neighborhood of st (use lp norm to define the distance between them) that can minimize the reward, which can be formulated as an optimization problem as follows:

To solve the above optimization problem (6), we use the gradients descent method to minimize the loss function. In a white-box attack, the loss function J of the adversarial state sadv t is designed to realize the optimization objective:

Similarly, the transfer-based black-box attack is also developed to generate the adversarial sample that minimizes J. Nevertheless, it uses the network parameters of the known DRL model θ to generate sadv t against the unknown DRL model with parameters θ′ .

3 Proposed Attacks and Defense Mechanisms

For the application of DRL-based UVLS, there are significant differences in adversarial examples generation mechanisms between pixel values investigated in the ML community and power system observations for the following reasons. First, the objective function of adversarial attack against the DRLbased UVLS attempts to influence the decisions of the percentage of load shedding on the control buses during test periods so as to induce more severe stability issues such as cascading failure or even widespread blackout. However, most adversaries against classification models in the ML community aim to misclassify images. Second, observations in the DRL-based UVLS model are not as unified as pixel information but rather the voltage and load percentage information with more complex physical meanings. For example, bus voltages are mutually restrained, making it more challenging to manipulate observations considering stealthiness.

On this basis, integrated with the characteristics of power systems, we propose the concept of recovery-targeted adversarial attacks for the DRL-based emergency control schemes as well as the corresponding distance-based robust defense mechanisms to improve the robustness of the model. An overview of the RT-attacks and DB-defense mechanisms for the DRL-based emergency control scheme proposed in this paper is outlined in Fig.1.

3.1 Recovery-Targeted Adversarial Attacks

In the DRL-based UVLS scheme, the output of DRL models is the action, which refers to the percent of load shedding on control buses. To this end, the targeted adversarial action aadv that the attacker would like the system operator to execute should realize the following optimization objective:

Note that in this work, we simplify the procedure to calculate the optimal targeted adversarial action via solving the above problem and directly consider the actions of no load shedding for UVLS, denoted by anls, to ensure that the system will not recover to a stable state after an emergency.

On this basis, we propose adversarial attacks against the UVLS scheme, including the recovery-targeted FGSM (RT-FGSM) family and recovery-targeted Jacobian-based Saliency Map Attack (RT-JSMA) family.

RT-FGSM Family

FGSM [Goodfellow et al., 2014] is one of the simplest and fastest gradient-based attack methods that crafts adversarial examples by calculating gradients of model outputs to inputs [Wang and He, 2021]. For the UVLS, the objective functions J1 of the proposed RT-FGSM can be defined as:

Then, the FGSM adversarial example can be designed by minimizing J1 with a one-step update:

where f(·) is a function to keep state satisfy physical constraints of power systems, such as load constraints and voltage constraints; ω denotes a small positive value. On this basis, the proposed RT loss function J1 can be employed in all FGSM families, such as Iterative Fast Gradient Sign Method (I-FGSM) [Kurakin et al., 2016], Momentum Iterative Fast Gradient Sign Method (MI-FGSM) [Dong et al., 2018].

RT-JSMA Family

Different from the FGSM, the JSMA algorithm [Papernot et al., 2016] is a targeted attack. Meanwhile, JSMA pursues to reduce the number of modified input dimensions as much as possible. Thus, we attempt to make agents' actions on a particular control bus more prone to decision errors than other buses through JSMA attacks. The JSMA algorithm [Papernot et al., 2016] is another type of gradient-based attack, which uses l0 norm as the distance metric.

Aiming at the DRL-based UVLS scheme, the implementation process of the proposed RT-JSMA algorithm can be divided into the following three steps. First, we define the loss function as the l1 norm between the action at time t and no load shedding action anls, then calculating the forward derivative as the Jacobian matrix:

where sjt is the jth value of the state st, M is the dimension of the state. Then, the second step is to construct adversarial saliency graph Gbst :

where b is the targeted attack bus; Γ is a searching domain, Γ = {1, ..., T}. Then, we find two states dimensions, sm1 t , sm2 t , which have the greatest impact on the objective function J b 2 for the current st. Note that searching domain Γ will remove m1, m2, if sm1 t , sm2 t are beyond limits f(·).

The adversarial state is iteratively updated until it reaches the maximum distortion γ. During the iteration, sm1 t , sm2 t are modified with an appropriate disturbance β.

It is imperative to highlight that, in this study, we propose a new adversarial attack called recovery-targeted MomentumJSMA (RT-MJSMA) that combines RT-JSMA with the idea of momentum. Different from RT-JSMA, RT-MJSMA uses the previous gradient information to modify the state at each iteration step k and calculates the adversarial saliency graph Gbst defined by Eq.16 , where gij is jth row, ith column element of g defined by Eq.15. The whole algorithm of the proposed RT-MJSMA is given in Appendix B.2.

Note that all the above-proposed approaches can be directly used as transfer-based attacks for the black-box settings.

3.2 Distance-based Robust Defense Methods

To mitigate the threats arising from the attacks mentioned above, it is of great importance to investigate the corresponding defense mechanisms to enhance the robustness of the DRL-based UVLS model so that it can still produce reliable control actions under various types of attacks. To this end, we propose two distance-based robust defense methods, named the distance-based robust defense padding method (DB-RDP) and the distance-based robust defense translate method (DBRDT), which utilize the characteristics of historical states to modify inputs st. The detail can be seen in Appendix B.3.

DB-RDP

The key idea of the proposed DB-RDP method is to pad the last observation ot with historical observation. If st is attacked, the attacker is more likely to modify the recent observation ot, especially in iterative attacks and local attacks. This observation has the greatest influence on action decisions, and its value is also an important indicator of the reward function. Meanwhile, the historical observation is similar with ot if the state is not attacked due to the short simulation sample time. Thus, the furthest observation from oadv t is more likely to be the nearest true value of ot and l2 norm is selected as the distance metric in order to find the furthest observation ot-T+d1, as shown below:

Then we pad ot with ot-T+d1 and the defense observation can be represented as:

Moreover, if no attack is applied in the DRL model, sdef t is similar to st, which makes the DRL-based UVLS scheme still perform well in no attack setting.

DB-RDT

The proposed DB-RDT method differs from the distancebased padding method in constructing the defense observation. More specifically, it exchanges ot with ot-T+d1:

To study the feasibility of proposed defensive methods, we use parameter p to control the percentage of our defense methods in the test episode.

where D(·) is the function corresponding to our defense methods.

Note that this work proposes two distance-based robust defense methods, which utilize the characteristics of historical states to modify inputs states. If the attacker is aware of the proposed attack mitigation scheme and factors this in the attack planning, one possibility is that the attacker will choose to reduce the attack preference and make more uniform changes to all state variables, resulting in a significant decline in attack performance; another possibility is that the attacker will make more preferred changes to a historical observation, but the modified observations may not influence the performance of our methods. Moreover, the design of defense probability p also ensures the dynamic characteristics and makes it difficult for the attackers to bypass it.

4 Experiments and Analysis

This section discusses results related to the proposed adversarial attack and defense methods on three DRL-based emergency control schemes (i.e., SAC-, DQN- and PPO-based) with four designed evaluation indicators. All the experiments have been conducted based on the RLGC open-source platform [Huang et al., 2019] with the IEEE 39-bus system [Athay et al., 1979]. The detailed information is shown in Appendix A and C.

4.1 Experiment Setup

Baselines

We use three DRL baselines exposed by OpenAI Lab, including DQN for the discrete action space, PPO and SAC for the continuous action space. All networks are publicly available 1 . In our setting, the state includes voltage magnitudes at buses 4, 7, 8, 18, and the percentage of remaining load served by buses 4, 7, and 18. The action is load shedding percentage at buses 4, 7, and 18. The model output is a binary vector in the discrete action space, where 0 represents no load shedding actions, and 1 represents 20% load shedding actions at corresponding buses. In the continuous action space, the action value is within -0.5 to 0, meaning the specific load shedding percentage value. It should be noted that simulations will break early when the voltage in observation buses is below the standard recovery for more than 4 seconds.

Evaluation Metrics

Considering the perspectives of effectiveness, stealthiness, and instantaneity of our adversarial attack methods, four evaluated metrics are given as follows: (1) Average Episodes Reward (AER). It represents the average cumulative rewards under the attack within test episodes; (2) Bad Recovery Ratio (BRR). It compares the voltage recovery with the standard recovery [Huang et al., 2019]. We record the duration if the voltage recovery line is lower than the standard recovery line; (3) Average Modification of Observation (AMO). It evaluates the stealthiness of adversarial attacks. We calculate the l1 norm between the adversarial state sadv t and the original state st; (4) Average Time Consumption (ATC). It evaluates the instantaneity of adversarial attacks. Because DRL decisionmaking in emergency control is real-time, adversarial example generations can not be time-consuming. Therefore, we introduce this indicator to evaluate the average time to make adversarial samples in each test step.

4.2 Adversarial Attacks Performance

This subsection aims at analyzing: 1) the influence of whitebox attack on SAC model; 2) the influence of transfer-based black-box attack on PPO and DQN models.

White-box Attacks. In the white box cases, the severe impacts of the proposed RT-adversarial attacks and their comparison with existing un-targeted attacks are given in Table 1. As can be seen, compared with the case of no attack, injecting the adversarial examples generated via the proposed RT-methods can significantly decrease the effectiveness of the executed control actions, indicated by approximately nine times lower and 35.78% higher metrics values in terms of the average AER and average BRR across all five RT-attacks. In particular, the RT-IFGSM and RT-MJSMA methods have reduced their AER metric values by more than ten times. Meanwhile, conventional attacks only result in the reduction of AER by between 1.5 and 3 times, and the BRR almost no change. Nevertheless, the metric values of average perturbation AMO and generation time ATC for the proposed RTattacks and conventional attacks can be retained at the same level. To sum up, it can be demonstrated that our RT-attacks have more practical significance in exploring the vulnerabilities of the DRL-based emergency control model.

1 https://github.com/hill-a/stable-baselines

Black-box Attacks. To study transferability, we use RTattack examples generated based on the SAC model (continuous action space) into the DQN model (discrete action space) and the PPO model (continuous action space). As shown in Fig.2, black-box RT-attacks still perform pronounced attack effects in DQN and PPO with an average of 8.46 times and 3.71 times decreasing rewards of AER. In addition, the BRR of them are increased by average 27.25% and 14.98%, respectively, when compared with the cases of no attack. One of the key conclusions stemming from the results is: black-box attacks against DQN are more severe than PPO, indicating that the transferability of continuous transfer into discrete action space is higher than the opposite. Furthermore, the proposed RT-JSMA family exhibits stronger transferability with average 2.692 higher rewards decreasing than the RT-FGSM family. The performances of RT-attacks in the DQN model under one test episode are presented in Figure 3 (a) and (b). From the recovery line, the black-box attack effect is not up to the white-box attack, which is consistent with common sense.

4.3 Defense Methods Performance

This part aims to evaluate the performance of the proposed defense mechanisms under both the white-box and black-box RT-attacks. As indicated in Table 1, the metric values of AER have significantly improved after applying the DB-RDP and DB-RDT methods with about 3.75 and 3.08 times average increases across different attacks, respectively. Moreover, Fig. 2 shows that the proposed DB-RDP successfully mitigates the impacts of all RT-attacks in the black-box setting, as indicated by the approximately 7.76 and 3.81 times decreased AER values for the DQN- and PPO-based UVLS models, respectively. In addition, we observe that all recovery lines are all over the blue standard recovery line from Fig. 3 (c) and (d). Overall, the above results illustrate that the proposed defense mechanisms can enhance the robustness of the DRL-based emergency control model to obtain an effective control action to recover the power system back to a stable state.

Ablation Study. Furthermore, we conduct a series of ablation experiments to study the impact of defense application proportion p on white-box adversarial attacks. The defense results of DB-RDP and DB-RDT on the SAC model are presented in Fig. 4 (a) and (b), respectively. As the proportion of defense application p increases, attacks' effectiveness gradually weakens or is even invalid. For the DB-RDP method, the BRR of the DRL-based UVLS scheme drops to less than 5% when it is fully applied (i.e., p = 1).

5 Conclusion and FutureWork

This paper explores the vulnerability of DRL-based emergency control for low carbon power systems, which aims to fill the knowledge gap between the ML and power system communities. In particular, we design five recovery-targeted adversarial attacks against the DRL-based UVLS scheme, named RT-FGSM family and RT-JSMA family. Considering characteristics of observations in the power system, we propose two distance-based robust defense methods, namely DB-RDP and DB-RDT. Based on the open-source platform, RLGC, case studies are conducted to evaluate the performance of the proposed adversarial attacks on the SAC model with the white-box setting and, the DQN model, the PPO model with the black-box setting. The key insights observed from the experiment results include: (1) DRL-based emergency control scheme indeed has security risk, and the proposed RT-attack algorithms can lead to more severe impacts regarding their effectiveness, stealthiness, instantaneity, and transferability; (2) the proposed active defense mechanisms can alleviate the vulnerability brought by the attacker, and at the same time, these defense mechanisms do not weaken the model performance in the attack-free DRL model.

In the future, we will explore the vulnerability considering power system physical constraints and extend to a wider range of power system applications. Furthermore, the proposed approaches can also be generalized to other complex domains (e.g., smart manufacturing systems, intelligent transportation systems) by considering their physical dynamics or characteristics.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 52161135201, U20A20159, and 62103371.