A Novel Demand Response Model and Method for Peak Reduction in Smart Grids -- PowerTAC

1 Introduction

Load balancing is one of the most prevalent problems in energy grids, which occurs when there is a sudden surge of consumption (i.e., during peak hours) and the demand goes beyond the normal working range of supply. The sudden surge in demand leads to multiple issues: (i) peak demands put an added load on electricity generating companies (GenCo) to supply additional energy through fast ramping generators to fulfill the energy requirement of the customers (agents). (ii) The grid needs to support such dynamics and peak demand. The ramping up of the generators results in higher costs for distribution companies (DC). Typically, daily peak demands are approximately 1.5 to 2.0 times higher than the average demand [U.S. EIA, 2014]. As per one estimation, a

5% lowering of demand during peak hours of California electricity crisis in 2000/2001 would have resulted in 50% price reduction [International Energy Agency, 2003]. Thus, it is paramount to perform load balancing in the grid efficiently.

A promising technology for load balancing is a smart grid. It is an electricity network that supplies energy to agents via two-way digital communication. It allows monitoring, analysis, control, and communication between participants to improve efficiency, transparency, and reliability [Techopedia.com, 2021]. The smart grid technology is equipped with smart meters capable of handling the load in the smart grid by advising the agents to minimize energy usage during heavy load scenarios. The smart grid system can effectively balance the load by incentivizing agents to shift their energy usage to non-peak timeslots by signaling them the updated tariffs, commonly known as demand response (DR).

DR involves DC offering the agents voluntarily monetary incentives to optimize their electricity load. There are many approaches, such as auction-based mechanisms [Zeng et al., 2015; Zhou et al., 2015] and dynamic pricing [Goudarzi et al., 2021] to achieve DR. The major challenge with these approaches is that different agents may respond differently to the given incentives. Thus, to increase agent participation, it becomes crucial to learn their reaction toward these incentives. Learning agents' behavior is challenging due to the uncertainty and randomness that creeps in due to exogenous factors like weather [Shweta and Sujit, 2020; Li et al., 2018]. Works like [Shweta and Sujit, 2020; Li et al., 2018] consider a very simplistic model - when DC offers to an agent incentive more than what it values, the agent reduces every unit of electricity it consumes with a certain probability independent of the incentive. This probability is termed as reduction probability (RP) [Jain et al., 2014; Shweta and Sujit, 2020]. RPs are learned using multi-armed bandit (MAB) solutions. There are three primary issues with these approaches. (i) Agents' valuations need to be elicited [Jain et al., 2014; Shweta and Sujit, 2020], which adds additional communication complexity, (ii) agents reduce all with RP else nothing, and (iii) RPs do not change with incentives. In the real world, an increase in incentives should lead to an increase in RP. Our work considers the model where the RP is a function of incentives offered and not a constant for an agent, and reduction is not binary.

To model RP as a function of incentive, we need to carry out experiments with smart grids. However, any DR technique (or such experiments) proposed for a smart grid should also maintain the grid's stability. The only way to validate that the proposed technique would not disrupt the grid operations while achieving DR is to test it on real-world smart grids, which is practically impossible. Nevertheless, Power Trading Agent Competition (PowerTAC) [Ketter et al., 2013] provides an efficient and very close-to real-world smart grid simulator intending to facilitate smart grid research. We first perform experiments with PowerTAC to observe the behavior of different agents for the offered incentives. With rigorous experiments, we propose our model EXPRESPONSE. We observe that the agents respond quickly to the incentives; however, more incentives may not substantially increase reduction guarantees. Different agents may have a different rate of reduction (RR) to incentives that determine how fast RP changes w.r.t. incentives. It also models the consumer valuation for one unit of electricity. A higher RR corresponds to the case where a consumer values the electricity less (for example, a home consumer). In contrast, a lower RR value indicates that the consumer values the electricity higher (for example, an office consumer).

We propose an optimization problem for the DC to maximize the expected peak reduction within the given budget. We then provide an optimal algorithm, namely MJSEXPRESPONSE, for the case when the reduction rate (RR)s of the agents are known. When RRs are unknown, we employ a standard MAB-algorithm, MJSUCB-EXPRESPONSE, to learn RRs. Our experiments with synthetic data exhibit sublinear regret (the difference between the expected reduction with known RRs and the actual reduction with MJSUCBEXPRESPONSE). With this success, we adopt it for PowerTAC set-up and experimentally show that it helps in reducing peak demands substantially and outperforms baselines such as distributing budget equally across all agent segments. In summary, the following are our contributions,

• We propose a novel model (EXPRESPONSE) which mimics smart grid agents' demand response (DR) behavior by analyzing agents' behavior in a close-to realworld smart grid simulator, PowerTAC.
• We design an offline algorithm to optimally allocate the budget to agents to maximize the expected reduction.
• We design an online algorithm based on a linear search method to learn the RR values required to calculate optimal allocation in the offline algorithm. We further show that the proposed algorithm exhibits sub-linear regret experimentally.
• We evaluate the proposed algorithm on the PowerTAC platform - close to a real-world smart grid system. Experiments showcase the proposed algorithm's efficacy in reducing the demand peaks in the PowerTAC environment (14.5% reduction in peak demands for a sufficient budget).

2 RelatedWork

Many demand response methods are available in the literature. Some of the popular ones include time-of-day tariff [Ramchurn et al., 2011; Jain et al., 2013], direct load control [Hsu and Su, 1991], the price elasticity of demand approach (dynamic pricing) [Chao, 2012] approaches. These approaches are quite complex for the agents as the price keeps changing. It can lead to agent confusion due to uncertain supply, volatile prices, and lack of information. Due to the complexity involved in these methods, many recent works have focused on providing incentives to the agents, which make them shift their load from peak hours to non-peak hours [Park et al., 2015; Jain et al., 2014].

In the literature, many techniques for providing incentives primarily focus on the setting where when given an offer (incentive), the consumer can either reduce or choose not to reduce the consumption. For example, DRmechanism in [Jain et al., 2014; Shweta and Sujit, 2020] considered a setting where each consumer was associated with two quantities: (i) valuation per unit of electricity, which represents how much a consumer values the unit of electricity, and (ii) acceptance rate, which denotes the probability of accepting the offer if a consumer is given the incentive more than his/her valuation. The authors then proposed a Multi-Armed Bandit mechanism that elicits the valuation of each consumer and learns the acceptance rate over a period of time. Similar approaches were also considered in [Ma et al., 2016; Ma et al., 2017; Methenitis et al., 2019; Li et al., 2018]. All the above models, in principle, assume that the acceptance rate is independent of the incentives given to the agents. In practice, this assumption does not hold. The acceptance rate ideally should increase with the increase in incentives. To the best of our knowledge, this paper considers the dependency of increased incentives on the acceptance rate for the first time, esp. in MAB-based learning settings. In principle, the paper considers the problem of an optimal allocation of the budget to different types of agents to maximize the overall peak reduction.

Two sets of works aim to maximize the peak reduction under a budget constraint. (i) With a mixed integer linear programming (MILP) approach [Chen et al., 2020], and (ii) with an efficient algorithm by drawing similarities from the min-knapsack problem [Singh et al., 2021]. Other than that, there are a few tariff strategies for PowerTAC environment which mitigates the demand peaks by publishing tariffs to incentivize customers to shift their non-priority electricity usage to non-peak timeslots [Chandlekar et al., 2022; Demijan et al., 2022; Ghosh et al., 2019]. However, none of this technique talks about DR in detail.

3 Preliminaries andMathematical Model

In a smart grid system, distributing companies (DC) distribute the electricity from GenCo to agents (household customers, office spaces, electric vehicles, etc.) in the tariff market. The customers are equipped with autonomous agents/bots to interact with the grid. Hence, we refer to customers as agents henceforth. Depending on their type, each agent exhibits a certain usage pattern which is a function of a tariff offered by the DC for most agents. We consider N = {1, 2, . . . , n} agents available to prepare for DR at any given timeslot. Here, n denotes the number of types of consumers in the tariff market.

A DR model can further incentivize agents, offering ci to agent i, to shift their usages from peak to non-peak timeslot. However, agents may do so stochastically, based on external random events and the offered incentives. For each agent i, this stochasticity can be modeled by associating the probability of reducing demand in the desired timeslot as i. We call this probability as reduction probability (RP) pi(ci). Note that the reduction in electricity consumption at peak slot for agents is not binary. For example, an agent with the usage of 10 KWh and RP (pi(ci)) of 0.6 would reduce its usage by 6 KWh in expectation. The general intuition is that higher incentives lead to a higher probability of accepting the offer, reducing the load in peak hours. Typically the DC has a limited budget b to offer discounts. It aims to achieve the maximum possible peak reduction within the budget.

First, we need to model the agent's RP function pi(·). We need a simulator that can efficiently model real-world agents' usage patterns and the effects of DR on their usage patterns. PowerTAC [Ketter et al., 2013] replicates the crucial elements of a smart grid, including state-of-the-art customer models. Below, we explain experimental details and observations from the PowerTAC experiments that helped us to come up with our novel model of the RP function.

3.1 Modelling the Reduction Probability (RP) Function Inspired from PowerTAC

PowerTAC Set-up. The PowerTAC simulates the smart-grid environments in the form of games that run for a fixed duration. The standard game duration is around 60 simulation days, which can be modified to play for an even longer duration. The simulation time is discretized into timeslots corresponding to every hour of the day. For each game, the PowerTAC environment randomly selects the weather of a real-world location, and the agents mold their usage pattern based on the selected weather in the game. During the game, DC aims to develop a subscriber base in the tariff market by offering competitive tariffs, which could be fixed price (FPT), tiered, time-of-use (ToU) or a combination of all. The DC also satisfies the energy requirement of their subscriber base by buying power in the wholesale market by participating in day-ahead auctions. There are different types of customers in PowerTAC. But, we focus on PowerTAC's consumption agents - who consume electricity and aim to learn their RP function.

Experimental Set-up. We perform the following nine sets of experiments to model the RP function. We play 10 different games for 180 simulation days for each experimental set-up and report the statistics averaged over these 10 games. For each experiment, we make DC publish a tariff at the start of the game and keep the same tariff active throughout the game. The initial tariff rates depend on the DC electricity purchase cost and may vary from game to game.

FPT Set-up to Identify Peak Slots. We make DC publish an FPT and record each consumption agent's true usage pattern without any external signals from DC. Based on the true usage pattern of each agent, we identify the potential peak demand hours in a day. Figure 1 shows the usage pattern of a PowerTAC agent in response to the FPT; in this figure, the hours 7 and 17 have the peak usages during the day. The rate value of the FPT is derived by adding a profit margin in the DC's electricity purchase cost. Next, we study the agents' response to different tariffs. To this, we consider the ToU model, where different prices are proposed at different times. These prices, however, are the same for all agents.

ToU Set-up. In ToU tariffs, the rate charged for each unit of electricity consumed can vary depending on the time of the day. The ToU tariffs are designed so that the agents get discounts during non-peak hours and no/little discounts during peak hours. The average rate of the tariffs across all timeslots remains the same as the previous FPT-set-up. Essentially, all the ToU tariffs have the exact same area under the curve (AUC) as the FPT. We perform such an experiment for the remaining 8 sets by offering discounts in each set; we give x% discount on non-peak timeslots compared to the price in peak timeslots. Here x ∈ {1, 2, 5, 7.5, 10, 15, 20, 30}. Figure 1 explains how we move from an FPT (Fig. 1(a)) to a ToU tariff (Fig. 1(c)) by offering a certain discount and keeping the AUC the same for all the tariffs. Based on the discount level, the agents modify their usage patterns ((Fig. 1(b,d)), and we collect the usage data of each agent for each of the sets.

To analyze the effects of various discounts on agents' usage patterns, we pick the top two peak hours in the day for each agent. Then, we calculate the difference between the electricity usage during FPT-Set-up and electricity usage during discounted ToU-Set-up for both peak slots. We do this for all eight sets numbered from 2 to 9. We can view the discounted tariffs as a DR signal for the agents to shift their non-priority usages from peak to non-peak timeslots. Below, we show the observations of the DR experiments for a few selected agents.

Figure 2a and Figure 2b show the DR behavior of three PowerTAC agents BrooksideHomes, CentervilleHomes and EastsideOffices for their top two peak demand hours (rescaled to visualize peak reduction as a probability function). The first two agents are household customers, whereas the last agent is an office customer. Analysing the plots gives a crucial insight into the agents' behavior. The agents reduce their usage by a great extent for the initial values of discount (1%, 2% and 5%) but cannot reduce their usage further even when offered a much higher discount; secondly, different agents follow the different rate of reduction.

Based on the PowerTAC experiments, we conclude that the

Algorithm 1: MJS-EXPRESPONSE Algorithm

Input: Budget b, n, RR vector λ Output: Final Allocation vector c 1 cost ← 0, c ← 0n; // initialization 2 while cost ≤ b do 3 d ← 1 // iterator 4 l ← 1 // index of agent with largest jump 5 while d ≤ n do 6 ∆cd+1 d ← (1 - e-λd(cd+1) ) - (1 - e-λdcd ) 7 ∆cl+1 l ← (1 - e-λl(cl+1) ) - (1 - e-λlcl ) 8 if ∆cd+1 d >∆cl+1 l then 9 l = d 10 d = d+1 11 cl = cl + 1 and cost = cost + 1 12 return c // final allocation

the current iteration, we calculate jump (∆) values for all the agents. We define ∆j+1 i value for each agent as the change in RP for a unit change in discount. For example, if an agent i has RP of (1 - e-λi(j+1) ) for discount ci = (j + 1) and RP of (1 - e-λij ) for discount ci = j, then the jump ∆j+1 i is the difference between these two probabilities. The algorithm finds an agent l having the maximum such jump for the current unit of reduction (denoted by ∆cl+1 l for agent l) and allocates the current unit discount to agent l-Maximum Jump Selection (MJS). Finally, the algorithm returns the allocation, which is the optimal distribution of the initial fixed budget, as shown in the below theorem.

Theorem 1. MJS-EXPRESPONSE is optimal.

Proof. For any discount vector c, the objective function in Equation 2 can be written as a sum of jumps ∆ji which denote the additional increase in reduction probability of consumer i when offered j units of discount compared to j - 1. i.e.

reduction probability function can be modeled by an exponential probability function and is given as:

Here, ci is a discount (or incentive) given to agent i, and λi is its reduction rate (RR). The proposed function depends upon the choice of λi; the higher value of λi generates a steeply increasing curve (as shown with λi = 0.5), while the lower λi value makes the curve increase slowly with each discount (as shown with λi = 0.1) as shown in Figure 2c. Let c = (c1, c2, . . . , cn) and λ = (λ1, λ2, . . . , λn) be vector of offered incentives and RRs.

3.2 EXPRESPONSE: The Optimization Problem

We assume that all the agents have the same electricity consumption in peak slots 1 . The aim is to maximize the expected reduction under a budget constraint. This leads to the following optimization problem:

Suppose the RR (λ) values are known. In that case, we present an optimal algorithm MJS-EXPRESPONSE to efficiently distribute the budget b among the agents to maximize the expected sum of peak reduction (Section 4.1). When RRs are unknown, we provide MJSUCB-EXPRESPONSE algorithm that estimates it (Section 4.2). The algorithm is motivated by multi-armed bandit literature [Jain et al., 2014; Shweta and Sujit, 2020] and uses the linear search over the possible range of values of RR.

4 Proposed Algorithms for EXPRESPONSE

This section proposes a novel algorithm to solve EXPRESPONSE. We discuss two settings: (i) perfect information that assumes the knowledge of RR, and (ii) imperfect information where RR values need to be learned over time.

4.1 Perfect Information Setting: Known RR

MJS-EXPRESPONSE (Algorithm 1) distributes one unit of budget to an appropriate agent in each iteration until the entire budget is exhausted. To decide the appropriate agent for

1 Agents consuming different amounts can be trivially modeled by duplicating agents

Algorithm 2: MJSUCB-EXPRESPONSE Algorithm

Input: Budget b, n, Batch Size bS, T Output: Allocation {ct}Tt=1 1 Initialize ˆλ, ˆλ+ randomly // n-dimensional vectors 2 Initialize offeredInst, successInst, offeredHist and successHist to 0 // 2D matrices of size n×b 3 t ← 0 4 while t < T do 5 {ct′ }t+bs t′=t = MJS-EXPRESPONSE(b, n, ˆλ+) 6 for i = 1 → n do 7 if ct(i) ̸= 0 then 8 offeredInst(i, ct(i)) += bS 9 successInst(i, ct(i)) += # Successes for agent i 10 Update Hist = {Hist, offeredInst, successInst} 11 Clear offeredInst, successInst 12 t ← t + bS 13 [ˆλ, ˆλ+] ← LinearSearch(Hist, n, b, t) 14 return {ct}Tt=1

Thus at the optimal solution, one unit is allocated to b highest jumps and 0 to other jumps. We now need to prove that the earlier jump is higher than the latter, i.e., ∆li ≥ ∆ji ∀l < j. The below lemma proves this for any agent i.

From the last equation, we have ∆ji ≥ ∆j+1 i .

Note that one can use KKT conditions and derive a set of linear equations to determine an optimal distribution of b. Our proposed algorithm is simple, determines an optimal solution in linear time, and has a time complexity of O(nb).

4.2 Imperfect Information Setting: Unknown RR

As RR of the agents are unknown in this setting, we estimate them based on the history of the agents, which consists of the agents' response during the past timeslots. For each agent, we store its historical behavior by keeping track of the offered history and success history; we estimate ˆλ and ˆλ+ through a routine LinearSearch(·).

MJSUCB-EXPRESPONSE. We start by initializing ˆλ and its UCB component ˆλ+, then estimating ˆ pi(ci) for each agent i and for each ci using Linearsearch(·) at each timeslot. LinearSearch(). Estimating RRs with the offered history and success history, we calculate ˆ pi(ci) = SuccHist(i,ci) OfferredHist(i,ci) , for each ci offered to the agent i. ˆ pi(ci) is then used to calculate candidate values of RR using Equation 1. Based on the candidate RR values, we determine ˆλi that minimizes the squared error loss between the historical probabilities and the probabilities calculated based on the Equation 1 i.e., ˆλi ∈ argminl ∑ci∈[b] ( ˆ pi(ci) - (1- e-lci ))2 , for each of the discount value. The RR value that achieves the least squared error loss is returned as the optimal RR after the current timeslot. We follow the same method for each of the agents. Algorithm 2 discusses our proposed MJSUCBEXPRESPONSE method in more detail, which takes budget b, n, batch Size bS, and T as inputs and returns ˆλs. Here, ˆλ, ˆλ+ denote estimated λ and its UCB version, respectively.

4.3 Experimental Evaluation of MJSUCB-EXPRESPONSE

To check whether the algorithm converges to the true RR, we conduct extensive analysis on a simplified version of a smart grid. Here, we discuss the experimental set-up to observe the regret of the proposed MJSUCB-EXPRESPONSE. Regret is the difference between total reduction with known RRs and total reduction with unknown RRs. In both experiments, we repeat the experiment 25 times, each instance having independently chosen λ. We report different statistics averaged over 25 iterations. We defer the detailed discussion of the second experiment to the Appendix.

Exp1- Effect of Batch Size. In this experiment, we keep the budget b and T constant, and vary batch sizes. This experiment shows the change in regret behavior as we change the batch sizes from low to high. Figure 3 compares average regret of MJSUCB-EXPRESPONSE over 25 iterations of varying true RR values of agents, with varying batch size, b = 5, and T = 5000000. The figure shows three subplots with batch sizes of 5, 10, and 50, respectively. Each subplot compares the regret values for 4, 5, and 6 agents, respectively, and shows sub-linear regret in the case of 5 and 6 agents for three different batch sizes. With an increased batch size to 50, even the case with 4 agents converges to sub-linear regret within a few timeslots.

Exp2- Effect of Budget and Relation w.r.t. T. In this experiment, we vary the budget and number of agents while keeping the number of iterations and time T constant. Figure 4 compares the regret and concludes that MJSUCBEXPRESPONSE achieves a sub-linear regret, and its peak reduction success rates are approximately the same as the optimal peak reduction success rates. We next show the performance of MJSUCB-EXPRESPONSE in PowerTAC.

5 MJSUCB-EXPRESPONSE in PowerTAC

We first explain how we model different customer groups in PowerTAC (More details are provided in the Appendix.) Modelling the Customer Groups. We begin by grouping the agents based on their electricity usage and create four groups, namely, G1, G2, G3, and G4, as shown in Table 1. We consider groups due to the limitations of PowerTAC, where we cannot publish individual customer-specific tariffs. We can only publish tariffs for customer groups having similar usage ranges. However, our proposed model and algorithms do not rely on any assumption of the existence of such groups and treat each consumer as a separate user (in Sections 3 and 4). We leave out the remaining PowerTAC agents as they do not use a considerable amount of electricity in the tariff market.

Designing the Tariffs for Each Group. For each group Gi, we publish ToU tariff ToUi such that agents from Gi subscribe to tariff ToUi, and no other group of agents subscribe to that tariff. To achieve this, we combine ToU tariffs with tier tariffs as follows. In PowerTAC, tier tariffs specify rate values and upper bounds on electricity usage below which the specified rates are applicable. However, if the usage goes beyond that particular bound, the agent has to pay the rate values associated with the next higher bound. As we have segregated the agents based on their usage range, for any targeted group, we offer standard ToU rate values for its particular usage range and high rates for the remaining ranges of electricity usage. Thus, a group of agents naturally like the tariff designed for their group as the other tariffs are way costlier for their usage pattern. At any moment in the PowerTAC game, we keep all four ToU tariffs active (one for each group); these tariffs keep getting updated based on the DR signals fromDC.

Adapting MJSUCB-EXPRESPONSE in PowerTAC. While proposing our model, we assume that agents are identical and have the same usage capability. Thus, maximizing the sum of probability would also result in maximizing reduction. However, for general smart grid settings such as PowerTAC, we modify our model by giving weightage to agents based on their usage percentage (market cap). Higher weightage is given to agents that can reduce the larger amount of energy. We modify MJS-EXPRESPONSE to introduce weights proportional to groups' contribution to electricity usage for each group. weights = {4, 2, 1, 1} to groups {G1, G2, G3, G4}, respectively in our experiments. We still use c = MJS-EXPRESPONSE(·) (Line 5, Algorithm 2) to find the group that can fetch the highest increase in the probabilities as shown in Algorithm 1. While allocating discounts to the groups, instead of allocating a 1 unit of budget to each group, we weigh the unit with the group's weight. For example, if G1 gets selected for the discount, we assign a 4 unit discount instead of 1. We call this way of allocation as WEIGHTEDMJS-EXPRESPONSE. It may help to assign weights to the groups as assigning weights will allocate discounts proportional to their peak reduction capacity. For instance, 10% reduction in G1 would reduce more peak demand than 10% reduction in G4.

Creating Baseline. To compare the performance, we consider the baseline of uniformly allocating the budget to all the groups. This leads to publishing group-specific tariffs with equal discounts. We record the peak reduction efficiency and reduction in capacity transactions from the baseline strategy. We then use the recorded information as a benchmark to evaluate MJSUCB-EXPRESPONSE performance. Furthermore, we compare the efficacy against the strategy when we do not provide groups with any DR signals.

Evaluation Metrics. Finally, we evaluate MJSUCBEXPRESPONSE's performance on two metrics, (i) MJSUCB-EXPRESPONSE's peak demand reduction capability, which indicates how much percentage of peak demand reduction MJSUCB-EXPRESPONSE achieved compared to the benchmark strategies, and (ii) the reduction in capacity transaction penalties that suggest how effectively MJSUCB-EXPRESPONSE can mitigate such penalties compare to the benchmark strategies.

Capacity Transactions. In PowerTAC, capacity transactions are the penalties incurred by the DC if the agents subscribed to their portfolio contribute to the peak demand scenarios. These huge penalties are a way to penalize the DC for letting the agents create demand peaks. Thus, as opposed to the previous section where we analytically show MJSUCBEXPRESPONSE exhibits a sub-linear regret, here in PowerTAC experiments, we aim to reduce capacity transaction penalties of DC using MJSUCB-EXPRESPONSE.

5.1 Experiments and Discussion

Experimental Set-up. We perform multiple experiments with different initial budgets. We play 8 games in each set with approximately 28 simulation weeks (total 210 weeks). For each set, we start the experiments by randomly initializing RR values for each group and calculate the budget allocation based on WEIGHTEDMJS-EXPRESPONSE as well as MJS-EXPRESPONSE (line 5 in MJSUCB-EXPRESPONSE), called as MJSUCB-EXPRESPONSE-W and MJSUCBEXPRESPONSE-UW, respectively. As explained in Section 5, for each of the 4 groups, we have four ToU tariffs. We keep the same tariffs active for 3 simulation days and invoke the MJSUCB-EXPRESPONSE at the end of the 3rd day. Based on the success probabilities, we update the offeredHist and successHist, and calculate the next set of ˆλ and ˆλ+ values for each group. Using the new ˆλ+, we calculate the next demand allocation and publish the new tariffs as explained earlier. While publishing new tariffs, we revoke the previous ones; thus, only 4 tariffs are active at any time in the game. This 3 days process constitutes a single learning iteration (t). To calculate the success probability of each tariff, we played 10 offline games without any discount to any group and noted down the top two peak timeslots. Let xi,1 and xi,2 denote per group usage during those peak timeslots. Then, we compute the success probability as pi,1 = (1 - yi,1/xi,1) and pi,2 = (1 - yi,2/xi,2), with yi,1 and yi,2 denoting group 1 and 2 usage respectively. pi is then set as pi = pi,1+pi,2 2 . We perform 2 sets of experiments with b = 15% and b = 7.5%. We define a scalar value that gets multiplied by the discounts to generate fractional discounts.

Observations and Discussion. Table 2 shows the cumulative peak usages under MJSUCB-EXPRESPONSE and benchmarked (baseline) method for the top 2 peaks of groups G1 to G4. As shown in the table, for the overall 210 simulation weeks of training in PowerTAC, MJSUCB-EXPRESPONSE cumulative peak usage for peak1 is similar to the baseline method for both weighted and unweighted allocations, while slightly worse than the baseline for the peak2. The observation is consistent for the budget values 15% and 7.5%. However, if we focus on only the last 10 weeks of training, MJSUCB-EXPRESPONSE's peak usage reduction capabilities are visible. Both weighted and unweighted allocations achieve cumulative peak reduction close to 14.5% concerning No Discount peak usages for peak1 and b = 15%, which is almost 5 times better than the baseline while maintaining similar performance as the baseline for peak2. Similarly, MJSUCB-EXPRESPONSE achieves significant improvement for b = 7.5% too for peak1 by reducing the peaks 3 to 4 times better than baseline. Furthermore, as shown in Table 3, capacity transaction penalties in the last 10 weeks are significantly lower than the No Discount and baseline. Due to DR signals, agents sometimes shift some of the demand from peak1 to peak2 or cannot reduce any demand from peak2. However, if the overall system's performance is observed with the help of capacity transaction penalties in PowerTAC experiments, the penalties are significantly lower than the baseline, reinforcing the efficacy of MJSUCB-EXPRESPONSE in the PowerTAC environment.

6 Conclusion

The paper proposed a novel DR model where the user's behavior depends on how much incentives are given to the users. Using the experiments on the PowerTAC real-world smart grid simulator, we first showed that agents' probability of accepting the offer increases exponentially with the incentives given. Further, each group of agents follows a different rate of reduction (RR). Under the known RR setting, we proposed MJS-EXPRESPONSE which leads to an optimal allocation of a given budget to the agents, which maximizes the peak reduction. When RRs are unknown, we proposed MJSUCBEXPRESPONSE that achieves sublinear regret on the simulated data. We demonstrated that MJSUCB-EXPRESPONSE is able to achieve a significant reduction in peak demands and capacity transactions just within 200 weeks of simulation on PowerTAC simulator.