Carbon Market Simulation with Adaptive Mechanism Design

1 Introduction

Climate change has emerged as a pressing worldwide concern [Schmalensee et al., 1998], significantly imperiling global ecosystems, economic systems, and sociopolitical stability. The United Nations reports that in developing regions, one in ten individuals subsists on less than US$ 1.90 daily [Nations, 2023b], with 2.2 billion people deprived of access to safely managed potable water resources [Nations, 2023c]. The burgeoning climate crisis amplifies these challenges, as worldwide temperature escalations provoke droughts and rising sea levels, exacerbating famines and enhanced forced displacements [Nations, 2023a].

In 2016, 196 nations endorsed the Paris Agreement to mitigate climate change collaboratively. However, pursuing transnational environmental targets often conflicts with shortterm interests, requiring mechanisms to reconcile local and international objectives [UNFCCC, 2015]. Carbon markets exemplify such mechanisms [UNFCCC, 1997; Wara, 2007; Zhou and Li, 2019], incentivizing economic agents to curb emissions. The cap and trade format, predominant in carbon markets, involves allocating and trading allowances [Goulder and Schein, 2013; Schmalensee and Stavins, 2017; Zhou and Wang, 2016; Hepburn, 2007]. Economic agents must possess sufficient allowances to offset emissions or face penalties for surplus emissions. The cap and trade system sets a predetermined limit on allowances within an economy, with a central authority introducing and allocating allowances based on specified objectives. While this policy helps balance efficiency and fairness, determining the optimal allocation remains challenging in general economic contexts. The high-dimensional dynamics of the carbon market, influenced by rational, self-interested, and far-sighted economic agents, lead to market simulation reliance on models like CGE (computable general equilibrium) [H¨ubler et al., 2014; Tang et al., 2016; Bi et al., 2019] or ABM (agent-based modeling) [Tang et al., 2017; Zhou et al., 2016; de Sousa, 2021] frameworks, employing simplifying assumptions that are arduous to validate, such as production and trading behaviors.

∗Corresponding to: Baoxiang Wang

Given the unique nature of the carbon market, we integrate the AI Economist [Zheng et al., 2022; Trott et al., 2021] to simulate market dynamics. Our adaptive mechanism design framework, employing hierarchical, model-free MARL, mimics the carbon market. Lower-level enterprise agents engage in realistic economic activities, such as emitting carbon dioxide, trading emission credits, and investing in emission reduction projects. Higher-level government agents analyze diverse allocation strategies to achieve balanced efficiency and fairness, leading to significant carbon emission reductions. The framework demonstrates the conduct of rational, self-interested, and far-sighted agents within the carbon market. We emphasize that our approach is not a simple transfer of the AI-Economist from taxation to carbon credit allocation. Simulating the carbon market is challenging due to limited data, fluctuating regulations, and non-market factors.

To validate our simulator, we conducted comparisons with several widely adopted indicator allocation approaches at the firm level [Zhou and Wang, 2016]. The simulation results indicate reasonable action responses by enterprise agents to these allocation policies. Additionally, numerical findings demonstrate that government agents, through MARL, effectively discover allocation policies capable of balancing productivity, equity, and carbon emissions. Our primary contributions encompass: 1) We propose a systematic carbon market simulator featuring carbon credits allocation and trading, and achieve realistic carbon economy simulation based on hierarchical, model-free MARL. 2) We implement several widely adopted indicator allocation approaches at the firm level as baselines. 3) We observe that learning-based allocation policies possess the potential to effectively balance productivity, equity, and carbon emissions.

2 CarbonMarket Modeling

We present a carbon market framework which exhibits a hierarchical structure, consisting of the higher-level government RL agent and lower-level enterprise RL agents. Consequently, it is referred to as a hierarchical modelfree MARL framework, which has also been known as a manager-worker architecture [Shu and Tian, 2019; Ma and Wu, 2020]. Concretely, the issues encountered by higher-level government agents can be modeled as a standard Markov decision process [Bellman, 1957] Mh = (S,A,P,R, γ). The problems encountered by the lowerlevel enterprise agents can formally defined by a partially observable stochastic game [Hansen et al., 2004] Gl = (S, A|I| i=1 , R|I| i=1 , P, O|I| i=1 , I). And both government and enterprise are aimed to obtain optimal policies π that maximize their expected return Eai∼πi,a-i∼π-i,s′∼P [∑t γ t ri t ].

2.1 Model Details

Observation space O. In real world, enterprise can observe their own attributes s (skills: enterprise size, R&D capabilities), assets x (income xi , carbon emission level xl and carbon emission credit xc ), and acquire request data in the carbon market from the news. While the government can collect data from all enterprises and access current market prices Wt. Additionally, because we model the economics activities based on a Gather-Trade-Build game [Zheng et al., 2020], which effectively models general market-based behaviors, the position information O should also add to both observation. Therefore, the government's observation space is Og = {Wt, s, x, O}, while the enterprise agent i can observe Oi = {si, xi, Oi}, i ∈ {1, ..., I}.

Action space A. The government and enterprises are make decisions in different timescales (Figure 1). When enterprise agents act, they have the options to Produce, Invest, Trade and Move. Moveover, carbon emission credits are virtual resources which can be overdrafted. At the end of each year, when an enterprise settles carbon emission credits overdraft, the overdrafted portion will be penalized according to a unit price of p. The action of the government agent is a I + 1 dimensional discrete vector. At the beginning of each period, the government decides the allocation of credits for each enterprise as well as the total credits for the period.

Reward functionR. Each enterprise's objective is to maximize utility which is defined as [Debreu and others, 1954]:

where z is income that the sum of enterprise's coins, cl is labor's weight, and l is also the cumulative labor during all previous time steps. Inspired by [Zheng et al., 2020], the government's objective is to maximize social welfare, which is defines as multiplication of productivity, equality and attenuation coefficient of excess carbon emissions:

where ce is a coefficient and ee is total excess emissions.

2.2 Model Calibration

The structural parameters of simulation were calibrated to meet the following objectives: 1) We calibrate the model of carbon emission reduction investments to possess similar attributes of risk, delay, and other behaviors observed in real-life investment activities [Lee, 2020]. Additionally, we calibrate parameters to ensure that the carbon price changes caused by carbon emission reduction investments are aligned with price tiers set within the carbon trading market. 2) The penalty for excess carbon emissions is also calibrated such that the excess penalty is only offset when a certain level of carbon emission reduction investment by enterprises, alongside green project investments by all enterprises, is achieved. 3) Furthermore, ensure that the outcomes resulting from various behaviors of enterprise agents are on the same temporal and economic scale.

3 Simulation and Visualization

Upon the completion of carbon market modeling, we can employ MARL to train the government (enterprise) agent(s), enabling them to exhibit behavior that closely resembles realworld scenarios, thereby achieving a realistic simulation of the carbon market.

3.1 Baseline Allocation Policies

We utilize the indicator approach [Zhou and Wang, 2016] to allocate the proportion of total carbon credits for each enterprise annually, with Emission (also called grandfathering or GF [Zetterberg et al., 2012]), Emission intensity (also called benchmarking or BM [Groenenberg and Blok, 2002]), and Enterprise size selected as indicators (shorten as SI). Due to the large temporal scale of our simulation spanning 10 years, the government agent needs to allocate the total carbon emission credits for each year over the 10-year period. We refer to the global carbon emission historical data and future forecasts provided by the IPCC [Climate, 2023] to establish emission scenarios on a large temporal scale: we call this scenario Convex. Additionally, we provide scenarios where the annual emission targets decrease gradually over time (Decreasing), and scenarios where the annual emission targets remain constant over time (Flat).

3.2 MARL Allocation Policy Training

MARL training aims to discover an allocation policy that maximizes the Government's reward r, while also finding a balance between productivity, equality, and carbon emissions under the designated economy-climate coefficient ce .

For the joint optimization of enterprise and government policies, we first initialize the parameters of enterprise agents to those trained under government policies based on Flat and Enterprise size as the indicator (SI) scenario. Subsequently, the parameters of government policies are randomly initialized. During training, we utilize the PPO algorithm [de Witt et al., 2020] under the RLlib framework [Liang et al., 2018]. Additionally, we experiment with various hyperparameters for both enterprise and government agents, including learning rate and entropy regularization. Following training with 400 million samples, we find both enterprise and government agents to converge to stable policies, which effectively balance productivity, equity, and carbon emissions. (Figure 2)

3.3 Visualization

We have also implemented visualization for the simulator, which provides a detailed breakdown of each time step within an episode. Through this visualization tool, we can gain a deeper understanding of and intuitive comparison between different baselines and MARL strategies (Figure 3).

4 Closing Remarks

In this paper, enterprise and government agents participate in carbon market simulations via MARL-based adaptive mechanism design. By fine-tuning the government's reward function, we can exploit the adaptability to strike a balance between various economic and climate objectives. Unlike the commonly used indicator approach, MARL-based agents can incorporate more comprehensive information, enabling them to formulate more personalized and diversified allocation strategies. We also illustrates the practicality of employing hierarchical model-free MARL for carbon market simulation. It envisions the potential of machine learning to contribute to global emission reduction endeavors. However, the proposed simulator still needs to be improved, notably the absence of empirical modeling for emissions reduction investments made by enterprises. Consequently, future simulations can enhance their realism by integrating more real-world data.

Ethical Statement

In our development of the carbon market simulator, we adhere to principles of transparency, integrity, and fairness, ensuring compliance with the highest ethical standards while advancing understanding in environmental economics. We prioritize privacy, equity, and social responsibility throughout our research and development process. However, it's important to acknowledge that our simulator may not encompass all aspects of the real world. As such, we do not endorse the use of learned policies for actual policy making.

Acknowledgements

Baoxiang Wang is partially supported by National Natural Science Foundation of China (62106213, 72394361).