Enhancing Cooperation through Selective Interaction and Long-term Experiences in Multi-Agent Reinforcement Learning

1 Introduction

The emergence of cooperation is fundamental to human civilization and evident in various biological and multi-agent systems [Rand and Nowak, 2013; Kram´ar et al., 2022]. However, maintaining cooperation in social dilemmas is challenging due to the conflict between individual interests and collective welfare [Sigmund, 2010]. A key focus in cooperation research involves understanding the conditions under which individuals favour altruism over self-interest, with reciprocity identified as a crucial element. In this context, reciprocity is examined through repeated interactions limited to

Version with Appendix: https://arxiv.org/abs/2405.02654

direct neighbours due to physical or social constraints, where the behaviours of individuals are shaped by the actions of their counterparts [Perc et al., 2013]. Studies using evolutionary game theory (EGT) demonstrate that network structure can promote altruistic behaviour through spatial reciprocity, clustering similar strategies and reducing exploitation by free-riders [Santos et al., 2006; Wang et al., 2013]. Furthermore, considering individuals' capability to interact with neighbours selectively, recent research emphasizes the significance of dynamic interaction mechanisms in the evolution of cooperation [Su et al., 2022; Rand et al., 2011; Sylwester and Roberts, 2013]. Such neighbour selection behaviours notably transform learning dynamics, thereby allowing agents to modify their interaction structures in response to evolving cooperative scenarios.

Despite advancements, explanations for the spontaneous emergence of cooperative behaviours and selective interactions remain limited. Traditional game theory and EGT emphasize social learning, where agents imitate nearby successful strategies [Sigmund et al., 2010; Li et al., 2016], overlooking learning through trial-and-error [Metcalfe, 2017]. Additionally, empirical studies employing experience-based learning face challenges in developing enduring strategies, particularly due to the complexities encountered in large population iterations [McKee et al., 2023]. To unravel the intricate coevolutionary dynamics of agent behaviours, there is an increasing interest in leveraging advanced deep reinforcement learning (RL) algorithms [Perolat et al., 2017; Du et al., 2023; Willis et al., 2023]. These methods are not only used to examine agents' decision-making processes but also to investigate the emergence of their behaviours [K¨oster et al., 2022]. However, RL agents typically optimize personal policies, which may lead to a reduction in global optimization [Lowe et al., 2017]. Although studies introducing mechanisms like reputation [Anastassacos et al., 2020] and moral rewards [Tennant et al., 2023] to address these issues, these specialized approaches exhibit restricted applicability. Therefore, it is crucial to have a deeper understanding of the learning dynamics of agent behaviours and encourage cooperation.

In this study, we construct a computational model using deep Q-learning (DQN) [Mnih et al., 2015] to explore how agents can simultaneously learn both interaction and dilemma strategies from their long-term experiences. These agents, modelled as artificial neural networks, learn behavioural policies and obtain rewards in a spatial Prisoner's Dilemma Game (PDG) setting within a multi-agent reinforcement learning (MARL) environment. Unlike previous studies that depended on predefined social norms or explicit external incentives, our approach highlights the significance of temporal factors and historical information in influencing agent decision-making. Initially, agents have no prior knowledge regarding the actions or game states, hindering their ability to assess the consequences of their actions and respond effectively. Throughout the training process, they must learn the causality between their actions, observations, and rewards from local environment observation. To aid this learning process, we introduce a utility function that integrates self-learning and social learning, reflecting the interplay of personal preferences and the influence of others in shaping human behaviour.

The experimental results demonstrate that RL agents trained in our framework effectively differentiate between cooperative neighbours and those who are free-riders. Their preference for building connections with cooperators bolsters network reciprocity, contributing to the formation of strategy clusters in network-structured populations. This finding aligns with existing EGT research, emphasizing the importance of strategy assortment in promoting cooperation. Moreover, the trained agents achieve superior cooperation levels and greater average payoff compared to the EGT baseline. Further, we observe that increased efficiency in learning is correlated with the length of memory experiences. For a detailed comparison between our RL model and EGT approaches, refer to the Supplementary Information (SI) A.2.

Our work offers three key contributions. Firstly, it reveals the coevolutionary dynamics of cooperation and interaction strategies within a spatial PDG framework, demonstrating that RL agents can learn effective interaction mechanisms to enhance network reciprocity and cooperation. Secondly, it sheds light on how extensive long-term experiences positively influence group cooperation. Finally, the MARL training environment we developed sets the stage for future explorations into diverse aspects of pro-social cooperative behaviour.

2 RelatedWorks

2.1 Evolutionary Game Theory

EGT is crucial for exploring the evolution of cooperation among self-interested individuals. It expands on traditional game theory by considering extended interactions and strategy dynamics, exploring the emergence and stability of cooperative behaviours. Notably, Nowak [2006] identifies five mechanisms central to understanding cooperation evolution. One vital aspect noted is the emergence of cooperation on network structures, wherein individuals predominantly interact with their immediate neighbours [Perc et al., 2017].

Inspired by the dynamic nature of social interactions, numerous studies have explored the coevolutionary dynamics of cooperation by integrating strategy evolution with network changes. In this domain, the concept of network assortativity has been highlighted, revealing that agents with similar strategies often connect, thereby boosting group cooperation [Tanimoto, 2013; Ren and Zheng, 2021]. Research like Su et al. [2022] explores how cooperative strategies evolve and spread, particularly in unidirectional interactions, shedding light on shaping social interactions to promote cooperation. However, these often neglect the self-learning aspect of agents, which is crucial in real-life where directly copying strategies is impractical. To address this gap and accurately depict cooperation emergence, we applied the MARL framework, enabling agents to learn both dilemma and interaction strategies through environmental observation independently.

2.2 Human Experiments

While EGT is essential for examining evolutionary trajectories and conditions favouring cooperation, incorporating empirical data brings psychological nuances to these models [K¨obis et al., 2019], which focus on imposed interaction structures and psychological mechanisms beyond laboratory settings [Rand and Nowak, 2013]. Surprisingly, behavioural studies indicate that participants in structured settings tend to randomly change strategies instead of copying higher-payoff neighbours [Traulsen et al., 2010], disrupting the clustering process and rendering cooperation less advantageous. Addressing this, Rand et al. [2011] found that dynamic interactions enhance multilateral cooperation by encouraging links with cooperators over defectors, promoting strategy clustering. However, such laboratory experiments often encounter challenges in terms of scalability and struggle with complex, larger-scale, or long-term scenarios [Moffatt et al., 2009].

2.3 Multi-agent Reinforcement Learning

Recent RL applications also focus on understanding the emergence of cooperation by integrating spatial and temporal dynamics relevant to realistic scenarios [Ren and Zeng, 2023; Vinitsky et al., 2023; Tennant et al., 2023], moving beyond traditional matrix games through the fusion of complex incentive structures [Leibo et al., 2017; Jaques et al., 2019; Abeywickrama et al., 2023]. Studies have applied the intrinsic trial-and-error learning characteristic of RL to reformulate agent interaction strategies. For instance, Anastassacos et al. [2020] demonstrate that RL agents can learn an interaction strategy akin to Tit-for-Tat when partner selection is present, aiding in the maintenance of cooperation. Meanwhile, McKee et al. [2023] used a graph neural network-based agent as a social planner, demonstrating the ability of deep RL to foster coordination and cooperation in a group.

Concurrently with our work, Ueshima [2023] employed two distinct Q-networks for each agent, differentiating interaction and dilemma strategies. However, our approach varies as follows: (1) we extend the model to allow each agent to interact with four potential neighbours, unlike their paired interaction focus; (2) we incorporate environments with an explicit interaction structure, considering network reciprocity; (3) they consider single-round observation input, while we take into account the agents' long-term experiences.

3 Background

3.1 Prisoner's Dilemma Game

The PDG is a fundamental paradigm in EGT [Rapoport et al., 1965], representing a typical decision-making scenario where agents balance individual benefits against collective welfare.

Fundamentally, the PDG is characterized as a symmetric matrix game, representing interactions between pairs of individuals within a population. Each participant faces a choice: to cooperate (C), incurring a cost c while providing a benefit b to others, or to defect (D), avoiding the cost while exploiting those who cooperate (with b > c > 0). The corresponding payoff matrix can be summarized as

where mutual cooperation yields a reward R = b - c, while mutual defection result in P = 0. Unilateral cooperation against a defector incurs a cost S = c, wherease the defector gains T = b. This payoff matrix forms four classical game structures [Wang et al., 2015]: PD, chicken, harmony and stag hunt, each defined by specific payoff inequalities. In the PD, the conditions T > R > P > S and 2R > T + S hold. Our model employs a weak PDG with T = b (1 ≤ b ≤ 2), R = 1, and P = S = 0 [Nowak and May, 1993]. Here, the parameter b directly assesses the strength of the dilemma, given c = 0. In a single-shot PDG, defection is the dominant strategy, leading to a Nash Equilibrium of defection, despite cooperation could yield a Pareto improvement. Considering that players often engage in repeated iterations with the same counterparts, the conventional PD can be extended to the iterated prisoner's dilemma (IPD) format. In this work, the dilemma strategy of agent i at timestep t is represented by a two-dimensional unit vector adi (t), with adi = [1, 0]T indicating cooperation and adi = [0, 1]T signifying defection.

3.2 MARLMarkov Game

Within MARL, the IPD is conceptualized as a multi-agent extension of Markov decision processes (MDPs), termed partially observable general-sum Markov games [Littman, 1994]. Here, agents have observations limited to their local environment. Formally, a N-player MDP is defined by the tuple M = ⟨S, {Ai }i∈N , T , γ, R⟩, where S is a set of joint states for all agents, A1, . . . ,AN represent joint actions, and R is the reward function. The function O : S×{1 . . . , N} → Rd maps each player's d-dimensional view of the state space. In a given state, each agent i selects an action from Ai, and the dynamics of MDP are determined by the stochastic transition function T : S × A1 × · · · × AN → △(S), where △(S) represents the set of discrete probability distributions over S. The objective of each agent i is to learn a policy πi : Oi → △(Ai) to maximize its extrinsic reward ri(s, a1 , . . . , a N ) based on the agent's own observation, simplified as π(ai |oi ). This optimization is conducted while adhering to the joint policy of all agents. The longterm γ-discounted payoff for agent i under the joint policy -→π = (π1, . . . , πN) from an initial state s0 can be defined as:

where γ ∈ [0, 1] represents the temporal discount factor, and T denotes the time horizon. Policies are optimized using trial-and-error interactions within the MARL environment to maximise cumulative long-term rewards.

3.3 Deep Q-Network

As an extension of Q-learning, DQN stands out as one of the most popular off-policy Deep RL algorithms, which utilizes an independent deep neural network to estimate Qvalues [Mnih et al., 2015]. Departing from the traditional tabular representation for Q-values of state-action pairs, DQN utilizes a parametrized Q-function Qθ(s, a) to approximate Q-values. Each Q-network is parameterized by θ, representing the weights of the neural network. This approach incorporates the utilization of a replay memory buffer D to store past experiences and a target Q function Q to mitigate the risk of overestimating Q-values. The learning process for the optimal action-value function Q∗ involves minimizing the loss.

DQN can be directly extended to a multi-agent setting by having each agent i learn an independent Q function denoted as Qi : Oi ×Ai →R. In line with the traditional Q-learning approach, DQN also adopts an ϵ-greedy policy to promote exploration. The policy for the i-th agent is parameterized as:

where U(Ai) signifies a sample drawn from the uniform distribution over the action space Ai.

4 Methodology

This model employs value-based optimization utilizing the DQN approach within a MARL environment. Throughout the training, agents interact with neighbours, exhibiting cooperative or defective behaviours across various episodes. As illustrated in Fig. 1, each agent i aims to learn a joint policy πi concerning dilemmas and selection actions, informed by their local observations and long-term experiences. A comprehensive explanation of our methodology follows. 1

4.1 Game Environment

Agents in our experiment are situated within an L×L square lattice with periodic boundary conditions. They are placed at specific spatial coordinates and can engage in interactions limited to their von Neumann neighbourhood. The graphical representation employs vertices to denote agents and edges to indicate the relationships between an agent and its four neighbours. The action space for each agent encompasses two strategies: the overall dilemma strategy and the specific interaction selection strategy. This dual contributes to the formulation of the agent's policy πi expressed as:

where πsi refers to the interaction selection policy dictating whether to interact with its neighbours, while πdi is the dilemma policy that guides the agent in choosing between a cooperative or defective strategy.

1 Code: https://github.com/itstyren/InteractionMARL-Coop

Interaction and Dilemma Actions

During each timestep of an episode, agents participate in multiple pairwise interactions within the IPD game framework, consisting of two phases: neighbour selection and dilemma strategy determination. In the first phase, agent i chooses game partners through an interaction selection action denoted as asi ∈ {0, 1}4 , informed by previously observed game experiences with its neighbours. This selection action ais is represented by the following bitstring of size 4:

Specifically, a bit value of 1 for ajsi signifies agent i choosing to interact with neighbour j, whereas a 0 implies no interaction. For example, a selection action of as = (1, 0, 0, 1) implies interaction only with the first and last neighbours. Importantly, actual interaction occurs only when both players decide to interact with each other, i.e., when ajsi = aisj = 1.

In the second phase, each player selects either cooperation or defection as their dilemma strategy. Following the interaction phase, paired co-players engage in one round of PDG pairwise, utilizing the action pair [adi , adj ] where ad ∈ [C,D]. Critically, the chosen binary dilemma strategy remains consistent across all interacted neighbours. In other words, a player cannot cooperate with one neighbour while defecting against another neighbour simultaneously.

Game Formulation and Reward

According to their selected dilemma and interaction actions, agent i receives an accumulative payoff at each timestep t by participating in multiple rounds of the PDGs with its currently interacted neighbours, as shown by:

where nti = ∑j∈Ωi ajsi × aisj denotes the number of interacted neighbours for agent i at timestep t, and Ωi is the set of neighbors. In this model, we incorporate learning from previous interactions by employing a weighted moving average of past payoffs [Danku et al., 2019]. Thus, the final payoff of agents i at each timestep not just based on the current round's payoff but also includes payoffs from the past m rounds:

where α is a parameter that controls the rate of weight decay with increasing m, indirectly determining the memory length M. To effectively assess the emergent behaviours, M is restricted by the condition M = min{n | αn < 0.01}. With α = 0, agents focus only on the current round, indicating short memory. Conversely, as α → 1, agent memory extends to include all previous timesteps, capturing a comprehensive history of experiences.

In EGT research, the Fermi rule is commonly employed to model the dynamics of strategy evolution [Szab´o and T˝oke, 1998], reflecting social learning wherein neighbours tend to imitate the most successful observed policy. A detailed description is provided in SI B. To adapt these imitation dynamics to the RL context, we have modified the utility function of agent i to align with game payoffs, as formulated below:

where ˜adi represents a counterfactual dilemma action, condition on the actual action ad taken by agent i. The function ωi(·) returns the number of neighbours performing a specific action. The termR(˜atd , ats ) denotes the average payoff associated with the counterfactual action ˜atd across the population at timestep t. Essentially, the agent raises a retrospective question: "Would a different past action have led to a more advantageous outcome?" This setting integrates aspects of social learning and RL, allowing agents to compare global information from group about the performance of different actions with their own localized experiences.

4.2 Training Approach

In our multi-agent PDG framework, the training methodology aligns with the well-established DQN approach, typically used in single-agent tasks. Our focus is on formulating a joint policy (πs, πd), which utilizes the combined action utilities to calculate the Q-loss for each policy network, guiding both dilemma strategy and interaction selection processes. See SI A.1 for a detailed elucidation of the training procedure.

Network Architecture

Each agent in our independent MARL setup is equipped with a memory buffer, storing experiences from the last M rounds, encompassing a record of both the agent's own actions and those of adjacent agents. The agent updates its memory at each timestep with recent feedback from the local environment, ensuring an accurate representation of its state. The selection and dilemma networks process inputs from longterm neighbour interactions and prior dilemma strategies, respectively. Additionally, both networks consider the dilemma strategies of four neighbouring agents, with actions represented through one-hot encoding and sequenced together.

In the selection phase, agents evaluate the state ss ∈ R2×16×M , and in the dilemma phase sd ∈ R2×5×M . They operate with two Q-networks configurations, formulating policies independently and without sharing parameters across agents. The architecture of each Q-network includes a dual-layer perception with 32 hidden units and employs the tanh activation function for nonlinear transformations.

Experiment Setup

During the training stage of our RL experiments, we employ a centralized training with decentralized execution approach [Lowe et al., 2017]. This method allows agents to access global information regarding the average payoff for potential action in the PDG among the population, thereby facilitating an effective evaluation of their action utilities.

The training involved 900 agents, each equipped with two neural networks, ensuring a broad representation of cooperation dynamics at the population level. To generate experiences for agents, 10 parallel arenas were established. In each arena during every experimental trial, agents engage in interactions with their neighbours over 6, 000 episodes, each comprising 10 timesteps, resulting in a total of 60, 000 steps. At the end of each episode, sampled trajectories for agents were aggregated and subsequently forwarded to the respective learner. We compute the gradient by using the Adam optimizer [Kingma and Ba, 2014] with a linear annealing schedule of learning rate. To enhance the efficiency of the Q-learner in learning from experience replay, we also implement a common practice known as prioritized experience replay [Schaul et al., 2015] within the DQN framework. Additional details on hyperparameters, please refer to the SI A.2.

5 Results and Discussion

In this section, we present the outcomes of our experimental investigations, which provide evidence supporting the hypothesis that incorporating the interaction selection action with the dilemma action through RL promotes network reciprocity and cooperation evolution in a spatial PDG setting.

5.1 Experimental Setup

Our experiment employs a square lattice setup, randomly assigning agents as cooperators or defectors with equal likelihood. The primary evaluation metric is the fraction of cooperative agents in the population, representing the achieved level of overall cooperation. For robustness, we average the outcomes of the final 10 episodes over the entire training duration. All experiments were replicated five times to ensure replicability. Unless otherwise stated, a memory weight of 0.6 is assigned, incorporating experiences from the previous four rounds as network input.

5.2 Promotional Effect of RL on Cooperation

To evaluate the effectiveness of our proposed method, we first train a population to learn a combined policy, including both dilemma and interaction strategies under various dilemma strength conditions. Figure 2 demonstrates that the application of RL in coordinating interaction and dilemma strategies enables the population to sustain a high cooperation level successfully. Notably, this approach proves robust, maintaining its efficacy even in scenarios characterized by increased dilemma intensity. As shown, the MARL system transitions from a complete cooperation phase to a mixed strategy phase exclusively, when the dilemma strength b > 1.2.

For comparison, the evolutionary outcomes of conceptually similar models from existing literature are used as a benchmark [Danku et al., 2019]. Within EGT framework, agents evaluate equivalent lengths of past payoffs and emulate the most successful dilemma strategy observed in their neighbourhood (a detailed description of EGT methodology, refer to the SI B). It is noteworthy that even when the dilemma intensity is reduced to b = 1.1, only 54% EGT agents opt for cooperative strategies. Moreover, ablation experiments detailed in SI D.3 demonstrate that agents utilizing RL to learn dilemma strategies exclusively underperform in comparison to the model proposed in this study. These results suggest that lacking selective interaction in the PDG and the mere imitation of neighbouring successful strategies are insufficient to achieve optimal performance within the population. Additional MARL-based benchmarks are reported in SI D.5.

5.3 Evolutionary Dynamics of Cooperation

We next investigate the evolutionary dynamics and outcomes of overall dilemma strategies within the population, focusing on the evolution trajectories and average payoffs in four representative dilemma conditions. Figure 3(a) reveals a rapid initial decline in the population's cooperation level as the dilemma intensity increases. However, RL agents employing additional interaction strategies exhibit two evident phases in their learning process, aligning with observations from previous EGT experiments [Wang et al., 2013]: the END period and the EXP period. During the END period, cooperative agents resist the invasion of defectors, and successful cooperators convert those defectors into cooperators in the EXP period. In our experiments, the former period is characterized by a rapid decrease in cooperation levels in the first 3, 000 training step, followed by a phase where these levels rise unless defectors completely dominate in the early stages. Consequently, at the end of the EXP period, the cooperation level in the population significantly decreases, falling from 0.987 to 0.294 as b rises from 1.20 to 1.26.

The dilemma strength also significantly affects the distribution of individual payoffs, leading to a bifurcated process. As shown in Figure 3(b), the group Gini Coefficient exhibits a temporal evolution, initially increasing and subsequently decreasing, hinting at a correlation between payoff equality and the evolution of cooperation. Notably, a large fraction of cooperators contributes to high levels of group equality. In Figures 3(c)-(e), the focus is on evaluating the average payoff for the population, as well as the separate payoffs for cooperative and defective individuals. There is a general decrease in the average payoff as the dilemma increases, which aligns with expectations, given that cooperators are primarily the contributors to the group payoff. Cooperative individuals, however, show greater resilience to tougher dilemma conditions. Specifically, with an increase in dilemma strength from b = 1.20 to 1.22, the average payoff per episode decreases from 2.67 to 2.25, but this can be lessened by adopting an effective interaction mechanism, resulting in a smaller decrease in cooperative payoff from 3.52 to 3.41, compared to a substantial drop for defectors from 1.44 to 1.10. This trend indicates the potential of RL agents to develop interaction policies for dilemma scenarios, mitigating adverse effects on their payoff. For detailed information on the average payoff from the trained population, refer to Table S1 in the SI.

5.4 Efficiency of Learned Interaction Patterns

To elucidate the role of interaction selection in network reciprocity and cooperation, we analyze strategy connection and distribution patterns in Figures 4 and 5. Through participation in PDG with selective neighbours, RL agents develop policies for distinguishing cooperators from defectors, thereby boosting spatial reciprocity and cooperation. Figure 4(a) depicts how the disparity in average connections between cooperators and defectors increases during the initial half of the total timestep. This leads to a preference for forming connections with cooperators, irrespective of the dilemma strategy chosen by agents. In Figure 4(b), we evaluate the actual link connections across different link types during the first 10 training episodes, signifying frequency where the chosen co-player reciprocally opts for interaction within the same round. As illustrated, the occurrence of interaction between two neighbouring individuals who both employ the defective strategy (DD link) is merely 45.48%. In contrast, interactions between two cooperators (CC link) can increase to as high as 77.29%. For measurement metrics of agent interactions, see SI A.3.

We next provide intuitive evidence regarding the previously described learned interaction policy and its role in enhancing spatial reciprocity by illustrating the spatial coevolution of the dilemma and interaction strategies within the population over time. Initially, in the END phase depicted in Figures 5(a) and (e), cooperators resist the invasion of defectors by forming small clusters, yet lack an effective neighbour selection strategy. As training progresses, RL agents learn to adapt their interaction strategies in response to neighbouring dilemma strategies, enhancing the influence of spatial reciprocity in promoting the evolution of cooperation. Figures 4 and 5 (f)-(h) show that, in the EXP phase, individuals within the cooperative cluster show a higher tendency to engage in the PDG with nearby cooperators, while those at the periphery are more likely to avoid interactions with defectors. This selective interaction mechanism favours cooperators and limits the payoff obtained from free-riding behaviours. Consequently, defectors gradually switch their dilemma strategies, leading to the expansion of cooperative clusters.

5.5 Role of Long-termExperiences

In prior experiments, RL agents use experiences from the last four rounds as network input. Finally, we investigated the influence of varying memory lengths on their learning dynamics, with findings presented in Figure 6. Three memory weights were assessed, with α varying from 0.6 to 0, representing input scenarios based on observations from the last, second, and fourth rounds, respectively. Notably, a positive correlation is observed between the group cooperation level and the memory length of trained agents. Figure 6(a) illustrates that, for instance, when dilemma strength increased from 1.2 to 1.22, agents recalling four-time steps maintained group cooperation effectively. Conversely, those relying solely on current information saw a substantial decrease in cooperation levels, dropping from 0.64 to 0.19.

Furthermore, Figure 6(b) echoes the findings of Figure 4, demonstrating that successful cooperation benefits from a preference for efficient communication with neighbouring cooperators. Moreover, the efficiency of interaction selection is also influenced by the length of agent memory. Longer observation inputs are shown to improve the average interaction ratio among cooperators in the END and initial EXP stages, which supports network reciprocity and contributes to the formation and development of cooperative clusters. Interestingly, our findings also reveal that RL individuals consistently and effectively avoid interactions with defectors, irrespective of memory length and dilemma intensity (detailed in SI D.2). Finally, experiments conducted in SI D.4 demonstrate that the dual Q-network configuration outperforms its single Q-network counterpart, highlighting the advantage of our proposed dual network approach.

6 Conclusion

Our computational model enables agents to interact selectively with their neighbours and protect cooperative behaviours from antisocial influences by MARL and iterative trial-and-error in simulations. Unlike existing RL cooperation studies, our approach does not rely on predefined social norms or external incentives [Ueshima et al., 2023; Tennant et al., 2023]. By integrating a spatial PDG into the training environment, we extend focus from paired interactions to those within a spatial structure, enabling a deeper analysis of how long-term learning impacts the coevolution dynamics of cooperation and selective interaction.

Our findings reveal that trained agents are capable of differentiating between neighbouring cooperators and defectors by observing information from their immediate surroundings, which enhances the network reciprocity and the associated group cooperation. Consistent with theoretical models [Szolnoki and Chen, 2020], our findings suggest that the performance of the learned interaction mechanism in promoting cooperation is attributed to its ability to help populations form homogeneous strategic clusters. These clusters are crucial for resisting invasions by defectors, especially in the early stages of development. Additionally, we confirm a positive correlation between agent memory length and the effectiveness of interaction selection, which in turn, aids the evolution of cooperative behaviors [Park et al., 2022].

In conclusion, we emphasize the significance of understanding the learning dynamics in interaction selection and their contribution to fostering cooperation. These insights offer new insights into the emergence of cooperation within social dilemmas. Moreover, the computational framework developed here has broader implications, providing a versatile tool for investigating and examining mechanisms behind the evolution of cooperation in spatially structured environments. Integrating psychological complexity in our training framework emerges as a promising direction for future research. Understanding these mechanisms may provide solutions to social dilemmas and strengthen cooperation within both human societies and artificial intelligence systems.

Acknowledgments

The authors would like to thank anonymous reviews for their constructive comments. We also appreciate the insightful discussions with Xiaomei Mi and Yang Li related to this paper, and are grateful Haotong Zhang and Runqi Zhao for their support with computational resources. Finally, we acknowledge the assistance given by Research IT and the use of the Computational Shared Facility at The University of Manchester.