AI-Olympics: Exploring the Generalization of Agents through Open Competitions

1 Introduction

In recent years, AI-based decision-making has gained much attention in both academia and industries. Particularly, Reinforcement Learning (RL) has matured rapidly yielding highlevel performance in various tasks in games [Schrittwieser et al., 2019], robotics [Kober et al., 2013], advertising [Cai et al., 2017], etc. However, many of these carefully trained agents have been criticized for their poor generalization abilities when applied to a slightly difference task [Leibo et al., 2021; Gorsane et al., 2022; Agapiou et al., 2022]. Implementing an adaptive agent that can perform equally well on a series of tasks is still an open and challenging problem. On the other hand, assessing an agent's generalization skills normally relies on benchmark suits encompassing various environments [Packer et al., 2018; Kirk et al., 2023], tasks [Leibo et al., 2021; Oh et al., 2017] and opponents [Leibo et al., 2021; Agapiou et al., 2022], which are often static and may prove inflexible amid the rapid evolution of agents.

To advance the research in this domain, we have developed a Python-based two-dimensional physical game engine AI-Olympics environment, accompanied by a variety of scenarios. Meanwhile, we innovatively conducted a series of AI competitions utilizing diverse scenarios created within the framework for a more generalized evaluation. In this paper, we provide a brief introduction to AI-Olympics environment and the accompanying scenarios, as well as the corresponding competition series and the evaluation of different generalization skills. A demonstration video is available at https://youtu.be/SFXRe1JI6C8.

∗Corresponding Author

2 AI-Olympics Environment

AI-Olympics environment is a two-dimensional physical simulator built from scratch with minimal dependencies, utilizing Pygame [Shinde, 2021] for visualization. The design ensures flexible deployment and extensibility. The underlying engine replicates the physical dynamics of agents and elastic collisions between geometric shapes, including straight lines, curves, and circles. Leveraging this game engine, various scenarios and tasks can be generated, spanning different configurations. Three essential characteristics define the environment: (1) continuous control with partial observation, (2) zero-sum, and (3) multi-tasking.

2.1 Mobility and Observability

Each agent is depicted as an elastic circle on the map, equipped with the ability to exert impetus and torque, albeit at the cost of internal energy, enabling movement and interaction with the environment. Running out of energy results in a loss of control over the agent. Within the game environment, agents can collide and experience friction, thereby altering their states accordingly. As illustrated in Figure 1, the agent can perceive its surroundings within a limited range, allowing it to observe any other object within the map. Other objects on the map are represented as geometric shapes, each distinguished by its unique color to denote specific characteristics. As a result, we can construct various maps with recognizable items placed, thereby enhancing intelligent AI recognition and generalization across scenarios.

2.2 Zero-sumGames

Typically, each game in AI-Olympics contains two controllable agents with conflicting interests, rendering it a twoplayer zero-sum game extensively explored in prior research [Silver et al., 2016; Silver et al., 2017; Song et al., 2024]. Both agents share the same mobility and observability for fair competition, and each can seek to gain an edge by obstructing pathways. Such uncertainty regarding the opponent's strategy during the evaluation phase can pose a significant challenge to the participants, encouraging them to consider generalization skills.

2.3 Multiple Designed Tasks

The game engine is crafted to generate diverse scenarios featuring common items on the map, allowing for an exploration of how agents can generalize their skills across tasks. We have developed six unique scenarios, each comprising a competitive sports task with different objectives and involving two agents with conflicting interests. These scenarios are:

Running game. Running game involves two agents exploring a maze and seeking to reach the goal line (finish line) as fast as possible. Figure 2 gives three example maps with common map elements. For example, the black line represents a sticky wall where the agent loses speed when collisions, and the red dashed line represents the goal line. The first agent that reaches the goal line wins the game.

Other single games. Five additional sports-related games are depicted in Figure 3. Wrestling game has the opposite objective compared to the Running game-each agent should keep itself away from the goal line (border line). In Curling game, players engage in a turn-based format where they strategically "throw" their rocks towards the central goal, aiming to position them as close to the center as possible. Both Table-Hockey and Football share similar map layouts, requiring players to score goals while also playing defensively. However, in Table-Hockey, agents are restricted to movement within their half of the field, while in Football, agents enjoy greater freedom of movement. In Billiard game, the objective of each agent is to pocket balls of the assigned color.

Integrated games. The integrated game combines 4-6 distinct scenarios into one unified task. Two players engage in consecutive games, with the victor of the majority of games as the series winner. While each game features unique objectives, they are underpinned by shared elements including agent movement control and map items. Consequently, a well-trained agent can adeptly identify critical elements within its environment, enabling efficient exploration and navigation to overcome obstacles.

3 AI-Olympics Competition Series

AI-Olympics competition series adopts the tasks outlined in Section 2.3 as themes and is evaluated by our online platform, Jidi (http://www.jidiai.cn). The platform offers live rankings and hosts AI competitions akin to Kaggle (https:// www.kaggle.com/), specifically focusing on decision-making challenges. The objective of the competition series is to provide testbeds and promote practical engineering exploration of agents' generalization abilities. Notably, as shown in Table 1, we have conducted several preliminary contests, two of which were partnered with the IJCAI committee to gain further attention.

3.1 Evaluation Protocol

Platform Jidi offers online evaluation services for submitted agents in various simulated environments. The registered users submit their code files on the website, then the platform performs back-end evaluation and updates the results on the front-end webpage. Jidi is able to provide real-time ranking which is helpful for participants to test their agents. For AIOlympics competition series, since the games are all zerosum, Jidi performs a Swiss-system Tournament for player pairing during the evaluation process, which has been used in various types of competitions including sports tournaments and academic events [Csat´o, 2013].

3.2 Generalization Abilities Assessment

During the preliminary competition, we utilize individual scenarios as testbeds and invite participation in each specific task. Additionally, we explore an integrated scenario wherein the agent engages in all games sequentially. Throughout the competition, agents are assessed for their generalization abilities from various perspectives.

Map generalization. In the Running game competition, agents are required to efficiently navigate the track map and reach the goal line (finish line) before their opponent. The competition spans multiple rounds of evaluation, with newly designed maps added to the pool in each round. This ensures that the submitted agents are tested not only on familiar maps but also on unseen layouts. These new maps feature identical elements, as depicted in Figure 2. A proficient agent must adeptly interpret the elements on the map and adjust its movement strategy accordingly, thereby demonstrating its generalization abilities across various maps.

Scenario generalization. In the integrated games competitions, agents are required to compete in multiple sequential sports scenarios, each characterized by a distinct layout and objective described in Section 2.3. To prevent the agent from memorizing the scenario sequence, we employ random shuffling of orders, forcing the agent to identify the current scenario during the evaluation phase. Furthermore, a future extension may involve the addition of new tasks in each evaluation round, placing more emphasis on the agent's ability to generalize across scenarios.

Opponent generalization. Throughout the competition series, the evaluation protocol requires the player not only solve multiple tasks but also compete with various opponents. Playing the same game with different opponents can yield disparate outcomes. Consequently, participants must take into account the strategies employed by other players. This can be achieved by reviewing replays of other players on the website and designing counter-strategies that can be generalized across various opponents.

4 Analysis of the Submitted Agents

During our competition series, some interesting findings echo our motivation as well as provide insights into the field.

Enhancing map generalization through increased data. In the Running game competition, at the beginning there are two maps in the candidate pool, then in subsequent rounds two additional maps are introduced into the pool, each with an increased probability of selection. As the competition progresses, we observe a consistent improvement in the mobility and completion rates of participants' agents on newly introduced maps. This phenomenon is attributed to participants incorporating the new maps into their training, thereby expanding the breadth of their training data through techniques such as data augmentation [Shorten and Khoshgoftaar, 2019]. Consequently, this broader scope of training data contributes to the enhancement of agents' generalization skills over time.

Improving performance via scenario-specific strategies. Despite the random order of scenarios, many agents still manage to apply a targeted policy to each scenario in the integrated game. In some scenarios such as Curling, which demands more intricate skills, participants demonstrated a notable trend for adapting their strategies to the current game situation. For instance, some agents will stop in front of the goal line (hog line) and observe the situation first and then move backward to leave space for "throwing". This aligns with our motivation to run the competition which is to ask the agent to act adaptively in the game instead of reacting to experience.

Gaining advantages in competitive games through aggressive actions. The presence of opponents introduces varying degrees of challenge. In scenarios emphasizing competition, we observe certain agents exhibiting interference behaviors. For example, in some maps of Running game, agents actively engage in tactics such as (1) intentionally colliding with opponents to slow their progress by pushing them against walls, or (2) strategically maneuvering to obstruct the paths of others during turns.

Employing diverse AI-related methods. After discussing with several participants, we have gained valuable insights into the methodologies they used to tackle the tasks at hand, including:

• reinforcement learningwith computer vision techniques;
• learning from demonstration to accelerate training;
• self-play framework for robustness;
• complex reward shaping to guide behavior;
• historical data summarization for long-term planning;
• curriculum learning to ease the training process;
• heuristic methods to fill up the opponent pool.

In practice, participants often utilized a combination of these methods when implementing their agent. In contrast to stateof-the-art algorithms in academia, the competition underscored the practicality of machine learning methods and offered valuable engineering insights to the field of game AI.

5 Conclusion

AI-Olympics competition series has provided a unique platform for participants to showcase their expertise in game AI. Through diverse challenges, participants have demonstrated their proficiency in addressing complex scenarios and adapting to dynamic environments. The competition's emphasis on practicality and agent generalization abilities underscores the complexity of game AI research nowadays and brings insights for further research endeavors. In the future, we will keep exploring the generalization skills of agents and continue to gather community support to tackle the problem.

Acknowledgments

Haifeng Zhang thanks the support of the National Natural Science Foundation of China, Grant No. 62206289.