Reinforcement Learning for Athletic Intelligence: Lessons from the 1st “AI Olympics with RealAIGym” Competition

1 Introduction

Benchmarks and competitions have proven extremely successful in computer vision and machine learning for driving algorithmic innovation [Deng et al., 2009]. In robotics, a number of simulated benchmarking environments gained popularity for evaluating learning algorithms [Brockman et al., 2016; James et al., 2020; Mittal et al., 2023; Al-Hafez et al., 2023]. However, only a handful of real-robot standardized environments are reliable, easy to simulate, and cheap to reproduce. Importantly, different environments have different focus, e.g., locomotion [Grimminger et al., 2020; Feng et al., 2023], finger-based manipulation [Funk et al., 2021; G¨urtler et al., 2023], etc. Our RealAIGym project [Wiebe et al., 2022b] 1 offers a suite of canonical underactuated systems

1 https://dfki-ric-underactuated-lab.github.io/real-ai-gym/

(simple pendulum [Wiebe et al., 2022a], AcroMonk [Javadi et al., 2023], hopper [Soni et al., 2023]) for benchmarking learning and control algorithms for athletic intelligence on real hardware with a user-friendly Python API. By opensourcing both the software and the hardware, we aim to establish a real-world equivalent to the well-known OpenAI Gym [Brockman et al., 2016].

The "AI Olympics with RealAIGym" competition 23 is based on a Double Pendulum system (see Fig. 1) in the RealAIGym project. The Double Pendulum [Wiebe et al., 2023] can operate in two modes: Pendubot when the actuator in the shoulder joint is active but the elbow is passive, and Acrobot with passive shoulder and active elbow. The challenge consists in performing a swing-up and stabilization from the free-hanging position to the upright position. For scoring, the Acrobot and Pendubot tasks are treated as two separate tracks.

2 https://ijcai-23.dfki-bremen.de/competitions/ai olympics/

3 https://youtu.be/eYDH1v1FqF8

The competition was held in two phases. In the simulation phase, the participants were provided model parameters, a simulation environment, and the scoring metric to rank the behaviors. The submitted controllers were evaluated with a performance score and a robustness score. The performance score, calculated in the simulation and in the hardware phase, evaluates how well the swing-up is performed and is given by a weighted sum of swing-up time, energy cost, maximum torque, integrated torque, torque cost, torque smoothness and velocity cost and is multiplied with the swing-up success rate. The robustness score evaluates the sensitivity of the controllers to model inaccuracies, measurement noise, torque noise, torque response, and time delay.

The teams who achieved high simulation and robustness scores were advanced to the hardware phase, where they were given up to 20h remote-access to a real double pendulum system to train and test their controllers. The final controllers were evaluated on the same PC using the performance score over 10 consecutive executions to provide the final ranking.

2 Algorithms

In this section, three families of RL methods that participated in the competition are presented, together with a baseline optimal control approach for comparison.

2.1 Model-based RL: MCPILCO

MC-PILCO (Monte Carlo - Probabilistic Inference for Learning COntrol) [Amadio et al., 2022] is a model-based policy gradient algorithm that relies on Gaussian Processes (GPs) to learn the system dynamics from data. Let xt and ut be, respectively, the state and input of the system at step t. A cost function c(xt) encodes the task to be accomplished. A policy πθ : x → u that depends on the parameters θ selects the inputs applied to the system. The objective is to find policy parameters θ∗ that minimize the cumulative expected cost:

MC-PILCO performs several successive attempts to solve the desired task, also called trials. Each trial consists of three main phases: (i) model learning, (ii) policy update, and (iii) policy execution.

In the model learning step, previous experience is used to derive a one-step-ahead stochastic model of the system dynamics using Gaussian Process Regression. The policy update step aims at minimizing the cost in eq. (1) w.r.t. the policy parameters θ. The expectation in eq. (1) is approximated based on the GP dynamics derived in (i) and Monte Carlo simulation. Finally, in the last step, the current optimized policy is applied to the system and the collected samples are stored to update the model in the next trials. Examples of MC-PILCO applications have been reported in [Amadio et al., 2023] and [Turcato et al., 2023].

2.2 Model-free RL, Actor-Critic Methods: SAC

A potential solution in the field of model-free reinforcement learning involves combining the Soft Actor Critic

(SAC) [Haarnoja et al., 2018] algorithm with the Linear Quadratic Regulator (LQR). The SAC algorithm is utilized to train a reinforcement learning (RL) agent for swing-up tasks, while the LQR controller is responsible for stabilizing the system at its highest position. The transition between the SAC and LQR occurs on entering the Region of Attraction (RoA) of the LQR controller, which is numerically approximated [Maywald et al., 2022]. A three-stage reward function is designed to steer the agent into the LQR controller's RoA, facilitating the switch. The initial stage involves a quadratic function that penalizes state errors and torque usage. Upon the end-effector reaching a specific threshold line, an additional reward is introduced. Furthermore, a substantial reward is granted when the state is within the LQR's RoA.

2.3 Model-free RL, Value-basedMethods: DQN

Deep Q-Network (DQN) [Mnih et al., 2015] is the seminal algorithm in the field of Deep RL, which has demonstrated that end-to-end controllers with high-dimensional observation spaces can be learned by trial and error. DQN is a valuebased method that learns the optimal action-value function Q(x, u) such that the optimal action can be obtained by optimization over u. To learn Q, DQN iteratively applies the Bellman operator, which is a contraction mapping and hence its successive application leads to a fixed point. The theory guarantees that the fixed point of the Bellman operator is the optimal action-value function corresponding to the optimal policy, yielding the maximal reward / minimal cost.

DQN is known to work well in discrete action spaces. Since the action space of Pendubot is 1-dimensional, we consider discretizing it into 9 bins. A logarithmic discretization centered around zero yields better performance in practice, because the agent requires more actions around zero to control the pendulum close to the upright position. The same rewards function as for the SAC algorithm is used for training DQN, and the LQR controller provided by RealAIGym is employed to stabilize the Pendubot when it enters the RoA.

2.4 Model-based Optimal Control: TVLQR

An optimal control (OC) controller serves as a baseline for the learning-based controllers in the competition. The controller utilizes three versions of the linear quadratic regulator (LQR). The basis is a trajectory (udt , xdt ) computed with the trajectory optimization technique iterative LQR (iLQR) [Li. and Todorov., 2004] and a time-varying LQR (TVLQR) [Tedrake, 2022] is used to stabilize the system towards that trajectory during the execution. This results in the control law at time t:

with the linear feedback matrix Kt. The controller switches to a LQR control once its RoA is entered.

3 Results

The results of the competition are listed in Table 1 and visualized in Fig. 2. All data, figures and videos of the experiments can be found online 4 . Swing-up attempts are evaluated based on the performance score (simulation and hardware) and the robustness score (simulation) as introduced in section 1.

4 https://dfki-ric-underactuated-lab.github.io/real ai gym leaderboard/

3.1 Acrobot

Two submitted controllers (MC-PILCO and SAC) passed the simulation stage for the Acrobot swingup problem. MCPILCO achieved a high performance score (0.869), beating the optimal control baseline (0.8) and a mediocre robustness score (0.595). SAC's performance score is a little lower (0.811) while the robustness score is significantly higher (0.82) and closer to the OC baseline (0.861). Despite the high performance and robustness scores, the SAC team did not succeed in transferring the controller to the real hardware. A successful swing up was not achieved. The MCPILCO team, on the other hand, retrained their policy on the real hardware and demonstrated a 100% success rate in the 10 evaluation trials resulting in a hardware score of 0.817 which is only slightly lower than the OC baseline (0.821). MCPILCO's mediocre robustness score indicates that a policy trained with MC-PILCO has limits when exposed to dynamics which have not been experienced during training. However, the notable performance of MC-PILCO on the real hardware shows its applicability even for real dynamics when directly trained under these conditions.

3.2 Pendubot

On the Pendubot, MC-PILCO, SAC and DQN passed the simulation stage of the competition. MC-PILCO scored the highest in the simulation (MC-PILCO: 0.891, SAC: 0.876, DQN: 0.815, TVLQR: 0.827) while SAC showed the highest robustness score only secondary to TVLQR (MC-PILCO: 0.852, SAC: 0.986, DQN: 0.226, TVLQR: 0.950). MCPILCO again was retrained on the real hardware and achieved a 100% success rate with a hardware score of 0.839. SAC achieved a 40% success rate and a hardware score of 0.298. Note that the SAC policy was not retrained on the hardware but only in a simulation with added noise. DQN could not be successfully transferred to the real system. The OC baseline has an 80% success rate and a 0.547 score.

4 Conclusion

We introduced a competition to evaluate the performance and robustness of learning-based vs. optimal control approaches. From the first competition conduction, we conclude that learning-based approaches provide a strong alternative to model-based optimal control, being able to learn from a few samples on the real system. The model-based MC-PILCO outperformed the model-free methods SAC and DQN, showing a more reliable performance on the real Pendubot system than the baseline TVLQR method. We hope this challenge inspires further research into control algorithms for athletic intelligence; in particular, tackling generalization to different initial conditions, robustness to perturbations and handling of unmodelled dynamic effects.

Ethical Statement

There are no ethical issues.

Acknowledgments

The competition organizers at DFKI-RIC (FW, SV, FK, SK) acknowledge the support of M-RoCK project funded by the German Aerospace Center (DLR) with federal funds (Grant Number: FKZ 01IW21002) from the Federal Ministry of Education and Research (BMBF) and is additionally supported with project funds from the federal state of Bremen for setting up the Underactuated Robotics Lab (Grant Number: 201342-04-1/2023-4-1). TV, BB, and JP acknowledge the grant "Einrichtung eines Labors des Deutschen Forschungszentrum f¨ur K¨unstliche Intelligenz (DFKI) an der Technischen Universit¨at Darmstadt" of the Hessisches Ministerium f¨ur Wissenschaft und Kunst. This research was supported by Research Clusters "The Adaptive Mind" and "Third Wave of AI", funded by the Excellence Program of the Hessian Ministry of Higher Education, Science, Research and the Arts. Parts of the calculations for this research were conducted on the Lichtenberg high-performance computer of the TU Darmstadt. TV and BB also thank Daniel Palenicek and Tim Schneider for their support while performing the experiments on the IAS group's computing cluster at TU Darmstadt.

Contribution Statement

NT, ADL, GG, RC and DR constitute the winning team and contributed to the MC-PILCO controller. CZ, AS and MZ formed the runner-up team which implemented the SAC+LQR controller. TV and BB provided the DQN implementation. FW, SV and SK implemented the baseline controller and benchmarking software. SK and BB conceptualized and executed the competition at the IJCAI venue. All authors contributed to the preparation of this manuscript.