Energy-Efficient Slithering Gait Exploration for a Snake-Like Robot Based on Reinforcement Learning

1 Introduction

Snake-like robots, as a class of hyper-redundant mechanisms, carry the potential of being one kind of promising mobile robotic applications that are capable of traveling and performing tasks in diverse environments, such as disaster rescue, underwater exploration, and industrial inspection [Liljeb¨ack et al., 2012]. Since snake-like robots can only carry limited energy resources for field operations, it is important to develop energy-efficient gaits to reduce the impact of the power constrains. On one hand, optimizing the power consumption can prolong the service time of a robot and maximize its locomotion performance at the same time. On the other hand, an efficient energy system may in return allow us to design a more lightweight robot or add other functional components [Tesch et al., 2009]. However, it is challenging to design energyefficient gaits for snake-like robots on the basis of their redundant degrees of freedom (DOF) and the complex interactions with the environment [Tucker, 1975].

Since the first snake-like robot was built in 1972, researchers have been working constantly on designing more advanced snake-like robots [Liljeb¨ack et al., 2012] and sophisticated gaits for robots with different types of mechanical configurations or terrains. Meanwhile, slithering gait has been considered as the most promising gait for snakelike robots to perform autonomous locomotion tasks, which imitates the serpentine locomotion of real snakes [Hu et al., 2009]. Hirose first used the serpenoid curve to control a snake-like robot, which was an effective approach by imitating the real snake movement [Hirose, 1993]. Ma proposed another model serpentine curve to describe the locomotion of snakes by modeling their muscle characteristics and achieved a higher locomotive efficiency than the serpenoid curve by running simulations [Ma, 1999]. On the basis of these snakelike movement curves, the gait equation, as a robust and effective method, works as an abstract expression of gaits of a snake-like robot by describing joint angles as parameterized sinusoidal functions [Tesch et al., 2009]. It allows for the emergence of complex behaviors from low-dimensional representations with only few key parameters, greatly expanding their maneuverability and simplifying user control. With this method, researchers developed several biological gaits for snake-like robots to move in the indoor and outdoor environment [Melo and Paez, 2014].

However, optimizing these parameterized gaits for the purpose of energy saving is difficult and limited, since they are confined to those abstracted gait parameters and only few studies have been reported. Crespi et al. adopted a heuristic optimization algorithm to rapidly adjust the travel speed of the robot [Crespi and Ijspeert, 2008]. Tesch et al. used the Bayesian optimization approach to regulate those openloop gait parameters for snake robots, which made the robot move faster and more reliable [Tesch et al., 2011]. Gong et al. proposed a shape basis optimization algorithm to simplify the gait design parameter space and came up with a novel gait in granular materials [Gong et al., 2016]. Even so, all these works still focus on optimizing the gait on the basis of the parameterized gait generation system, and have very limited effect on further improving gait efficiency.

This gait optimization task, however, corresponds to a complex control problem due to two primary reasons [Liljeb¨ack et al., 2012]. The extrinsic challenge comes from the complex dynamic interaction between the ground and the redundant mechanism with many degrees of freedom. Therefore, it is extremely important to model precisely and rapidly. The intrinsic challenge is how to synchronize and coordinate all the body joints to exhibit a proper motion pattern integrally, which is expected to be both robust and efficient.

As an emerging technology, reinforcement learning (RL) reveals the nature of the learning process of locomotion in animals that can offer a model-free learning to master new skills or adapt to diverse environments. On this basis, this work offers a novel alternative to design the slithering gait of a snakelike robot based on reinforcement learning technology. Our main contributions are summarized as follows. First, we define the energy-efficiency metrics and introduce the parameterized slithering gait design method as the baseline, which is optimized by using a grid search method and the Bayesian optimization method in terms of energy efficiency. Second, we propose a gait controller using the state-of-the-art RL algorithm PPO in simulation. The learned gait exhibits surprising similarity to the natural movement of real snakes. Third, the results demonstrate that the learned gait successfully outperforms the parametrized slithering gait in terms of energy efficiency at a range of velocities.

2 RelatedWork

Unlike other land animals, snakes achieve diverse locomotion gaits by twisting their bodies on various terrain and exhibit undulatory locomotion. To imitate similar efficient movement, most snake-like robots are controlled by the kinematics-based method, which can be regarded as a process to simplify parametric representations of the snake-like trajectories [Hirose, 1993; Ma, 1999; Tesch et al., 2009]. However, this method limits gait efficiency by only manually tuning those parameters, even without considering that it is timeconsuming and inefficient.

Although there are few studies on optimizing the gaits for snake-like robots, studies on optimizing the locomotion gaits of other kinds of robots have been reported for the purpose of energy saving. Initially, researchers adapted techniques for multidimensional function optimization tasks to design efficient gaits, such as evolutionary algorithms [Chernova and Veloso, 2004] or policy gradient search algorithms [Kim and Uther, 2003; Kohl and Stone, 2004]. However, these algorithms are usually plagued by local optima, which makes the process slow, inefficient, and manually intensive. Afterwards, gait optimization methods based on prior knowledges are further investigated. Lizotte et al. first optimized the speed and smoothness of the gait with Gaussian process regression on a quadruped robot[Lizotte et al., 2007]. Calandra et al. used Bayesian optimization approach to regulate those openloop gait parameters for bipedal robots, which made the robot move faster and robuster [Calandra et al., 2014]. Even though these optimized gaits outperform the hand-coded gaits, they are still very inefficient forms of locomotion compared with the natural movement achieved by animals.

As an intelligent trial-and-error learning method, RL brings new solutions for free gait generation tasks without knowing precise models or prior knowledge. Initially, RL was not widely used in the domain of robotics, because robots often require high-dimensional, continuous states and actions [Kober et al., 2013]. However, with more and more advanced RL algorithms coming out, many robotic implementations are able to handle complicated tasks, such as gait generations [Peng et al., 2017], dexterous manipulation [Rajeswaran et al., 2017], and autonomous driving [Long et al., 2018]. In [Cully et al., 2015], two prototype experiments are shown in which RL can help robots recover from damage and adapt quickly like animals do. A hexapod robot learns to walk fast and straight with broken or missing legs. And a robotic arm learns to reach its previous position goal with one or more stuck joints. Afterwards, RL technologies are increasingly used in free gait generation tasks. Schulman et al. [Schulman et al., 2017] implemented their PPO algorithm on a collection of benchmark robotic locomotion tasks from 2D swimmer to 3D humanoid robot. Except for simply using RL-based methods to generate gaits for robots, they have been used to learning energy-efficient gaits. Kormushev et al. [Kormushev et al., 2018] used RL to optimize the vertical center-of-mass trajectory and the walking pattern of a bipedal robot to exploit the passive compliance for the purpose of energy efficient walking. Yu et al. [Yu et al., 2018] proposed an RL-based approach for creating low-energy and speed-appropriate locomotion gaits for four types of walking, including biped walking, quadruped galloping, hexapod walking, and humanoid running.

In this paper, we focus on using RL to learn slithering gaits for a snake-like robot with redundant degrees of freedom, so that the learned gaits can outperform those parameterized gaits in terms of energy efficiency.

3 Models andMetric Definition

In this section, we first introduce the snake-like robot model used for exploring different gaits. Then, we present our energy efficiency metric for comparing different gaits based on our robot model.

3.1 Snake-like Robot Model

We model and simulate our snake-like robot in MuJoCo [Todorov et al., 2012], which is a physics engine offering fast and accurate robot simulation environment. The snake-like robot model used in this study is inspired by the ACM snake-like robot [Hirose, 1993], which uses eight joints and nine identical modules. The first module is used as the head module and equipped with a vision sensor for performing other tasks. Figure 1 shows the technical drawings of the first two modules of the snake-like robot model. The model acts in a dynamic environment, therefore its weight, friction on the ground, and actuator power are critical to enable availability of those values. A uniform density of 600 kg/m3 is set for all components of the model. This density is selected based on the robot developed by Dowling [Dowling, 1996], which has about the same value including all mechanical and electrical components.

Actuated Joints

The snake-like robot adopts actuated joints to bend its body to slither forward. All the joints are modeled as a servo motor and rotate along the perpendicular direction to the ground in a range of [-90◦ , 90◦ ]. The force is limited to a range of [-20, 20] in Newton, which is adequate to propel the modules at a reasonable speed and strength. The maximum actuator torque can be calculated by multiplying the actuator force with the gear length of 0.175 m, which is the half length of a module.

It is worth mentioning that a configuration with torque motor is also tested and works as well as the servo motors. For comparison purposes, the servo motor type is chosen because the gait equation controller also depends on this input type, which will be further explained in Section 4.

Passive Damping Wheels

Each robot module is equipped with two passive wheels, which are used to imitate the anisotropic friction property of the snake skin. Like the movement of real snakes, those passive wheels enable a minimum friction in the direction of rotation and a high friction in the lateral direction. To enable the torque energy costs at all the joints, a constant damping property is added at each wheel joint. This constantly decelerates the rotation of the wheels, which imitates a low friction in forward direction. Otherwise, the robot will just form itself in a straight line and roll forward to gain distance while avoiding joint torque energy costs.

3.2 Energy EfficiencyMetric

Mobile robots have to conserve their battery power so they can operate for long periods of time. Our goal is to design gaits that are capable to move in an energetic and economical manner and are still versatile to move at a range of velocities [Dowling, 1997].

Power Profiles

The power profile is one set of the average power for each module of the robot during a test run, which offers detailed information to help improve the gait generations. For example, we can observe that which modules consume more power and therefore are more likely to heat up fast or broken.

For a snake-like robot with N joints, the averaged power ¯ Pj for the j th actuator is the averaged absolute value of the product of the torque τj and its angular velocity ˙φj during a run with k steps. Thus, our first metric can be expressed as

Power Efficiency

Based on the power consumption of each module of the robot, the total power consumption P of all N actuators at each time step is calculated by

where the torque τj is the product of its applied force fj and its gear constant parameter hj (the length of the actuator). The model uses actuators with a limited force of fmax as maximum force in both directions. With this property, the normalized power consumption ˆ P is calculated as

This normalized power consumption ˆ P will be further used for reward definition.

With the variables of energy consumption P and the velocity v, several efficiency metrics can be calculated. A usual way is to calculate the Cost of Transport (COT), which is the power P or energy E divided by the mass m, the gravity g and the distance d or velocity v:

This unit-less measure is also able to compare the efficiency between different mobile systems. Since the same robot model and the same environmental properties are compared, those constants only scale the results and would not provide any additional insight for the comparison. Thus, the second metrics is to calculate the Averaged Power per Velocity (APPV) as

Similar power efficiency metrics can be found in [Tesch et al., 2009] and [Saito et al., 2002].

4 Baseline Examples

This section provides two baseline examples, where the parameterized gait equation controller is presented to generate the slithering gait for our snake-like robot. By searching a grid of gait parameters with fixed intervals, we try to find out the best energy-efficiency gaits at different velocities that can be acquired by this controller. Then we use the Bayesian optimization algorithm to explore better parameter combinations in the range of searching grid, since the searching grid is relatively sparse.

4.1 Gait Equation Controller

The gait equation method represents a kinematic locomotion controller that uses a mathematical equation to describe its gait. In this work, an undulation gait equation extended from is used for the purpose of comparison [Bing et al., 2017]. The gait equation controller is modeled as

φ(n, t) presents the joint angle value at time t, where n is the joint index and N is the joint amount. λ and ω are the spatial and temporal frequency of the movement, respectively. The spatial frequency represents the cycle numbers of the wave and the temporal frequency represents the traveling speed of the wave. A is the serpentine amplitude and x and y are the constants for shaping the body curve.

To ensure a fair comparison, we use the grid search method to generate a variety of gaits and find out those parameter combinations with the best power efficiency at different velocities. The grid search method generates a Cartesian product from the parameters in Table 1, resulting in 6480 parameter sets. Then, each motion parameter set gets tested by running 1000 steps in the simulation environment. For each run, the first 200 time steps are ignored and the remaining 800 time steps are evaluated for collecting experimental data. This is done because it has been observed that the snake robot needs about 200 time steps to accelerate and then moves at a steady speed.

4.2 Bayesian OptimizationMethod

Since the searching grid is relative sparse, we further use the Bayesian optimization algorithm to search for better parameters that may be located in those grid intervals. This strategy has been used for optimizing snake-like robot gait parameters in [Tesch et al., 2011] and other kinds of robots [Lizotte et al., 2007; Calandra et al., 2014].

The purpose of this method is to find a group of optimized parameters in (6) to achieve minimum power consumption at different velocities. The boundaries of these parameters are set as same as the value boundaries in Table 1. According to the gait equation controller, the maximum speed of the robot is mainly bounded by its temporal frequency ω. Therefore, we perform the optimization process twelve times when ω is uniformly sampled from [0, 3.0] at a step size of 0.25, which results in 12 trials. For each search, the exploration and exploitation samples are set as 10 and 100 for avoiding the local minimum.

4.3 Baseline Performance

The power consumption and the corresponding velocity results from the grid search algorithm and the Bayesian optimization algorithm are shown in Figure 2 as a point cloud of parameter sets using blue dot and orange diamond markers, respectively. The lowest points at different velocities in the point cloud have the highest efficiency. As we can see, the power requirement grows linearly with the increasing velocity. Most results from the Bayesian optimization controller (orange diamond marker) almost match the bestenergy-efficiency points from the grid search method, which proves the applicability of the algorithm. And some points even exhibit better efficiency especially when the desired velocity is higher, which shows the capabilities of the Bayesian optimization algorithm for choosing better parameters for gait generation tasks. However, there are two outliers with low efficiency, which may be caused by falling into local minimum values during the optimization process.

In short, we demonstrate that the Bayesian optimization algorithm is a quick and effective method to find a proper parameter set for achieving desired velocity, especially when most of the parameter sets in the searching space are distributed in the low velocity area (0 m/s ∼ 0.1 m/s).

5 Proposed RL-Based Controller

We begin this section by introducing the key ingredients of our reinforcement learning-based controller. Then the RL network architecture and the training configuration are introduced as well.

5.1 Reinforcement Learning Setup

Below we describe the details of the observation space, the action space, and the reward function.

Observation Space

The RL agent receives the environmental information via the observation space at each step. A proper choice of the observation space parameters is critical in RL, since the agent needs the right set of information to learn the causality of the given rewards based on its actions.

The observation space oti is given in Table 2. The joint position φj and its angular joint velocity ˙ φj are required to learn the locomotion and represent the proprioceptive awareness of the robot. The head link velocity v1 helps to sense its global velocity, which offers better movement awareness. In order to learn a power efficient gait, the sense of energy consumption is necessary. Therefore, the actuator torque of each joint τj is provided and can be interpreted in combination with ˙ φj to determine the total power usage. The specified target velocity vt is passed to the environment and can be dynamically changed. This is required to control the velocity of the robot. In summary, an overall 26-DOF observation space is used in this work.

Action Space

The action space ati of the environment has 8-DOF with finite continuous values in the range of [-1.5, 1.5], which linearly translates to a corresponding joint angle φ in range of [-90◦ , 90◦ ]. Each action represents eight actuator angle positions of the servo motors. A uniform setup with servo motors is chosen so the environment between the two controllers can be compared. There has been no significant difference noticed between a servo and a torque setup.

Reward Function

The objective of this experiment is to learn a power efficient locomotion for a variety of specified velocities. Therefore, the energy consumption and the difference between the actual model velocity and the target velocity are the main criterias to find a successful behavior. The challenge is to combine and weigh the variables into one numerical reward for each time step. Therefore, we first split the power efficiency and velocity criteria into two normalized reward function components.

First, a normalized reward is defined to maintain the specified velocities. The objective is to reach and maintain the target velocity vt by comparing it with the head velocity v1. The following function represents the velocity reward:

The parameter a1 = 0.2 influences the spread of the reward curve, by defining the x-axis intersections with x = vt ± a1. a2 = 0.2 affects the changes of the curves gradient. If |vt - v1| = 0.0, the velocity reward rv has to be the maximum value 1.0.

Second, the normalized value of the total power usage ˆ P in (3) is used to determine the power efficiency reward component rP , which is represented by

Here, rmax is the maximum reward value and b1 = 0.6 is the slope of the curve. The power efficiency is influenced by the desired target velocity. Thus, the normalized rmax represents this influence by limiting the maximum value of rP .

Last, the rewards from the velocity rv and the power efficiency rP are combined to form the overall reward r:

This equation replaces the rmax in (8) with rv. With that, the maximal power efficiency depends on the absolute value of the difference between the desired velocity and the robot velocity.

5.2 Network Architecture

Given the input (observation oti ) and the output (action ati ), we now elaborate the policy network mapping oti to ati . We design a fully connected 2-hidden-layer neural network as a non-linear function approximator to the policy πθ. The input layer has the same dimension as the action space oti . Both hidden layers have 200 ReLU units and the final layer outputs the joint position commands for the robot. To train the network, the PPO algorithm adapted from [Schulman et al., 2017] is used and our codes are open-sourced at here 1 .

5.3 Training Configuration

To enable the learning of different velocities, the parameter vt is changed by iterating over 0.05, 0.1, 0.15, 0.20, and 0.25 for each episode while training. Meanwhile, to simplify the beginning of the learning process the first 100 episodes are trained with a fixed target velocity of 0.1 m/s.

We train our policy network on a computer with an i7-7700 CPU and a Nvidia GTX 1080 GPU. Based on the learning curve a total of 3 million time steps (about 1400 updates) are used for training. With the environment settings of 50 ms per time step, the training takes about 42 hours in total simulation time and 2 hours in wall clock time for the policy to converge to achieve a robust performance.

6 Results and Comparisons

In this section, we first describe the performance of the gaits generated by the RL controller. Then, we compare our gaits to the scripted slithering gaits in terms of energy efficiency.

6.1 Results

In this study, 45 target velocities in the range of [0.025, 0.25] with a step interval of 0.005 are used for the evaluation of performance. Simulation results demonstrate that the PPO controller has succeeded in learning a gait from scratch without knowing any prior locomotion skills. Two exemplary gait patterns are shown in Figure 3. Surprisingly, we find that the PPO controller can adapt its gait pattern according to its desired speed like a real snake. The montages in the first row present a concertina-like gait at a low speed of 0.05 m/s and the second row resembles a slithering-like gait at a higher speed of 0.25 m/s. In nature, snakes usually take the concertina locomotion at a low speed and switch to the slithering locomotion at a fast speed. In the second row, we also observe that the snake-like robot executes waves from the head to tail through a movement pattern of lateral undulation. This learned undulation wave is even smoother than the parameterized serpentine curve and drives the robot to move with better power efficiency.

1 https://github.com/BZSROCKETS/RL Snake

The power consumption results of the learned gait are marked with red points in Figure 4. The data depicts a linear relationship between the travel velocity and the power consumption, which is in line with the physical law Power = Force × Speed. There is only one point with higher power consumption when the velocity is around 0.02 m/s. Importantly, this also reveals the adaptability of the learning approach for generating gaits in a range of velocities. It can also be observed that the mean velocities do not exactly align with the specified interval of 0.005, especially at higher target velocities. The reason for this is the difficulty to achieve the exact ratio between holding the right velocity and performing the corresponding power-efficient locomotion.

6.2 Comparison

We first have a look at the energy metric based on the averaged power per velocity. After putting the power efficiency data of the gait equation controller (See Figure 2) and the PPO controller together, we can clearly conclude that the PPO controller has a much better power efficiency at a range of velocities (See Figure 4). As the velocity grows, the advantage of the PPO controller for saving energy is even more obvious. Taking the velocity of 0.15 m/s as an example, the PPO controller can save 35% ∼ 65% energy consumption depending on the parameter sets chosen by the gait equation controller.

We then discuss the power profile for each module of the robot. The power profiles of the slithering gait at the velocity of 0.25 m/s generated from the parameterized equation controller and the PPO controller are shown in Figure 5. It should be noted that we only present the results of the parameterized equation controller with the best energy efficiency at this speed. For the parameterized equation controller, the first joint consumes the most power since it rapidly adjust its head to aim at the front direction. Other module joints use similar power around 5mW to 5.7mW. For the PPO controller, the head joint still expends more energy than the average power consumption. Besides, the joints close to the body center consume more power than the far-ends of the body. This is because the body trunk modules exert more strength to twist its body to generate the locomotion, which shows that the PPO controller generates more natural slithering gait compared to the parameterized slithering gait.

The PPO controller improves on the traditional kinematicbased controller in two ways. On one hand, the traditional gait equation controller is based on the kinematics and describes the gait movement with no influence of physical forces such as friction or damping. Meanwhile, it only extracts several critical parameters to represent the gait, which is in fact a highly dimensional interaction with the environment. Although this parameterized description simplifies the control task, it inevitably brings difficulties for designing more sophisticated gaits despite of using optimization technologies, especially when the parameter space will grow exponentially with the increasing joint numbers. On the other hand, the RL method shows its effectiveness in the ability for solving this kind of complex control problem, since it is trained directly in the dynamic environment. It is able to generate an undulation gait that not only imitates real snakes, since natural evolution is not a perfect process but a compromise result. Therefore, it can explore its limitations and keep improving its behavior under its hardware constraints. This is a remarkable achievement because the algorithm has to handle a high degree of freedom without prior information about the environment or any locomotion behavior. In short, the PPO controller is able to overall outperform the equation controller in terms of power efficiency at a range of velocities.

7 Conclusion

Designing power-efficient gaits for snake-like robots remains a challenging task, since they come with redundant degrees of freedom and have complicated interactions with the environment. In this paper, we present a novel gait design method based on reinforcement learning. The learned gait has shown to have much better energy efficiencies at different travel velocities compared to the current kinematic-based method even after choosing those parameters using the grid search and Bayesian optimization algorithm. Our work contributes and serves as an exploration for designing sophisticated moving patterns for snake-like robots. Our future work will aim at designing gaits based on reinforcement learning for those kinds of snake-like robots without passive wheels.