Function Approximation for Reinforcement Learning Controller for Energy from Spread Waves

1 Introduction

Coastal ocean waves offer a consistent and clean energy source that can help in reducing carbon emissions from power generation. To enhance energy capture and competitiveness, a new design with three generators on three interdependent legs has 2) been developed, utilizing translational and rotational motions. However, this complex design poses challenges for traditional control methods, and the variability of waves in terms of direction, frequency, and height further complicates the process. Additionally, the spinning yaw motion causes high mechanical stress, necessitating measures to prevent costly maintenance in offshore environments.

∗ corresponding author

This work focuses on the CETO 6 Wave Energy Converter (WEC) system, which is currently deployed in Australia and planned for deployment in Europe. Traditional engineering approaches, such as the spring damper, were ineffective in optimizing energy capture efficiency and stress reduction due to the complexities involved. To address these challenges, the paper explores the use of Reinforcement Learning (RL) with a Multi-Agent Reinforcement Learning (MARL) controller and a Proximal Policy Optimization (PPO) training approach. The study evaluates different neural network architectures for the actor and critic and introduces a highperforming STrXL transformer architecture. The goal is to enable optimal control of the generators on the power takeoffs (PTOs) to maximize power generation while minimizing mechanical stress and reducing yawmotion, thereby reducing maintenance costs in offshore environments.

The main contribution of this paper can be summarized as follows:

• First ever implementation of RL controller for 3legged Spread Waves.

No published work has ever addressed this complex realworld problem for an industrial WEC.

• Looks beyond the common RL hyper-parameter tunings and PPO optimizations, into the different function ap-

proximations for the policy and critic networks like Fully Connected Neural Network (FCN), LSTM, and Transformer variants which can effectively model the sequential nature of system dynamics better.

• Explore and investigate various gated bypass options in and around the transformer block for easier trainability and faster convergence with RL which are key bottlenecks in using transformers with RL.
• Propose a transformer block architecture STrXL which performs better and trains faster than state-of-the-art GTrXL for this problem- as represented in Figure 1(b) and Figure 10.
• Investigate the effects of various depths of the network for the function approximators.

2 Background and RelatedWork

2.1 Wave Energy Converter (WEC)

The CETO 6 WEC is a wave energy converter that uses a cylindrical Buoyant Actuator (BA) to convert the chaotic motions of the ocean into linear motions. The BA is secured to the seabed through three mooring legs, each of which terminates on one of the three power take-offs (PTOs) located within the BA. The PTOs resist the extension of the mooring legs, thereby generating electrical power. The optimal timing of the PTO forces resisting the wave excitation force is key to maximizing WEC performance. Various control strategies like damping control, spring damper control, latching control, and model predictive control exist, attempting to get as close as possible to the optimal force function.

2.2 OceanWaves and SpreadWaves

Regular waves also called "monochromatic waves", are simplified waves that comprise of only one period. In other words, each wave is the same as the previous one and the next one and looks like a perfect sinusoidal. Though the ocean at times does exhibit conditions close to regular waves, they mostly have a theoretical interest, as they allow to target the response of a system to a particular frequency excitation

Irregular waves are a superimposition of regular waves of different periods. Which periods are superimposed and how much energy is contained within each period is described by the energy spectrum. Wind waves have a spectrum that peaks at periods below 7 seconds, whereas ground swell is represented by a spectrum that peaks above 7 seconds.

Spread Waves: The above descriptions do not mention wave directionality. In the open ocean, waves rarely come from one single direction, but rather from a range of directions as shown in figure 3. In practice, this means that the waves don't look like straight lines propagating toward the observer, but are rather short-crested. The spread factor is representative of how wide a directional range of waves is propagating within. These are called spread waves.

2.3 Spring Damper Benchmark Controller (WEC)

The PTO is composed of a mechanical spring and an electrical generator. The damping component is akin to a reactive braking torque against the input shaft, driven by the wave energy source. The captured energy equals the braking mechanical work done by the generator minus losses. The average mechanical power ( ¯ Pm) generated by each PTO is the average of the product of the generator force (Fgen) and the leg extension/retraction velocity (vpto).

2.4 RelatedWork

RL has been applied to continuous control tasks for different applications [Duan et al., 2016] [Sarkar et al., 2022b], [Sarkar et al., 2022a], [Sarkar et al., 2021], and model-free RL outperforms many model-based non-RL controls. For wave energy, the bulk of the research on machine learning has focused on wave energy converters (WECs) in academic settings with simplified mechanical designs. A pioneer in this area, [Anderlini et al., 2017a] used RL in simple one-legged WEC in an academic setting to control the PTO damping and stiffness coefficients for discrete sea states. [Anderlini et al., 2017b] also applied least-squares policy iteration for resistive control of a nonlinear model of a WEC. [Anderlini et al., 2018]

also used RL to obtain optimal reactive force for a two-body heaving point absorber with one degree of freedom. [Yin and Jiang, 2023][Wu et al., 2022] implemented a real-time nonML control for a simple point-absorbing WEC by designing a wave condition preview-based nonlinear controller, in which the identification of the wave condition was done using an LSTM identifier. [Li et al., 2022] also uses LSTM to predict the short-term wave force and a 4 th order state-space model to represent the hydrodynamic behavior of the WEC under irregular waves in time-domain with verification using an Autoregressive ANN network. A non-RL technique that utilizes artificial neural networks has also been proposed ([Anderlini et al., 2016]) to generate power through WECs. [Anderlini et al., 2020] makes use of Deep RL for real-time control of a WEC in continuous action space. However, most of these techniques have been applied to one degree of freedom point absorbers.[Gambarelli, 2022] gives an overview of one-body and two-bodies point absorbers with different controls, and concludes that RL techniques are both effective and viable for controlling the parameters of PTO from an academic perspective. [Zou et al., 2022] is another example of applying simpler RL DQN algorithm to a point absorber WEC on a small scale. [Trigueiro et al., 2022] applies Delayed Deterministic Policy Gradient(TD3) RL to Mutriku plant WEC which is of Oscillating Water Column (OWC). In this paper, we apply multiagent RL to the industrial WEC CETO 6 platform, which has a complex mechanical design of three legs with three generators to capture energy from braking against the motion for all 6 degrees of freedom. Never before have RL controllers been used for spread waves under these situations. With waves simultaneously coming from different directions, the asymmetry makes it even more challenging. With the PPO RL controller with fully connected neural network (FCN) RL function approximation, we had limited success. However, in this work, we investigate a variety of ML model architectures for the actor and the critic in the PPO, which can better learn the sequential characteristics of the WEC. Our studies show that these sequential models outperform the FCN by a significant margin. Also, we investigate the different architectural modifications that the transformer model needs to effectively train and converge in a MARL design which has been a major challenge. The skip STrXL trains a lot faster and has a performance that exceeded the state-of-the-art GTrXL. CETO 6 WEC platform is one of the top WEC solutions for the Europe Wave project and can operate at depths where the wave energy is more persistent and takes wave energy closer to financial viability.

3 Reinforcement Learning forWave Energy Converters

3.1 RL AlgorithmExploration

After exploring different RL algorithms like Deep QLearning (DQN) [Mnih et al., 2013], Soft Actor-Critic (SAC) [Haarnoja et al., 2018], and Asynchronous Advantage ActorCritic (A3C) [Mnih et al., 2016], we limited our focus to Proximal Policy Optimization (PPO) ([Schulman et al., 2017]) for this study. We found through evaluation that PPO outperformed other models for this WEC controller. DQN

has problems with continuous action space for WEC, and both SAC and A3C had training stability issues and performed poorly. We fine-tuned the PPO implementation to achieve stable convergence of MARL. But these refinements were not enough for the best performance without refining the underlying function approximation of the actor and critic.

3.2 Multi-Agent RL Choice

In WECs, the generators mounted on the individual legs tend to generate different amounts of energy based on the orientation of the mechanical structure and variation of wave directionality. Simpler, single-agent RL with multiple actions failed to control the WEC effectively and resulted in poor performance. To accommodate the heterogeneity, we needed to use separate agents (MARL) to control the individual generators on each of the legs. This enabled convergence to a better policy. The architecture is shown in Figure 4.

3.3 RL State and Action Design

We validated the inclusion of states with ablation studies of total rewards. Figure 5 shows the attributes included in the state. The continuous action space (ai) for the individual RL agent indexed "i" is defined by the reactive force (Fgeni ) for the controlled generator.

3.4 Reward: Cooperation vs. Competition for Agents

Even though each RL agent controls the generator on an individual leg, it needs to consider power contributions from all the generators in its reward for overall WEC optimization. An individual RL agent can be cooperative or competitive based on the effectiveness of the trade-off. The reward function for power is represented in Equation 3.

Here, Pown is the generator's power being controlled, and Pother is power from other generators, and η = team coefficient. A negative "η" implies adversarial contributions of power from the other legs in the reward for the competitive, while a positive value implies cooperation. With a combined Bayesian hyperparameter search for the multi-agent, we achieved the best performance with η of 0.8 for the agents for the back legs and -0.6 for the agent for the front leg.

3.5 Design for Trust

The spinning yaw motion of the voluminous buoy of WEC causes the tether connections to wear out and has potential maintenance implications. The yaw is high for angled wavefronts w.r.t. the axis of symmetry of WEC. The incentive to reduce yaw is added to the total reward of the RL agents as a weighted addition of the power:

where α is a tunable yaw penalty hyper-parameter (lower the stronger), and Preward is defined in Equation 4. This led to significant improvements in yaw reduction compared to the currently deployed spring damper controller.

3.6 Exploration

Exploration is key to ensuring that the policy does not converge to a local optimum. PPO explores by sampling from the action distribution. However, this is dependent on the standard deviation predicted by the policy. We tried two exploration techniques - parameter space noise [Plappert et al., 2017] and state entropy maximization with random encoders [Seo et al., 2021]. According to our studies, parameter space resulted in better performance than just sampling from the action distribution.

4 Function Approximation for RL

The periodic nature of ocean waves and the inertia and springtype response of the WEC require a system model which can represent and process sequential information. So the RL policy and critic function approximators need to have an architecture to support this, unlike the default feed-forward network. LSTMs can represent long-term information using a recurrent architecture. The Transformers have tremendous success in the ability to process a sequence by explicitly ensuring interaction between the elements of the sequence with the attention mechanism. However, unlike LSTMs, they are limited by the limited sequence length. The Tr-XL architecture solves this problem by keeping a memory of hidden states corresponding to previous sequences. But unlike supervised learning tasks like Natural Language Processing and Computer Vision, the difficulty in training remains a key challenge of using transformers in RL.

4.1 Function Approximation for Policy and Critic Network

Refinements beyond the RL agent algorithm and hyperparameters depend on function approximators (FA), such as the most suitable deep neural networks, to leverage the exploration-exploitation trade-off and hence efficiency at the core of RL. Function approximation blends statistical estimation issues with dynamic optimization issues, resulting in the need to balance the bias-variance tradeoffs that arise in statistical estimation with the exploration-exploitation tradeoffs that are inherent in RL. These changes are inspired by the objective to combine long-term behavior from past observations and future observations of wave states from sensors placed further into the ocean, into the representation of the current state with the predictive power of the short-term memory transformer architectures.

4.2 Function Approximators Explored forWEC

We investigated the WEC controller performance and speed of convergence of fully connected neural networks (FCN), LSTMs, and Transformers of varying depths ([Hochreiter and Schmidhuber, 1997; Vaswani et al., 2017; Parisotto et al., 2020; Lim et al., 2021]). Transformers with the attributes like multi-head attention, temporal convolution network, and large contextual horizon with relative position encoding, are ideally suited for PPO function approximation for WEC.

4.3 Exploring Transformer Architectures for Function Approximation

We also explored the effect of variation of gated bypass for Transformer FA on the stability and speed of training in the RL setting, as the standard transformer models are too unstable to train and learn outside supervised learning. With identity map reordering in TrXL-I the layer normalization is placed on only the input stream of the submodules ([Parisotto et al., 2020]). This enables an identity map from the transformer's input at the first layer to the transformer's output after the last layer, unlike canonical transformers, where there are a series of layer normalization operations that nonlinearly transform the state encoding. The state encoding is passed un-transformed to the policy and value heads, enabling the agent to learn a Markovian policy at the start of training. In our case, the reactive behaviors need to be learned before memory-based ones can be effectively utilized. We also found that GRU style multiplicative interactions with powerful gating mechanisms in place of the residual connections within the transformer block helped stabilize learning and improved performance.

4.4 Skip Transformer-XL (STrXL) Architecture

Unlike supervised learning tasks ([Vaswani et al., 2017]), using traditional transformers as function approximators for a RL task is extremely difficult to optimize, as established by [Parisotto et al., 2020]. Inspired by residual network architecture ([He et al., 2016]) and expanding earlier work of [Parisotto et al., 2020], we propose a variant of the transformer "Skip Transformer-XL" or STrXL, that enables faster convergence during training specifically in an RL setting represented in Figure 6. An additional bypass connection with a gating layer around the transformer block helps accelerate training convergence when compared to the previous methods like GTrXL, as presented in Figure 10 showing the training progression of these models. The GRU gating adds to the non-linearity. The equations below represent the Gated Recurrent Unit type gating.

The STrXL block can be represented with the following equations:

The Gated Recurrent Unit (GRU) type gating:

The input to the transformer block is E(l-1) The output of the transformer block is E(l) , where l is the layer index. The multi-head attention (MHA) block gated output:

The MLP block gated output:

The STrXL gated output:

5 Experiments

The CETO 6 wave energy converter (WEC) platform simulator was used to accurately model the mechanical structure, the mechanical response, the electro-mechanical conversion efficiency with losses for generator and motor modes, and the fluid dynamical elements of the wave excitation.

Wave data such as the distribution of principal time periods, height, and spectrum were collected from Albany, and Garden Island in Western Australia, Armintza in Spain Biscay Marine Energy Platform, and Wave Hub on the north coast of Cornwall in the United Kingdom. The wave generator model used in simulation uses a well-established ocean wave spectrum like Jonswap, which accurately models the heterogeneous components in ocean waves, letting the simulator sample the waves for training and evaluation. For evaluation, we used 1000 episodes for each principal wave period and height, where each episode covers 2000 sec of continuous wave data in steps of 0.2 sec for RL loop and 0.05 sec (4x) for simulation response. Each training run has roughly

50 million steps for convergence, with 2000 training runs required for hyper-parameter optimization and model search with early stops. For regular operation, we show results of a median wave height of 2m for the entire wave frequency spectrum spanning time periods of 6s to 16s.

6 Results

The power generated by the baseline spring damper controller with resonant spring constant and damping constant is used as a reference for evaluation to estimate RL controllers' gain of energy capture as a percentage improvement. We use both the complex spread waves and unidirectional waves of different time periods and heights as shown in Figure 3. We used the same seed for sampling waves for episodes between RL and SD for evaluation.

6.1 Energy Capture Gains and Variations with PPO Function Approximation

Table 1 shows that for spread waves, the MARL with STrXL performs on an average 22.1% better than the baseline spring damper (SD) controller. In contrast, the LSTM performs 14.3% better and FCN performs 11.6% better on average for the entire range of principal wave periods 6s to 16s. Even though the STrXL is 1.5% better than GTrXL overall, it trains much faster with high stability. Table 3 shows the variance of the energy capture for the spread waves with the RL controller.

Table 2 show that for 0° uni-directional frontal waves, the MARL with STrXL performs on an average 25.2% better than the baseline spring damper (SD) controller, while the LSTM performs 22.2% better and FCN performs 18.8% better on an average for the entire range of wave time periods 6s to 16s. For angled waves of 30°, the MARL with STrXL (28.8%) performs much better than LSTM (21.6%) on average.

6.2 Mechanical Stress and YawMinimization under Normal and Survival Conditions

Figure 3 shows the results of yaw reduction with the reward shaping for aggressive Yaw minimization for α=0.2, where 80% weightage is given to penalty for rms yaw, and 20% weightage is given to energy capture maximization. For the entire range of wave time periods of 11s to 16s, where yaw is a significant problem of default SD controller, the yaw is reduced by more than 99%, significantly reducing mechanical stress with huge maintenance savings. The power generation increased with aggressive yaw control as an indirect effect. Even under dangerous conditions with an extreme wave height of 7m. Table 4 shows that the PPO with STrXL function approximation reduces the yaw by over 98.6% for the critical range of wave time periods from 11s to 16s, where the baseline SD controller faces high yaw.

6.3 VariationWith a Depth of Models for Function Approximators

Figure 9 shows that in the best-performing STrXL transformer model, the performance peaks at a depth of 3. For LSTM, both the depths of 2 and 3 yielded similar results. However, for FCN a depth of 2 has an overall better performance, even though the FCN performs worse than STrXL, GTrXL, and LSTM.

6.4 Training Convergence for Different FAModels

Figure 10 compares the average RL environment steps required for convergence for different Transformer function approximators across several time periods. STrXL converged the fastest compared to GTrXL and TrXL-I.

7 Conclusions

The proposed MARL controller with STrXL transformer function approximation yields22.1% gain over the baseline Spring Damper controller (SD) on an average for the entire spectrum of spread waves, boosting energy production with revenue implications. The MARL also helped reduce mechanical stress, which impacts maintenance and operating costs, and actively mitigated the adverse effects of high waves. This is the first published paper where an RL controller can successfully control a 3-legged WEC for spread waves while beating the baseline spring damper controller.

We found that robust RL function approximation sequence models of suitable architectures and depths are key to achieving higher performance for complex real-life use cases like WEC, and RL refinements alone cannot do that. Also, the biggest challenge with the highest-performing transformers is training convergence, and GRU-gated bypass inside and around the transformer block help solve this problem. The proposed novel STxRL architecture trains faster and performs better than the state-of-the-art GTrXL. The STrXL model and PPO function approximation exploration and analysis for MARL will help others to stabilize training convergence for complex RL control systems.