Guide to Control: Offline Hierarchical Reinforcement Learning Using Subgoal Generation for Long-Horizon and Sparse-Reward Tasks

1 Introduction

Deep reinforcement learning (RL) has achieved remarkable success in a range of domains, such as robotics [Kalashnikov et al., 2018], games [Silver et al., 2016], and autonomous driving [Balaji et al., 2019]. However, these works require a large amount of online interaction with the environment, and in real-world applications, online interaction may be limited due to high risk and cost. To address this problem, offline RL methods have emerged, which use previously logged data without online interaction. Unlike online RL, the value overestimation problem on out-of-distribution actions can be fatal because correcting the overestimation is impossible in an offline setting[Fujimoto et al., 2019]. Recent studies have produced promising results via various regularization techniques which conservatively mitigate the distributional shift problems [Wu et al., 2019; Kumar et al., 2020; Kostrikov et al., 2021].

∗Corresponding author

In the real world, however, we face difficulties of complex and long-horizon tasks with sparse rewards which are still challenging to solve with conventional offline RL algorithms. Previous studies have shown the effectiveness of breaking down these long and complex problems into simpler subtasks with a hierarchical structure for online RL [Nachum et al., 2018; Zhang et al., 2020; Bagaria et al., 2021]. Among the various design of hierarchy, goal-conditioned hierarchical RL allows a high-level policy to sequentially generate subgoals at regular intervals and a low-level policy to learn to reach the generated subgoals. A key element of a subgoal-based hierarchical RL is stable and compatible training between the high-level and low-level policies [Wang et al., 2022]. Specifically, a high-level policy should provide reasonable subgoals that a low-level policy can easily reach. In a fully offline setting, it is difficult to determine whether a low-level policy actually reaches a subgoal generated by a high-level policy during the learning process due to the absence of direct validation in the environment. In addition, offline RL learns from fixed datasets which may include a mix of task-agnostic or sub-optimal trajectories [Fu et al., 2020], while online RL can update a policy or a value function with sufficient exploration for a specific task.

In this work, we propose a novel offline hierarchical RL algorithm, Guider, to address the above challenges. Our primary contribution is pre-training a latent variable model to extract the latent distribution of reachable subgoals from an offline dataset. With the pre-trained prior distribution model, we can effectively apply additional regularization while training the subgoal generation policy. Within the constraints of the reachable subgoal distribution in latent space, the highlevel policy learns to generate subgoals that ensure a gradual approach to the final goal while maximizing the cumulative rewards. The low-level policy learns to reach a subgoal generated by the higher level through relabeled rewards of hindsight goal relabeling. We benchmark our method on a variety of simulated continuous control tasks including robot locomotion, navigation, and manipulation. Extensive experiments show that our subgoal generation method can successfully break down a complex and long-horizon task into eas-

(a) Offline dataset

(b) Illustrative example of subgoal generation

Figure 1: (a) Task-agnostic (random starts and goals) trajectories in the Antmaze-large offline dataset. (b) An example of the subgoal generation scenario. Guider learns hierarchical policies from the offline data. For a new longer trajectory test (unseen in the offline dataset), a high-level policy can sequentially generate a subgoal (the star mark) toward the new final goal (the flag mark) and a low-level policy can focus on reaching each generated subgoal.

ier short-horizon goal-reaching problems. In most evaluation tasks, Guider outperforms prior offline RL baselines.

2 RelatedWorks

2.1 Offline Reinforcement Learning

Offline RL methods mainly tackle the distributional shift issue between a learned policy and a behavior policy that is utilized to collect an offline dataset. Prior works constrain the learned policy to stay close to the offline data distribution via explicit regularization [Wu et al., 2019; Fujimoto et al., 2019], conservative value learning [Kumar et al., 2020], and importance sampling [Nachum et al., 2019]. Several studies employ advantage-weighted supervised learning approach, which implicitly imposes constraints to the offline dataset. Compared with other types of regularization methods, these implicit constraints avoid excessive conservative updates and allow effective online fine-tuning [Nair et al., 2020; Kostrikov et al., 2021]. Recently, [Yang et al., 2022] has shown promising results on multi-goal manipulation tasks by adapting the advantage-weighted method coupled with detailed weighting scheme and hindsight relabeling. Although previous methods effectively address the distributional shift problem, a substantial challenge of accomplishing long-horizon and complex tasks, particularly with sparse reward, still remains. We propose a hierarchical approach in offline RL using subgoal generation which efficiently improves performance on such challenging problems. From the perspective of a high-level policy, we impose additional regularization toward prior distribution that has been pre-trained via unsupervised learning, in order to avoid generating infeasible subgoals. At the same time, a low-level policy learns to reach relatively short-horizon subgoals with an implicit regularization.

2.2 Hierarchical Reinforcement Learning

Hierarchical reinforcement learning (HRL) aims to solve complex long-horizon tasks by breaking them into more tractable subtasks with multi-level policies. In particular, goal-conditioned HRL in which a high-level policy presents subgoals has shown great potential in a variety of sparse reward problems. The effectiveness of goal-conditioned HRL

relies on the reasonable subgoal generation, i.e., the highlevel policy should provide subgoals that the low-level policy can reach. Recent studies have improved the learning efficiency of goal-conditioned HRL by employing hindsight correction [Nachum et al., 2018; Levy et al., 2019], adversarial learning [Wang et al., 2022], and representation learning of subgoal with contrastive objective [Li et al., 2021] under interaction with the environment. However, these works need extensive online interaction with the environment, which could be expensive and risky in real-world applications. In contrast, our method focuses on training with only offline data including trajectories that may be irrelevant to the evaluation tasks or collected by sub-optimal policies, without additional access to the environment.

Some recent studies proposed offline HRL methods using temporally extended skills [Lynch et al., 2020; Ajay et al., 2021]. These hierarchical skill learning methods train low-level action policies as reconstruction-based decoders of state-action sequences. These works are similar to our method of utilizing unsupervised pre-training and high-level training in latent space. However, we separately train lowlevel policy via value-based reinforcement learning, which provides a considerable advantage when using sub-optimal mixed data.

3 Preliminaries

We consider a continuous control problem formulated as a goal-conditioned Markov decision process (MDP), denoted by a tuple (S, G,A, p, r, γ), where S is a state space, G is a goal space, A is an action space, p(s′ |s, a, g) is a transition probability, r(s, a, g) is a reward function, and γ ∈ (0, 1] is a discount factor. The objective is to obtain an optimal policy π that maximizes the expected discounted return Eπ[∑∞t=0 γ t rt]. The goal-conditioned task generally provides sparse reward and the reward function is defined as:

where sG ∈ G is a state that is mapped to goal space, and ϵ is a given threshold from the environment. The goal space is defined according to the environment and can be a subset of the state space. For instance, the goal space of a robot navigation task is given as a location of the robot torso, while the state space is composed of tens of proprioceptive features (e.g. coordinates, angles, velocities). For offline RL settings, we assume that the agent uses only previously collected data without any interaction with the environment during the training process.

We consider the goal-conditioned HRL framework consisting of a high-level policy πh(z|s, gfinal) and a low-level policy πl(a|s, gsub, gfinal), where z is a subgoal in a latent space, gsub is a subgoal decoded into a goal space, and gfinal is the final goal of the downstream task. The high-level policy generates a subgoal every c-steps, and the low-level policy executes an action in each of the steps to reach the generated subgoal. In the following section, we show how to train the latent variable model which captures the reachable subgoals within c-steps from the current state, and utilizes the pre-trained model to regularize the subgoal generation policy.

4 Method

In this section, we describe our method Guider, which trains subgoal generation policy within the constraints of reachable subgoal distribution in latent space. Our objective is to ensure the low-level policy easily reaches a subgoal generated by the high-level policy, while both policies are trained to avoid distributional shifts from the offline dataset without any additional data collection or online interaction. We first introduce how the overall framework is structured, and elaborate on each training part.

4.1 Model Overview

Our model consists of three parts of training: (1) pretraining the latent variable model which embeds reachable subgoals in latent space via unsupervised learning (2) training the high-level policy to generate a subgoal that progressively approaches a given final goal (3) training the lowlevel policy that reaches the subgoal generated at the highlevel. We train the latent variable model comprised of encoder qψ(z|sGt , sGt+c ), decoder pθ(sGt+c |sGt , z), and trainable prior model ρω(z|sGt ). We relabel the offline dataset with the pre-trained encoder qψ(z|sGt , sGt+c ) for the high-level policy training, and the high-level policy πh(z|sGt , gfinal) is regularized with the pre-trained prior distribution ρω(z|sGt ). The low-level policy πl(a|st, gsub, gfinal) is trained by relabeled rewards for reaching the subgoals. At test time, the high-level policy generates a subgoal every c-steps, and the low-level policy selects an action for each step toward the generated subgoal. The architecture of our model is illustrated in Figure 2.

4.2 Latent Variable Model for Reachable Subgoals

We train the latent variable model in order to learn the subgoal generation policy in the latent space of subgoals that can be reached within c-steps from the current state, and not in the entire goal space. We sample pairs of random states and cstep next states from the offline dataset and train the encoder qψ(z|sGt , sGt+c ) and the decoder pθ(sGt+c |sGt , z) to reconstruct the next state in a goal space after c-steps, while conditioned on the current states. Additionally, we train subgoal prior model ρω(z|sGt ) to obtain the latent distribution when a current state is given. We jointly train the components of the latent variable model by maximizing the following objective function based on evidence lower bound (ELBO):

The subgoal prior model is trained to cover the latent distributions of reachable subgoals from the current state, and the posterior distribution from the encoder is regularized toward the prior by the KL loss term. β is a weighting parameter for the regularization. We implement a balanced training of both parameterized prior and posterior networks with alternately stopped gradient flow at different learning rates following [Hafner et al., 2020]:

where sg(·) indicates a stopped gradient flow and τ is the weighting parameter.

4.3 High-level Subgoal Generation Policy

We apply the pre-trained latent variable model to train the high-level subgoal generation policy πh(z|sG , gfinal). In the beginning, we relabel the offline dataset using the trained encoder. An action at is replaced by an embedded latent subgoal z ∼ qψ(·|sGt , sGt+c ). The next state of the high-level transition is relabeled as the state mapped to the goal space after c-step sGt+c . In addition, we relabel the original goal with a random future state after the c-step, employing hindsight relabeling which improves the learning efficiency of a goal-conditioned policy [Andrychowicz et al., 2017]. The relabeled reward ˜ r(sGt , z, sGt+c , ˜ gfinal) for relabeled goal ˜ gfinal is computed by equation 1. Consequently, an original transition tuple (st, at, rt, st+1, gfinal) in the offline dataset is relabeled to (sGt , zt, ˜ rt, sGt+c , ˜ gfinal).

We optimize the subgoal generation policy πh(z|sG , gfinal) to maximize value function Q(sG , z, gfinal) while regularized toward the pre-trained subgoal prior ρω(z|sG ) following the proposed objective:

Minimization of the reverse KL divergence constrains the learned policy to generate subgoals within a learned subgoal space that is reachable in c-steps from the current state. Our regularized policy optimization objective can be applied to an off-the-shelf offline RL algorithm. Here, we opt for Conservative Q-Learning (CQL), an extensively used offline actorcritic algorithm [Kumar et al., 2020]. A maximum entropy regularization term min log πh(z|s, gfinal) of the original CQL policy objective which encourages diverse behaviors is replaced with our KL regularization. This regularization plays an important role in learning hierarchical policies in offline conditions where it is impossible to know whether the lowlevel policy can reach the generated subgoals. We will show how the additional regularization improves the quality of the generated subgoals and resulting performance in section 5.

4.4 Low-level Subgoal Reaching Policy

Our subgoal generation method can be used in conjunction with any low-level goal-conditioned policy learning algorithm such as goal-conditioned supervised learning (GCSL) [Ghosh et al., 2021] or goal-conditioned version of existing offline RL methods. Since our hierarchical framework decomposes complex tasks into relatively easy shorthorizon goal-reaching problems, the lower-level policy can be learned effectively with a simple policy learning algorithm. We introduce a simple advantage-weighted supervised learning method adapted for our goal-conditioned hierarchical framework.

The lower-level policy πl(a|s, gsub, gfinal) should learn to choose an optimal action to reach a guided subgoal gsub while avoiding the distributional shift problem of offline RL. Therefore, we update the policy by maximizing advantage A(s, a, gsub, gfinal) with KL divergence regularization toward

Algorithm 1 Guider

Require: Dataset D, subgoal generation period c. Initialize: Encoder qψ, decoder pθ, prior ρω, high-level policy πh, high-level value function Qh, low-level policy πl, low-level value function Ql.

• 1: for M iterations do
• 2: Sample mini-batch: {(sGt , sGt+c )} ∼ D
• 3: Update ψ, θ, ω using objective in eq. 2
• 4: end for
• 5: for N iterations do
• 6: # High-level policy training
• 7: Sample mini-batch: {(st, st+c, gfinal)} ∼ D
• 8: z ∼ qψ(sGt , sGt+c )
• 9: ˜ gfinal ∼ sGi (t + c < i ≤ T)
• 10: Relabel the mini-batch: {(sGt , z, ˜ rt, sGt+c , ˜ gfinal)}
• 11: Update Qh to minimize TD error
• 12: Update πh using objective in eq. 4
• 13: # Low-level policy training
• 14: Sample mini-batch: {(st, at, rt, st+1, gfinal)} ∼ D
• 15: ˜ gsub ∼ sGj (t < j ≤ t + c)
• 16: Relabel the mini-batch: {(st, a, ˜ rt, st+1, ˜ gsub, ˜ gfinal)}
• 17: Update Ql to minimize TD error
• 18: Update πl using objective in eq. 6
• 19: end for
• 20: return Trained policies πh, πl.

the behavior policy πβ(s, a, gsub, gfinal), using the following optimization formulation:

where ϵ is a threshold value. The above constraint optimization problem is solved by enforcing the Karush-KuhnTucker conditions following prior works [Peng et al., 2019; Nair et al., 2020]. The implicitly constrained policy optimization objective is derived as:

We train the low-level policy with goal-relabeled data similar to the high-level policy training. However, the relabeling strategy is slightly different considering our subgoal generation framework. We relabel the subgoal gsub to random future states within c-steps as ˜ gsub = sGi for t < i ≤ t+c+1, since our high-level policy is trained to generate subgoals reachable within c-steps. This modified relabeling strategy effectively accelerates training the low-level policy. The whole training process of our proposed method is summarized in Algorithm 1.

5 Experiments

Through our experiments, we aim to answer the following questions: (1) Can Guider be effectively applied to a variety of long-horizon and sparse-reward tasks? (2) Can Guider learn stitching knowledge from a fixed dataset including taskagnostic trajectories or mixed sub-optimal behaviors? (3) Does our proposed latent subgoal prior model help generate a reasonable subgoal and improve the performance of offline hierarchical RL? To answer the first and second questions, we evaluate the Guider against a variety of benchmarks dealing with stitching and narrow distribution challenges of offline RL. We present ablation studies to answer the third question.

5.1 Environments and Datasets

We evaluate our method on diverse continuous control tasks in three simulated robotic domains: maze navigation, arm manipulation, and kitchen manipulation. All these environments provide sparse reward feedback.

Antmaze

The antmaze domain requires an agent to control a quadruped robot to navigate to a designated goal position. We use diverse dataset from the D4RL benchmark [Fu et al., 2020]. The dataset is composed of trajectories from random start locations to random goal positions, which are irrelevant to the target task. Therefore, the agent is required to stitch the parts of suboptimal trajectories to find the optimal path to the evaluation goal. The domain contains three maze layouts (umaze, medium, large). In particular, the long-horizon large task is substantially challenging to solve with a conventional offline RL approach.

Fetch

The fetch domain contains four robotic arm manipulation tasks: FetchPickAndPlace, FetchPush, FetchSlide, FetchReach. The tasks are to place defined objects at randomly given target goal positions from random initial states. We use expert and mixed dataset for each task. The mixed dataset contains a mixture of trajectories collected by 90% random policy and 10% expert policy provided from [Yang et al., 2022].

Kitchen

The tasks of kitchen domain are to control a robotic arm interacting with several household items to reach a desired goal configuration. D4RL benchmark [Fu et al., 2020] introduces three types of datasets collected by human demonstrations. The complete dataset consists of a relatively small number of trajectories performing all subtasks in order. The partial dataset contains a mix of successful trajectories and partially performed trajectories. The partially performed trajectories contain a subset of the desired configuration as well as subtasks irrelevant to the target task. The mixed dataset consists of only partially performed trajectories.

5.2 Baselines

We compare Guider against prior approaches in goalconditioned supervised learning, offline goal-conditioned RL, and offline hierarchical RL.

• GCSL [Ghosh et al., 2021] is a goal-conditioned supervised learning method using hindsight goal relabeling.
• WGCSL [Yang et al., 2022] is a state-of-the-art offline goal-conditioned RL algorithm based on advantageweighted supervised learning, using hindsight goal relabeling.
• CQL+HER [Kumar et al., 2020] is a modified version of offline RL algorithm CQL which learns the outof-distribution value function conservatively, with hindsight goal relabeling added.
• OPAL [Ajay et al., 2021] is a state-of-the-art offline hierarchical RL method based on skill discovery. The high-level policy directs which skill to use, and the lowlevel policy executes a sequence of actions for the skill.

Implementation details and hyperparameters for our methods and baselines are provided in supplementary material A.

1 The results for OPAL on antmaze and kitchen tasks are taken from [Ajay et al., 2021]. Since the official source code of OPAL is not available, the results for fetch tasks are from our reimplementation based on the paper.

5.3 Overall Results

The overall results of our experiments are reported in Table 1, and we present the learning curves in Figure 6 in the supplementary material. Our method Guider achieves the best performance on 12 out of 14 tasks. Specifically, the results show that Guider outperforms prior methods by a large margin on challenging offline RL problems such as long-horizon tasks or using mixed datasets.

The antmaze and kitchen are considerably challenging tasks as they require learning policies for reaching longhorizon goals from sparse rewards. Besides, all antmaze datasets and mixed dataset of kitchen are composed of taskagnostic datasets, which means that the tasks can not be accomplished by imitating certain trajectories in datasets. The agent must learn to stitch meaningful sub-trajectories from datasets to solve new longer trajectories at testing. Our proposed method effectively addresses these challenges by generating adequate subgoals which are easily reachable. Figure 3 shows the generated subgoals from our high-level policy on evaluations of each task. It is remarkable that Guider shows a success rate of over 80% in antmaze-large, where prior methods without the hierarchical approach succeed at less than 20%. We observe that our subgoal-based hierarchical method also outperforms OPAL, the skill-based hierarchical approach. We will discuss the difference between the two approaches through an ablation study in the next subsection.

Another important challenge in offline RL is learning an optimal policy from mixed data including optimal and suboptimal trajectories. The agent should learn near-optimal policies from the datasets consisting of only a small portion of expert trajectories and the rest of the sub-optimal or random trajectories. In our experiments, the mixed datasets of fetch and the partial dataset of kitchen tasks deal with the above challenge. Although the fetch-mixed dataset contains 10% of expert trajectories and kitchen-partial contains only 3.2% of optimal trajectories, Guider significantly surpasses the average return of the dataset.

5.4 Ablation Studies

Different Low-level Policies

We conduct an ablation study to investigate how our subgoal generation method improves performance on a long-horizon task. We train the unsupervised latent variable model and the high-level policy of our proposed Guider framework on the antmaze-large task, in conjunction with diverse low-level policy learning methods. Figure 4 shows a significant improvement in performance for all types of low-level policy learning when combined with Guider's subgoal generation method. This indicates that Guider takes long-horizon tasks that are difficult to solve with existing methods and turns them into simple tractable problems with guided subgoals.

Interestingly, we find that Guider combined with behavior cloning (BC) leads to performance similar to OPAL. OPAL learns skill-based hierarchical policies, where the low-level policy extracts a temporally-extended sequence of primitive actions by behavior cloning loss [Ajay et al., 2021]. On the other hand, Guider originally learns low-level goal-reaching policy based on trained value function. This reinforcement learning process at the lower level facilitates additional improvement than imitating actions in the dataset. We conjecture that the different approach in low-level policy learning accounts for the superior performance of Guider to OPAL, although both methods use hierarchical architectures.

Regularization Methods

We also examine the importance of our proposed regularization strategy in subgoal generation and performance. We compare three different regularization methods while highlevel policy training on the antmaze tasks.

• Guider is our proposed method implemented on top of CQL with additional regularization toward the prior ρω(z|sGt ).
• Guider w/o prior learns the high-level policy without any additional regularization. It is also implemented on top of CQL.
• Guider w/o CQL is implemented on top of Soft ActorCritic (SAC) [Haarnoja et al., 2017]. SAC is an online actor-critic algorithm without any regularization towards the behavior policy of the offline dataset. We added regularization with the prior ρω(z|sGt ).

As shown in Table 2, the performance of high-level policy training considerably decreases without prior regularization. We visualize the generated subgoals and the arrived position on evaluation in Figure 7 in the supplementary material. We observe that the agent reaches the generated subgoal in practice only when prior regularization is imposed. Otherwise, the subgoal is generated too far or at an invalid location in relation to the final goal. These infeasible subgoal generations result in decreased success rate.

Subgoal Generation Period

We conduct experiments with varying subgoal generation period c to investigate the influence of this hyperparameter. As shown in Table 3, the subgoal generation period does not critically affect the performance of Guider. However, we understand that too short a subgoal generation period insufficiently benefits from a temporal abstraction of the hierarchical architecture, and too long a period makes it difficult to reach the subgoal with the low-level policy. We chose the appropriate range of the hyperparameter considering the episode length of tasks. In the case of antmaze, it shows the best performance around c = 50, which is 1/20 of the episode length of 1000. Additional results on the other tasks are provided in the supplementary material. More sophisticated ways to determine the subgoal generation period can be studied for future work.

Latent Subgoal vs. Reconstructed Subgoal

We investigate the effectiveness of using a decoder to reconstruct a latent subgoal generated by a high-level policy. Our proposed model learns a low-level policy conditioned on the subgoal in goal space, not in latent space. During the evaluation, a generated latent subgoal from the high-level policy is decoded into a goal space before being provided to the low-level policy. However, we do not claim that using a decoder is mandatory for our method. As shown in Figure 5, the difference between Guider with and without a decoder is not significant, although slightly better performance is achieved with the decoder. Our experiments are conducted on environments possessing relatively low-dimensional spatial goal space, which provides clear information. Considering the property of these environments, learning goal-reaching policy in goal space is favorable. On the other hand, one could expect that learning low-level policy conditioned on latent subgoal improves performance on high-dimensional observations such as RGB

images, as suggested in prior works [Rafailov et al., 2020; Hafner et al., 2022]. Through these empirical results, we suggest that the general framework of Guider can be extended to diverse high-dimensional observations.

6 Conclusion

In this work, we propose Guider, an offline hierarchical reinforcement learning method that learns to generate subgoals at the high level and reach the generated subgoals at the low level. Our unsupervised pre-training of the subgoal prior distribution in latent space can effectively regularize the subgoal generation policy. The generated subgoal can be easily reached by simple low-level policies. Empirical studies show that our proposed method outperforms prior offline RL methods on long-horizon and sparse-reward tasks. An interesting direction of future work would be generating meaningful subgoals from high-dimensional space such as offline RGB images. We also plan to design a flexible subgoal generation model where the generation period can vary depending on the situation.

Acknowledgments

This work was supported partly by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea Government (MSIT) (No. 2022-0-01045, Self-directed Multi-Modal Intelligence for solving unknown, open domain problems), (No. 20220-00688, AI Platform to Fully Adapt and Refect PrivacyPolicy Changes), and (No. 2019-0-00421, Artifcial Intelligence Graduate School Program(Sungkyunkwan University)).