DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards

1 Introduction

Exploration is an important aspect of reinforcement learning (RL), as suggested by the famous exploration-exploitation trade-off [Sutton and Barto, 2018] wherein an agent that only exploits with the current policy would be stuck and fail to improve its policy anymore due to the lack of novel experiences. Effective exploration is non-trivial, especially in tasks where environmental rewards are sparse. Relying on unstructured exploration (e.g., ϵ-greedy or randomized probability matching [Sutton and Barto, 2018; Scott, 2010]) requires an exponential number of samples and is unlikely to achieve a satisfactory level of exploration. Manually designing dense rewards with domain knowledge has exhibited promising results in several areas where RL has significantly progressed, such as game-playing and robotics [Mnih et al., 2015; Baker et al., 2019; Hafner et al., 2020]. However, given the huge amount of knowledge and effort required, designing such dense rewards is only feasible in a handful of tasks, and effective exploration thus remains a challenge.

To tackle this issue, several works have proposed guiding the exploration with intrinsic rewards or rewards that are internal to agents, including ICM [Pathak et al., 2017], RND [Burda et al., 2019], NGU [Badia et al., 2020], and NovelD [Zhang et al., 2021]. In these methods, the intrinsic reward is devised to encourage visiting states that are likely to be more novel, where novelty is defined as either the distance between the current and past observations or the difference between model predictions and realities. Although these works have shown encouraging empirical results leading to better exploration efficiency, the relationship between the observed novelty and the agent's actions has not yet been explicitly decoupled. In other words, effectively handling trivial novelties remains unaddressed, as they are rooted in the stochasticity in the environment's dynamics and have little to do with the agent's exploration capabilities (e.g., the "noisy TV" problem [Pathak et al., 2017]).

This paper bridges this gap with DEIR (Discriminativemodel-based Episodic Intrinsic Reward), a novel intrinsic reward design that considers not only the observed novelty but also the effective contribution brought by the agent. In DEIR, we theoretically derive an intrinsic reward by scaling the novelty metric with a conditional mutual information term between the observation distances and the agent's actions. We make the intrinsic reward tractable by using a simple formula serving as the lower bound of our primary objective. DEIR is thus designed to distinguish between the contributions to novelty caused by state transitions and by the agent's policy. For computing the proposed reward, we devise a discriminative forward model that jointly learns the environment's dynamics and the discrimination of genuine and fake trajectories. We evaluated DEIR in both standard and advanced MiniGrid [Chevalier-Boisvert et al., 2018] tasks (grid-world exploration games with no extrinsic reward until reaching the goal) and found that it outperforms existing methods on both (see Figure 1). To examine DEIR's generalization capability in tasks with higher dimensional observations, we also conducted experiments in ProcGen [Cobbe et al., 2019; Cobbe et al., 2020] (video games with procedurally generated levels that require planning, manipulation, or exploration) and found that it delivers a state-of-the-art (SOTA) performance in all selected tasks. Finally, we performed an indepth analysis of our method through additional experiments to clarify the effectiveness of DEIR and its components.

Our contributions can be summarized as follows. (1) Our method, theoretically grounded, effectively decouples the stochasticity in the environment and the novelty gained by the agent's exploration. (2) Our method empirically outperforms others in this line of work by a large margin, especially in advanced MiniGrid tasks, but not limited thereto. It also has potential applications for a variety of other tasks. (3) Our method is easy to implement and use, and provides state representations useful for potential downstream objectives.

2 RelatedWork

Exploration through intrinsic rewards is widely studied in RL literature, with many works falling into one of two categories: Prediction error-driven methods that are motivated by the differences ("surprises") between predictions and realities, and Novelty-driven methods that seek novel agent observations.

Prediction error-driven methods. It is typical for prediction error-driven methods to learn a model of the environment's dynamics and use it to make predictions for future states. Large deviations between the predictions and the realities suggest regimes where the model is insufficiently learned. Intrinsic rewards, positively correlated to prediction errors, encourage the agent to explore more in those states. One of the representative works in this category is ICM [Pathak et al., 2017], which jointly trains both the forward and the inverse transition models to capture the environment's dynamics better, but only uses the forward model's prediction errors to generate intrinsic rewards for the agent. Prediction errordriven methods require approximating the environment's dynamics with neural networks, which is especially difficult in high-dimensional spaces. Still, as demonstrated by ICM, training with auxiliary tasks seems to be worth the effort.

Novelty-driven methods. Recent novelty-driven methods usually define a distance metric between observations, and then formulate this distance as intrinsic rewards to encourage more exploration in RL roll-outs. Early works include count-based methods, which record how many times distinct states are visited and use the count differences as intrinsic rewards [Bellemare et al., 2016; Ostrovski et al., 2017; Tang et al., 2017]. However, due to their simplicity, countbased methods have difficulty in tasks featuring continuous states or high-dimensional observations. To overcome this problem, neural networks have been introduced to encode the states and observations. For example, RND [Burda et al., 2019] defines the distance as the difference between the outputs of a parameter-fixed target neural network and a randomly initialized neural network. In this approach, the former network is used to be distilled into the latter one, effectively "evolving" a distance metric that adjusts dynamically with the agent's experience. In NovelD [Zhang et al., 2021], one of the latest works, RND is applied to evaluate the distances between pairs of observations, in which the boundary between explored and unexplored regions is defined as where the distance is larger than a predefined threshold and large intrinsic rewards are provided when the agent crosses the boundaries. In this way, NovelD encourages the agent to explore in a manner similar to breadth-first search and has already demonstrated a state-of-the-art (SOTA) performance on many MiniGrid tasks. Never-Give-Up (NGU) [Badia et al., 2020] introduces the inverse model from ICM in its episodic intrinsic reward generation module. Their final intrinsic reward is based on the Euclidean distances of the K-nearest embeddings of the recently visited states. In practice, measuring novelty often requires analysis of the distribution of an environment's states. Even so, "noisy TV"-like problems can still occur, where novelty in the observation space is primarily due to the stochasticity in an environment's dynamics. This prevents the agent from achieving meaningful explorations.

Our method incorporates the advantages of the two categories: while we explicitly encourage our agent to seek novel observations, we also rely on a discriminative model to construct a conditional mutual information term that scales novelty in the observation space, incorporating the model-based prediction task from prediction error-driven methods. Unlike conventional novelty-driven methods, our conditional mutual information scaling term effectively eliminates the novelties rooted in the environment's stochasticity other than those brought by the agent's explorations. It has granted us better performances (see Figure 1). To illustrate the difference from existing prediction error-driven methods, our model learns both the environment's dynamics and the capability to tell the genuine and fake trajectories apart. Consistent with the results reported in contrastive learning studies [Laskin et al., 2020; Agarwal et al., 2021], the discriminative nature of our model encodes the observations in a space closely related to the underlying tasks and thus enables us to measure the distances between observations more accurately.

3 ProposedMethod

3.1 Background and Notations

A Markov decision process (MDP) [Sutton and Barto, 2018] can be defined as a tuple (S,A, r, f, P0, γ), where S is the state space, A is the action space, r(st, at, st+1) is the reward function, f(st, at) is the environment's transition function, P0 is the distribution of the initial state s0, and γ ∈ [0, 1] is the reward discount factor. The goal is to optimize a policy π : S × A → R so that the expected accumulated reward Es0∼P0 [∑t γ t r(st, at, st+1)] is maximized. However, in a partially observable MDP (POMDP), st ∈ S is not accessible to the agent. A common practice is to use π(at|τt) for the policy instead, where τt = {o0, o1, · · · , ot} is an approximation of st. Many works implement this using recurrent neural networks (RNN) [Rumelhart et al., 1986] to best utilize available historical information.

We adopt proximal policy optimization (PPO) [Schulman et al., 2017] to learn the agent's policy. With PPO as a basis, we propose an episodic intrinsic reward that helps decouple the novelties introduced by the agent from those by the environment. We also introduce a discriminative forward model to learn better state representations in partially observable environments. Following popular studies, our reward function is designed to be the weighted sum of extrinsic (those from the environment) and intrinsic (those from curiosity) rewards: r(st, at, st+1) = rE t +β · rIt , where rE t and rIt (both functions of (st, at, st+1)) are respectively the extrinsic and intrinsic rewards at time step t, and β is a hyperparameter.

3.2 Episodic Intrinsic Reward

Scaling the Novelty

For a pair of states (st, si), ∀i ∈ [0, t) and the action at, we want a novel state (valuable for exploration) to have a large distance between observations (ot+1, oi) while that distance is closely related to the action at. Intuitively, it is crucial to distinguish novelty rooted in the environment's stochasticity from novelty brought by an exploratory action of the agent. This leads to our primary objective, which is to maximize

as a product of the distance dist(ot+1, oi) (denoted as Dt+1,i) between observations and conditional mutual information

With the Bretagnolle-Huber inequality [Bretagnolle and Huber, 1978], we have

where P(x) = p(x|st, si, at) and Q(x) = p(x|st, si) are defined for simplicity, and dTV(P,Q) = 12 ∥P -Q∥1 is the total variation between P and Q.

Note that, in deterministic environments (including partially observable cases), (a) P is a unit impulse function that has the only non-zero value at Dt+1,i, and (b) we can naturally assume the Dt+1,i|st, si ∼ Exp(λ = 1/dist(st, si)) to match the distance between observations and that of their underlying states. Thus, we devise a simplified surrogate function for the mutual information, as

Substituting it back to the objective, we obtain

To make it tractable, we simplify the right-hand side to

which is a lower bound for the original objective. It is simpler and empirically performs as well as or even better than Equation 2. We speculate that improving the minimum value is crucial to improving the expectation value. A detailed derivation is provided in Appendix A.1.

Intrinsic Reward Design

To maximize our objective through the lower bound in Equation 3, we propose our intrinsic reward in an episodic manner:

where eOBS t is the embedding of observation ot, eTRAJ t is the embedding of trajectory τt at time step t in an episode, dist is the Euclidean distance between two embeddings, and ϵ is a small constant (10-6 in our experiments) for numeric stability. Note that the complete information regarding st is inaccessible in POMDPs, so eTRAJ t is commonly used as a proxy of st. All these constitute a metric space of observations where distance is defined. Both eOBS t and eTRAJ t are computed by a discriminative forward model, as detailed in Section 3.3.

Our intrinsic rewards are episodic, as they are created from the observations seen in a single episode. Arguably, episodic rewards are generally compatible with lifelong rewards and can be used jointly. Still, in this work, we focus on the former for simplicity and leave such a combination for future studies.

3.3 Learning a Discriminative Model

We train a neural network to extract embeddings eOBS t and eTRAJ t from observations in high-dimensional spaces. Existing studies have adopted auxiliary tasks in which the forward and inverse models are two representatives to obtain better embeddings suitable for exploration. We propose an improved auxiliary task suitable for exploration in POMDPs inspired by contrastive learning [Laskin et al., 2020]. Concretely, our proposed model learns the environment's dynamics and discriminates the genuine trajectories from the fake. Figure 2 illustrates the architecture of our model.

Model Definition

Our discriminative model is denoted as Dsc(ot, at, ox) : O × A × O′ → [0, 1], where ot, at are defined as above and ox ∈ {ot+1, ot′} is either the next observation ot+1 (positive example) or an observation ot′ that has been observed recently and is randomly selected at time step t (negative example) for each training sample. In short, the model estimates the likelihood that the input ox is a positive example.

Specifically, to efficiently retrieve fake examples with decent diversity during RL roll-outs, we maintain recent novel observations with a first-in-first-out queue Q so that ot′ can be randomly selected from all observations saved in Q. This queue is maintained by continuously adding newly retrieved "novel" observations to replace the oldest observations. Here, ot is considered "novel" only if rI t is not less than the running average of all intrinsic rewards. In addition, when sampling fake observations from Q, we always sample twice and keep only ones that differ from the true ones. These measures lead to a very high ratio of valid samples with distinct true and fake observations in training.

Proposed Architecture

The proposed intrinsic reward is based on the observation embeddings eOBS t and the trajectory embeddings eTRAJ t generated by the discriminator. From input to output, the discriminator Dsc is formed by a convolutional neural network (CNN) [LeCun et al., 1998], a recurrent neural network (RNN) [Rumelhart et al., 1986], and a multi-layer perceptron (MLP) [Rosenblatt, 1958] output head. We adopt gated recurrent units (GRU) [Cho et al., 2014] in the RNN module to reduce the number of trainable parameters (compared to LSTM [Hochreiter and Schmidhuber, 1997]). As shown in Figure 2, the CNN takes the observations ot and ot+1 as input in parallel and outputs two observation embeddings. These observation embeddings are then fed into the RNN, together with the RNN's previous hidden state ht-1. In addition to the updated hidden state ht, the RNN outputs the embeddings of two trajectories starting from the beginning of an episode and ending at time steps t and t + 1, respectively. Finally, the two trajectory embeddings with the action at are fed into the MLP for predicting the likelihood. RNN hidden states are saved as part of the discriminative model's training samples. We adopt PPO for learning the policy, as it refreshes the experience buffer more frequently than other algorithms. Therefore, we do not apply specialized methods to renew the hidden states within one episode.

Mini-Batches and Loss Function

During training, each mini-batch consists of two types of samples: half of them positive and half of them negative. Both types are picked from the agent's recent experiences. The discriminator is trained with a binary cross-entropy loss function [Murphy, 2022] as in ordinary classification tasks.

4 Experiments

We address the following questions through our experiments:

• Is DEIR effective in standard benchmark tasks and can it maintain decent performance in advanced, more challenging settings?
• Is our design decision in DEIR generally applicable to a variety of tasks, and particularly, can it generalize to tasks with higher dimensional observations?
• How significantly does each technical component in DEIR contribute to the performance?

4.1 Experimental Setup

We evaluate DEIR using the following two popular procedurally generated RL benchmarks. (1) MiniGrid [ChevalierBoisvert et al., 2018], which consists of 20 grid-world exploration games featuring different room layouts, interactive objects, and goals. An agent needs to learn a specific sequence of actions to reach a final goal with its limited view size. Valid actions include picking up a key, unlocking a door, unpacking a box, and moving an object. No extrinsic reward is given until the goal. (2) ProcGen [Cobbe et al., 2020], which consists of 16 games with 64 × 64 × 3 RGB image inputs, each of which requires a certain level of planning, manipulation, or exploration skill to pass. Each episode is a unique game level with randomly initialized map settings, physical properties, enemy units, and visual objects. The agent needs to learn policies that can be generalized to unseen levels.

Environments and networks are by default initialized with 20 seeds in each MiniGrid experiment and three seeds (from the full distribution of game levels) in each ProcGen experiment. All experimental results are reported with the average episodic return of all runs with standard errors. We performed hyperparameter searches for every method involved in our experiments to ensure they have the best performance possible.

We also performed sensitivity analyses on two key hyperparameters of our method, namely, the maximum episode length and the maximum observation queue size, and found that both work well with a wide range of candidate values.

Note that, while our experiments in this paper focus on POMDPs, we find that the performance of DEIR is superior to existing exploration methods in many fully observable tasks as well, which deserves further analysis in future works.

4.2 Evaluation Experiments

Evaluation in Standard MiniGrid

We first evaluated the performance of DEIR in four standard MiniGrid tasks, where the agent's view size is a 7 × 7 square and no other constraints are applied. The mean episodic returns of all exploration methods are shown in Figure 3 (first row). In FourRooms and MultiRoom-N6, which are simple tasks, DEIR and the existing methods all exhibited a decent performance. DoorKey-16x16 and KeyCorridorS6R3 are more complex tasks featuring multiple sub-tasks that must be completed in a particular order, requiring efficient exploration. In these complex tasks, DEIR also learned better policies faster than the existing methods.

We also compared the performances of DEIR and NovelD on the most difficult MiniGrid task ObstructedMaze-Full (see Figure 5). Its difficulty is due to the highest number of subtasks that need to be completed in the correct order among all standard MiniGrid tasks. So far, it has been solved by only few methods, including NovelD, albeit in an excessive amount of time. To reach the same SOTA performance, DEIR required only around 70% of frames as NovelD, for which we also conducted hyperparameter searches and exceeded its original implementation by requiring fewer training steps.

Evaluation in Advanced MiniGrid

We further evaluated the robustness of DEIR on 12 MiniGrid tasks with advanced (more challenging) environmental settings, in which the following modifications were made to the standard environments (see examples in Figure 1):

Reduced view sizes. The agent's view size is reduced from the default 7 × 7 grids to the minimum possible view size of 3 × 3 (81.6% reduction in area). Consequently, the agent needs to utilize its observation history effectively.

Noisy observations. At each time step, noises are sampled from a Gaussian distribution (µ = 0.0, σ = 0.1) in an element-wise manner and added to the observations that are first normalized to [0, 1]. With this change, each observation looks novel even if the agent does not explore.

Invisible obstacles. Obstacles are invisible to the agent but still in effect; that is, the agent simply perceives them the same way as floors, but cannot step on or pass through them. This requires the agent to have a comprehensive understanding of the environment's dynamics, beyond the superficial observable novelty.

The results in Figure 3 (second to fourth rows) clearly show that DEIR was significantly more robust than the existing methods in all advanced settings. The performance difference was especially large when only noisy observations were presented. Compared to the results in standard tasks, DEIR lost at most 25% of its returns by the end of the training, while other methods lost nearly all of their returns in 75% of the noisy-observation tasks. Note that our noisy-observation task is similar to the "noisy-TV" experiments [Burda et al., 2019] but features increased difficulty due to changing more pixels. In ICM [Pathak et al., 2017], up to 40% of the observation image was replaced with white noise. According to published source code, NovelD was originally tested in a scenario where at most one special object in an observation switches its color among six hard-coded colors when triggered by the agent, i.e., most parts of the observation image remain unchanged.

Generalization Evaluation in ProcGen

We evaluated the generalization capability of DEIR under various reward settings and environmental dynamics using the ProcGen benchmark. Four tasks (on hard mode) were selected on the basis of their partial observability and demand for advanced exploration skills. We used the same CNN structure as in IMPALA [Espeholt et al., 2018] and Cobbe et al.'s work [Cobbe et al., 2020] for the agent's policy and value function, and a widened version of the CNN used in DQN [Mnih et al., 2015] for the dynamics model of each exploration method. The results in Figure 4 show that DEIR performed better than or as well as other exploration methods and could successfully generalize to new game levels generated during training. The results also suggest that the proposed model and intrinsic reward are universally applicable to a variety of tasks, including those with higher-dimensional observations. In addition, we confirmed that the performance of DEIR was consistent with the training results reported in previous studies [Cobbe et al., 2020; Cobbe et al., 2021; Raileanu and Fergus, 2021].

4.3 Ablation Studies

To better understand DEIR, we created an advanced KeyCorridorS6R3 task with all three modifications proposed in Section 4.2 (view size: 3, standard deviation of noise: 0.3, obstacles: invisible), and conducted ablation studies to analyze the importance of (1) the conditional mutual information term and (2) the discriminative model.

Conditional Mutual Information Scaling

To analyze the importance of the conditional mutual information term proposed in Equation 1, we evaluated the performances of our DEIR agent and a variant without the mutual information term (see Figure 6(a) and (b)). As we can see, DEIR performed significantly better than the latter, which demonstrates the importance of our intrinsic reward design.

We utilize RNN in our model to capture temporal information in trajectories, which enables a more accurate representation to be learned. Thus, we further examine the effect of RNN on its own by training a separate agent with the discriminator only (see Figure 6(c)) and found that using RNN alone barely brings any benefit compared to Figure 6(b). Thus, we are confident that the mutual information scaling term indeed contributes to all of the performance improvements.

Discriminative Model

The performances of the DEIR agents driven by the inverse model and the forward model are shown in Figure 6(d) and (e), respectively. We applied the conditional mutual information term and RNN to both, and used the same tuned hyperparameters as in our previous experiments. Compared with Figure 6(a), our discriminative model-driven agent presented an evident advantage over the agent trained with the inverse model alone, while the forward model-driven agent completely failed to learn any meaningful policy in the task (due to the fact that the forward model is notorious for being weak to noise [Pathak et al., 2017]), suggesting that to achieve an advanced performance, the discriminative model is indispensable for learning state representations in POMDPs. We believe the main contribution stems from the trajectory embeddings learned by the discriminative model, and we present further analysis results in Section 4.4 for more insights.

4.4 Effectiveness of Learned Embeddings

To obtain further insights into the impact of the learned embeddings on the final performance, we conducted an experiment to compare three model variants. Concretely, we trained a forward-only, an inverse-only, and a discriminative model using the same data sampled by a vanilla PPO agent in standard DoorKey-8x8, where an agent needs to find a key to open a door that leads to the goal in another room. Following Alain and Bengio's work [Alain and Bengio, 2016], we devise five auxiliary supervised learning tasks. Given the learned trajectory embeddings eTRAJ , each task predicts an important temporal or spatial metric that is directly related to the game's progress or the agent's position. Technically, the ground-truth metrics are retrieved from the game's backend, and training stops after the agent reaches a near-optimal policy. The results in Figure 7 demonstrate that our discriminator-based method learns the most helpful embeddings for predicting important temporal and spatial metrics. This explains why it can benefit downstream objectives in RL, including exploration.

In comparison, embeddings from the forward and inverse models did not perform as well as ours, which is consistent with the findings in Figure 6. We hypothesize that the inverse model is less motivated to include historical data in embeddings because, by design, it can reliably infer actions merely from its always-factual inputs. On the other hand, the forward model relies too much on visual details, so its embeddings do not carry much information crucial to underlying tasks [Gulrajani et al., 2017; Goyal et al., 2017].

5 Conclusion

Training RL agents to explore effectively is a challenging problem, especially in environments with sparse rewards. A promising approach is to augment the extrinsic rewards with novelty-driven intrinsic rewards. However, focusing only on the novelty of observations is insufficient because an agent may incorrectly recognize the stochasticity in the environment's dynamics as novelties brought by its explorations. In this work, we proposed scaling the observation novelty with a conditional mutual information term that explicitly relates the agent's actions to the distances between observations, and learning a discriminative model that gives better intrinsic rewards. Compared with baselines, our method delivers outstanding performances in both standard and advanced versions of MiniGrid tasks. Also, it demonstrates general applicability to a variety of tasks with higher-dimensional inputs (such as those in ProcGen). As future work, we envision research on continuous action spaces and multi-agent settings.

A Technical Appendix

A.1 Intrinsic Reward Derivation

Given a pair of states (st, si), ∀i ∈ [0, t), a pair of observations (ot+1, oi), and an action at from an episode's history, we optimize the agent's policy with an intrinsic reward

The first term of J is the distance between observations ot+1 and oi, as used in previous studies [Blundell et al., 2016; Pritzel et al., 2017; Badia et al., 2020]. The second term of J is the conditional mutual information between dist(ot+1, oi) and at given st, si. Our intuition behind such design is that the policy achieves novelty by maximizing the distance between observations (ot+1, oi) and letting the distances be closely related to at, the action taken in st. This would efficiently eliminate the novelties rooted in the environment's stochasticity other than those brought by agent explorations.

The second term of J can be derived as (denoting Dt+1,i ≜ dist(ot+1, oi) for simplicity).

which is because

I(X; Y |Z)

We seek to simplify J, which contains an intractable evidence term in the Kullback-Leibler (KL) divergence DKL. To do so, we devise a surrogate function for the KL divergence by connecting it with total variation (TV). Denoting P(x) = p(x|st, si, at) and Q(x) = p(x|st, si), the Bretagnolle-Huber inequality [Bretagnolle and Huber, 1978] gives

where dTV(P, Q) = 12 ∥P-Q∥1 is the total variation between P and Q. Since dTV(P,Q) monotonically increases w.r.t. to KL divergence DKL(P∥Q), we use dTV as the surrogate function. Furthermore, given st, si, at in any deterministic environment (including POMDPs), st+1, ot+1, and Dt+1,i are uniquely determined. P is thus a unit impulse function that has the only non-zero value at Dt+1,i:

where D denotes the set of all possible observation distances. Also, by the definition of Q, we have ∫D Q(x)dx = 1 and Q(Dt+1,i) ≤ 1. Given those properties and Scheff´e's theorem [Tsybakov, 2008], we obtain

We derive the KL divergence's lower bound using dTV, as

where -log(2) is constant, and we effectively maximize the last term -log (Q(Dt+1,i)). Since Dt+1,i ≜ dist(ot+1, oi) is the non-negative distance between two observations ot+1 and oi, and dist(ot+1, oi) is related to dist(st, si), we assume Dt+1,i approximately follows an exponential distribution with a mean of the distance between its closest underlying states st and si. The intuition here is that limited observability leads to similar observations with minor variations, especially during early training phases when exploration is crucial. The marginal distributions of observation distances observed in our experiments also confirmed this. By setting Dt+1,i|st, si ∼ Exp(λ = 1/dist(st, si)) with an expected value of 1/λ = dist(st, si), we obtain

With all of the above, we finally define J's lower bound as J ≥ dist(ot+1, oi)

Note that the full distributions of observations and states are difficult to sample. By keeping only the dominating term dist(ot+1, oi)/dist(st, si) and relaxing the expectation to the minimum, we further simplify Equation A2 to

where ot+1, oi, and st, si are random variables in Equations A1 and A2 but realized values in Equation A3. We find that Equation A3 is much simpler than Equation A2, yet delivers a performance that is just as good or even better. Equation A3 is equivalent to the intrinsic reward proposed in the main text (Equation 4).

A.2 Implementation Details

Our implementations are based on Stable Baselines 3 [Raffin et al., 2021] and the official code of existing methods (if available). Our source code is available at https://github.com/ swan-utokyo/deir. More details about our algorithms, benchmarks, hyperparameters, network structures, and experimental results can be found at https://arxiv.org/abs/2304.10770.