MOSER: Learning Sensory Policy for Task-specific Viewpoint via View-conditional World Model

1 Introduction

Deep reinforcement learning (RL) has achieved remarkable successes in challenging domains, from game-playing [Mnih et al., 2015; Schrittwieser et al., 2020] to drug discovery [Pereira et al., 2021]. Many RL works now focus on learning skills directly from visual inputs like images and videos, which are common in real-world settings such as robotics manipulations [Zhu et al., 2020; Zhan et al., 2021] and autonomous driving [Hu et al., 2022]. Typically, autonomous agents acquire visual observations through static, fixed cameras to train policies [Levine et al., 2018]. In contrast, humans dynamically adjust their eye movements to find optimal viewpoints that provide the most helpful information to facilitate task completion [Dodge, 1903].

∗Corresponding author.

In most reinforcement learning environments, camera parameters are predefined by the designers without much concern about their explicit impact on subsequent learning. Different views provide varying amounts of task-relevant information, thus optimal viewpoint design requires effort and depends on expert knowledge. To provide a straightforward intuition of this, we conduct a proof-of-concept experiment. We quantify this effect by collecting image trajectories from well-trained policies under six viewpoints in the Mujoco Walker2d environment [Todorov et al., 2012]. We measure the mutual information (MI) between image observations and ground truth states using the MINE 1 method [Belghazi et al., 2018]. As shown in Figure 1, overall, the policies trained from viewpoints with high mutual information have relatively high test performance, indicating that viewpoint is crucial for the final RL performance. On the other hand, many studies explore the multi-view RL paradigm [Li et al., 2019; Kinose et al., 2022] to improve performance with more cameras, but at the cost of increased computation and storage. This raises an intriguing question: can we design an algorithm that automatically determines the optimal viewpoint for RL tasks, thus improving the final performance without expert knowledge or additional cameras?

1 It is worth noting that the MINE method is only used in the proof-of-concept experiment, and not used to estimate the MI in main experiments. More details are in Appendix D.1.

This problem poses two challenges: 1) we need to train a stable motor policy that adapts to the real-time changes of the observation viewpoint; 2) we need to provide suitable signals that guide the sensory policy to find a good viewpoint. We propose a new method, the MOdel-based SEnsor controlleR (MOSER), that simultaneously learns a view-conditional world model to simulate the environment, a sensory policy to control the camera, and a motor policy to complete the task. The view-conditional world model overcomes the two challenges by: 1) incorporating viewpoint information in the observation decoding process to exclude the view-related information from the latent states; 2) providing three types of intrinsic rewards from its dynamics model, reward model, and image decoder to direct the sensory policy toward the optimal viewpoint.

We evaluate MOSER on eight tasks in two environments: DeepMind Control Suite [Tassa et al., 2018] and Jaco Arm [Laskin et al., 2021]. MOSER outperforms the baseline methods on almost all tasks and approaches the performance of model-based multi-view RL methods on most tasks. We conduct ablations to analyze the effect of different design choices and interpret the adapted camera parameters in each task. These experiments demonstrate that MOSER can find optimal viewpoints during RL training to improve performance across diverse tasks.

Our contributions can be summarized as follows:

1. To the best of our knowledge, we are the first to introduce the sensory policy into model-based RL approaches for autonomously changing camera parameters.
2. We propose an effective MOdel-based SEnsor controlleR (MOSER) method that simultaneously learns the view-conditional world model, sensory policy, and motor policy to find the task-specific viewpoint.
3. Our approach performs excellently on several locomotion and manipulation tasks, demonstrating the significance of actively changing viewpoints in visual RL tasks.

2 RelatedWork

Paradigm of Visual Reinforcement Learning. In many real-world problems, agents can only observe highdimensional inputs, such as images or videos, rather than the underlying states. Previous works have defined this problem from various perspectives; for example, POMDP [Hausknecht and Stone, 2015] models it as a partially observable problem. HCMDP [Ma et al., 2022] considers high-dimensional input as a combination of mappings of task-relevant state and task-irrelevant contexts. Block MDP [Zhang et al., 2020] assumes that observation generation is the concatenation of noise and state variables. However, these definitions neglect an important factor - how images are captured. Different perspectives can lead to different visual representations of the same object, directly affecting the taskrelevant information in observations. We propose a Viewconditional Partially Observable Markov Decision Processes (VPOMDPs) assumption incorporating viewpoint factors into the observation generation process.

Multi-view Reinforcement Learning. Many studies exploit observations from multiple cameras to improve RL task performance. MVRL [Li et al., 2019] proposed a framework and model-free/based solutions for multi-view reinforcement learning. Multi-view Dreaming [Kinose et al., 2022] uses contrastive learning to embed multi-view observations in a shared latent space. MV-MWM [Seo et al., 2023] trains a masked multi-view autoencoder and world model. The model-free LookCloser [Jangir et al., 2022] integrates multi-view observations via cross-view attention. The Fuse2Control [Hwang et al., 2023] learns the underlying state space model from multi-view observation sequences with an information-theoretic method, showing the effectiveness in aggregating task information across many views. While these approaches outperform single-view methods, they increase computation and storage costs and may not guarantee the coverage of the best viewpoint. In contrast, our method receives observations from a movable camera from one viewpoint at a time instead of multiple cameras.

Active Vision in Reinforcement Learning. Actively seeking optimal viewpoints is essential for agents performing visual RL tasks. Previous work has enabled control of hand and eye movements to avoid occlusions and complete manipulation tasks [Cheng et al., 2018; Grimes et al., 2023]. SUGARL [Shang and Ryoo, 2023] proposes active perception of acquired images by selectively focusing on certain image regions. The recent work SAM-RL [Lv et al., 2023] uses a differentiable physics simulator and rendering and learns a sensing-aware Q function for sensory and motor action selection. Our method fundamentally differs from previous work: (1) We use a sensory policy decoupled from the motor policy and change camera parameters guided by intrinsic signals. (2) We efficiently train the motor policy inside a Viewconditional World Model, simulating the environment under viewpoint changes.

3 Preliminary

View-conditional Partially Observable Markov Decision Processes. A standard RL problem can be defined as a Markov decision process (MDP), which is a tuple of M = (S,A, P, p, r, γ), where S is the state space, A is the action space, P : S × A → S is the transition function, p is the initial state distribution, r : S × A × S → R is the reward function, and γ is the discount factor. The goal of RL is to learn an optimal policy π⋆ (a|s) that maximizes the expected cumulative discounted return RM(π) = Es0∼p(·),at∼π(·|st),st+1∼P(·|st,at) [∑∞t=0 γ t r(st, at, st+1)]. To solve RL problems with incomplete observations such as high-dimensional images, prior works [Hafner et al., 2019; Hafner et al., 2020; Yarats et al., 2021b; Bharadhwaj et al., 2022] adopt partially observable Markov decision processes (POMDPs) assumption which additionally introduces an observation space O and generates observations by an emission function ϕ : S → O. We modify the emission function as ϕ(o|s, c) and reformulate a new assumption named view-conditional partially observable Markov decision processes (VPOMDPs), where observations are generated based on the ground-truth state s and the sensor parameter c.

Model-based Reinforcement Learning. Model-based reinforcement learning (MBRL) learns a model of the environment's transition dynamics p(st+1|st, at) and reward function r(st, at) from experience and leverages this model to plan actions by searching over possible future states [Janner et al., 2019; Wang et al., 2019]. Model-based methods enable sample-efficient learning as the agent can learn from the simulated environment instead of the real one [Moerland et al., 2023]. This is especially beneficial when the input is highdimensional images [Hafner et al., 2019; Hafner et al., 2020; Hafner et al., 2021], as MBRL can obtain a low-dimensional surrogate environment to reduce the storage cost and improve the learning efficiency. Our method adopts a model-based approach to learn the dynamics of the environment and obtain an abstract state representation based on historical information, stabilizing policy training under varying viewpoints.

Soft ActorCritic. Soft Actor-Critic (SAC) [Haarnoja et al., 2018; Haarnoja et al., 2019] is an off-policy actor-critic deep RL algorithm based on the maximum entropy framework that optimizes a stochastic policy to maximize expected reward while also maximizing entropy. The agent learns a policy network πθ(at|st) and two Q-networks Qψ(st, at) to optimize the expected return J(π) = E(st,at)∼ρπ [r(st, at) + αH(π(·|st))], where ρπ is the state-action marginal distribution, r(st, at) is the reward, and H represents entropy [Haarnoja et al., 2018]. The policy and Q-functions are updated by backpropagating the gradients of the expected return via the reparameterization trick, while the temperature parameter α is automatically adjusted to ensure sufficient exploration [Haarnoja et al., 2019]. The two Q-networks mitigate positive bias and improve training stability. We choose SAC, a typical off-policy algorithm, to update the sensory policy since it can efficiently utilize the collected sensory data. Any off-policy algorithm can instantiate our sensory policy.

4 Methods

In this section, we first introduce the model components and training objective of the View-conditional World Model (VWM) (Section 4.1). We then detail the designed intrinsic rewards for sensory policy training from three aspects, active mutual information maximization, next-frame prediction, and future reward maximization (Section 4.2). Finally, we use the actor-critic algorithm to train the motor policy by imagining in the learned world model (Section 4.3). We present an overview of our training process in Figure 2a, the detailed algorithm in Appendix B, the network architectures in Appendix D.3, and the hyper-parameters in Appendix F.

4.1 Training View-conditional WorldModel

To enable the agent to train stably under varying viewpoints, we propose the View-conditional World Model (VWM), a variant of the world model [Ha and Schmidhuber, 2018;

Hafner et al., 2019; Hafner et al., 2020]. We model the image decoder of the VWM as the emission function under the VPOMDPs assumption to separate the viewpoint information from the latent state. The image decoder pϕ (ˆ ot|st, ast-1 ) reconstructs the original image oct from the current latent state st and the previous sensory action ast-1 . The other components of VWM are as follows:

The encoder infers latent state st from the previous latent state st-1, previous motor action amt-1 , and current observation oct . The dynamics model predicts the latent state st without observation oct , enabling the model to forecast future states in the latent space. The reward decoder estimates rewards for possible future states. We jointly optimize the VWM parameters by minimizing the negative variational lower bound [Hafner et al., 2020] as below:

where β is a hyperparameter to weigh the KL-divergence term. The detailed derivation of the loss is in Appendix C.

4.2 Learning Sensory Policy

The desiderata of the optimal view captured by the camera for real-world RL tasks are threefold. First, the view should be informative, containing rich proprioceptive information about the ground truth state. Second, the view between adjacent frames should exhibit temporal coherence, as consistent and continuous visual inputs can stabilize the world model training and policy learning. Finally, the view should inform high potential future rewards for the policy. In this section, we formulate these desiderata into three types of intrinsic rewards that encourage the sensory policy to take proper actions.

Active Mutual Information Maximization (AMIM). Based on the result of the proof-of-concept experiment in Figure 1, viewpoints containing rich information about the ground-truth state benefit policy training. We employ the sensory policy to select actions that adjust camera parameters to maximize the mutual information between the image observations oc and latent states s, conditioned on the previous states and actions.

Directly optimizing this mutual information is challenging. We can decompose it into two terms:

Equation (3) tells us that maximizing the original mutual information is equivalent to maximizing the above information gain, which maximizes the diversity in latent state space and minimizes the uncertainty after obtaining the observation under a selected viewpoint. We omit the sensory action ast in the above two entropy terms for two reasons: (i) in the VWM's latent state space, predicting the next state st+1 does not require sensory actions; (ii) ast only affects the generation of the image observation oct+1 and can be absorbed into it. Therefore, to maximize the mutual information, we only need to minimize the second term in Equation (3), which depends on the sensory policy's actions. The sensory policy should select the action that produces the most informative observation oct+1 , reducing the uncertainty of the posterior distribution as much as possible. To encourage this behavior, we define the AMIM reward as:

where oct+1 is generated by the underlying emission function ϕ(oct+1 |st+1, ast ) of the environment, and ast is determined by the sensory policy. We can effectively estimate the posterior entropy in Equation (4) by the encoder qψ(st+1|st, amt , oct+1 ) of VWM, which outputs the mean and standard deviation of a Gaussian distribution.

Next-Frame Prediction (NFP). The key challenge of exploring viewpoints is that timely changing camera parameters can cause inconsistent observations for model and policy learning. To alleviate abrupt changes in adjacent observations, the sensory policy should output predictable actions that can help predict the next frame from the latent state. Based on this, we design the NFP reward as the log-likelihood of the next frame prediction by the VWM's image decoder, as follows:

Future Reward Maximization (FRM). The sensory policy should take actions that generate image observations that imply high rewards. To achieve this, we use the VWM's reward decoder Rϕ to predict the potential reward associated with the selected observation, without introducing any additional modules. We formulate the FRM reward as a prediction of the potential task reward on the image observation oct+1 , which is controlled by the sensory action ast and the latent state st+1:

Sensory Policy Training. We update the sensory policy through the SAC algorithm [Haarnoja et al., 2018] based on the weighted sum of three intrinsic rewards (ra , rn , and rp ). In principle, the sensory policy can be optimized with any off-policy RL algorithm. The critic of the sensory policy fits the value function based on the bootstrapped intrinsic rewards and guides the actor to select actions based on the current camera parameters ct. The components of the sensory policy are shown below:

Intrinsic Reward:

Sensory Actor:

where α1, α2, α3 are the weights of different intrinsic rewards. The detailed training algorithm of the sensory policy is in Algorithm 2 of Appendix B, and the values of α1, α2, α3 are in Table 3 of Appendix F.

4.3 LearningMotor Policy

We adopt the behavior learning mechanism of Dreamer [Hafner et al., 2020] to optimize the motor policy within the VWM. We only train the motor policy on the imagined trajectories inside the VWM to improve sample efficiency. Specifically, the critic of the motor policy estimates the value function using the decoded reward and bootstraps, and the actor

samples actions based on the imagined latent state, maximizing the expected return.

5 Experiments

We conduct several experiments to answer the following research questions (RQs):

1. Effectiveness (RQ1): How effective is MOSER in continuous control tasks?
2. Ablation (RQ2): How do the critical designs of MOSER contribute to the final performance?
3. Specialization (RQ3): Can MOSER identify specific viewpoints for different environments and tasks?
4. Optimality (RQ4): Can the best camera parameters found by MOSER improve the performance of the current algorithms?

5.1 Experiments Setup

Locomotion. We select four visual control tasks from DeepMind Control Suite [Tassa et al., 2018]: cheetah run, walker walk, reacher easy, and quadruped run. These tasks pose challenges like sparse rewards, contact dynamics, and 3D scenes. The shape of image observation is 64×64×3 for all tasks.

Manipulation. We test the method's ability to control cameras and adapt to different goals while performing simple manipulation tasks on the Jaco Arm, a 6-DOF robotic arm with a three-finger gripper. The Jaco Arm has been used in previous unsupervised RL works [Laskin et al., 2021]. This environment is challenging due to the sparse reward, as shown in prior work [Laskin et al., 2021; Yarats et al., 2021a]. We consider four tasks with different target positions: top left, top right, bottom left, and bottom right. The shape of observations is the same as that in the Locomotion tasks.

Settings of camera parameters. As depicted in Figure 2b, each sensory state is represented by the tuple (d, β, h) in a spherical coordinate system, where d is the distance from the target to the camera, β is the azimuth angle, and h is the elevation angle. For both locomotion and manipulation tasks, the sensory state space is continuous, initialized at (3, 90,-45) and bounded within the ranges of [0, 10] for d, [0, 180] for β, and [-90, 90] for h. The sensory action space is also continuous, spanning [-1, 1] in all three dimensions. The examples of camera parameters are shown in Appendix A.

Baselines. We compare our method with several baselines on both DMControl Suite and Jaco Arm:

• SUGARL: A visual reinforcement learning method that trains a sensory policy to actively select the optimal region from the original image observation as the input for the motor policy [Shang and Ryoo, 2023].
• MF-joint: A model-free RL method that jointly trains sensory and motor policies based on the environmental reward. We use the DrQ algorithm [Yarats et al., 2021b] as the backbone, which is a state-of-the-art model-free visual RL method.
• MB-free: A model-based RL method that learns a sensory policy with the environmental reward and trains the motor policy inside the model. We employ the DreamerV2 algorithm [Hafner et al., 2021], an MBRL method that plans through a learned world model.
• MF-multiview: A model-free RL method with multiview observation. We choose the DrQ algorithm [Yarats

et al., 2021b] as the backbone and concatenate the firstand third-person image observations at each step as inputs.

• MB-multiview: A model-based RL method with multiview observation. We select the DreamerV2 algorithm [Hafner et al., 2021] as the backbone and process the inputs the same way as MF-multiview.

5.2 Effectiveness (RQ1)

Locomotion. As shown in Figure 3, MOSER enhances performance in tasks with single fixed-camera observations, outstrips the model-free multi-view approach, and rivals the model-based multi-view method in nearly all tasks. The MFjoint method struggles with task execution and viewpoint optimization due to fluctuating observations from changing perspectives. Interestingly, without explicit sensory rewards, the MB-free method outperforms the MF-joint method, possibly because its long-term predictions enable accurate proprioceptive state estimation and facilitate motor control under viewpoint changes. The SUGARL method, although it seeks ideal image regions, is hindered by static camera settings. MOSER exceeds SUGARL across all tested environments by actively selecting the best viewpoints. It also surpasses the MF-multiview in four environments, suggesting that a single optimal viewpoint can be as informative as multiple ones. While MOSER nears the performance of the MB-multiview method in three tasks, it lags in Cheetah Run. Notably, the MB-multiview benefits from utilizing twice the amount of data compared to MOSER, as it receives two observations from distinct views at each step. This substantial increase in data volume significantly contributes to its performance advantage. Our objective is not to supplant the multiview approach with MOSER universally but to enhance the efficacy of existing visual RL methods by a learnable viewpoint.

Manipulation. In Figure 4, the MOSER method outperforms all other methods with fixed camera observation inputs, including single-view and multi-view, on Jaco Arm. This is because the manipulation task requires the agent to control the arm to reach the goal, which demands dynamic viewpoint changes to capture the relative position between the arm and the goal and contribute to policy adaptation. Multi-view methods perform significantly better than single-view methods on the Jaco Arm task since multiple viewpoint inputs can provide richer task-relevant information. Overall, MOSER achieves superior results by actively selecting suitable viewpoints to optimize the raw input for the motor policy.

5.3 Ablation (RQ2)

To evaluate the contribution of different components of MOSER, we conduct ablation studies on the viewpoint condition, the designed intrinsic rewards, and the frequency of executing sensory actions.

Impact of viewpoint condition. To examine the influence of viewpoint conditioning, we modify the image decoder in VWM to reconstruct the image observation from the latent state only. In Figure 5, MOSER with WM shows a significant performance drop on multiple tasks compared to the original

(a) Ablation study.

MOSER, especially on the Quadruped Run task. This indicates that incorporating the viewpoint condition in the world model of MOSER is essential for stable training and achieving high performance.

Effects of intrinsic rewards. To investigate the individual effects of different intrinsic rewards - AMIM, NFP, and FRM - we conduct ablation studies on each separately within DMC environments. In Figure 5a, removing the AMIM reward leads to a significant performance decrease in the DMC tasks, indicating that a good viewpoint needs to maximize the amount of information about the proprioceptive state. Eliminating the NFP reward results in a high variance of performance, revealing that predicting the following observation from the previous state and current sensory action helps smooth and continuous viewpoint changes. Removing FRM results in a remarkable performance drop across environments, confirming good viewpoints should enable high rewards. Each component is crucial to learning good viewpoints across diverse control tasks. We present the change of these intrinsic rewards during training in Appendix E.4.

Impact of sensory action frequency. Different sensory action frequencies can affect training stability and final performance. We investigate the effects of different sensory action frequencies on experimental performance. The results in Figure 5b show that agents prefer high sensory action frequency to adapt to the environment in complex environments such as Quadruped Run. In contrast, in 2D locomotion environments like Cheetah Run and Walker Walk, changing camera parameters every 10 steps leads to optimal performance, suggesting that low sensory action frequency is more suitable for these environments.

5.4 Specialization (RQ3)

We present the camera parameters selected by MOSER for four DMC tasks in Figure 6a, illustrating the preference for different viewpoints in each environment. Cameras are positioned closer to the agent in tasks like Cheetah Run, Walker Walk, and Reacher Easy compared to Quadruped Run, to accommodate the need for a broader view in 3D locomotion tasks. Variations in azimuth angles across tasks highlight the unique optimal viewpoints required for each. Elevation angles, consistently negative, suggest a universal preference for a bottom-up perspective to observe leg movements. These findings underscore MOSER's capability to tailor viewpoints to specific tasks.

Figure 6b details the camera parameters selected by MOSER for four Jaco Arm tasks, revealing shared and distinct viewpoint preferences within the same environment under varied objectives (the red balls). All four tasks display similarities in camera distances and elevation angles while differing in azimuth angles, notably demonstrating symmetry between the Top and Bottom tasks (with the y-axis as the axis of symmetry). Such variations in azimuth are attributable to the target's horizontal positioning, influencing the optimal viewpoint. These results confirm MOSER's proficiency in discerning both common and subtle task characteristics to determine ideal camera perspectives.

5.5 Optimality (RQ4)

We validate the optimality of the searched camera parameters in Quadruped Run, a challenging 3D locomotion task. Specifically, we compare three environmental settings: (1) camera parameters chosen by MOSER, (2) original default camera parameters, and (3) two distinct cameras with default views. We run DreamerV2 and DrQ algorithm on these three environments with all other settings identical and record the episodic return for 100K, 250K, and 500K steps. In Figure 7, all methods exhibit performance gains to varying degrees in the environment with camera parameters selected by MOSER. With pre-training viewpoints by MOSER, the performances of both algorithms surpass that of human-defined multiviews. These results reveal that learning from an optimal viewpoint can improve the performance of current visual RL algorithms, thus corroborating our initial motivation.

2 Angles in degrees can be converted to radians by multiplying by π 180 .

6 Conclusion

In visual RL research, learning from a fixed single-view observation requires expert knowledge to determine the viewpoint, and learning from a multi-view input will incur additional computing and storage costs. To tackle this problem, we introduce the view-conditional partially observable Markov decision processes assumption to incorporate the view-related parameters into the emission function and develop the view-conditional world model based on this assumption. We propose the MOdel-based SEnsor controlleR (MOSER) method, which learns the sensory policy to control the camera parameters and optimizes the motor policy to complete the RL task simultaneously. We evaluate the performance of MOSER on four locomotion tasks and four manipulation tasks. Moreover, we conduct ablation studies to analyze the contribution of viewpoint conditioning and different intrinsic rewards. We explore the suitable sensory action frequency in diverse tasks. MOSER can select task-specific viewpoints and improve the performance of the visual RL methods with a single fixed camera.

One limitation of MOSER lies in its performance on specific tasks compared to the MB-multiview method. The latter benefits from a richer data regime with two distinct images as input at each step. Our objective is not to replace the multi-view method with MOSER but to offer a possible improvement of autonomously selecting optimal viewpoints when using model-based multi-view methods. We will explore this further in future work. Furthermore, our current experiments involve task-centric observations, such as those in DMControl environments. Robotics manipulation and some navigation tasks typically require more flexibility from the camera. We plan to explore other scenes [Yang et al., 2023] beyond the Jaco Arm task to demonstrate MOSER's capabilities. Additionally, real-world scenarios often present distracting inputs impacting sensory and motor policy optimization. One straightforward approach is to separately model task-relevant and irrelevant dynamics [Fu et al., 2021; Wan et al., 2023] within the view-conditional world model, without modifying other aspects of MOSER. Further investigation of this is a promising direction for future work.

Acknowledgments

We thank Han-Jia Ye, Shaowei Zhang, Minghao Shao, Yucen Wang, Ziyuan Chen, and Olivia Yan for their valuable discussions. This work was partly supported by the National Science and Technology Major Project under Grant No. 2022ZD0114805 and partly by the Postgraduate Research & Practice Innovation Program of Jiangsu Province.