Vision Language Navigation with Knowledge-driven Environmental Dreamer

1 Introduction

Vision-Language Navigation (VLN) task [Anderson et al., 2018a] requires an agent to navigate in an embodied environment following a natural language instruction. This task is closely connected to many real-world applications, such as household robots and rescue robots [Zhu et al., 2021]. The VLN task is challenging since it requires an agent to master diverse skills, such as vision-language alignment, sequential vision perception, and long-term decision-making. The key to the VLN task is to perceive the panoramic visual scene, comprehend natural language instructions sequentially, and make actions step-by-step.

Learning a robust navigation policy is a long-study problem in the Artificial Intelligence community. Previous methods [Fried et al., 2018; Wang et al., 2019; Chen et al., 2021; Qiao et al., 2022] attempt to have made great progress in improving the ability to perceive the vision and language inputs and learn a robust navigation policy [Liu et al., 2021; Hao et al., 2020; Hong et al., 2021]. However, the limited scale of training data introduces a large bias between training environments and testing environments, which severely impacts navigation performance. The most widely used VLN dataset, Room-to-room dataset [Anderson et al., 2018b], contains only 22K instruction-path pairs from 90 house scenes. However, the space of possible navigation paths and visual observations increases exponentially along with the path length. Therefore, the learned navigation policy can easily overfit the seen scenes and is hard to generalize to the unseen scenes. For example, if an agent learns navigation in a scene with red chairs and a blue sofa, it may be confused when it is tested in an unseen scene with blue chairs and a red sofa.

In this paper, we propose a model named Knowledgedriven Environmental Dreamer (KED), which is able to generate unseen environmental scenes without extra data annotations. The environmental dreamer is built based on an autoencoder model, which consists of an encoder and a decoder. The encoder learns to disentangle two latent encodings from an image view: a texture encoding and a structure encoding. The decoder receives these two latent encodings and generates a synthetic image to represent a panoramic view of an agent in an unseen environment. The environmental dreamer generates an unseen environment by three steps: 1) we divide house scenes into rooms; 2) we extract texture encodings and structure encodings of each room; 3) we randomly match the texture encodings and the structure encodings from different house scenes and decode them to generate unseen house scenes. An overview of our Knowledge-driven Environmental Dreamer method is shown in Figure 1.

Different from previous image generation works [Park et al., 2020; Zhu et al., 2020b; Karras et al., 2021] that synthesize unseen images, our work is required to generate unseen environments. Generating unseen environments is challenging because we need to ensure texture consistency and structure consistency of the generated environments. The texture consistency indicates that the nearby views should have similar textures. If a sofa could be observed from several positions, the sofa should have the same color in the views of these positions. The structure consistency means the house structure shown in nearby views should be consistent. If a pillar could be observed from several positions, the pillar should have the same shape in the views of these positions. Therefore, we introduce semantic regularization objectives in order to ensure texture consistency and structure consistency. First of all, we suggest that all panoramic views in the same room, which are represented as all nodes in a connected block, should be transferred by the same texture. Second, we propose that the semantic information of an original panoramic view and the generated view should be consistent. Third, we suggest that a trajectory sampled in a generated house scene should contain the same semantic information as the original trajectory.

Our experiments show that the navigation agent with KED outperforms the previous state-of-the-art method on various VLN benchmarks, such as R2R, R4R, and RxR. The results also demonstrate that using the augmentation data generated by KED is able to significantly reduce the performance gap between seen and unseen environments, which dramatically improves the overall navigation performance. Our ablation study shows that the proposed augmentation method outperforms other augmentation methods at the same augmentation data scales. The visualization results show that our proposed regularization objectives are able to ensure texture consistency and structure consistency and generate environments with higher-quality images.

2 RelatedWork

Embodied Navigation Environments. have been proposed for navigation learning. House3D [Wu et al., 2018] is the first indoor environment for navigation. AI2THOR [Kolve et al., 2017] is an interactable indoor environment. The Active Vision dataset [Ammirato et al., 2017] consists of dense scans of 16 different houses. Matterport3D [Anderson et al., 2018b], Gibson [Xia et al., 2018; Xia et al., 2020] and Habitat [Savva et al., 2019] propose high-resolution photo-realistic panoramic view. However, the scales of these environments are small since collecting data for rendering house scenes is expensive.

Vision-language Navigation. Anderson et al. [Anderson et al., 2018b] propose the Room-to-Room (R2R) dataset, which is the first Vision-Language Navigation (VLN) benchmark [Chang et al., 2017]. To address the VLN task, Fried et al. propose a speaker-follower framework [Fried et al., 2018] for data augmentation and reasoning in a supervised learning context, along with a concept named "panoramic action space" that is proposed to facilitate optimization. Wang et al. [Wang et al., 2019] demonstrate the benefit of combining imitation learning [Bojarski et al., 2016; Ho and Ermon, 2016] and reinforcement learning [Mnih et al., 2016; Schulman et al., 2017]. Due to the success of BERT [Devlin et al., 2018], researchers have extended it to learn visionlanguage representations in VLN. PRESS [Li et al., 2019] applies the pre-trained BERT to process instructions. PREVALENT [Hao et al., 2020] pre-trains an encoder with imagetext-action triplets to align the language and visual states, while VLN-BERT [Majumdar et al., 2020] fine-tunes ViLBERT [Lu et al., 2019] with instruction-trajectory pairs. Hong et al. [Hong et al., 2021] implement a recurrent function to leverage the history-dependent state representations based on previous models. However, the agent learned with limited training scenes can overfit seen textures, and hard to generalize to unseen scenes where objects look different.

Style transfer. requires blending two objects, one containing content and the other containing style, to integrate into a new object. Many classic works of style transfer [Hertzmann et al., 2001; Johnson et al., 2016; Tenenbaum and Freeman, 2000] focus on obtaining human faces of different styles [Hertzmann et al., 2001; Tenenbaum and Freeman, 2000] as well as art works [Chen et al., 2017; Gatys et al., 2016]. Later, with the potentiality of style transfer being revealed, it has been applied to some challenging tasks like navigation [Zhu et al., 2020c] and super-resolution [Johnson et al., 2016]. However, the application of this technique to vision language navigation is not trivial since it is difficult to generate unseen scenes which vary in size, shape, and texture. Compared with previous works that generate a synthetic scene [Zhao et al., 2021] or introduce segmentation data [Koh et al., 2021], our work makes the first attempt at generating photo-realistic scenes without additional prior.

3 Knowledge-driven Environmental Dreamer

3.1 Problem Setup

In the Vision-and-Language Navigation (VLN) task, an agent follows a natural language sentence and predicts actions A = {a1, ..., an} step-by-step to reach the target. A navigation environment is formulated as a graph G = {V, E} consisting of nodes and edges. At each node, a navigation agent is able to observe a panoramic image view I. The agent takes an action a to navigate from one node to another via an edge.

3.2 Learning to Dream

Here we introduce the framework and the objective functions for learning the environmental dreamer. The detailed framework is shown in Figure 2. Our environmental dreamer consists of an encoder E and a decoder G. The encoder takes a panoramic view as input and outputs a texture encoding f T and a semantic encoding f S . The decode is used to decode a concatenated feature with a texture encoding and a semantic encoding as a panoramic image view.

Reconstruction and Adversarial Objectives. For each image I in a minibatch, we calculate a reconstruction loss as:

In addition, we introduce an adversarial objective with a discriminator D to improve the quality of the generated image:

This objective encourages the discriminator D to distinguish fake image and encourage the generator G to generate high quality image to deceive D.

Decomposition Objective. is adopted to help our model to learn feature decomposition. Different from classical autoencoder framework, we divide the latent feature encoding f into two parts: a texture encoding f T and a semantic encoding f S . The texture encoding is a shallow but wide feature map and the semantic encoding f S the deep but narrow feature map. We sample two images IA and IB from different house scenes and optimize the decomposition objective:

where [·] represents concatenation. IC and ID are unseen synthetic images generated by G. The decomposition objective enables the encoder to learn to represent texture encoding and semantic encoding respectively. The encoder and decoder are jointly optimized by these objectives.

3.3 Regularizing Environmental Consistency

Texture Consistency Objective. We propose the texture consistency objective to ensure that all panoramic images in the same room are consistent. Firstly, we adopt a neural mapping network M to map the texture encodings of the original image IB and the generated image IC as feature vectors that represent the global texture information. We use a cosine similarity loss to maximize the similarity of the global texture vectors:

Semantic Consistency Objective. One difficulty in our environmental generation task is to ensure that all objects should be easily distinguished without distortion, especially small objects at a distance. To address this problem, we propose to leverage the knowledge from the visual encoder of the CLIP [Radford et al., 2021] model. We use the CLIP model to encode the original image IA and the generated image IC as features, and use Mean Absolute Error (MAE) to penalize the error between the features:

The semantic consistency objective encourages the environmental dreamer to generate panoramic views that contain diverse objects with new textures. It solves the problem that the model focuses on style transfer while ignoring the generation of detail objects.

Instruction Consistency Objective. In addition to visual objectives, we suggest that the semantic information contained in the trajectory of a generated scene should be consistent with which of the original scene. Firstly, we train a "speaker" model P using the instruction-trajectory pairs as in [Fried et al., 2018]. The "speaker" model encodes a sequence of image observations following a trajectory and decodes a natural language sentence to describes the image sequence. Secondly, we sample a trajectory from an original room A and a trajectory from a generated room C, and use the "speaker" model to describe the trajectories by natural

Algorithm 1 Selecting key vertexes

Input: Two environments GA = {VA, EA}, GB = {VB, EB},

• encoder E, decoder D 1: ˜ VA = {v | top 10% v in VA ordered by BC(v)}; 2: ˜ VB = {v | top 10% v in VB ordered by BC(v)} 3: RA = ConnectedBlock(VA - ˜ VA); 4: RB = ConnectedBlock(VB - ˜ VB) 5: for rA in RA do 6: rB = random(RB) 7: vB = random(rB) 8: f TB , f SB = E(vB) 9: for vA in rA do 10: f TA , f SA = E(vA) 11: vC = D([f TB , f SA ]); 12: VC = VC ∪ {vC}; 13: end for 14: end for 15: for vA in ˜ VA do 16: vB = random(˜ VA) 17: f TA , f SA = E(vA) 18: f TB , f SB = E(vB) 19: vC =D([f TB , f SA ]) 20: VC = VC ∪ {vC} 21: end for 22: return GC = {VC, EC}

language sentences and penalize the error by a cross-entropy loss:

The wA,i denotes the ground truth word at position i in room A, and ICi denotes the view of position i in room C. We regularize the instruction consistency objective in the "teacherforcing" paradigm. This training paradigm gives the previous ground truth words to the "speaker" model to reduce the accumulating error. The weights of the "speaker" model are fixed in optimizing the instruction consistency objective.

Loss Reweighting for Adaptive Optimization. If the model focuses on regularization too early in the training process, the performance of image translation can be reduced. Therefore, we design a loss reweighing mechanism to encourage the model to focus on the image translation task at the beginning and pay more attention to regularization after it performs the image translation task well:

Above all, we sum up all objectives to jointly train the environmental dreamer:

3.4 Generating Training Data for Navigation

To describe our method for generating augmented training data for the vision-language navigation task, we first generate views of house scenes to build environments. Then we sample instruction-trajectory data from the generated house scenes.

As described in the Problem Setup section, we model a navigation environment as a graph. We pick out the key nodes according to the betweenness centrality (BC). A node is considered a critical node if it has a betweenness centrality score in the top 10%. The connected sub-graphs divided by key nodes are regarded as rooms.

We generate a new house scene by generating new views room-by-room in three steps: 1) we randomly sample a scene A to provide semantic information and a scene B to provide texture information; 2) for each room in scene A, we encode a random view from rooms in scene B as f TB and use this texture encoding to decode all semantic features of that room to generate a new room; 3) for each key node in scene A, we use the texture encoding of a random key node in scene B to generate a new view for that key node. A detailed demonstration of this process is shown in Algorithm 1. The reason we only use the texture encoding of scene B instead of all the scenes in the dataset is that it prevents the textures from changing too much in the new scene. Ideally, our method is able to generate more than 8K new scenes. Due to the limit of computation resources, we generate 270 scenes in practice, 3 times more than the original scenes provided by the R2R dataset.

Next, we randomly sample trajectories in the generated unseen scenes and use the "speaker" model to describe these trajectories to generate instruction-trajectory pairs. At last, we generated 500K instruction-trajectory pairs, 25 times the size of the original data, which greatly enriches our training data.

3.5 Learning to Navigate

We adopt a joint optimization policy that uses imitation learning and reinforcement learning.

ImitationLearning. forces the agent tomimic the behavior of its teacher. Our agent learns from the teacher action a∗t for each step:

where a∗t is a one-hot vector indicating the ground truth.

Reinforcement Learning. We implement the A2C algorithm [Mnih et al., 2016] to maximize the total reward of navigation, whose loss function is formulated as:

where At is a scalar representing the advantage defined in [Mnih et al., 2016].

4 Experiment

Dataset and Environments. We evaluate our navigation agent on three VLN benchmarks: Room-to-Room (R2R), Room-for-Room (R4R), and Room-Across-Room (RxR). The original training set provided by R2R consists of 61 environments and 14,025 instructions. We augment the training dataset by 500K instruction-trajectory pairs from 270 generated unseen scenes.

Evaluation Metrics. We use a large number of metrics for evaluation, such as Trajectory Length (TL), Navigation Error (NE), Success Rate (SR), and Success rate weighted by Path Length (SPL). In the evaluation of R4R and RxR, we compare our method with previous state-of-the-art methods on Coverage weighted by LS (CLS), Normalized Dynamic Time Warping (nDTW), Success weighted by normalized Dynamic Time Warping (sDTW).

Results on VLN Standard Benchmark. In this section, we compare our method with several other representative methods on the standard R2R benchmark. We apply our KED method on two backbone models: a seq-to-seq model with cross-modal attention [Wang et al., 2019], and a recurrent VLN-BERT model [Hong et al., 2021]. Tab. 1 shows that both backbone models trained with our augmentation data outperform the models without augmentation data. Our method improves the SPL performance of the RCM model by 10.7% on validation seen, 2.0% on validation unseen, and 1.8% on test split. Our method improves the SPL performance of the recurrent VLN-BERT model by 5.0% on validation seen, 2.5% on validation unseen, and 2.7% on test split. The KED based on the VLN-BERT model outperforms the previous state-of-the-art model [Qiao et al., 2022] by 0.9% on validation seen, 1.1% on validation unseen, and 1.4% on test split. Above all, the agent built on the recurrent VLN-BERT model and learned with our augmentation data outperforms all previous methods and achieves the state-of-the-art method on the standard R2R benchmark.

Results on R4R and RxR. Here we investigate if our method works on other navigation benchmarks, such as R4R and RxR. The result are shown in Table 2 and Table 3. We compare KED with other augmentation methods: 1) pairwise data augmentation (Speaker-follower); 2) feature-wise data augmentation (Envdrop); 3) and scene-wise data augmentation (KED). Our KED method significantly outperforms the other two augmentation methods on all metrics in the R4R dataset and RxR dataset, which validate the effectiveness of our method.

Ablation of Knowledge-driven Dreamer (KED). In this section, we ablate different versions of our model and compare how much performance improvement the navigation agent is able to achieve by each part. Results are shown in Table 4. By comparing 1 and 2 , we find that the training data generated by the baseline model is able to improve the navigation performance. We discover that 3 significantly outperforms 2 , which infers that the regularization objective largely improves the quality of training data. 3 is trained with 90 scenes. The performance of this model improves if it is trained with more data like 4 , which reveals that our data is beneficial and large-scale scene-wise augmentation data can reduce the training-testing domain gap.

Ablation of Speaker. Here, we ablate the impact of the speaker for our data augmentation method based on the RCM backbone, as shown in Table 5. The KED method without a speaker is able to improve the navigation performance on both validation sets. More performance improvement can be get by applying the speaker, which indicates that the speaker is useful in our data augmentation method.

Quality of Generated Scenes. In Figure 3, we compare the generated panoramic views between the model with regularization objectives and the baseline model without regularization objectives. It turns out that our model with regularization objectives have several advantages: 1) they have fewer fragments and "dirty" regions, such as 1 , 4 , 5 ; 2) the objects are clearer and easier to distinguish, such as 2 , 6 , 8 ; 3) the shadows and light effects are more realistic. For example, the baseline model produces shades of orange in 7 when there is no orange object or light source in the view. We conclude that is because the baseline model mistakenly transfer the texture information to the generated image. 4) the generated views contain more detailed objects, especially the small objects in the distance. In 3 , 6 and 8 , the objects generated by the baseline model cannot be seen clearly while our model is able to generate detailed objects. 5) The objects in the generated view look more realistic. In 7 , the clock is transferred to another texture but the numbers are still clear. In contrast, the numbers generated by the baseline model have large distortion.

5 Conclusion

In this paper, we propose Knowledge-driven Environmental Dreamer (KED) to generate unseen scenes for agents to learn. KED is required to ensure texture consistency and structure consistency in generating new scenes. We propose three novel reconstruction objectives that leverage knowledge from pertained CLIP model and the "speaker" model to regularize the optimization of KED. Our experimental results reveal that the augmentation data generated by KED is able to significantly improve the performance of the navigation agent. Both quantitatively and qualitatively analysis infers that the knowledge-driven environmental dreamer is able to generate high-quality augmentation data.

Ethical Statement

Positive Impacts. Our work proposes a method to improve navigation performance, which contributes to realworld robotic applications. Household robots and rescue robots are required to navigate following a natural language instruction in an unseen house. Our work enables robots to complete navigation tasks accurately and efficiently without high-precision maps or hard-coded rules. Our work has a wide range of industrial applications, which could make today's robots more intelligent.

Risks and Ethical Concerns. In this paper, we use an openly published dataset, named Matterport3D [Chang et al., 2017] for our research. Collecting 3D house scenes require consent from house owners to avoid violation of privacy. There is a local government "general data protection regulation" that a researcher needs to obtain prior ethical committee approval before conducting the investigation. As the endto-end framework is vulnerable to abuse, security protection mechanisms should be incorporated to protect its embedded algorithm from hacking. If multiple languages are to be incorporated, then additional unified textual information protection techniques should be developed to alleviate natural language processing risk.

Acknowledgements

This work was supported in part by National Key R&D Program of China under Grant No. 2020AAA0109700, Nansha Key RD Program under Grant No.2022ZD014, Guangdong Outstanding Youth Fund (Grant No. 2021B1515020061) and the Fundamental Research Funds for the Central Universities, Sun Yat-sen University under Grant No. 22lgqb38.