Look Wide and Interpret Twice: Improving Performance on Interactive Instruction-following Tasks

1 Introduction

There is a growing interest in the community in making an embodied AI agent perform a complicated task following natural language directives. Recent studies of vision-language navigation tasks (VLN) have made significant progress [Anderson et al., 2018b; Fried et al., 2018; Zhu et al., 2020]. However, these studies consider navigation in static environments, where the action space is simplified, and there is no interaction with objects in the environment.

To consider more complex tasks, a benchmark named ALFRED was developed recently [Shridhar et al., 2020]. It requires an agent to accomplish a household task in interactive environments following given language directives. Compared with VLN, ALFRED is more challenging as the agent needs to (1) reason over a greater number of instructions and (2) predict actions from larger action space to perform a task in longer action horizons. The agent also needs to (3) localize the objects to manipulate by predicting the pixel-wise masks. Previous studies (e.g., [Shridhar et al., 2020]) employ a Seq2Seq model, which performs well on the VLN tasks [Ma et al., 2019]. However, it works poorly on ALFRED. Overall, existing methods only show limited performance; there is a huge gap with human performance.

In this paper, we propose a new method that leads to significant performance improvements. It is based on several ideas. Firstly, we propose to choose a single instruction to process at each timestep from the given series of instructions. This approach contrasts with previous methods that encode them into a single long sequence of word features and use soft attention to specify which instruction to consider at each timestep implicitly [Shridhar et al., 2020; Singh et al., 2020a]. Our method chooses individual instructions explicitly by learning to predict when the agent completes an instruction. This makes it possible to utilize constraints on parsing instructions, leading to a more accurate alignment of instructions and action prediction.

Secondly, we propose a two-stage approach to the interpretation of the selected instruction. In its first stage, the method interprets the instruction without using visual inputs from the environment, yielding a tentative prediction of an action-object sequence. In the second stage, the prediction is integrated with the visual inputs to predict the action to do and the object to manipulate. The tentative interpretation makes it clear to interact with what class of objects, contributing to an accurate selection of objects to interact with.

Moreover, we acquire multiple agent egocentric views of a scene as visual inputs and integrate them using a hierarchical attention mechanism. This allows the agent to have a wider field of views, leading to more accurate navigation. To be specific, converting each view into an object-centric representation, we integrate those for the multiple views into a single feature vector using hierarchical attention conditioned on the current instruction.

Besides, we propose a module for predicting precise pixelwise masks of objects to interact with, referred to as the mask decoder. It employs the object-centric representation of the center view, i.e., multiple object masks detected by the object detector. The module selects one of these candidate masks to specify the object to interact with. In the selection, selfattention is applied to the candidate masks to weight them; they are multiplied with the tentative prediction of the pairs of action and an object class and the detector's confidence scores for the candidate masks.

The experimental results show that the proposed method outperforms all the existing methods by a large margin and ranks first in the challenge leaderboard as of the time of submission. A preliminary version of the method won the ALFRED Challenge 2020 1 . The present version further improved the task success rate in unseen and seen environments to 8.37% and 29.16%, respectively, which are significantly higher than the previously published SOTA (0.39% and 3.98%, respectively) [Shridhar et al., 2020].

2 RelatedWork

2.1 Embodied Vision-Language Tasks

Many studies have been recently conducted on the problem of making an embodied AI agent follow natural language directives and accomplish the specified tasks in a threedimensional environment while properly interacting with it. Vision-language navigation (VLN) tasks have been the most extensively studied, which require an agent to follow navigation directions in an environment.

Several frameworks and datasets for simulating real-world environments have been developed to study the VLN tasks. The early ones lack photo-realism and/or natural language directions [Kempka et al., 2016; Kolve et al., 2017; Wu et al., 2018]. Recent studies consider perceptually-rich simulated environments and natural language navigation directions [Anderson et al., 2018b; Chen et al., 2019; Hermann et al., 2020]. In particular, since the release of the Room-toRoom (R2R) dataset [Anderson et al., 2018b] that is based on real imagery [Chang et al., 2017], VLN has attracted increasing attention, leading to the development of many methods [Fried et al., 2018; Wang et al., 2019; Ma et al., 2019; Tan et al., 2019; Majumdar et al., 2020].

Several variants of VLN tasks have been proposed. A study [Nguyen et al., 2019] allows the agent to communicate with an adviser using natural language to accomplish a given goal. In a study [Thomason et al., 2020], the agent placed in an environment attempts to find a specified object by communicating with a human by natural language dialog. A recent study [Suhr et al., 2019] proposes the interactive environments where users can collaborate with an agent by not only instructing it to complete tasks, but also acting alongside it. Another study [Krantz et al., 2020] introduces a continuous environment based on the R2R dataset that enables an agent to take more fine-grained navigation actions. A number of other embodied vision-language tasks have been proposed such as visual semantic planning [Zhu et al., 2017; Gordon et al., 2019] and embodied question answering [Das et al., 2018; Gordon et al., 2018; Wijmans et al., 2019; Puig et al., 2018].

1 The ALFRED Challenge 2020 https://askforalfred.com/EVAL

2.2 ExistingMethods for ALFRED

As mentioned earlier, ALFRED was developed to consider more complicated interactions with environments, which are missing in the above tasks, such as manipulating objects. Several methods for it have been proposed so far. A baseline method [Shridhar et al., 2020] employs a Seq2Seq model with an attention mechanism and a progress monitor [Ma et al., 2019], which is prior art for the VLN tasks. In [Singh et al., 2020a], a pre-trained Mask R-CNN is employed to generate object masks. In [Corona et al., 2020], a modular architecture is proposed to exploit the compositionality of instructions. These methods have brought about only modest performance improvements over the baseline. A concurrent study [Singh et al., 2020b] proposes a modular architecture design in which the prediction of actions and object masks are treated separately, as with ours. Although it achieves notable performance improvements, the study's ablation test indicates that the separation of the two is not the primary source of the improvements. Closely related to ALFRED, ALFWorld [Shridhar et al., 2021] has been recently proposed to combine TextWorld [Cˆot´e et al., 2018] and ALFRED for creating aligned environments, which enable transferring high-level policies learned in the text world to the embodied world.

3 ProposedMethod

The proposed model consists of three decoders (i.e., instruction, mask, and action decoders) with the modules extracting features from the inputs, i.e., the visual observations of the environment and the language directives. We first summarize ALFRED and then explain the components one by one.

3.1 Summary of ALFRED

ALFRED is built upon AI2Thor [Kolve et al., 2017], a simulation environment for embodied AI. An agent performs seven types of tasks in 120 indoor scenes that require interaction with 84 classes of objects, including 26 receptacle object classes. For each object class, there are multiple visual instances with different shapes, textures, and colors.

The dataset contains 8,055 expert demonstration episodes of task instances. They are sequences of actions, whose average length is 50, and they are used as a ground truth action sequence at training time. For each of them, language directives annotated by AMT workers are provided, which consist of a goal statement G and a set of step-by-step instructions, S1, . . . , SL. The alignment between each instruction and a segment of the action sequence is known. As multiple AMT workers annotate the same demonstrations, there are 25,743 language directives in total.

We wish to predict the sequence of agent's actions, given G and S1, . . . , SL of a task instance. There are two types of actions, navigation actions and manipulation actions. There are five navigation actions (e.g., MoveAhead and RotateRight) and seven manipulation actions (e.g., Pickup and ToggleOn). The manipulation actions accompany an object. The agent specifies it using a pixel-wise mask in the egocentric input image. Thus, the outputs are a sequence of actions with, if necessary, the object masks.

3.2 Feature Representations

Object-centric Visual Representations

Unlike previous studies [Shridhar et al., 2020; Singh et al., 2020a], we employ the object-centric representations of a scene [Devin et al., 2018], which are extracted from a pretrained object detector (i.e., Mask R-CNN [He et al., 2017]). It provides richer spatial information about the scene at a more fine-grained level and thus allows the agent to localize the target objects better. Moreover, we make the agent look wider by capturing the images of its surroundings, aiming to enhance its navigation ability.

Specifically, at timestep t, the agent obtains visual observations from K egocentric views. For each view k, we encode the visual observation by a bag of N object features, which are extracted the object detector. Every detected object is associated with a visual feature, a mask, and its confidence score. We project the visual feature into Rd with a linear layer, followed by a ReLU activation and dropout regularization [Srivastava et al., 2014] to obtain a single vector; thus, we get a set of N object features for view k, V k t = (vk t,1 , . . . , vk t,N ). We obtain V 1 t , . . . , V K t for all the views.

Language Representations

We encode the language directives as follows. We use an embedding layer initialized with pretrained GloVe [Pennington et al., 2014] vectors to embed each word of the L step-bystep instructions and the goal statement. For each instruction i(= 1, . . . ,L), the embedded feature sequence is inputted to a two-layer LSTM [Hochreiter and Schmidhuber, 1997], and its last hidden state is used as the feature si ∈ Rd of the instruction. We use the same LSTM for all the instructions with dropout regularization. We encode the goal statement G in the same manner using an LSTM with the same architecture different weights, obtaining hG ∈ Rd .

3.3 Instruction Decoder

Selecting Instructions

Previous studies [Shridhar et al., 2020; Singh et al., 2020a] employ a Seq2Seq model in which all the language directives are represented as a single sequence of word features, and soft attention is generated over it to specify the portion to deal with at each timestep. We think this method could fail to correctly segment instructions with time, even with the employment of progress monitoring [Ma et al., 2019]. This method does not use a few constraints on parsing the step-bystep instructions that they should be processed in the given order and when dealing with one of them, the other instructions, especially the future ones, will be of little importance.

We propose a simple method that can take the above constraints into account, which explicitly represents which instruction to consider at the current timestep t. The method introduces an integer variable mt(∈ [1, L]) storing the index of the instruction to deal with at t.

To update mt properly, we introduce a virtual action representing the completion of a single instruction, which we treat equally to the original twelve actions defined in ALFRED. Defining a new token COMPLETE to represent this virtual action, we augment each instruction's action sequence provided in the expert demonstrations always end with COMPLETE. At training time, we train the action decoder to predict the augmented sequences. At test time, the same decoder predicts an action at each timestep; if it predicts COMPLETE, this means completing the current instruction. The instruction index mt is updated as follows:

where pat-1 is the predicted probability distribution over all the actions at time t - 1, which will be explained in Sec. 3.4. The encoded feature smt of the selected instruction is used in all the subsequent components, as shown in Fig. 1.

(a) For action predictions

(b) For object predictions

Figure 2: An example illustrates how we reinitialize the hidden states of the two LSTMs in the instruction encoder by smt to predict actions and objects tentatively when mt = mt-1 + 1 (mt = 4).

Decoder Design

As explained earlier, our method employs a two-stage approach for interpreting the instructions. The instruction decoder (see Fig. 1) runs the first stage, where it interprets the instruction encoded as smt without any visual input. To be specific, it transforms smt into the sequence of action-object pairs without additional input. In this stage, objects mean the classes of objects.

As it is not based on visual inputs, the predicted actionobject sequence has to be tentative. The downstream components in the model (i.e., the mask decoder and the action decoder) interpret smt again, yielding the final prediction of an action-object sequence, which are grounded on the visual inputs. Our intention of this two-stage approach is to increase prediction accuracy; we expect that using a prior prediction of (action, object class) pairs helps more accurate grounding.

In fact, many instructions in the dataset, particularly those about interactions with objects, are sufficiently specific so that they are uniquely translated into (action, object class) sequences with a perfect accuracy, even without visual inputs. For instance, "Wash the mug in the sink" can be translated into (Put, Sink), (TurnOn, Faucet), (TurnOff, Faucet), (PickUp, Mug). However, this is not the case with navigation instructions. For instance, "Go straight to the sink" may be translated into a variable number of repetition of MoveAhead; it is also hard to translate "Walk into the drawers" when it requires to navigate to the left/right. Therefore, we separately deal with the manipulation actions and the navigation actions. In what follows, we first explain the common part and then the different parts.

Given the encoded feature smt of the selected instruction, the instruction decoder predicts the action and the object class to choose at t. To be precise, it outputs the probability distributions pia t (∈ RNa ) and pio t (∈ RNo ) over all the actions and the object classes, respectively; Na and No are the numbers of the actions and the object classes.

These probabilities pia t and pio t are predicted separately by two LSTMs in an autoregressive fashion. The two LSTMs are initialized whenever a new instruction is selected; to be precise, we reset their internal states as hia t-1 = hio t-1 = smt for t when we increment mt as mt = mt-1 + 1 ( see the example in Fig. 2). Then, pia t and pio t are predicted as follows:

where Wia ∈ RNa×d , bia ∈ RNa , Wio ∈ RNo×d , and bio ∈ RNo are learnable parameters; Ea maps the most likely action into the respective vectors according to the last predictions pia t-1 using a dictionary with Na ×d learnable parameters; Eo does the same for the object classes. The predicted pia t and pio t are transferred to the input of these LSTMs at the next timestep and also inputted to the downstream components, the mask decoder and the action decoder.

Now, as they do not need visual inputs, we can train the two LSTMs in a supervised fashion using the pairs of instructions and the corresponding ground truth action-object sequences. We denote this supervised loss, i.e., the sum of the losses for the two LSTMs, by Laux. Although it is independent of the environment and we can train the LSTMs offline, we simultaneously train them along with other components in the model by adding Laux to the overall loss. We think this contributes to better learning of instruction representation smt , which is also used by the mask decoder and the action decoder.

As mentioned above, we treat the navigation actions differently from the manipulation actions. There are three differences. First, we simplify the ground truth action sequence for the navigation actions if necessary. For instance, suppose an instruction "Turn left, go ahead to the counter and turn right" with a ground truth action sequence "RotateLeft, MoveAhead, MoveAhead, MoveAhead, MoveAhead, RotateRight". The repetition of MoveAhead reflects the environment and cannot be predicted without visual inputs. Thus, by eliminating the repeated actions, we convert the sequence into the minimum-length one, "RotateLeft, MoveAhead, RotateRight", and regard it as the ground truth sequence, training the instruction decoder. Second, as there is no accompanied object for the navigation actions, we use the object-class sequence "None, None, None" as the ground truth. Third, in the case of navigation actions, we do not transfer the outputs pia t and pio t to the mask decoder and the action decoder and instead feed constant (but learnable) vectors pia nav ∈ RNa and pio nav ∈ RNo to them. As the instruction decoder learns to predict the minimum-length action sequences as above, providing such predictions will be harmful for the action decoder. We avoid this by feeding pia nav and pio nav .

3.4 Action Decoder

The action decoder receives four inputs and predicts the action at t. The inputs are as follows: the encoded instruction smt , the output pia t and pio t of the instruction decoder 2 and aggregated feature vt of visual inputs, which will be described below.

Hierarchical Attention over Visual Features

As explained in Sec. 3.2, we use the multi-view object-centric representation of visual inputs. To be specific, we aggregate N ×K outputs of Mask R-CNN from K ego-centric images, obtaining a single vector vt. The Mask R-CNN outputs for view k(= 1, . . . ,K) are the visual features (vk t,1 , . . . , vk t,N ) and the confidence scores (ρkt,1 , . . . , ρkt,N ) of N detected objects.

2 These are replaced with pia nav and pia nav if argmax(pia t ) is not a manipulation action, as mention above.

To do this feature aggregation, we employ a hierarchical approach, where we first search for the objects relevant to the current instruction in each view and then merge the features over the views to a single feature vector. In the first step, we compute and apply soft-attentions over N objects for each view. To be specific, we compute attention weights αk s ∈ RN across vk t,1 , . . . , vk t,N guided by smt as

where W k s ∈ Rd×d is a learnable matrix, for k = 1, . . . ,K. We then apply the weights to the N visual features multiplied with their confidence scores for this view, yielding a single d-dimensional vector as

where ρkt,n is the confidence score associated with vk t,n .

In the second step, we merge the above features v1 t , . . . , vK t using gated-attention. We compute the weight αk g (∈ R) of view k(= 1, . . . ,K) guided by smt as

where Wg ∈ Rd×d is a learnable matrix. Finally, we apply the weights to {vk t }k=1,...,K to have the visual feature vt ∈ Rd as

As shown in the ablation test in the appendix of our Arxiv version, the performance drops significantly when replacing the above gated-attention by soft-attention, indicating the necessity for merging observations of different views, not selecting one of them.

Decoder Design

The decoder predicts the action at t from vt, smt , pia t and pio t . We employ an LSTM, which outputs the hidden state hat ∈ Rd at t from the previous state hat-1 along with the above four inputs as

where [; ] denotes concatenation operation. We initialize the LSTM by setting the initial hidden state ha0 to hG, the encoded feature of the goal statement; see Sec. 3.2. The updated state hat is fed into a fully-connected layer to yield the probabilities over the Na + 1 actions including COMPLETE as follows:

where Wa ∈ R(Na+1)×d and ba ∈ RNa+1 . We choose the action with the maximum probability for the predicted action. In the training of the model, we use cross entropy loss Laction computed between pat and the one-hot representation of the true action.

3.5 Mask Decoder

To predict the mask specifying an object to interact with, we utilize the object-centric representations V c t = (vc t,1 , . . . , vc t,N ) of the visual inputs of the central view (k = c). Namely, we have only to select one of the N detected objects. This enables more accurate specification of an object mask than predicting a class-agnostic binary mask as in the prior work [Shridhar et al., 2020].

To do this, we first apply simple self-attention to the visual features V c t , aiming at capturing the relation between objects in the central view. We employ the attention mechanism inside the light-weight Transformer with a single head proposed in [Nguyen et al., 2020] for this purpose, obtaining ¯ AV c t (V c t ) ∈ RN×d . We then apply linear transformation to ¯ AV c t (V c t ) using a single fully-connected layer having weight W ∈ RN×d and bias b ∈ Rd , with a residual connection as

where 1K is K-vector with all ones.

We then compute the probability pmt,n of selecting n-th object from the N candidates using the above self-attended object features along with other inputs smt , pia t , and pio t . We concatenate the latter three inputs into a vector gm t = [smt ; pia t ; pio t ] and then compute the probability as

where Wm ∈ R(d+Na+No)×d is a learnable matrix. We select the object mask with the highest probability (i.e., argmaxn=1,...,N(pmt,n )) at inference time. At training time, we first match the ground truth object mask with the object mask having the highest IoU. Then, we calculate the BCE loss Lmask between the two.

4 Experiments

4.1 Experimental Configuration

Dataset. We follow the standard procedure of ALFRED; 25,743 language directives over 8,055 expert demonstration episodes are split into the training, validation, and test sets. The latter two are further divided into two splits, called seen and unseen, depending on whether the scenes are included in the training set.

Evaluation metrics. Following [Shridhar et al., 2020], we report the standard metrics, i.e., the scores of Task Success Rate, denoted by Task and Goal Condition Success Rate, denoted by Goal-Cond. The Goal-Cond score is the ratio of goal conditions being completed at the end of an episode. The Task score is defined to be one if all the goal conditions are completed, and otherwise 0. Besides, each metric is accompanied by a path-length-weighted (PLW) score [Anderson et al., 2018a], which measures the agent's efficiency by penalizing scores with the length of the action sequence.

Implementation details. We use K = 5 views: the center view, up and down views with the elevation degrees of ±15◦ , and left and right views with the angles of ±90◦ . We employ a Mask R-CNN model with ResNet-50 backbone that receives a 300 × 300 image and outputs N = 32 object candidates. We train it before training the proposed model with

800K frames and corresponding instance segmentation masks collected by replaying the expert demonstrations of the training set. We set the feature dimensionality d = 512. We train the model using imitation learning on the expert demonstrations by minimizing the following loss:

We use the Adam optimizer with an initial learning rate of 10-3 , which is halved at epoch 5, 8, and 10, and a batch size of 32 for 15 epochs in total. We use a dropout with the dropout probability 0.2 for the both visual features and LSTM decoder hidden states.

4.2 Experimental Results

Table 1 shows the results. It is seen that our method shows significant improvement over the previous methods [Shridhar et al., 2020; Singh et al., 2020a; Singh et al., 2020b] on all metrics. Our method also achieves better PLW (path length weighted) scores in all the metrics (indicated in the parentheses), showing its efficiency. Notably, our method attains 8.37% success rate on the unseen test split, improving approximately 20 times compared with the published result in [Shridhar et al., 2020]. The higher success rate in the unseen scenes indicates its ability to generalize in novel environments. Detailed results for each of the seven task types are shown in the appendix in our Arxiv version 3 .

3 Our Arxiv version is available: https://arxiv.org/abs/2106.00596

The preliminary version of our method won an international competition, whose performance is lower than the present version. It differs in that (pia t , pio t ) are not forwarded to the mask decoder and the action decoder and the number of Mask R-CNN's outputs is set to N = 20. It is noted that even with a single view (i.e., K = 1), our model still outperforms [Shridhar et al., 2020; Singh et al., 2020a] in all the metrics.

Sub-goal success rate. Following [Shridhar et al., 2020], we evaluate the performance on individual sub-goals. Table 2 shows the results. It is seen that our method shows higher success rates in almost all of the sub-goal categories.

4.3 Ablation Study

We conduct an ablation test to validate the effectiveness of the components by incrementally adding each component to the proposed model. The results are shown in Table 3.

The model variants 1-4 use a single-view input (K = 1); they do not use multi-view inputs and the hierarchical attention method. Model 1 further discards the instruction decoder by replacing it with the soft-attention-based approach [Shridhar et al., 2020], which yields a different language feature satt at each timestep. Accordingly, pio t and pia t are not fed to the mask/action decoders; we use gm t = [satt; hat ]. These changes will make the method almost unworkable. Model 2 retains only the instruction selection module, yielding smt . It performs much better than Model 1. Model 3 has the instruction decoder, which feeds pio t and pia t to the subsequent decoders. It performs better than Model 2 by a large margin, showing the effectiveness of the two-stage method.

Model 4 replaces the mask decoder with the counterpart of the baseline method [Shridhar et al., 2020], which upsamples a concatenated vector [gm t ; vt] by deconvolution layers. This change results in inaccurate mask prediction, yielding a considerable performance drop. Model 5 is the full model. The difference from Model 3 is the use of multi-view inputs with the hierarchical attention mechanism. It contributes to a notable performance improvement, validating its effectiveness.

4.4 Qualitative Results

Entire Task Completion

Figure 3 shows the visualization of how the agent completes one of the seven types of tasks. These are the results for the unseen environment of the validation set. Each panel shows the agent's center view with the predicted action and object mask (if existing) at different time-steps. See the appendix in our Arxiv version for more results.

Mask Prediction for Sub-goal Completion

Figure 4 shows an example of the mask prediction by the baseline [Shridhar et al., 2020] and the proposed method. It shows our method can predict a more accurate object mask when performing Slice sub-goal. More examples are shown in the appendix in our Arxiv version. Overall, our method

(b) Ours shows better results, especially for difficult sub-goals like Pickup, Put, and Clean, for which a target object needs to be chosen from a wide range of candidates.

Figure 4: The prediction masks generated by Shridhar et al.and our method where the agents are moved to the same location to accomplish Slice sub-goal.

5 Conclusion

This paper has presented a new method for interactive instruction following tasks and applied it to ALFRED. The method is built upon several new ideas, including the explicit selection of one of the provided instructions, the two-stage approach to the interpretation of each instruction (i.e., the instruction decoder), the employment of the object-centric representation of visual inputs obtained by hierarchical attention from multiple surrounding views (i.e., the action decoder), and the precise specification of objects to interact with based on the object-centric representation (i.e., the mask decoder). The experimental results have shown that the proposed method achieves superior performances in both seen and unseen environments compared with all the existing methods. We believe this study provides a useful baseline framework for future studies.

Acknowledgments

This work was partly supported by JSPS KAKENHI Grant Number 20H05952 and JP19H01110.