MuEP: A Multimodal Benchmark for Embodied Planning with Foundation Models

1 Introduction

With the tremendous success of ChatGPT [Wu et al., 2023a], foundation models trained on web-scale data using self-supervision have garnered increasing attention from the community. Foundation models with the great emergent abilities have demonstrated significant performance improvements when adopted for downstream tasks, including fluent interaction [Touvron et al., 2023; OpenAI, 2023], sophisticated literary works creation [Waisberg et al., 2024], image captioning [Alayrac et al., 2022], and code generation [Chen et al., 2021]. These advancements hold great promise for enhancing the reasoning and planning abilities of advanced embodied agents [Yang et al., 2023b; Yang et al., 2023c; Brohan et al., 2022]. Motivated by this, numerous recent studies have utilized foundation models, such as large language models (LLMs) and large multimodal models (LMMs), across a range of tasks as embodied agents, including environmental grounding [Ahn et al., 2022; Driess et al., 2023], vision-language navigation [Brohan et al., 2022; Mu et al., 2023], and task planning [Yao et al., 2023;

∗Corresponding author. This work was done during the internship at JD Explore. The source code and datasets are available: https://github.com/kanxueli/MuEP.

Shinn et al., 2023]. Figure 1 provides an example of embodied planning driven by different foundation models with vision-language and text-only observations.

Despite the increasing number of recent studies on embodied AI with foundation models, a significant challenge persists due to the absence of a comprehensive benchmark for evaluating embodied planning abilities in complex and multimodal scenarios. Many existing benchmarks, including [Wijmans et al., 2019; Ebert et al., 2021; Brohan et al., 2022; Wu et al., 2023b], suffer from limitations such as limited diversity, single modality, and coarse metrics. These drawbacks make them insufficient for a thorough and nuanced evaluation of the capabilities of embodied agents. Moreover, collecting large-scale datasets with multi-round interactions between embodied agents and real-world environments is expensive and time-consuming. As a result, existing embodied datasets are typically gathered on a small scale, as exemplified by the TEACh project, which collected 4,365 crowdsourced data samples at a total cost of $105K [Padmakumar et al., 2022]. To address this challenge, a cheap and scalable solution is automatic data generation. For exampl e, several studies have undertaken data generation and annotation using LLMs and expanded existing datasets [Wang et al., 2022; Xu et al., 2023]. Inspired by this, there are also initial attempts in the field of embodied intelligence such as TaPA [Wu et al., 2023b] and RoboGen [Wang et al., 2023]. However, TaPA may suffer from the issue of illusion in finetuned models due to the mismatch between detected objects and ground truth, while RoboGen utilizes popular generative models (e.g., Midjourney and Zero-1-to-3) to synthesize diverse tasks and simulated environments but struggles for a stable generation of complex and contextually consistent training scenarios.

In this paper, we present MuEP, a comprehensive multimodal benchmark for embodied planning. The main dataset construction pipeline is shown in Figure 2, which includes metadata collection, task generation, task filtering, and multimodal demonstration. Specifically, we first collect a small subset of metadata, including scene/task descriptions from the ALFworld [Shridhar et al., 2021] simulator, where the Planning Domain Definition Language (PDDL) [Aeronautiques et al., 1998] and TextWorld [Cˆot´e et al., 2019] are utilized to align the text with the agent's egocentric visual observations during exploration. We then employ LLMs to generate different tasks via the in-context learning strategy, i.e., the collected examples as the context are provided to be part of the prompt. By doing this, we can easily generate a large number of different tasks, with those imperfect samples removed according to several heuristic filtering criteria. Lastly, the rule-based planner [Hoffmann and Nebel, 2001] is utilized to generate multimodal data at each demonstration episode. Overall, the MuEP benchmark currently encompasses 14,927 expert demonstration episodes, spreading across 108 varied household scenes and corresponding to 176,593 image-text pairs. Each scene includes diverse objects with variations in shapes, textures, and colors, enriching scene settings and task complexity.

We adopt five distinct metrics to thoroughly evaluate popular foundation models on the proposed MuEP. These include three commonly used metrics: Success Rate (SR), Interaction Step (IS), and Goal-Condition Success (GCS), as well as two additional carefully designed metrics, namely Language Compliance (LC) and Reasoning Disorientation Index (RDI). Notably, the newly introduced metrics specifically target the agent's capabilities in generating compliant structured actions and effectively replanning while avoiding cognitive pitfalls, respectively.

We consider eight representative and open-sourced foundation models for embodied planning, employing the parameter-efficient fine-tuning (PEFT) strategies [Mangrulkar et al., 2022; Dettmers et al., 2023]. As shown in Figure 3, these models include LLMs (i.e., LLaMA-2 [Touvron et al., 2023], OPT [Zhang et al., 2022], and Bloomz [Muennighoff et al., 2022]) and LMMs (i.e., BLIP2 [Li et al., 2023], InstructBLIP [Dai et al., 2023], LLaMA-Adapter-v2 [Gao et al., 2023], MiniGPT-4[Zhu et al., 2023], and LLaVA [Liu et al., 2023]). Experimental results show that: 1) Foundation models based on textual representations of environments usually outperform their visual counterparts, suggesting a gap in embodied planning abilities with multimodal observations; 2) Structured control language is crucial for successful embodied task execution; 3) Agents necessitate increased interaction steps to learn and adapt to new dynamic scenes effectively.

In summary, our contribution is twofold: 1) We propose a multimodal benchmark for embodied planning. 2) We evaluate popular LLMs and LMMs on this benchmark. We expect the research presented in this paper to benefit the development of the embodied AI community.

2 RelatedWork

Embodied AI with Foundation Models. Emerging trends in AI research show the extension of foundation models from basic language tasks to embodied decision-making [Vemprala et al., 2023; Yang et al., 2023a]. Recent research utilizes LLMs for grounding in embodied planning tasks within interactive environments [Ahn et al., 2022; Huang et al., 2022; Huang et al., 2023; Lin et al., 2023]. Some approaches like React [Yao et al., 2023] and Reflexion [Shinn et al., 2023] integrate chain-of-thought [Wei et al., 2022] into embodied agents, enabling them to formulate autonomous problemsolving procedures. Many researchers also propose to refine agent capabilities of reasoning and decision-making in embodied environments through fine-tuning with pre-collected data [Xiang et al., 2023; Mu et al., 2023]. Meanwhile, the advancement in LMMs has been pivotal in more integrated and sophisticated systems, including BLIP2 [Li et al., 2023], InstructBLIP [Dai et al., 2023], LLaMA Adapter v2 [Gao et al., 2023], MiniGPT-4 [Zhu et al., 2023], and LLaVA [Liu et al., 2023]. However, there is still an urgent need for research to test foundation-model-based agents in embodied planning. This paper focuses on testing the ability of opensourced foundation models on embodied multimodal tasks.

Embodied Planning Benchmark. Over the past several years, the emergence of embodied interactive environments has played a pivotal role in the evaluation of Embodied AI [Kolve et al., 2017; Gan et al., 2020], paving the way for groundbreaking approaches in visual navigation [Ramakrishnan et al., 2020; Gan et al., 2021], visual-language tasks [Anderson et al., 2018a; Deitke et al., 2022], and embodied question answering [Das et al., 2018a; Zhou et al., 2023]. ALFWorld [Shridhar et al., 2021] combines the interactive textbased game engine TextWorld [Cˆot´e et al., 2019] with ALFRED [Shridhar et al., 2020], a dataset for vision-language tasks in embodied environments, creating a platform for developing AI agents capable of understanding and interacting in complex scenarios using both text and visuals. In this paper, we utilized ALFWorld scenes for testing and its annotated tasks to generate new ones with LLM. While there is an ongoing trend towards the enlargement and enhancement of these datasets, the substantial cost and resource investment required to adapt them for embodied tasks continues to present a significant obstacle. For instance, RT-1 [Brohan et al., 2022] utilized 13 robots over 17 months to collect approximately 130K episodes of data. Recent studies aim to bridge this gap by leveraging LLMs [Bubeck et al., 2023] to generate training data [Wang et al., 2022; Xu et al., 2023], where the research most closely related to ours is TaPA [Wu et al., 2023b]. In contrast to our approach of directly acquiring metadata from the simulator, TaPA utilizes an object detector to generate lists of objects from images captured within the simulator, a process prone to inaccuracies. Moreover, TaPA relies on the evaluation of action plans generated by task planners (LLMs) by 30 volunteers, deviating from an automated and objective evaluation system. Additionally, existing benchmarks mainly use Success Rate (SR) and Interaction Step (IS) as primary evaluation metrics [Shridhar et al., 2021; Deitke et al., 2022], while these coarse metrics fall short in providing a systematic evaluation of emerging embodied agents. To address this limitation, we further introduce two novel fine-grained metrics: Language Compliance (LC) and Reasoning Disorientation Index (RDI).

3 The MuEP Dataset

This section presents the details of MuEP.

3.1 Dataset Construction

The main dataset construction pipeline consists of four stages: metadata collection, task generation, task filtering, and multimodal demonstration, as follows.

Metadata Collection. We first extract metadata from the ALFWorld [Shridhar et al., 2021], including scene information, available tasks, and item attributes and states, as shown in Figure 2-Step 1. We use PDDL, a standard planning language, and the TextWorld engine to programmatically align the agent's egocentric visual observations with accurate textual descriptions during exploration. PDDL encodes the textual representations of each visual scene in our dataset, similar to the approach used in ALFWorld. TextWorld then generates corresponding textual environments from this PDDL encoding, ensuring that textual and visual information are consistently aligned. Furthermore, we also gather information about the inventory, attributes, and states of objects within the scene. This information acts as constraints to reduce hallucinatory outputs and the occurrence of invalid tasks. In contrast to TaPA [Wu et al., 2023b], which relies on object detection/segmentation, our method directly obtains error-free metadata from Ai2-THOR's [Kolve et al., 2017] configuration. This approach not only streamlines the process but also significantly enhances the accuracy and quality of the generated tasks.

Task Generation. For each piece of metadata collected in the previous step, we explore LLM's in-context learning capability for task generation. Considering the impressive performance of GPT, we incorporate the text-davinci-003 model from OpenAI. We adopt an example-based prompt to enhance task generation, which involves querying with specific examples, including scene information, object dictionary, the target number of tasks, and 30 manually crafted task examples along with their types. The object dictionary includes operational attributes of items (e.g., pickupable, receptacle, and cookable), along with quantity details. These attributes form constraint conditions during task generation, ensuring relevance and feasibility. Specifically, we prompt LLM to generate different tasks that can be executed within that particular scene. To account for different room types linked to distinct task types, we have created four distinct prompt templates. Through iterative queries with diverse scenes and object dictionaries, we can generate an extensive array of tasks that capture relevant object attributes and scene characteristics.

Task Filtering. While prompt templates have proven effective, occasionally, some generated tasks may not meet practical standards. We thus introduce a four-fold criterion to remove imperfect tasks: 1) the task is incompatible with the room scene, e.g., the "Heat & Place" category of tasks will be filtered when meeting in an environment that contains no items used for heating, such as microwaves; 2) the task involves invalid items; 3) the task involves inadequate item attributes, e.g., "put a bread in book" is invalid because, in the Ai2-THOR environment, books are not considered receptacles - they lack the necessary attribute for holding objects; 4) the task is already completed in the scene. For instance, if a "keychain" is already placed on the "shelf" in the current scene, the task "put a keychain on the shelf" will be removed.

Multimodal Demonstration. We utilize a parallel textvisual environment to construct the dataset, which allows the simultaneous acquisition of visual observations from Ai2THOR and corresponding textual observations in TextWorld, as depicted in Figure 2-Step 4. PDDL serves as a bridge between the visual and the textual environments. We employ a rule-based planner [Hoffmann and Nebel, 2001] to generate action sequences. Inaccessible to the agent during inference, the planner relies on metadata and encodes the environment as fully observable states that contain perfect knowledge of world dynamics. Figure 4 illustrates the data format as {Scene, Instruction, Action Sequence, Textual State, Visual State}. "Scene" details the agent's current environment, "Instruction" specifies the task, "Action Sequence" records the planner's steps, and "Visual State" and "Textual State" provide visual and textual representations of the post-action environment, respectively. This design is wellsuited for training both LLM and LMM agents because it enables multi-round interactions and extends task horizons in dynamic scenes.

3.2 Dataset Metrics and Statistics

This subsection first explains the evaluation metrics and then summarizes the statistics of MuEP by comparing with previous embodied datasets.

Evaluation Metrics. To form a more nuanced evaluation, MuEP provides five types of metrics, including three commonly used metrics success rate, interaction step, and goalcondition success, and two additional designed metrics, language compliance, and reasoning disorientation index. All metrics are complementary to each other and can comprehensively test the capabilities of foundation models.

Success Rate (SR) refers to the ratio of the number of successful plays and the total plays. Task completion requires two vital conditions: the target location and the target state. For example, the task "placing a heated potato on a table" is deemed successful if the potato is in a heated state and located on the table.

Interaction Step (IS) measures the number of steps an agent interacts with the environment to complete a task. This metric evaluates the efficiency of an agent's interactions in achieving the task objectives. Fewer interaction steps indicate a better capability for reasoning and planning.

Goal-Condition Success (GCS) measures how close an agent is to entirely completing a task by counting the completion of all the sub-goals. For instance, the task "placing a heated potato on a table" includes three sub-goals: finding the potato, heating it, and placing it on the table. If the agent only puts the potato on the table without heating it, the task's GCS score is 2/3 or about 66%, indicating partial success.

Language Compliance (LC) was designed to measure the structuredness and compliance of the action generated by foundation models, providing a crucial tool for assessing their capability in delivering effective and syntactically correct action commands. Foundation models exhibit fluid and free expressive capabilities, enhancing human-machine interaction with naturalness. Nevertheless, their capacity to generate strictly formatted language in embodied tasks remains unclear. To address this issue, we introduce the 'Language Compliance' metric. The metric LC is given as LC = aˆ/a∗ , where aˆ is the number of valid actions and a∗ is the total number of actions.

Reasoning Disorientation Index (RDI) reflects whether an agent persistently takes repetitive actions in response to the same environmental feedback. Specifically, the RDI quantifies the tendency of an agent to resort to repetitive actions or iterative, ineffective solutions when faced with consistent environmental feedback. The RDI is calculated by observing the agent's actions in specific tasks and checking for repetitive or cyclical patterns. The metric RDI is given as RDI = prdi/pfail, where prdi is the number of planning with RDI occurrences and pfail is the total failed tasks. A higher RDI value indicates a higher likelihood of the agent encountering cognitive traps, necessitating enhanced reasoning strategies.

Dataset Summary. MuEP comprises 15,247 demonstration episodes, corresponding to 176,593 image-text pairs, where Figure 4 illustrates the multimodal data. A comparison between MuEP and other embodied planning datasets is shown in Table 1. MuEP outshines others with its extensive scale of over 170k+ instances, robust generative task support, high-quality virtual scene interactions, and superior metrics for dynamic scene evaluation. Furthermore, MuEP can be used to evaluate embodied agents using both LLMs and LMMs.

4 Experiments

This section details and analyzes our experiments.

4.1 Baseline Choices

To assess the performance of mainstream open-source large models in embodied tasks, we fine-tuned and analyzed foundation models on the MuEP dataset. For a fair comparison, all models are at around the 7B magnitude.

LLMs: Our first choice is the LLAMA-2-7B [Touvron et al., 2023] model from the LLAMA family, known for its range of models with 7B to 65B parameters and its high quality. This selection balances model performance with resource efficiency, aiming to reduce training time and resource consumption. Further exploring the LLM domain, we included the OPT-6.7B [Zhang et al., 2022] model, comparable in size to LLAMA-7B, to assess the impacts of using different foundational models. Additionally, the Bloomz-3B [Muennighoff et al., 2022] model was trained to explore the performance of models with varying parameter magnitudes.

LMMs: We selected five representative LMMs, including BLIP2 [Li et al., 2023], InstructBLIP [Dai et al., 2023], LLaMA Adapter V2 [Gao et al., 2023], MiniGPT-4 [Zhu et al., 2023], and LLaVa [Liu et al., 2023]. These models integrate pre-trained image encoders with large language models and underwent extensive pre-training on massive image-text pairs. This process aligned text-image characteristics, setting a strong foundation for subsequent finetuning. All models were then fine-tuned on diverse, smallscale instruction-following datasets, enhancing their basic instruction-following capabilities. Table 3 includes detailed configurations of all baseline models in our experiments.

4.2 Experimental and Implementation Details

Experimental Settings. We conduct extensive experiments on 134 #Unseen and 140 #Seen test tasks from the original ALFworld to evaluate the performance of foundation models and also to compare with previous works. #Seen tasks entail executing in familiar rooms from training, yet with different instantiations of object locations, quantities, and visual appearances, whereas #Unseen tasks involve new tasks in entirely unknown rooms, characterized by different objects and scene layouts not encountered during training. Each model is evaluated under a constraint of a maximum of 30 steps per task.

Implementation Details. The overall model fine-tuning framework is illustrated in Figure 5. For LMMs, core components like the ViT, Q-Former, and LLM itself were frozen to maintain stability, with the projection layer fine-tuned. Notably, the LlaMa adapter v2 model only underwent finetuning on the bias of adapter layers [Gao et al., 2023]. LMMs incorporate two main types of inputs: text instructions (i.e., task directives and historical actions) and image observations. Specifically, the historical actions provide crucial contextual information for understanding the extent of task completion and aids in the agent's subsequent reasoning process. Concurrently, the visual inputs offer a first-person perspective of the agent's external environments. If not otherwise stated, we always adhere to the prompt templates used in the original works during fine-tuning. As illustrated on the right side of Figure 5, we employed Q-LoRA [Dettmers et al., 2023] to fine-tune all LLMs efficiently. LLM's input consisted exclusively of text-based instructions and environmental observations. Despite representing different modalities, visual and textual observations were treated as equivalent in our dataset. All experiments were accelerated by four Tesla V100 GPUs.

4.3 Results and Discussions

As shown in the main results Table 3 and Figure 3, for LLM agents, OPT achieved the best performance in terms of SR (i.e., 79.10%). However, OPT and Bloomz models showed higher tendencies toward reasoning disorientation, as reflected by their RDI scores. Conversely, LlaMA2-7B demonstrated the best performance in IS and RDI, but its lower SR suggested limited stability in task completion even compared with Bloomz-3B. Regarding LMM agents, MiniGPT-4 outperformed others in both #Seen and #Unseen scenarios in terms of the SR. However, it required more interaction steps to complete tasks, suggesting a higher frequency of task completion but with increased interactions. In contrast, InstructBLIP, while slightly underperforming in SR, had the most outstanding IS metric, indicating higher efficiency in task execution. Overall, although LLMs demonstrate significantly superior performance in terms of SR, there is notable potential for improvement in terms of LC.

To offer insights into the capacities of the model for each task category, a more comprehensive evaluation per task type is presented in Tables 2 and 4. This comprehensive comparison not only pinpointed specific weaknesses in the model's capabilities but also shed light on the difficulty levels associated with all tasks. For instance, we find that the 'Pick two & place' task was particularly challenging for all models, with the highest success rate of 52.94% achieved. Additionally, the success rate of LlaMA-2-7B's in the task of 'Pick two & place' was 0% (#Seen tasks), primarily due to its extremely low LC score of 4.17%, highlighting the importance of improving LlaMA-2-7B's in terms of the LC ability.

Several key observations regarding the performance disparities between LLMs and LMMs are as follows: 1) Foundation models based on textual representations often tend to outperform their visual counterparts. This is mainly because embodied agents based on LLMs directly leverage PDDL, while there are still gaps for current LMMs to capture comparable information from visual representations. 2) Generating formatted control language is of great importance for embodied task completion. The LC metric highlights the significance of the formatted control language. Our results suggest that higher LC values in LLMs correspond to increased success rates (SR) and Goal-Condition Success (GCS). 3) New environments present heightened challenges for task planning. The IS metric reflects the number of interaction steps. Our results suggest that agents operating in "Unseen" scenarios (unfamiliar environments) typically require significantly more steps to complete tasks compared to those in "Seen" scenarios. 4) According to Table 3, LLM agents demonstrated significantly better performance in #Unseen scenarios, whereas LMM agents exhibited consistent performance across both scenarios. We owe this observation to variations, such as the difference in object location and appearance. They would not be reflected in the current textual representation of scene information, yet they can be noticed in visual information. Therefore, the LLM-based agents, when meeting tasks in seen scenes, tend to output actions they have already learned, even though the generated control languages are ineffective after multiple action failures, leading to worse RDI and LC metrics and further affecting the SR metric.

5 Conclusions and FutureWork

We introduce MuEP, a comprehensive multimodal benchmark for embodied planning that tackles the challenges posed by diversity, modality limitations, and the coarse metrics in existing benchmarks. Our evaluation of recent foundation models highlights the superior performance of text-based models over their visual or multimodal counterparts in tasks related to embodied planning. Moreover, our newly proposed metrics, LC and RDI, offer valuable insights into agents' action generation and cognitive planning capabilities. These insights serve as crucial guidance for future research in multimodal embodied agent planning. We envision MuEP as a pivotal benchmark, contributing to the advancement of embodied agents in higher-level reasoning and planning abilities. In the future, we plan to expand the benchmark in terms of scale, diversity, and complexity. Additionally, we aim to investigate the disparity between textual and visual modalities in embodied planning tasks.

Acknowledgments

This work was partially supported by the Major Science and Technology Innovation 2030 "New Generation Artificial Intelligence" key project (No. 2021ZD0111700), National Natural Science Foundation of China under Grant 62172354, Yunling Scholar Talent Program of Yunnan Province under Grant K264202230207, and Yunnan Provincial Key Laboratory of Media Convergence Open Project Funding Project Number: 220235203.