Odyssey : Empowering Minecraft Agents with Open-World Skills

1 Introduction

Developing autonomous agents capable of performing openworld tasks represents a significant milestone towards achieving artificial general intelligence [Reed et al., 2022; Driess et al., 2023]. These open-world tasks necessitate that agents interact with complex and dynamic environments, make decisions based on incomplete information, and adapt to unexpected events. Early reinforcement learning agents [Tessler et al., 2017; Oh et al., 2017] have demonstrated limited knowledge in such open-world setting. Furthermore, these agents often struggle with long-term planning, which is crucial for the fulfillment of intricate goals. Recent breakthrough of Large Language Models (LLMs) [Hu et al., 2021; Achiam et al., 2023; Touvron et al., 2023] have shown the potential to revolutionize various fields such as healthcare [Zhang et al., 2023b; Yang et al., 2024b], robotics [Huang et al., 2022; Singh et al., 2023], and web services [Deng et al., 2023; Iong et al., 2024], attributed to its capability on endowing agents with expansive knowledge and sophisticated planning akin to human reasoning [Wei et al., 2022; Wang et al., 2024a; Liang et al., 2023]. However, the development of LLMs in open-world tasks remains challenging due to the need for well-defined environments and measurable benchmarks [Wang et al., 2023a; Qin et al., 2023].

∗Corresponding author.

The popular Minecraft game features a vast and diverse world with various biomes, terrains, and resources, making it an ideal testbed for evaluating the capabilities of autonomous agents in the open-world setting. To facilitate the development of generalist agents in this setting, MineRL [Guss et al., 2019] and MineDojo [Fan et al., 2022] introduced simulation benchmarks built upon the sandbox Minecraft environment. The seminal work, Voyager [Wang et al., 2023a], proposed an LLM-based agent to drive exploration in Minecraft. Subsequently, there has been a surge of efforts to leverage the superior performance of LLMs to extend the capabilities of such Minecraft agents [Wang et al., 2023b; Zhou et al., 2024; Wang et al., 2023c; Qin et al., 2023]. Despite recent advancements, existing works mainly focus on solving basic programmatic tasks, often considering the ObtainDiamond task as the ultimate challenge [Guss et al., 2019]. Basic programmatic tasks refer to those constrained by the explicit dependencies following the Minecraft tech-tree, such as collecting materials and crafting tools. Such tasks inherently only assess the ability of LLMs to prioritize crafting steps within a limited task space, rather than their potential for complicated and diverse solutions. This limitation arises from the narrowly defined set of actions available to agents (e.g., mouse and keyboard), which necessitates learning skills from scratch. Since Minecraft is fundamentally resource-based, an agent must first learn to collect adequate resources and tools to engage in creative play, which limits the exploration of diverse gameplay options. Moreover, methods like Voyager [Wang et al., 2023a] heavily rely on the powerful GPT-4 for high-quality solutions, imposing a substantial cost burden on researchers who prefer open-source models.

In this work, we introduce ODYSSEY 1 , a novel framework that equips LLM-based agents with advanced open-world skills, enabling efficient interaction and exploration within the Minecraft environment. ODYSSEY allows agents to move beyond basic programmatic tasks and focus more on complex open-world challenges. As shown in Fig. 1, ODYSSEY comprises three key contributions:

1. We develop an LLM-based interactive agent with an openworld skill library, encompassing 40 primitive skills that serve as underlying interfaces and 183 compositional skills tailored for complex and diverse tasks in an openworld setting. A recursive method improves skill execution by checking prerequisites. The ODYSSEY agent consists of a planner for goal decomposition, an actor for skill retrieval and subgoal execution, and a critic for feedback and strategy refinement.
2. We fine-tune the LLaMA-3 model [Touvron et al., 2023] for Minecraft agents using a comprehensive questionanswering dataset. This involves generating a largescale training dataset with 390k+ instruction entries from Minecraft Wikis, fine-tuning various sizes of the LLaMA3 models using LoRA [Hu et al., 2021], and evaluating them with a custom multiple-choice dataset.

1 The Odyssey is a great ancient Greek epic poem attributed to Homer, which is now often used metaphorically to describe a long adventurous journey (Oxford English Dictionary).

1. We introduce a new agent capability benchmark to evaluate different aspects of agent performance in Minecraft, including the long-term planning task, the dynamicimmediate planning task, and the autonomous exploration task. Extensive experiments demonstrate that the proposed ODYSSEY framework provides a robust measure of agent effectiveness, showcasing the practical advantages of our framework using the open-source models.

It is worth noting that our focus is not to design a new LLMbased agent architecture. Instead, this work aims to provide a comprehensive framework for developing and evaluating autonomous agents in open-world environments, enabling them to explore the vast and diverse Minecraft world.

2 Open-World Skill-based Interactive Agent

ODYSSEY develops an LLM-based interactive agent with an open-world skill library, aiming to enhance the efficiency and adaptability of agents in complex Minecraft environments. The skill library comprises 40 primitive skills and 183 compositional skills, while the LLM-based agent employs a planner-actor-critic architecture to facilitate task decomposition, skill execution, and performance feedback. The architecture of the interactive agent is depicted in Fig. 2. Full skill and prompt details used in the LLM-based interactive agent are given in Appendix C.

2.1 Open-World Skill Library

Primitive skills. ODYSSEY encompass a series of underlying interfaces on top of Mineflayer JavaScript APIs [PrismarineJS, 2023], divided into two main categories: 32 operational skills and 8 spatial skills. This suite of skills exceeds the 18 primitive skills (all are operational skills) delineated in Voyager [Wang et al., 2023a]. Operational skills serve as foundational interfaces with parameterized input, such as mine(·) for material collection and craft(·) for tool crafting. Additionally, we pioneer 8 spatial skills that Voyager [Wang et al., 2023a] lacks, allowing for environmental interactions based on the agent coordinates. Given that our work is conducted within a text-based Minecraft environment [Wang et al., 2023a; Fan et al., 2022], spatial skills are crucial for handling tasks that require precise positioning and orientation, especially in the absence of visual input.

Compositional skills. We encapsulate primitive skills into higher-level ones, functioning to address a variety of basic programmatic tasks, such as mineDiamond and craftIronPickaxe. ODYSSEY classifies 183 compositional skills into types like mineX, craftX, plantX, breedX, cookX, etc. We use a recursive method to construct the skill library, simplifying complex task decomposition by ensuring prerequisites are met before skill execution. Taking mineDiamond as an example, if the agent lacks an iron pickaxe, it will recursively execute craftIronPickaxe. This indicates that our program internally manages the construction and execution order of skills through its recursive method, thereby avoiding the need for the agent to engage in additional planning.

To facilitate efficient retrieval of skills in the skill library, we first generate a description for each skill by calling the LLM and using the complete program code as a prompt. We then employ Sentence Transformer [Reimers and Gurevych, 2019] to encode the skill description. This method transforms text information into vector representations, facilitating semantic retrieval and enabling the agent to find the most relevant skill description based on the context provided.

2.2 Planner-Actor-Critic Architecture

LLM Planner. The LLM Planner is responsible for developing a comprehensive plan, facilitating efficient exploration through long-term goal decomposition. The LLM Planner breaks down high-level goals into specific low-level subgoals, each corresponding to a particular skill outlined in Sec. 2.1. By addressing each subgoal in the plan, the ultimate goal can be progressively achieved. The input prompt to the planner consists of several components: (1) Ultimate goals and behavioral constraints. For example, "My ultimate goal is to ... Propose the current task only when you ensure that you have all the necessary dependent items in inventory". (2) States of the agent. This reflects the interaction between the agent and environment, such as hunger and health values, position and nearby entities, etc. (3) Achievements of the agent. This includes the current inventory and unlocked equipment, as well as previously successful and failed tasks.

LLM Actor. In the execution phase, the LLM actor is invoked to sequentially execute the subgoals generated by the LLM planner within the Minecraft environment. This process utilizes the open-world skill library to achieve these subgoals. The mapping from high-level subgoals to executable skill code is accomplished through query context encoding and skill similarity retrieval. This process includes: (1) Query context. The text-based subgoals generated by the LLM planner are encoded by Sentence Transformer [Reimers and Gurevych, 2019] to vector representations as the query context. (2) Similarity matching. The vector similarity between the query context and the skill descriptions in the skill library is computed to determine semantic closeness. (3) Skill selection. The top-5 relevant skills with the highest scores are identified, and the actor selects the most appropriate code for execution within the environment based on their descriptions.

LLM Critic. During action execution, it is critical for an agent to document its experiences, especially noting successful outcomes and failure points. This is crucial in open-world planning to establish a feedback-informed system, which corrects initial plan discrepancies that can cause execution errors. For instance, achieving the animal breeding goal requires prerequisite crops for feed. The LLM critic can assess action effectiveness by comparing expected and actual outcomes, providing insights for refining future strategies. We categorize feedback into three types: (1) Execution feedback. This captures the progress of skill execution. For example, "No hoe in inventory. Craft a hoe first!" not only highlights the reason for failure in hoeing farmland but also provides a guideline to address this problem. (2) Selfvalidation. By presenting inventory changes post-action to the LLM critic, we empower it to validate whether the skill has achieved its subgoal, eliminating the need for manual checks. (3) Self-reflection. Simply confirming the completion of a subgoal is often inadequate for correcting planning errors. The LLM critic also serves as an analyst, deducing the cause of task failure by evaluating the current state of the agent and its environment. It then offers a critique, suggesting a more efficient strategy for task completion.

3 Fine-tune Minecraft LLM

To improve agent performance in Minecraft, we fine-tune the LLaMA-3 model [Touvron et al., 2023] using a large-scale Question-Answering (Q&A) dataset with 390k+ instruction entries sourced from the Minecraft Wiki. ODYSSEY presents an effective procedure for converting a foundation model into a domain-specific model, which involves dataset generation, model fine-tuning, and model evaluation. The detailed descriptions can be found in Appendix D.

Dataset Generation. We develop a GPT-assisted method to generate a Minecraft instruction dataset. First, we crawl relevant content from the Minecraft Wiki, excluding nonessential sections like history. The collected data is then categorized and separated into different files based on their content type. Then we use GPT-3.5-Turbo [OpenAI, 2023] with different customized prompts to generate diverse Q&A pairs. These Q&A pairs are categorized into four types based on the nature of the answers: short, normal, long, and boolean, yielding 390k+ entries. In contrast, the Wiki dataset released by MineDojo [Fan et al., 2022] only collects Minecraft Wiki pages, without refining the content and generating Q&A pairs for model training. STEVE [Zhao et al., 2023] introduces a non-public dataset with 20k+ Q&A pairs, which is smaller than our dataset in terms of scale and diversity.

Model Fine-tuning. We employ LoRA [Hu et al., 2021] for model fine-tuning, which is a parameter-efficient training technique. LoRA introduces small, trainable low-rank matrices to adapt a pre-trained neural network, enabling targeted updates without the need to retrain the entire model. Using LoRA, we fine-tune the LLaMA-3-8B-Instruct and LLaMA3-70B-Instruct models with our Minecraft dataset, resulting in the new models termed MineMA-8B and MineMA-70B.

Model Evaluation. InMinecraft, questions are often openended and can yield diverse answers; therefore, conventional evaluation metrics [Papineni et al., 2002; Lin, 2004] may fall short. Meanwhile, common benchmarks [Wang et al., 2018; Wang et al., 2019; Hendrycks et al., 2021] are not suitable for assessing the capabilities of expert models. Thus, we employed GPT-4 [Achiam et al., 2023] to generate two MultipleChoice Question (MCQ) datasets based on different themes and keywords related to Minecraft. These datasets can quantitatively evaluate the domain-specific expertise of models.

4 Agent Capability Benchmark

ODYSSEY presents a new benchmark for evaluating agent capabilities within Minecraft, offering three task types: long-term planning, dynamic-immediate planning, and autonomous exploration. It is notable that these tasks cannot be solved by any single skill but demand a sophisticated combination of multiple skills. These tasks are set in various Minecraft scenarios, with different tasks in the same scenario testing different agent capabilities. For example, in the cooking scenario, long-term planning requires formulating a complete plan to locate and hunt a specific animal, whereas dynamic-immediate planning involves selecting which nearby animal to cook based on the immediate environment. Please refer to Appendix E for more details.

Long-term Planning Task. We design a suite of combat scenarios to assess the long-term planning capability of agents, requiring them to craft appropriate weapons and equipment to defeat various monsters. These combat scenarios can be divided into single-type and multi-type monster scenarios. Agents must generate a comprehensive longterm plan, detailing the sequence of crafting the necessary weapons and equipment for the assigned combat task. Performance is measured by remaining health and time consumed during combat. After each battle, agents can iteratively optimize their plan, learning from previous outcomes to improve performance in subsequent rounds. To extend the scope of the long-term planning task beyond combat, we also adopt animal husbandry and cooking scenarios, where agents are required to formulate detailed plans for completing tasks related to specific animals.

Dynamic-immediate Planning Task. This task requires agents to dynamically generate and execute plans based on immediate environmental feedback. Thus, we design a suite of farming scenarios, where agents engage in activities like planting, cooking, and animal husbandry. Although some scenarios are similar to the long-term planning task, the dynamic-immediate planning task emphasizes reacting to real-time feedback like available resources and nearby animals. Performance is evaluated through task completion time and success rates.

Autonomous Exploration Task. To further test the exploratory capability of agents within open-world settings, we design an autonomous exploration task. In this task, agents are required to determine their subsequent objectives and execute the appropriate skills based on the game context. This task involves discovering and utilizing resources, while adapting to unexpected events such as encounters with hostile monsters. Agents must adapt to these challenges by developing strategies for resource management and task prioritization. The performance metrics include the number of distinct items obtained, the total items crafted, the recipes and advancements (R&A) unlocked, and the distance traveled.

5 Experiments

To demonstrate the effectiveness of the proposed ODYSSEY framework, we conduct experiments on basic programmatic tasks and the agent capability benchmark. Our simulation environment is built on top of Voyager [Wang et al., 2023a], providing a text-based interface for agents to interact with Minecraft. We only use GPT for initial data generation, but all experiments are conducted with the open-source LLaMA3 model, significantly reducing costs compared to GPT-4based skill generation methods [Wang et al., 2023a; Wang et al., 2023b]. Notably, we do not employ GPT-4 in Voyager due to the high cost, which we estimate would be in the thousands of dollars per experiment. Instead, we reproduce Voyager using GPT-4o-mini and GPT-3.5 for comparison. More details are provided in Appendix F. We aim to answer the following questions: (1) Can the open-world skill library improve the efficiency of agents in Minecraft? (Sec. 5.1). (2) How well do agents with different LLMs perform on the agent capability benchmark tasks? (Sec. 5.2). (3) What is the contribution of different components of the ODYSSEY agent to its overall performance? (Sec. 5.3).

5.1 Open-World Skill Library

To demonstrate the superior capability of our open-world skill library in Minecraft, we first tested it on 5 basic programmatic tasks. We conducted 120 repeated experiments on each task and recorded the average completion time for each task as well as the success rates at different time points. The results in Table 1 demonstrate that our open-world skill library efficiently handles basic programmatic tasks. Simple tasks achieve near-perfect success within five minutes. Even for difficult tasks like obtaining a diamond, success rates rise from 21.7% at five minutes to 92.5% at ten minutes, highlighting the effectiveness of the skill library.

5.2 Agent Capability Benchmark

We evaluate the LLM-based agent on the long-term planning task, the dynamic-immediate planning task, and the autonomous exploration task from the ODYSSEY benchmark. These tasks cover a variety of complex gaming scenarios and require diverse solutions.

Long-term Planning Task. The long-term planning task assesses the agent capability to directly formulate and execute comprehensive plans over extended periods. For example, in the combat scenarios, the agent is required to plan a list of weapons and equipment to craft based on the strength of different monsters, with the goal of defeating the monster in as short a time as possible. We compared the performance of our agent with both the fine-tuned MineMA-8B and the original LLaMA-3-8B, and also the performance of Voyager [Wang et al., 2023a] with GPT-4o-mini across these tasks. Moreover, we also evaluate the performance of single-round and multi-round planning. The single-round test results in Tab. 2 demonstrate that the fine-tuned MineMA-8B surpasses the original LLaMA-3-8B in terms of success rate and time efficiency, albeit at the cost of more LLM iterations. Moreover, our agent with MineMA-8B can outperform Voyager with GPT-4o-mini in most scenarios, indicating the effectiveness of our fine-tuning strategy. The multi-round test results in Fig. 3 show that the multi-round planning strategy significantly improves time efficiency. This improvement suggests that the agent is capable of iteratively refining its plans based on the outcomes of previous encounters, thereby boosting its performance in subsequent rounds.

Dynamic-immediate Planning Task. For the dynamicimmediate planning task, the agent is required to dynamically generate and execute plans based on immediate environmental feedback. We compared our MineMA model with different open-sourced LLMs, including GPT-4o, Qwen27B [Yang et al., 2024a] and Baichuan2-7B [Yang et al.,

2023]. Moreover, we evaluate the performance of the MineMA-8B and the MineMA-70B model to investigate the impact of model size on task performance. As shown in Tab. 3, the MineMA-8B model outperforms the Baichuan27B and Qwen2-7B models in terms of success rate and time efficiency. Moreover, the MineMA-70B model shows superior performance compared with the MineMA-8B model. Across all open-sourced LLMs, MineMA-70B demonstrates higher success rates and generally lower average execution times and LLM iterations. Additionally, our MineMA can achieve performance similar to that of GPT-4o.

Autonomous Exploration Task. In the autonomous exploration task, the agent is required to explore the Minecraft world freely without any specific goals. We compare our agent with different Minecraft agents (Voyager [Wang et al., 2023a] and DEPS [Wang et al., 2023b]) and different LLMbased agent techniques (ReAct [Yao et al., 2023] and AutoGPT [Significant-Gravitas, 2023]) on this task. Note that we reproduced different LLM-based agent techniques following the same settings as in Voyager [Wang et al., 2023a]. As shown in Fig. 4, our agent with the MineMA-8B model can achieve superior performance compared with all baselines, indicating that the agent can autonomously explore the Minecraft world without specific goals. It is notable that our agent with the MineMA-8B model can outperform Voyager [Wang et al., 2023a] with GPT-4o-mini or GPT-3.5.

5.3 Ablation Study

We conduct ablation studies on two core components of the ODYSSEY agent, including the LLM planner and the openworld skill library. The results are shown in Fig. 4. In the autonomous exploration task, the LLM planner is responsible for generating a comprehensive plan based on the openworld skill library. The ablation study demonstrates that the planner is indispensable for the agent to effectively navigate the complex Minecraft environment. Additionally, our experimental results indicate that the absence of the open-world skill library significantly degrades performance. Without the open-world skill library, the 8B LLMmodel alone is largely incapable of generating executable codes for the agent. This underscores the critical role of the open-world skill library in enabling the agent to perform complex tasks within the openworld setting of Minecraft.

6 RelatedWorks

Minecraft Agents. Minecraft agents have been widely studied in recent years to test the capabilities of autonomous agents in open-world environments. Previous works focused on training Minecraft agents with reinforcement learning [Tessler et al., 2017; Oh et al., 2017; Lin et al., 2022; Mao et al., 2022; Hafner et al., 2023] or imitation learning [Baker et al., 2022; Cai et al., 2023; Lifshitz et al., 2023], which are extensively used in the MineRL [Guss et al., 2019] competition to solve the ObtainDiamond task. With the rapid development of LLMs, numerous studies leverage LLMs to enhance agent capabilities [Zhang et al., 2023a; Zhu et al., 2023; Feng et al., 2023; Zhao et al., 2023; Zheng et al., 2023; Zhou et al., 2024; Li et al., 2024; Yu and Lu, 2024; Wang et al., 2024b; Cai et al., 2024]. Among these, several works [Li et al., 2023; Yuan et al., 2023; Wang et al., 2023c; Qin et al., 2023; Ding et al., 2023] employ LLMs to guide skill learning in Minecraft, enabling agents to act in a humanlike way. However, these methods mainly focus on learning primitive skills from scratch, lacking a reusable skill library. Voyager [Wang et al., 2023a] builds a skill library by allowing the LLM to write its own skills. However, Voyager must rely on GPT-4 for high-quality skill generation, incurring substantial costs. This expense can be prohibitive for many researchers. In contrast, ODYSSEY provides an openworld skill library that agents can call upon, achieving performance comparable to Voyager with GPT-4, but using only 8B LLMs. This makes ODYSSEY significantly more accessible and cost-effective, enabling LLM-based agents to efficiently generate complex policies for broader exploration.

Open-world Tasks. Open-world tasks have gained attention from research communities [Chevalier-Boisvert et al., 2018; Juliani et al., 2019; Shen et al., 2021; Du et al., 2023; Liu et al., 2023; Liu et al., 2024; Jiang et al., 2021; Jiang et al., 2022]. Minecraft, with its diverse tasks and mature game mechanics, has emerged as an ideal test-bed for open-world tasks. Built on Minecraft, MineRL [Guss et al., 2019] implements a simulation environment for agent learning. MineDojo [Fan et al., 2022] further extends MineRL with thousands of diverse tasks. MCU [Lin et al., 2023] collects a variety of atom tasks, offering a method to generate infinite tasks by combining the atom tasks. However, existing benchmarks mainly focus on providing basic programmatic tasks to evaluate agents learned from scratch. Our benchmark is built on top of the skill library, enabling the agents to bypass basic programmatic tasks and focus on complex challenges.

7 Conclusion

This work proposes ODYSSEY to empower agents with openworld skills in the Minecraft environment. The proposed open-world skill library enables the use of open-source LLMs as the foundation for agents to call upon skills, avoiding the high costs associated with previous work using GPT-4 [Wang et al., 2023a; Li et al., 2023; Qin et al., 2023]. The public availability of all datasets, model weights, and code will facilitate future research in the development of autonomous agents. We hope that ODYSSEY will inspire further innovation and progress in the field of autonomous agent. For future work, the open-source LLMs are now prone to generating hallucinations, leading to a decrease in agent performance. Thus, we will focus on employing retrieval-augmented generation to improve LLMs in Minecraft. Additionally, this work focuses on developing text-based LLMs in the context of Minecraft, with visual aspects currently out of scope. Looking ahead, we plan to integrate visual understanding into the skill library to enhance the agent capabilities.

Acknowledgments

This work was supported in part by the CCF-Baidu Open Fund under Grant No. CCF-BAIDU OF202410, in part by the Hangzhou Joint Funds of the Zhejiang Provincial Natural Science Foundation of China under Grant No. LHZSD24F020001, in part by the Zhejiang Province HighLevel Talents Special Support Program "Leading Talent of Technological Innovation of Ten-Thousands Talents Program" under Grant No. 2022R52046, in part by the Zhejiang Provincial Natural Science Foundation of China under Grant No. LMS25F020012, and in part by the advanced computing resources provided by the Supercomputing Center of Hangzhou City University.

Contribution Statement

Shunyu Liu, Yaoru Li, Kongcheng Zhang, Zhenyu Cui, and Wenkai Fang made equal contributions.