Using Planning for Automated Testing of Video Games

1 Introduction

Game testing is crucial for identifying bugs, optimizing performance, and ensuring a high-quality user experience. It involves scenario testing, UI refinement, and compliance checks, especially for online games. While modern testing blends automation and manual efforts, fully integrating automation without disrupting development remains a challenge [Politowski et al., 2022]. Our system expands automated testing with a planning-based approach.

We present an alternative to scripted testing. The most significant difference is that scripts, in conventional scripting, must be manually created and maintained for each test, whereas tests utilizing automated planning only require specifying the test goals. A plan can be used in the same way as test scripts in games. The planner creates an ad hoc path to the goal each time it is executed based on the game mechanics captured in the defined domain model. Furthermore, our system can react to unexpected events during the execution of a script and adjust it by re-planning from the current state and still possibly reaching the goal.

The presented system is being developed as a commercial product by Filuta AI, Inc. in collaboration with real game development studios as customers. It undergoes continuous evaluation and refinement based on their feedback to ensure it meets their needs.

2 Motivation

We focus on regression testing during game development, i.e., running frequent tests to ensure that recent changes did not introduce unintended glitches and the game still performs as expected. This is a particularly repetitive and dull task for a human tester but crucial for effective development. It is often the case that the longer the time difference between the introduction and discovery of the bug, the harder it is to fix.

2.1 What can be Tested with Planning

For regression testing, we can define a set of objectives that must be tested every time the game code is updated. These objectives can be translated into (planning) goals that need to be achieved by sequences of actions (plans) that can be generated automatically (by planners). The plans are then executed in the game in a similar fashion as (manually written) test scripts. On top of that, we can collect game performance telemetry data.

Besides regression testing, planning has the potential for investigating whether the game can be completed, i.e., each intended objective can be achieved from the starting state of each scenario, if the game has dead-ends, i.e., states that can be reached but it is impossible to complete the game from them. In modern game design, such situations are very undesirable and annoying for gamers. Additionally, planning can be also used to identify undesired shortcuts in the game.

The Planning Agent technology is a great fit for games that have random and non-deterministic nature due to its ability to adapt to changing conditions and the availability of replanning (changing the game script) at any point. Planning agents also inherently allow the automatic variation of the playthroughs by weighing the decision points (preferring the unexplored paths) so that the agent finds plans that are as different as possible from all the previous playthroughs while still achieving the goal. This principle is like Monte-Carlo Sampling, providing coverage of diverse gameplays even in infinitely large game worlds, where the only limitation is the number of hours budgeted for the explorative agents. This adaptability allows us to test objectives such as:

• Dialogue Choices / NPC 1 Interactions: Assuming the tree of narrative decisions is finite, the Planning Agents are expected to provide complete coverage of the narrative tree and capture anomalies such as suddenly inaccessible dialogue paths or unreachable story goals.
• Unscripted Event Responses: This can be a side effect of any other tests or a test by itself. Re-planning based on a changed world state is an integral part of Planning Agents and happens seamlessly. We can either detect deviations or re-plannings in all the other tests that can be used for this analysis, or we can design a test that covers a part of the game that changes dramatically and lets the Planning Agent reach a goal. We also measure the cognitive complexity of the dynamic changes and gameplay (how difficult the reasoning task about the dynamic change has been).

2.2 Applicability

The classical planning-based testing approach is most suitable for games with many discrete causal interactions such as roleplaying games (RPG), point-and-click adventures, strategy or city-building games, visual novels, or puzzles. It is less suitable for fast-paced action-oriented games such as sports games or driving simulations. For such games, reinforcement learning-based approaches, that have been studied in recent works, seem to be more suitable (see the Related Work Section of this paper). Overall, we observe that the reinforcement learning and the planning-based approaches can complement each other rather well.

3 Preliminaries

Automated planning concerns finding a plan - a sequence of actions that transform the world from an initial state to a goal state, i.e., a state where all goal conditions are satisfied. A planning problem instance consists of a domain definition and a task definition. The domain definition describes the environment and possible actions with their preconditions and effects. In STRIPS planning [Fikes and Nilsson, 1971], the environment is specified by predicates, and preconditions and effects of actions are conjunctions of predicates and their negations. The task specification consists of the description of the initial state and the goal conditions.

4 RelatedWork

Planning is rooted in search and, as such, has had a long history with games. At the beginning of AI, the focus was on games like chess [Newell et al., 1972] and checkers [Samuel, 1959], which initially relied on the search for solutions. Planning continued to dominate in the 80s and 90s with checkers and chess with Deep Blue [Hsu et al., 1990] and Chinook [Schaeffer et al., 1996]. Agent architectures and production systems added value, and soon, planning started to add value in games like bridge [Smith et al., 1998] and the class of Real-Time Strategy (RTS) games [Chung et al., 2005]. In [Duarte et al., 2020], they survey the history of planning and learning in games, covering the spectrum as well as diving into the lineage of planning from search, minimax and alpha-beta pruning, hierarchical task networks, and Monte Carlo Tree search, through classical planning, rapidly-exploring random trees, case-based planning, and behavior trees. Most of the work is focused on creating AI-driven opponents [Wurman et al., 2022], which are sometimes used to play both sides for evaluation, AI training, and testing.

1 Non-player character

Automated testing with AI has been a rising research focus more recently with work that has focused on agent-based approaches that include navigation mesh path-finding [Shirzadehhajimahmood et al., 2021], reinforcement learning agents for finding design and environmental defects [Ariyurek et al., 2019; Ferdous et al., 2022], reinforcement learning for load testing [Tufano et al., 2022], modeling of user interaction for boundary testing [Owen et al., 2016], search for test case generation [Ferdous et al., 2021a], and search for automated play testing [Ferdous et al., 2021b]. Despite planning uses in-game AI, we do not see its use in in-game testing more broadly beyond back to search. However, as evidenced by Bram Ridder's (AI Programmer for Rebellion) keynote talk at the 2021 AIIDE Conference on "Improved Automated Game Testing Using Domain-Independent AI Planning" [Riddler, 2021] and his 2021 GDC AI Summit talk "Automated Game Testing Using a Numeric Domain Independent AI Planner," planning techniques for game testing are beginning to be used in the games industry mixed in with calls for more AI automation of testing [Fray, 2023].

In [Volokh and Halfond, 2023] the authors proposed an automated approach for determining actions when conducting automated exploration for games. It is based on program analysis (slicing) of the game code. Although they are not using the usual planning formalism (like PDDL), they work with symbolic representation of states and actions and rely on SMT (satisfiability modulo theories) solvers to determine the set of applicable actions in a given state.

5 SystemDescription

A Planning Agent is an entity that generates a plan to reach the test goal, executes the plan, and then reports the outcome of the plan execution (e.g. passed or failed) and provides telemetry data. A planning agent requires a domain model that captures the mechanics of the game and a description of the current game state and a goal that needs to be achieved. The domain model is provided in Planning Domain Definition Language (PDDL) [Ghallab et al., 1998], while the initial and goal states are described using predicates. The planning agent utilizes the Unified Planning framework [Micheli et al., 2025] to interface with state-of-the-art planners and find plans.

Planning Agents are engine-agnostic and can be integrated with any game engine. Our most advanced integration is with Unity and Unreal engines. For both we developed a unique Filuta Plugin that automatically generates the data required by the planning agent. Each plugin can hook onto the Unity and Unreal objects to automatically collect information about the game environment. Each plugin also allows the Planning Agent's output to be integrated back into the game to play it. Based on the environment configuration, it sets the game up into an initial state, creates a plan to fulfill the (user-specified) test goal and executes it. In most tests, the only configuration the user must specify is the environment (map) on which the test will be executed. Everything is configured and viewed on a web application called Planning Dashboard. It is also used by the user to inspect the results of the tests.

Filuta AI allows for unattended testing even in nondeterministic environments. In case of any unforeseen circumstances in a game, such as the targeted item is collected by a different game character, the Planning Agent can automatically adjust, i.e., the Planning Agent can stop the execution, evaluate the current world state, and produce a new plan to reach the goal.

5.1 Instrument Once, Test All Code Versions

The most labor-intensive part of integrating the Filuta AI Gaming QA solution is mapping the actions from the domain (move, kill, collect, . . . ) back to the game code. Therefore we designed our plugins in a way that the game code itself does not need to be modified, and new code is only added on top of it. The plugins use reflection techniques to extract game state information. This allows the mapping needs to be developed only once, and unless there are significant changes to the game mechanics, it can remain the same throughout the whole game development process.

The original instrumentation of the real-time strategy game Silica [Bohemia Interactive, 2025] happened on a code version from September 2023. One of the initial groups of tests was to build all the available unit types in the game. The version from September 2023, however, did not implement some of the unit-building structures, so a player could not build some units yet, hence these goals were unachievable. When updating the code to a version from April 2024 (after 7 months of active development), no change was required to the Filuta AI instrumentation. All the tests that worked on the September 2023 version also work on the latest version without any amendments. Moreover, the structures to build the new units were implemented into the game and the Filuta Unity Plugin automatically registered them.

5.2 Addressing Hardness of PDDLModeling

Automated Planning is not commonly utilized by game developers. A major challenge in its adoption lies in the authorial burden of writing domain and problem descriptions in PDDL [Ghallab et al., 1998]. Crafting well-structured PDDL models is a specialized skill that develops over time, making efficient use of experienced modelers crucial. While Knowledge Engineering tools exist to support PDDL modeling [Simpson et al., 2007; Vaquero et al., 2013], the process remains somewhat of a "black art" [McCluskey et al., 2017].

To address this challenge, we investigate the practical applicability of Automated Planning Action Model Learning (AML) techniques. AML aims to synthesize domain definitions directly from logs containing state sequences, reducing the need for manual modeling. Numerous algorithms have been developed for this task [Wang, 1996; Aineto et al., 2019; Zhuo et al., 2010; Yang et al., 2007; Juba et al., 2021], and the MacQ project [Callanan et al., 2022] provides a recent overview along with new implementations of multiple AML techniques. Our research team is actively contributing to this field with ongoing work [Balyo et al., 2024a; Balyo et al., 2024b].

6 Demo Video

The demo video (available at https://youtu.be/ggRiXeQxMsE) showcases our system in action with two real game examples. The first is a real-time strategy game Silica by Bohemia Interactive [Bohemia Interactive, 2025] developed in the Unity Engine [Unity Technologies, 2025] and the second is the firstperson shooter Lyra [Epic Games, 2025a]. Lyra is a sample tutorial project developed alongside Unreal Engine [Epic Games, 2025b] to serve as a starting point for learning about the engine and creating new games in it. The video features gameplay footage from the planning agent, along with behindthe-scenes insights into test setup and evaluation using our web application, the Filuta Planning Dashboard.

7 Conclusion

We introduced the Filuta AI planning agent, which automates regression testing for video games using PDDL-based automated planning. This approach eliminates the need for manual test scripts, reducing workload while maintaining accuracy and adaptability. Integrated with Unity and Unreal, it enables seamless test execution and detailed result analysis.

Our evaluation across multiple games shows that planningbased testing effectively verifies gameplay mechanics and objectives. Its ability to adapt and replan in dynamic environments enhances robustness.

However, challenges remain, particularly in the complexity of PDDL modeling. To address this, we explored action model learning (AML) to generate domain models from gameplay logs, making automated planning more accessible. Our research continues to refine these techniques and streamline integration.

7.1 FutureWork

Moving forward, we aim to extend our system's applicability by testing it across a broader range of game genres. We're working closely with several video game studios to meet their real-world needs. Additionally, we plan to develop intuitive tools for domain modeling, such as graphical or no-code interfaces inspired by Unreal Engine's Blueprint system, to further lower the barrier for game developers. By enhancing usability and expanding coverage, we hope to drive wider adoption of automated planning in game testing, ultimately improving game quality and development efficiency.

Acknowledgements

Roman Bart´ak and Luk´aˇs Chrpa are supported by the project 25-18003S of the Czech Science Foundation.