Human-AI Interaction Generation: A Connective Lens for Generative AI and Procedural Content Generation

1 Introduction

Recent generative AI tools for text and image generation have created a large response in the general public and academic community. These models generally take prompts, often natural language instructions, as input. They then typically output content in whatever media format a tool focuses on in a single-shot manner. While this paradigm for generating content can work well in many instances, it is ill-suited to many others, leading in some part to the wide range of responses to these tools [Vimpari et al., 2023].

Generative AI tools did not invent the concept of generating content with computers. Procedural Content Generation (PCG) dates back to 1983 with the game Beneath Apple Manor for the Apple II, and has benefited from decades of research since then [Shaker et al., 2016a]. PCG refers generally to processes (procedures) for generating content with computers, and can be broken into constructive or rules-based methods, search-based methods [Togelius et al., 2011], and machine learning-based methods [Guzdial et al., 2022b], including deep learning methods [Liu et al., 2021]. Recent research has directly investigated applications of Large Language Models (LLMs) to typical PCG problems [Gallotta et al., 2024]. However, we can argue that generative AI tool and PCG are the same process at the highest level, as PCG is broadly defined as procedures for content generation. With this perspective, we can identify how existing PCG research and open research may benefit the future development of these tools.

In this paper, we formally define human-AI interactive generation as a lens to understand the commonalities of PCG and generative AI. We then identify three attributes of this lens that are less present in generative AI tools than existing PCG

research. We survey prior work related to these attributes and outline opportunities for future work.

2 Human-AI Interactive Generation

Procedural Content Generation (PCG) research has a number of frameworks or lenses for understanding content generation processes that involve humans [Shaker et al., 2016b; Guzdial et al., 2022a]. However, these frameworks tend to divide generation into autonomous (AI system alone) or mixedinitiative (AI and human together). We argue this is an artificial divide, when even "autonomous" generation processes still require human prompting or to otherwise start the process. We identify human-AI interactive generation as an alternative lens in terms of the below equation, meant to illustrate the commonalities between PCG and Generative AI paradigms. We intentionally leverage the basic function formulation common across many AI paradigms.

In the above equation, Y indicates the set of all sets of possible outputs across all generation processes. The theoretical concept of the set of all possible outputs for a single generation process is historically referred to as a possibility space [Rabii and Cook, 2023; Smith, 2017]. A single generation process is similarly referred to as a generator. For our lens, we can define the possibility space as Yg, the set of all possible outputs for a particular generator g. A particular output can then be rendered as y. For example, YChatGPT might indicate all possible combinations of strings up to the maximum size of a specific version of ChatGPT's output or YStableDiffusion might indicate all possible RGB images that Stable Diffusion can output.

G in the above indicates the space of all possible generators. To parallel possibility space we can refer to this as the theoretical generator space [Khalifa, 2020]. Though this is a less-investigated concept in PCG literature, it is related to the concept of a hypothesis space of potential generative models. We can then define an individual generator as g or the set of generators of a specific type as GType, such as GLLM for the set of all the possible generators that involve a Large Language Model. However, a generator g does not need to be a single model or AI approach, such as in the case of ChatGPT which leverages outside services like WolframAlpha as part of its generation process.

X then gives us the set of all possible inputs to all possible generators. A particular input x would then define a specific input, for example a specific prompt and context to an LLM or random noise for a constructive generator [Shaker et al., 2016a]. Our definition of X considers all possible inputs, even if the vast majority would not lead to valid outputs for a generator. We refer to the set of inputs that would lead to a valid output for a generator g as Xg. For example, Trajkova et al.'s "LuminAI" dance partner can be understood as a generator that takes in a recording of human dance xi and outputs a space of possible dances in the form of a projected character YLuminAI [Trajkova et al., 2024]. We could not feed this input xi to ChatGPT and expect it to function, but neither could we feed an appropriate prompt and context xj for ChatGPT into LuminAI. Thus Xg may change as the generator changes, as with the increasing prevalence of multimodal Generative AI tools.

In the above paragraphs, we have attempted to demonstrate how this simple human-AI interactive generation lens allows us to consider the commonalities between PCG and Generative AI systems. In addition, this lens allows us to identify several desirable attributes of these systems. We will dive further into the commonalities for PCG and generative AI by investigating these attributes.

Human-centered input alignment represents the extent to which Xg for a particular generator g aligns with human expectations for that generator. We contrast this to the general use of the term alignment which largely focuses on the behavior of a generator in terms of its output [Ji et al., 2023; Lopes and Bidarra, 2011]. Adaptability is the extent to which the generator g adapts to the user over a sequence of inputs (x0, x1, ... xn) and outputs (y0, y1, ... yn). This in turn is related to continual and lifelong learning, along with player adaptation research, but focuses specifically on generation processes [Parisi et al., 2019]. Novelty is the extent to which a generator g can produce outputs within Yg that go beyond its learned or authored distribution, in other words outof-distribution generation. This is related to several concepts from core machine learning, including out-of-distribution detection [DeVries and Taylor, 2018] and generalization [Zhang et al., 2021]. We will overview recent work and results related to these attributes in the following sections.

3 Human-centered Input Alignment

The most common valid input for Generative AI tools are prompts. While initially constrained to natural language, Generative AI tools are increasingly focused on increasing the number of valid input modalities, leading to modern multimodal models [Liu, 2023]. In other words, Generative AI tool developers seem to seek to expand Xg to reach X. However, we argue this actually runs counter to successful generation processes in many instances, thus our proposal of the human-centered input alignment attribute. This is part of the broader concept of human-centered design, which argues design processes should be driven by the needs of human users [Cooley, 2000]. This attribute then considers the extent to which the valid inputs for a generator correspond to user preferences over possible valid inputs.

This attribute can encourage us to consider possible inputs from the start, impacting the design of our models. In Sketchbetween, we developed a tool to color in the inbetween frames for hand drawn animation using a vectorquantized variational auto-encoder (VQ-VAE) [Loftsdottir and Guzdial, 2022]. We visualize inputs from human animators and Sketchbetween's outputs in Figure 1. To mirror the human design process, our generator takes in two colored keyframes and sketched in-between frames. While many Generative AI models could take an input image prompt like this they may struggle with consistent output [Saravanan and Guzdial, 2022; L¨owenstr¨om, 2024]. In comparison, SketchBetween outperformed more general image inpainting baselines. We have continued to find success when taking this tact, specifically designing generators to align their valid input sets with user expectations [Halina and Guzdial, 2021b; Cooper and Guzdial, 2023; Halina and Guzdial, 2023].

We provide an example of the consequences of not achieving this attribute in Anhinga, a puzzle design tool based on the game Snakebird [Guzdial et al., 2021]. In Anhinga, we included an exhaustive PCG system capable of generating variations of a current puzzle that maximally altered an input puzzle's difficulty while minimally altering its structure [Sturtevant et al., 2020]. Crucially, Anhinga would run this generative process any time a user made any change to a puzzle, functionally meaning that every single user change represented an input to our generator. Based on a human subject study, we found that designers using Anhinga did find the generator easy to use but significantly preferred a version of the tool without the system that responded to every user input. We consider this to be due to cognitive overload based on the rate of feedback. Of greater concern was that in a comparative ranking, designers believed a puzzle they made without the generator would be more challenging than one they made with it, but we found objective evidence for the opposite. This suggests that lacking human-centered input alignment may harm a user's ability to evaluate the outputs of a generation process.

4 Adaptability

Modern generative AI and PCG tools rarely adapt to a user in a significant fashion. In generative AI tools, this is due to the large financial and time cost of retraining models, which makes such an adaptation ill-suited to real-time interactive sessions. In PCG tools, significant adaptation requires an additional development effort in terms of modelling the designer, which may not be feasible except in a post-hoc fashion [Alvarez et al., 2022]. However, due to the diversity in the design practices of human designers, and more generally the differences across humans, we have found a strong positive response to models that adapt to users [Mahmoudi-Nejad et al., 2021].

We developed Morai Maker as a platform to study humanAI interactions in level generation and design for the game Super Mario Bros., we visualize its interface in Figure 2. We employed a turn-based interaction modality [Guzdial and Riedl, 2019], which meant that a human designer and an AI agent took turns placing down pieces of level structure, with the human ending their turn with the large "End Turn" button at the bottom right of the interface. We ran an initial comparative study between three non-adaptive machine learning agents (Markov Chain, Bayes Net, and LSTM), and found no significant difference between user preferences across the approaches [Guzdial et al., 2019], despite measurable differences between the possibility spaces Yg of each generator [Summerville, 2018]. In a follow-up study, we replaced these static, non-adaptive agents with a deep reinforcement learning agent that learned from implicit human designer feedback [Guzdial et al., 2018], and found that users could identify and preferred the adaptation [Guzdial et al., 2019].

We found further support for the importance of adaptability in human-AI interactive generation with KiaiTime. KiaiTime was a level design tool we developed for the game Taiko no Tatsujin [Halina and Guzdial, 2021a]. We ran a study with KiaiTime comparing the naive, implicit adaptation strategy we found success with in Morai Maker to Threshold Designer Adaptation (TA) [Halina and Guzdial, 2022]. In TDA, we learn a domain-dependent adaptation hyperparameter by approximating interactive design sessions from pre-existing outputs, treating these outputs as if they were fromKiaiTime Y ′KiaiTime . We found that designers enjoyed working with the more advanced adaptatibility approach and that this led to a greater variety of final outputs.

5 Novelty

Out-of-distribution generation exists as a term for approaches that alter a generator such that its generative distribution shifts to encompass an area outside of its initial generative distribution [Lotfollahi et al., 2020]. However, this approach typically relies on training data from a secondary distribution and a transfer learning methodology. We do consider this an example of the novelty attribute in human-AI interactive generation, but also look beyond it to cases where we may have no data from the out-of-distribution region that encompasses our desired outputs [Sarkar et al., 2023]. This is related to the distinction between P-creativity and H-creativity [Boden, 2009], we consider cases where examples of the kinds of output we wish to generate may not exist in history. While this may sound impossible, it is possible to approximate such a generator through secondary features of the desired outputs.

Combinational creativity, also called conceptual combination, is a common human cognitive practice for approximating novel knowledge by combining existing knowledge [Wisniewski and Gentner, 1991; Boden, 2010]. Existing approaches have attempted to replicate this process in computers, most commonly through an approach referred to as conceptual blending [Fauconnier and Turner, 2003]. Conceptual blending generally functions by combining two existing pieces of content to produce a novel piece of content, essentially acting as a generator g that takes two inputs and returns an output that represents their combination [Goguen and Harrell, 2004]. We instead propose taking two arbitrary generators gi and gj and combining them, which can be understood as an example of PCG via Knowledge Transformation (PCGKT) [Sarkar et al., 2023].

We initially proposed directly combining learned video game level generation models via combining learned Bayes Nets for Super Mario Bros. level generation via conceptual blending [Guzdial and Riedl, 2016]. We found that the "blended" Bayes Nets better-approximated blended game levels from human expert. We followed this work by proposing a novel combinational creativity algorithm we called conceptual expansion, which was better suited to dealing with matrix-based, machine-learned features.

We define a conceptual expansion as a linear combination of potentially repeated features F and α filters A. We can then optimize this combination of existing features and α filters to approximate novel features, typically through a search-based process. We visualize a conceptual expansion in a toy example in Figure 3. In an initial experiment replicating Bayes Net Super Mario Bros. level design task, we found that conceptual expansion outperformed conceptual blending and other PCG-KT approaches in terms of the variety of possible outputs, the size of the Yg set [Guzdial and Riedl, 2018c].

To move beyond video game levels, we next attempted to generate whole novel video games. We machine learned symbolic rule-based models of three video games from gameplay video, which we might now consider examples of world models [Guzdial et al., 2017]. We applied conceptual expansion over these world models, combining them to produce novel world models for unseen games. In an initial leave-one-out experiment, we found that conceptual expansions could better approximate held-out, human-authored games than relevant game generation baselines [Guzdial and Riedl, 2018a]. We then applied this approach to generate fully novel games, and found in a human subject study that participants ranked the generated games as more surprising than games authored by human expert game developers and equally as creative [Guzdial and Riedl, 2021].

We found deep neural networks (DNNs) to be an excellent fit for conceptual expansion due to their learned matrices of features. We initially found that conceptual expansion on DNNs could allow for low-data transfer learning for image classification and generation tasks [Guzdial and Riedl, 2018b]. We since have applied this approach to recombine features fromMusicVAE to create a model for generating Iranian Folk Music, which outperformed transfer learning baselines [Doosti and Guzdial, 2023].

We do not anticipate that conceptual expansion is the only approach for achieving novelty. However, we have demonstrated across a number of models and domains that it is a robust approach for approximating unseen or nonexistent knowledge in generation processes. We therefore identify it as a valuable comparison point for future approaches at truly novel content generation.

6 FutureWork

In this paper, we have surveyed recent work and findings related to three key attributes of human-AI interactive generation. However, there are still a large number of open problems related to each attribute. In addition, there are likely many more underexplored or even unidentified attributes of similar importance to these three.

For human-centered input alignment, the research thus far has been piecemeal, considering domain-dependent input modalities and evaluating their impact independently. There is the possibility of deriving a theory to guide this practice, but we anticipate full automation may be more useful. As with Threshold Designer Adaptation, we imagine extracting beneficial input modalities and other relevant design features from existing content or records of creative processes [Halina and Guzdial, 2022] We identify this as a similar impulse to automating science, with many of the same methods likely relevant [Waltz and Buchanan, 2009].

For adaptability, we have demonstrated consistent results supporting user preference for adaptive generators [Guzdial et al., 2019; Mahmoudi-Nejad et al., 2021; Halina and Guzdial, 2022]. However, we have evidence that adaptation can fail to outperform random behavior [Yu et al., 2021]. As such, it's clear that further research is required to differentiate between these cases. Outside of this more general need, there are specific questions in terms of when to adapt, how to adapt, and by how much. We anticipate significant overlap with other areas of human-AI interaction, such as teachable robots [Thomaz and Breazeal, 2008].

For novelty, we have identified that computational models of combinational creativity or conceptual combination can successfully lead to out-of-distribution generation without any training data [Guzdial and Riedl, 2016; Guzdial and Riedl, 2021]. We can more broadly consider these tasks to be related to zero-shot transfer learning. As such, we anticipate these approaches may be beneficial for more general zeroshot transfer learning tasks, and we have already found evidence for this in the financial domain [Mahajan and Guzdial, 2022]. We hope to more fully explore potential applications, particularly in low data domains like medicine, but we also anticipate more fundamental research into novelty in generation. In particular, we hope to determine how human users can best guide and benefit from novelty.

We have defined three beneficial attributes in terms of our human-AI interactive generation lens, informed by prior work. However, we do not believe this is an exhaustive set. In particular, each attribute focused on only one aspect of our lens, but we anticipate benefits from studying the interplay and relationships between these aspects.

7 Conclusions

In this paper we introduced the lens of human-AI interactive generation to identify connections between the Generative AI and PCG paradigms. Based on this lens we defined three attributes: human-centered input alignment, adaptability, and novelty. We surveyed recent work and results related to these attributes, and outlined opportunities for future work. Our hope is that this lens will contribute to an ongoing dialogue between Generative AI and PCG researchers, in order to share future contributions across both fields.

Acknowledgments

The work included in this article was funded through an NSERC Discovery Grant and Canada CIFAR AI Chair at the Alberta Machine Intelligence Institute (Amii). I am deeply grateful to all of my collaborators, students and otherwise, who made this work possible.