SandboxSocial: A Sandbox for Social Media Using Multimodal AI Agents

1 Motivation

Even with imperfect fidelity, modern Large Language Models (LLMs) are capable of replicating human-like behavior with an impressive degree of face validity. As a re-

∗Correspondence to: puelmatm@mila.quebec

Contributions. Amulti-agent simulator:

1. a real social media environment accessed by agents as an application on their phone that includes the ability to post and process multi-modal social media content;
2. observation functions of multi-modal media;
3. balanced social context specification through combining user-specified text files for configuration alongside mapping of exogenous real-world content in the form of agent profiles and news media;

1. an efficient and scalable implementation through a combination of cloud infrastructure, selective use of elaboration of context, and full parallelization of agent-specific computation; and
2. a custom, web-hosted analytics dashboard of simulation output, including all actions by all users on the platform, the within-agent planning information, and the results of the longitudinal probes of agent beliefs.

An example simulation of election disinformation is given.

2 Methodology &Features

Our multi-agent simulator is built using the Concordia generative agent modelling framework [Vezhnevets et al., 2023]. Concordia-based simulations rely on LLM-based elaboration of social situational context using structured text inference and summarization to simulate interactions of generative agent models in a grounded physical, social, or digital space. Concordia agents are component networks (memories, longterm objectives, perception of self/others etc.) whose state are updated with experience and are used as social context for LLMs to infer agent plans (e.g., What kind of person is X? What situation is X in now? What would a person like X do here?). A 'Gamemaster' agent abstraction orchestrates simulation control by evaluating attempted actions, handling events, and distributing their effects. In this main section, we outline some of the novel and desirable features we designed when building this simulator.

Highly configurable multi-agent simulation. Our simulator moves away from the complexities of Concordia's software library, allowing users and researchers to easily customize, run, scale, and analyze the results of multi-agent simulations. Specifically, we designed simulations to be configurable to any setting through structured text descriptions of the setting and the agents (see full version for examples). We add a survey system deployed longitudinally on the agent population to passively probe their in-context beliefs. We implemented a versatile and easily configurable base agent model having native Concordia components. We also provide a clear and simple way to add specific features dependent on a custom setting (the component graph for models used in our example is shown in fig. 2).

Social Media. We implement a virtual social media environment accessed by agents using a custom phone application within the existing phone library provided by Concordia. It integrates into the agent's experience in our simulation, providing the option to use the app and access the virtual platform. For demonstration, we tailor our platform implementation to Mastodon, a popular open-source social media platform where users interact (e.g., posting or liking content and following one another), though our implementation can be adapted to any open-source platform, not just Mastodon.

To demonstrate the benefits of tailoring the platform implementation, we set up a real Mastodon instance on a cloud server with a number of blank-slate users. The simulation begins by first building the agents within Concordia and then assigning the blank-slate Mastodon users to them, modifying user profile information, and seeding an initial followership network. Initial followership can be specified directly or realized from followership probabilities for pairs of agent roles. To populate the platform with initial content, each agent makes an introductory post or 'toot'. Agents then take full control of their accounts, which enables them to modify all aspects in the context of natural behavior on the platform.

When agents open Mastodon, they read their feed and choose from a set of actions ('post', 'follow', etc.). We include soft constraints on action selection through suggestions that bias towards configurable role-based action frequencies.

Multimodality. We allow agents to process multi-modal social media content by generating character-focused, context-dependent text impressions of images contained in social media posts that are stored with the agents' memory of viewing the post (see full version for the prompt).

World-mapping. A major obstacle to studying particular questions of social systems in models is how to specify all the details unrelated to target factors, which nevertheless influence them. To overcome the arbitrariness in seeding all the semantic content of such a complex social setting while still preserving the freedom to specify imagined and/or counterfactual aspects, we develop a methodology we call 'worldmapping' that adapts and maps real-world information into our simulation settings. In our demonstration, we map Reddit users into agent persona summaries based on their posts and multimodal news media from NewsAPI into a news account to promote discussion of realistic issues (see fig. 1). The latter is an example of the exogenous agent role that we designed, i.e. one whose actions are a fixed sequence. Use of this role allows us to control how much of the semantic content of the simulation is a direct reflection of real-world content.

Probes & Dashboard. To aid analysis of simulation results, we provide two add-ons. The first is a longitudinal survey system that queries each agent at each step with userspecified questions. In our example, we use questions resembling those found in political survey research [DeBell, 2023]. The second add-on is a custom, web-hosted analytics dashboard of simulation output. This includes time series of survey and platform interaction statistics, a highly annotated, episode-conditioned interaction network, and withinagent action plans. Additionally, all LLM prompts and responses can be loaded and easily parsed for more detailed inspection of the within-agent computation producing the observed behavior.

Demonstration: small town election Agents access the platform at random times. Agents have formative memories derived from Reddit-scraped user personas and shared context about the town. We add a news-agent, the local 'Storhampton Gazette' newspaper (see fig. 3), which injects modified real-world information (see full version for details).

Ethics Statement

While this simulation is grounded through the use of social media, it does not fully capture every aspect. Our current simulation makes use of Reddit, which has a higher concentration of North American users and culture. However, future works can use other social media platforms as well. The simulation should not be used with the expectation that it will match the real world perfectly. Rather, it should be used to gather intuitions, explore counterfactuals, and detect more plausible and interesting situations that could occur and warrant further investigation.

We follow responsible AI development standards that should guide researchers applying our simulator. For example, the simulation relies on Large Language Models (LLMs) and thus inherits any representational bias in the training data. Researchers should be cautious asking research questions that address aspects for which such bias could be large (e.g. under-represented demographic groups).

Simulations of elections and information integrity present potential dual use concerns. However, information operations and other tactics aimed at influencing elections are already widely deployed, so it is critical to develop reliable approaches towards ensuring healthy, democratic information ecosystems. Furthermore, while bad actors who aim to manipulate can test many strategies in the real world through direct evaluation, good actors are ethically constrained when performing controlled manipulation experiments geared at testing the effectiveness of candidate defenses. Consequently, simulations-which avoid the ethical concerns around human subjects-are critical to level the playing field, and ultimately develop trustworthy and robust solutions to information integrity that resist AI-powered, inauthentic and malign influence operations.

Acknowledgements

We thank Axiom Futures and Berkeley SPAR for connecting collaborators. This work was partially funded by the CIFAR AI Chairs Program and IVADO. Sneheel Sarangi and Gayatri Krishnakumar were supported by funding from Axiom Futures. Andreea Musulan was supported by funding from IVADO. Kellin Pelrine was supported by funding from the Fonds de recherche du Queb´ec.

Contribution Statement

Equal Research Contributions: Maximilian Puelma Touzel lead the project, making substantial contributions to direction and research questions, codebase engineering, writing, coordination/project management, and in general across the project. Sneheel Sarangi made critical idea and engineering contributions in getting the simulations running, scalable, measurable, and demonstrated. He also contributed to multiple other areas like direction and writing. Gayatri Krishnakumar did key analysis and improvements to the environment and agents, including the multimodal, world-mapping, and persona systems, as well as other engineering and input that helped get the simulations running well.

Busra Tugce Gurbuz implemented a multimodal news agent, developed various news types as a part of world mapping, and contributed engineering improvements that enhanced the environment, agents, and overall simulation performance. Austin Welch set up Mastodon and the initial integration with the simulation, set up other fundamental engineering infrastructure and procedures, helped refine the environment and agents, and made substantial contributions to initial direction. Andreea Musulan helped understand the literature, threat modeling, and developing news and malicious agents. Zachary Yang contributed to initial proposal, direction and framing. Hao Yu tested scalability approaches. Ethan Kosak-Hine and Tom Gibbs helped review literature and understand manipulation threats. Camille Thibault helped writing. Reihaneh Rabbany, Jean-Franc¸ois Godbout, Dan Zhao, and Kellin Pelrine advised the project. Godbout led the social science aspects of the research, and contributed key input on direction at all stages of the project.

Equal Advising: Dan Zhao made substantial contributions to direction including the multimodal agent idea, advising, and writing. Pelrine managed the project and contributed key input on direction across many stages, including the worldmapping idea and the key motivation that there are fundamental empirical barriers in this domain that need a new simulation paradigm to break.