Digital Avatars: Framework Development and Their Evaluation

1 Introduction

The performance of LLMs in role-playing has attracted significant attention, with platforms like Character.ai providing virtual character role-playing options. Current platforms are confined to text interactions, and responses in virtual character role-playing still fall short of authentically portraying intended real-world personas. Our paper introduces a comprehensive prompting strategy to increase humor, authenticity and favorability in LLM responses. We call it show don't tell prompting for digital avatars. As part of an initial starting prompt, we define an avatar's background and provide few-shot examples for relevant scenarios, creating response templates emphasizing language nuances and emotional tone. Like other state-of-the-art prompting strategies, our strategy only relies on this initial prompt [Wei et al., 2022; Touvron et al., 2023].

To better visualize our avatar responses, we developed a virtual environment capable of rendering complete responses from LLM output to voice and video. The outcome is a comprehensive end-to-end AI driven digital avatar framework - adept at listening (recording user questions), thinking (computing responses and rendering), and responding (displaying audio-video output) in a highly realistic manner aligned with the intended real-world persona's characteristics. While qualitative results showcase the effectiveness of our strategy, a quantitative metric is necessary. We adapt the usage of Crowd Score for our own purposes in an a evaluation tool we call Crowd Vote. The subsequent metrics, inspired by the original Crowd Score paper [Goes et al., 2022], assess sense of humor, favorability and authenticity in LLM responses. The contributions of this paper are summarized as follows:

• Our show don't tell strategy prompts LLMs to respond with a sense of humor and in an anthropomorphic manner, generating more interesting results as measured by our adaptation of Crowd Score.
• We adapt Crowd Score (code released here) to evaluate the competing AI driven digital avatars on their senses of humor, favorability and authenticity showing we outcompete baseline LLMs not prompted with show don't tell and even outcompete the real-world personas the avatars represent.
• We develop a novel visualization pipeline as a way of demonstrating the quality of our prompting results using an end-to-end digital framework. Shown here.

2 Approach

2.1 Prompt Mechanism

In this section, we propose the novel prompting strategy - show don't tell for digital avatars. Our strategy is similar to other few-shot LLM prompt strategies [Wei et al., 2022; Touvron et al., 2023]. Rather than solely giving various instructions (zero-shot learning), we give examples for the LLM to learn from directly (few-shot learning). In our initial prompt, we provide a large number of examples and we briefly define the avatar's role. We record in this initial prompt as many characteristic responses of the real-world persona as possible in various situations, such as reactions when challenged by reporters, or responses to pointed questions. Finally, to make the responses more lively and interesting, we incorporate elements of humor and entertainment into our final version of the prompts. For example, if an avatar's real-world persona appeared on a comedic show, we would sample jokes from their act hoping that a parody of the reallife persona distills relevant humor into the avatar persona.

2.2 An End-to-End Digital Avatar Framework

We develop an end-to-end AI pipeline to demonstrate the prompting strategy developed in this work. Using local implementations of various states of the art, we find the same qualitative trends regardless of baseline architecture. We believe we are the first to use an end-to-end AI driven digital avatar pipeline that includes a large language model for avatar speaking. This framework is presented in full in Figure 1.

Speech-to-Text and Text-to-Speech

Speech-to-text (STT) is the process of converting spoken language into text. The states of the art demonstrate impressive results in English and Mandarin [Amodei et al., 2016] and can be robust to noisy environments [Radford et al., 2023]. Text-to-Speech (TTS) is a technology that converts written text into spoken words using synthesized voice. WaveNet [Oord et al., 2016] is a deep learning approach that improved the naturalness of synthesized speech. Recently, FastSpeech [Ren et al., 2019], presented a novel network-based TTS model that maintained high-quality audio output efficiently.

Talking Face Synthesis

Talking Face Synthesis (TFS) aims to generate talking heads that are synchronized with input audio. Prior works [Prajwal et al., 2020; Cheng et al., 2022] adopt pre-trained experts in lip-audio synchronization to guide the training of Generative Adversarial Networks for achieving highly synchronized and high fidelity videos. These works suffer from unpleasant frame continuity and a lack of high-texture information. We propose a codebook-based TFS method that generates talking faces from both low and high semantic levels incorporated with audio and reference images. Our method takes advantage of the strong pre-trained codebook decoder to ensure a rich-textured and high-fidelity face generation.

Video Selection Algorithm

Depending on the end goals of our video demo we can take video data from different camera perspectives. For example, if we don't care about camera transitions, we use one camera angle for thinking and another for listening to the user. When processing is complete we cut to the second camera angle paired with lip synced avatar output. We also can provide seamless transitions between listening, thinking of a response, and answering. By labelling every point in a sample video when the real-world persona transitions from listening to talking, we can start our demos at these labelled positions. We can play the video in reverse to create listening data as a user asks a question. Then during processing we play the video sample forward and if necessary use slow motion and frame skipping to sync the end of processing to the start of the avatar answer where the real-world persona in the video sample begins talking.

2.3 EvaluationMetrics

Crowd Score

Crowd Score [Goes et al., 2022] introduced a novel method for evaluating humor by utilizing an LLM to serve as judges, offering a unique approach to gauge humor effectiveness. Crowd Score as it was originally implemented ranks a series of jokes from least to most humorous without much regard for where these jokes come from - whether a single LLM or multiple instances. Crowd Score used judging personalities that reflect preferences for types of humor including: "afilliative", "self-enhancing", "aggressive", and finally "self-defeating."

Introducing Crowd Vote

Our Crowd Score, that we refer to as Crowd Vote, asks judges [OpenAI, 2023] with different assigned personalities to pick a single response from many LLM candidates responding to the same prompt that best matches a condition like authenticity, favorability, or humor. A tally is kept for each candidate

Character-Joe Biden

Table 1: Response Comparison: Green text indicates the LLM has provided a humorous or interesting response, while Blue text indicates the LLM's reply is more characteristic of the persona of Joe Biden. Green and blue details likely originate from different few-shot samples seen in start prompt. The baseline LLM is capable of bringing these details into foreground, but show don't tell prompting near guarantees it.

as they respond to multiple prompts allowing for an overall comparison between competing LLM approaches. The frameworks presented here can work with any avatar persona. Yet for the purpose of this demo we use most often Donald Trump and Joe Biden. Because we are creating avatars for two politicians, we decided to judge our avatar responses on two conditions politicians often find themselves being judged on: authenticity, and favorability. Our prompting strategy and application of Crowd Vote are seen in our demo and code.

3 Experiments

The candidates: To judge our prompting strategy relative to a baseline LLM on humor and interest we include three candidates in our evaluation: a) Baseline LLM (zero-shot) b) Character.ai (few-shot) c) Our LLMwith show don't tell prompting (few-shot) and d) (when possible) the real-world avatar target. Please note that character.ai is a notable platform in the field of AI-driven digital avatars with over 20 million users, making it a very strong baseline.

3.1 Interest and Humor

We use our implementation of Crowd Vote to judge our avatar responses. You can see the results of our experiment in our demo slides. The vote percentages of the judges are 58.3% in favor of our show don't tell prompted LLM being most humorous, followed by abstaining judges at 20.8%, and then the baseline LLM, and Character.ai. We show the quality comparison of the response between a baseline LLM, Charater.ai, and our own work. Table 1 shows that even though the baseline LLM is powerful, without any few-shot examples in an initial prompt it often fails to display any sense of humor in its responses. Character.ai provides a straightforward and somewhat mundane response, depicting a simple loss of sunglasses due to a bike accident. While this response might be relatable, it lacks imaginative and humorous elements. Our LLM response is playful and self-deprecating, creating a humorous scenario where the sunglasses are misplaced in a common yet amusing way - on top of the head. This response also cleverly ties in the ideas of misplacing one's sense of humor and forgetfulness, adding additional layers of wit. The invitation to go on a sunglass hunt adds an engaging and interactive element to the answer, increasing its charm.

3.2 Authenticity and Favorability

To compare an avatar to their human counterpart one needs a fair comparison between avatar answers and real-world person answers. In the first Presidential Debate of the 2020 campaign season we were able to find 17 questions resulting in responses that could be compared between a Trump avatar or Biden avatar and all their respective competitors. We publish this small dataset with our code. Once we had a dataset that comprised real world Donald Trump and Joe Biden answers on a variety of topics, we asked our LLM candidates the same questions and we used Crowd Vote to compare our avatar answers to the real-world Donald Trump and real-world Joe Biden on metrics like authenticity and favorability.

The authenticity and favorability experiments used different Crowd Vote judges to vote on all the candidates' answers to the 2020 debate questions. For authenticity we asked "a pychologist", "a political commentator", "an American voter", "a close family member of the avatar" and the avatar's adversary. For our tests on favorability our judges span the typical American political spectrum: "far-right", "conservative", "centrist", "liberal", and finally "far-left". Figure 2 shows that in both tests our LLM initialized with our show don't tell prompting strategy out performs the popular character.ai, as well as the baseline LLM intitialized without show don't tell. Our Trump and Biden avatars actually outperform each of their real-world counterparts both in terms of authenticity and favorability. Both of our avatars won easily against all other LLM candidates and real-world personas and subsequently advanced to a "general election" that we hosted. Our avatar of Joe Biden won against our avatar of Donald Trump: 45% to 37.5% with 17.5% abstaining.

4 Conclusion

Our prompting strategy shows both qualitatively and quantitatively impressive results. To measure this, we proposed Crowd Vote adapted from Crowd Score. With Crowd Vote we demonstrate that our show don't tell prompting strategy generates avatar responses that out compete all competitors including the real-world personas they represent in terms of authenticity and favorability. To further demonstrate the anthropomorphic nature of our responses, we developed an endto-end pipeline agnostic to specific architecture that renders high quality avatar interactions.

Acknowledgements

This research was supported by National Science Foundation grant CNS-2312158.

Contribution Statement

Timothy Rupprecht and Sung-En Chang contributed equally to this work as first authors. Thanks to Pinrui Yu, and Priyanka Maan for helping with development.