MultiPar-T: Multiparty-Transformer for Capturing Contingent Behaviors in Group Conversations

1 Introduction

In order to develop AI agents that can co-exist with people in the real-world, it must be able to understand people's behavior in a multiparty (group) setting, as many common forms of important communicative behavior take place in small group settings. Accurate recognition and interpretation of multiparty behavior enables AI agents to support and facilitate group conversations across many domains, including lessons, business meetings, and collaborations at the workplace.

Recognizing and interpreting group behaviors is much more challenging than that of individual behaviors. Firstly, the system must perform well in recognizing individual behavioral cues. Secondly, it must do so simultaneously, while keeping track of every individual in the group. Finally, it must also recognize the subtle interactions that take place between group members as it can provide more insights into what is being communicated. Natural human conversations are interactively contingent, where people act and react in a coordinated fashion in turns [Kopp, 2010]. Consequently, understanding group behavior in multiparty conversations requires recognizing contingent behaviors between group members.

To address these challenges, we propose the MultipartyTransformer (MultiPar-T), a transformer model for multiparty behavior modelling, which is able to handle multiple streams of input data for all of the members of the group, unlike most previous works which focus solely on individual interactions without taking into account the relationship between people's behaviors. At the core of MultiPar-T is Crossperson Attention (CPA), which is used to attend to the behaviors of the entire group and capture contingent behaviors. Instead of using cross attention to discover alignment between two sequences of different modalities (i.e. vision and language) [Tsai et al., 2019] or differing views of a single visual input to learn multi-scale feature representations as in previous approaches [Chen et al., 2021], we propose and show that cross attention can be effectively used to capture contingent behavior between two behavioral sequences across pairs of people. CPA implicitly searches for how and when one person's current behavior is contingent on another person's past behavior, whereas previous approaches that do not take contingent behaviors into account. Via careful construction of the direction of attention, Crossperson Attention controls the direction of contingent behaviors it captures. Furthermore, applying multiple layers of crossperson attention allows the model to discover relationships between the contingent behaviors with the other parts of the target person's behavioral sequence. We also include a self-transformer to address cases where behaviors are non-contingent and need to rely solely on the target person's behaviors. In summary, given a target person's input behavioral sequence, MultipartyTransformer applies Crossperson Attention with the behavioral sequences of other members of the group in a pairwise manner to output an embedding that has contextualized information about the rich, social contingent interactions in reference to the target person's behaviors.

In order to measure the effectiveness of our proposed approach, we focus on the important task of engagement prediction in online learning activities. Engagement prediction is a task that requires understanding of contingent behaviors, as many studies demonstrate that the presence and lack of contingent behaviors influence people's engagement [Masek et al., 2021; Xu et al., 2022; Sage and Kindermann, 1999]. Furthermore, engagement detection in group settings is an important problem in developing AI systems that can gauge a group's interest to develop behavior policies and strategies to maximize the group's overall satisfaction with the AI agent's actions. We establish baselines to compare against the proposed model on a publicly available group engagement detection dataset in an online educational setting [Reverdy et al., 2022], where we find that MultiPar-T consistently outperforms previous approaches, across all levels of engagement. We provide in-depth ablation studies to show how each specific components in MultiPar-T contributes to the performance boost. Furthermore, we empirically show that the Crossperson Attention mechanism is able to discover contingent behaviors across pairs of people, which is especially important with the new EU policies [EUCommission, 2021] requiring explainability in affect recognition models.

Our contributions are summarized as follows: (1) We introduce the Multiparty-Transformer (MultiPar-T), a novel transformer model which can handle multi-stream multi-party data in group conversations. (2) The key component is Crossperson Attention, which is the first of its kind to reframe cross attention to discover contingent behavior between group members by controlling the direction of the attention. The output embedding prioritizes parts of the target person self's behavioral sequence that is contingent on another person other's behavior. (3) The inclusion of the self-transformer module further contextualizes the embedding with the target person's own behavior and handles non-contingent behaviors. (4) We run extensive experiments on a important and timely task of multiparty engagement detection in online educational setting and show that MultiPar-T significantly outperforms all previous state-of-the-art approaches 1 .

2 RelatedWorks

2.1 Contingent Behavior

Contingent behavior refers to a person's action that takes place as a response to another person's behavior. It falls under a broader umbrella of "interpersonal coordination" [Bernieri, 1988], which refers to people's behavioral adaptations that happen as a result of social resonance in natural interaction. An example of contingent behavior is mimicry and interactional synchrony. Furthermore, nonconscious contingent behavior acts as a "social glue" to enhance the naturalness and sympathy in conversation [Lakin et al., 2003]. Temporal coordination between people in communication is found in body movements and facial expressions [Bernieri et al., 1994]. Studies have shown that contingent and reciprocal interaction amongst groups of peers [Sage and Kindermann, 1999], teachers and students [Boyd and Rubin, 2006], caregivers and children [Masek et al., 2021; Chen et al., 2022], children and on-screen characters [Xu et al., 2022], robots and humans [Park et al., 2017; Chen et al., 2020b], influences individual engagement, motivation, and learning.

2.2 Modeling of Group Interactions

In small group interactions (at least 3 people), each person has their own attributes, and each member of the group communicates with each other via both nonverbal and verbal behaviors [Adams et al., 2006]. Graphical modelling of people's interactions have been explored in various tasks such as prediction of group performance [Lin and Lee, 2020], group behavior recognition [Yang et al., 2020], social interaction field modelling [Zhou et al., 2019], and social relation recognition [Li et al., 2020]. The above-mentioned works all involve representing each person's individual features as a node, and their interactions as their edges. There has been work that utilizes attention in modeling dyadic interactions [Curto et al., 2021]. However, our work addresses a more complex multiparty interaction setting with a novel set-up of cross attention which captures contingent behavior in all pairwise interactions.

1 The scripts used for all experiments can be found at:

https://github.com/dondongwon/Multipar-T

2.3 Engagement Prediction

At a high level, engagement is defined as a state of consciousness where a person is fully immersed in the task at hand [Ren, 2016]. Studies have investigated finer differences between specific types of engagement and shown that engagement is defined to be a multi-dimensional construct [Fredricks et al., 2004], composed of behavioral (e.g. [Griffin et al., 2008]), cognitive (e.g. [Corno and Mandinach, 1983]), emotional (e.g. [Park et al., 2012]), and attentional (e.g [Chapman, 1997]) engagement. In our work, we focus on perceived behavioral and emotional engagement. Previous approaches utilize CNN-LSTM models to predict engagement [Del Duchetto et al., 2020; Steinert et al., 2020]. More recently, models that use bootstrapping and ensembling are proposed in BOOT [Wang et al., 2019] and ENS-MODEL [Thong Huynh et al., 2019]. HTMIL uses a Bi-LSTM with multi-scale attention and clip-level and video-level objectives [Ma et al., 2021], and TEMMA [Chen et al., 2020a] utilizes a Resnet-Transfomer model. Unlike our work, previous approaches do not take into account the group setting; they focus on modeling individuals. Closest to our work in multiparty engagement prediction is the work of [Zhang et al., 2022], where they utilize a graph attention network (GAT) to contextualize social interactions between multiple people to estimate engagement in elderly multiparty human-robot settings. To the best of our knowledge, we are the first to utilize a transformer network to model group's behavioral contingencies for engagement prediction.

3 Problem Statement

We formulate the multiparty video-based engagement prediction problem as the following. We are given video clips of groups involved in an online learning activity. We split the videos into N interval clips of k frames. At any arbitrary time t, where t is the exact timestep in which we want to predict each individual's engagement value, we are given the [t - k, ..., t] interval of contextual video information; k is the number of frames we will use as context. Let P be the number of all participants in the video, for a person in the clip p ∈ [P], their corresponding contextual behavioral features can be viewed as Xtp = [xt-k p , . . . , xtp ], where xtp ∈ RF with dimension size F at the tth frame. For brevity, we will drop the t and assume it is arbitrarily fixed. The input with all of the group's features is a 3-D tensor, X = [x1, . . . , xP ] ∈ RP×k×F . For a target person: self, we train a model that takes as input X, which includes the target person's features xself as well as all other members' features [xother, ∀other ∈ P \ {self}] to predict the engagment value ˆ Yself ∈ (0, 1)C .

4 Methods

Here, we describe our proposed Multiparty-Transformer (MultiPar-T). We utilize the Crossperson Attention (CPA) module to discover the contingencies across time-series sequences of behaviors from pairs of people in the group.

4.1 Crossperson Attention

Given a pair of people, we want to capture their contingent behaviors. We state that target person self's behavior is contingent on person other's behavior if person self's behavior was likely to be influenced by person other's behavior (other→self). We posit that capturing contingent behavior across people can be handled with mechanisms that can capture alignment between sequences. Inspired by the multimodal transformer model [Tsai et al., 2019], which shows one effective method that can automatically align sequences of differing modalities is by using a scaled dot product cross attention [Chen et al., 2021; Vaswani et al., 2017], we propose that it could be applied to pairs of behavioral sequences, whichwe call CrosspersonAttention (CPA), to automatically discover contingent behavior and subtle social interactions.

Cross attention utilizes query Q, key K, and values V , where the first step is to find the importance of each key with respect to the query. The attention mechanism computes the dot product of the query with each key to obtain a weight for each key. These resulting weights represent the importance of each key to the query. The weights are then used to obtain a weighted sum of the values in the matrix, which is referred to as the context vector. Therefore, for the target person self, and another person other, we are given their time-aligned encoded visual representations Zself , Zother ∈ Rk×dx , where dx is the dimension size of the visual embeddings. Therefore, we construct the queries, keys, and values as the following: Querys as Qother = ZotherWQother , Keys as Kself = ZselfWKself , and Values as Vself = ZselfWVself , where WQother ,WKself ,WVself ∈ Rdx×dx are trainable weights.

We refer the readers to Figure 1(a) for a visual depiction. In Equation 4.1, the scaled softmax produces the attention weight between two people's temporal behavior inputs, which weighs the importance of person other's each behavioral timesteps with respect to the self's behavior. Specifically, the resulting weight is a k × k matrix, where k is the number of timesteps in the sequence. After the dot poduct with Vself , the Crossperson Attention from other to self CPAother→self(Zother,Zself) outputs an embedding which has captured the person self's behavior contingent on person other's behaviors. We highlight this is the reverse direction of cross attention compared to many previous works [Tsai et al., 2019; Curto et al., 2021], and a crucial distinction in capturing contingent behaviors as we discuss in Section 6.1. Furthermore, Crossperson Attention mechanism is performed with h multiple heads; we define this as CPAmulti other→self .

The outputs of each head of CPA are concatenated, then linearly projected with weight matrix: W multi ∈ R(h·dx)×dx .

4.2 Multiparty-Transformer

In order to successfully address the complex social interactions taking place in a group setting, we must properly represent each person's individual temporal features, address the group social interactions, then take into account the group's temporal nature. We describe in detail the individual components which are designed to tackle these challenges.

Individual Temporal Encoder: Convolutions and Positional Encoding We utilize 1D convolutional layers such that the convolution kernel convolves over the temporal dimension and each timestep in the sequence is contextualized by its surroundings. Furthermore, we further enforce the temporal structure by including the additive positional encoding (PE) used in [Vaswani et al., 2017]. Therefore, the individual temporal encoder, given target person self's input Xself is:

Conv1D includes a kernel that maps each individual's features into a common dimension dx.

Behavior Interaction Encoder: Crossperson Transformer & Self Transformer Crossperson Attention (CPA) is a core component of the M-layered Crossperson Transformer (CPT). CPAm,multi other→self refers to the multi-headed Crossperson Attention from person other to person self at the m-th layer. Following standard transformer operations [Vaswani et al., 2017], ˆ γm other→self refers to the intermediate output after the Crossperson Attention with residual connections. γm other→self refers to the final output of a cross-person transformer block after feedforward network (FFN) and residual connections. As the input to the first layer, γ 0 other→self = Zother. We refer the readers to Figure 1(b) for details.

With this formulation, CPA0other→self (Zother,Zself) discovers contingent behaviors in the first layer. Then, in the later CPT layers, CPA contextualizes the embedding by discovering correlations on how the contingent behavior is related to different parts of the target person's behaviors. We empirically show that standalone first layer CPT is not enough, and that further contextualization with multiple layers of transformer blocks is useful in Section 6.1.

In addition to all pair-wise Crossperson Attention across all other members of the group ∀other ∈ P \ {self}, CPAmulti other→self , we also compute self-attention in order to (1) account for how one's earlier behavior correlates with their current behavior and (2) handle cases where there are no contingent behavior information. This is equivalent to performing CPA with an equivalent query, key and value matrices (i.e. given a target person self we perform CPAmulti self→self (Zself , Zself )). Its usefulness is tested with ablation studies in Section 6.1.

Temporal Classifier Finally, we concatenate the outputs from the above-mentioned behavior interaction encoder, [·||·] refers to concatenation. The concatenated outputs are fed into an LSTM for k steps to enforce a stronger temporal structure. The resulting output is passed through fully connected layers (FFN) for the final prediction ˆ Yself ,

5 Experiments

5.1 Dataset

We utilize the RoomReader [Reverdy et al., 2022] as a benchmark to measure the performance of our proposed method against other baselines. RoomReader [Reverdy et al., 2022] is a corpus of multimodal, multiparty conversational interactions in which participants followed a collaborative online student-tutor scenario designed to elicit spontaneous speech. Online interaction settings are well-suited to study contingent behaviors as nonverbal online actions, indication of online presence, demarcation of others' contributions, and expressions of support affect interactants' experiences in videoconferencing [Park and Whiting, 2020]. Engagement is focused on off-task/on-task engagement, where the task at hand is led by the instructor. Roomreader consists of 5-person and 4-person settings. In our experiments, we focused on the 5-person setting which is the majority of the dataset (26/30 groups, 106/118 unique participants).

Engagement Classification RoomReader provides continuous annotations for engagement, where the labels range from [-2, 2]. Instead of regression, we define the task as a 4class classification, where labels between (1, 2] refer to high engagement, (0, 1]: lowengagement, (-1, 0]: lowdisengagement, (-2,-1]: high disengagement. Setting up the task in this way results in more interpretable evaluation metrics than regression losses (such as MAE, or MSE), and allows us to report categorical metrics conditioned on each class. Generally, it is well known that class imbalance is often severe for datasets with engagement labels [Del Duchetto et al., 2020; Dhall et al., 2020; Steinert et al., 2020]. We refer the readers to Figure 2 for the imbalanced distribution of labels. To counter the effects of class imbalance, we (1) oversample the infrequent class to balance the dataset and (2) train the models with a Focal Loss [Lin et al., 2017] that applies a modulating term to the cross entropy loss to focus learning on hard misclassified examples. yic refers to the ground truth labels, ˆ Yic is the probability prediction, and α is a hyperparameter that weighs how much easy samples should be down-weighted.

Combined, we find that the models are able to predict the infrequent classes and overcome the class imbalance problem, which can be seen in Figure 2.

Data Preprocessing For the input features, we utilize the normalized eye gaze direction, location of the head, location of 3D landmarks, and facial action units extracted via OpenFace [Baltrusaitis et al., 2018]. In addition, we extract framewise image features from the penultimate layer of Resnet50 [He et al., 2016]. The two features are concatenated per timestep to be used as input. The input feature dimension size per timestep is F = 2183. For each label at timestep t, we use 8 seconds worth of video context information, where the frame rate is 8 fps. We utilize k = 64 frames as input. We apply a sliding window with an interval of 1 second between each sample. In total, we have 184970 samples.

5.2 Baseline Models

We compare our proposed model with a family of baselines in engagement prediction, as well as action recognition. We run the newest versions of these models and report their scores on a unified benchmark 2 . We compare MultiPar-T to ConvLSTM [Del Duchetto et al., 2020], OCtCNN-LSTM [Steinert et al., 2020], TEMMA [Chen et al., 2020a], BOOT [Wang et al., 2019] and ENS-MODEL [Thong Huynh et al., 2019], GAT [Zhang et al., 2022], and MulT [Tsai et al., 2019]. For action recognition models, we compare our method with TimeSformer [Bertasius et al., 2021], SlowFast [Feichtenhofer et al., 2019] and I3D [Wang et al., 2018].

5.3 Implementation Details

We train our models on 2 NVIDIA GeForce GTX 1080 Ti with a batch size of 64 for 20 epochs. We use the AdamW [Loshchilov and Hutter, 2017] optimizer with an initial learning rate of 0.0001 with a scheduler that decays the learning rate by 0.1 every 5 epochs. We train on 16 groups' data, validate on 3 groups, and test on 1 group for 3 seeds. The model is exposed to different totally held-out subsets of groups for cross-validation. MultiPar-T can be used in real-time, where the inference time only takes 0.0981 ± 0.0029 seconds.

2 We also compare with other multilabel classification methodologies in an extended version: https://arxiv.org/abs/2304.12204

6 Results &Discussion

In this section, we discuss the quantitative and qualitative results of our experiments. We compare our approach MultiParT with state-of-the-art baselines. Then, we discuss the importance of the attention modules and the effects of their directions. Finally, we qualitatively demonstrate that Crossperson Attention has learned to recognize contingent behaviors.

6.1 Quantitative Results

Following previous works [Del Duchetto et al., 2020; Dhall et al., 2020; Steinert et al., 2020], we report accuracy and weighted-F1, which is the weighted mean of all per-class F1 scores considering each class's support in the data. Most importantly, we report the macro-F1, i.e., the unweighted mean of per-class F1. A high macro-F1 score demonstrates that the model performs well across all engagement classes regardless of its frequency in the dataset.

Comparisons Against State-of-the-Art In Table 1, stateof-the-art engagement prediction models are compared to MultiPar-T. MultiPar-T outperforms all state-of-the-art approaches in accuracy by 2.9%, weighted F1 by 3.0%, and Macro-F1 by 5.2%. For a closer look into how MultiParT performs for individual levels of engagement, we refer the readers to the right of Table 1, where we compare each model's performance for each class. Although other baselines result in comparable high accuracy and weighted-F1 scores, they fail to predict the infrequent disengagement class. MultiPar-T has significant performance gains (10% increase against best performing baseline) in the most challenging task of accurately predicting high and low disengagement, which consists of only 2% of the entire dataset. Moreover, comparing with the other group behavior encoding method, GAT [Zhang et al., 2022], we find that MultiPar-T outperforms across all metrics, highlighting that our method is a more effective method of capturing group behaviors. We also compare with an adaptation of the multimodal transformer (MulT) [Tsai et al., 2019] where we replace differing modality inputs with differing person's behavioral sequences, which is equivalent to MultiPar-T w/o Self Transformer and reversed CPA direction self → other in Table 2. The inclusion of the self-transformer and the configuration of the attention directions is a key component in modeling human multiparty behavior, different frommultimodal alignment, as we demonstrate via ablation studies in the following sections.

Importance of Crossperson Attention (CPA) and SelfAttention In Table 2, we present results where we ablate Crossperson Attention and Self-Attention from our models. We see that ablating Crossperson Attention leads to a significant drop in performance metrics, especially Macro-F1. The model struggles at harder, low-data disengagement instances. Therefore, the inclusion of Crossperson Attention, which allows the model to attend to how others are behaving in the group provides more context information for the model to differentiate harder cases. We also find that the ablation of selfattention leads to significant drops in performance metrics, as self-attention provides more information regarding one's own behavior, which is especially important when there are no contingent behaviors.

Importance of the Direction of CPA In predicting a target person self's engagement value, we hypothesized that the self's behavior contingent on other's behavior is an important predictor of engagement and disengagement. On the other hand, we hypothesized that the other's behavior contingent on the self's behavior would not be an important predictor. To test these hypotheses, we experiment with the directions of the Crossperson Attention mechanism. In Table 2, CPA: other → self refers to our attention direction, performing Crossperson Attention where the query corresponds to behaviors of other persons in the group, and the key and value correspond to the behavior of the target, self. The resulting embedding contains information about the self's behavior which is contingent on other's. Conversely, CPA: self → other, outputs an embedding with the other's behavior contingent on the self's behavior. This is similar to the cross attention set-up in [Tsai et al., 2019; Curto et al., 2021].

We refer the readers to the results for MultiPar-T with other → self and self → other. We find that MultiPar-T with our formulation of cross attention, CPA: other → self , performs significantly better, which indicates that it is important to explicitly set the direction of cross attention such that the output embedding prioritizes parts of target's behavioral sequence that is contingent on another person's behavior. Interestingly, when predicting low disengagement, MultiPar-T with CPA: self → other results in better performance. This shows that how the self impacts others could be a important predictor when predicting if the target person is disengaged.

Encoding Size & Transformer Layers In Figure 3, we first display results for ablations on varying number transformer layers M. The first layer of CPT encodes the self's behavior contingent on other. The later layers further contextualizes the contingent self's behavior with its own behavior again, allowing it to attend to other parts of its own behavioral sequence. We find that having multiple layers leads to a significant improvement in Macro-F1. Secondly, we display results for varying sizes of the embedding dimensions per timestep dx. We find that the optimal encoding dimension to encode behavior per timestep setting is dx = 100.

Comparisons against Action Recognition Models Following recent approaches in utilizing action recognition models in engagement prediction [Ai et al., 2022; Kim et al., 2022], we compare MultiPar-T to activity recognition models in Table 3. Training state-of-the-art action recognition models are computationally much more expensive, due to the fact that the models are trained on a time-series of raw pixels endto-end. Therefore, instead of applying an 8 seconds window with 1 second interval, we apply an interval of 8 seconds. Even with a modified set-up, there is a large discrepancy between the training time, one seed takes ∼ 150 minutes for a raw video-based models, compared to ∼ 20 minutes for an engagement prediction models. For a fair comparison, we train MultiPar-T with the same training settings and find that MultiPar-T performs better than other architectures given the same training conditions for scarcer disengagement samples.

6.2 Qualitative Analyses

Given the upcoming EU regulations [EUCommission, 2021] requiring explainability for any affect-related AI, an important facet of our work is that the resulting attention weights can be used as a way [Wiegreffe and Pinter, 2019] to explain why the model made this specific prediction for this specific timestep. Given an AI agent which utilizes MultiPar-T as its backbone engagement detection module, if a person inquires the agent why it thought that they were disengaged at a point in time, the AI agent could examine its Crossperson Attention weights and provide some rationale behind its prediction based on discovered contingent behaviors.

To demonstrate how Crossperson Attention (CPA) could be used to explain model predictions, in Figure 4, we visualize the attention weights from the first layer of the Crossperson Attention between Person self, (left) and Person other, (top) and show that it is able to capture contingent behavior between pairs of individuals' behaviors. The x and y axes refer to each person's behavior aligned to timesteps. We first provide attention weights that demonstrate the lack of contingency for comparison. The attention weights visualized in Figure 4(b) Diagonal Contingency, is a diagonal attention weight matrix, which indicates that Person self and Person other's behavior are only related at the exact same time steps. The attention weights visualized in Figure 4(c), Uniform Contingency, is the default behavior if we assume that all of Person other's past behavior is uniformly related to Person self's current behavior, demonstrated by the uniform color across each row, which indicates that the attention weights are uniformly distributed across the available timesteps. The upper triangular matrix is masked to encode the natural assumption that other's future behavior shouldn't affect self's current behavior.

Figure 4(a) shows the learnt Crossperson Attention weights, which indicates that Person self's behavior in timesteps 16-60 is contingent on Person other's behavior in timestep 20-25. We find that, after inspecting the video at the aligned timesteps, that Person other laughs during timestep 20 - 25 and starts to talk afterwards. Person self was initially distracted, but after they see Person other laughing at timestep 20 - 25, they look at Person other and starts listening. Hence, we find that Crossperson Attention has discovered meaningful contingent behavior between two people.

7 Conclusion

In this work, we study the challenging task of modelling human behaviors in a multiparty setting. We proposed a new Transformer-based model for multiparty behavior modeling,

Person other's Behavior

MultiPar-T. At the core of MultiPar-T is Crossperson Attention, which is designed to capture contingent behaviors. We compare MultiPar-T against previous approaches on a timely and challenging task of engagement prediction in online meetings and provide in-depth analysis and ablation studies. We see significant gains in performance (up to 10%) compared to previous approaches, where we find that controlling the direction of Crossperson Attention, including multiple layers of transformer blocks, and self attention blocks is crucial. We also demonstrate qualitatively that our model is able to find contingent behaviors. Our MultiPar-T is a novel approach to modeling contingent behaviors in multiparty conversation, a crucial problem in developing AI agents that can communicate with groups of people. We publicly share our code to enable research in multiparty interactions.

Limitations and Future Work Our evaluation benchmark Roomreader [Reverdy et al., 2022] is collected from online lessons, which is one specific context. Future works should test the generalizability of models in a larger number of datasets and benchmarks consisting of different contexts including in-person settings, various relationships, diverse cultures, and a wider range of age and number of participants. We believe that our proposed method would generalize across different group settings, environment, and tasks, as supported by literature reviewed in Section 2.1 that contingent behaviors play an important role in various interaction settings.

Acknowledgments

This work was supported by the IITP grant funded by the Korean Ministry of Science and ICT (No.2020-0-00842, Development of Cloud Robot Intelligence for Continual Adaptation to User Reactions in Real Service Environments). Yubin Kim was supported by Korean Ministry of Trade, Industry, and Energy (MOTIE), under Human Resource Development Program for Industrial Innovation (Global, P0017311) supervised by the Korea Institute for Advancement of Technology.