MultiPar-T: Multiparty-Transformer for Capturing Contingent Behaviors in Group Conversations

MultiPar-T: Multiparty-Transformer for Capturing Contingent Behaviors in Group Conversations

Dong Won Lee, Yubin Kim, Rosalind W. Picard, Cynthia Breazeal, Hae Won Park

Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence
Main Track. Pages 3893-3901. https://doi.org/10.24963/ijcai.2023/433

As we move closer to real-world social AI systems, AI agents must be able to deal with multiparty (group) conversations. Recognizing and interpreting multiparty behaviors is challenging, as the system must recognize individual behavioral cues, deal with the complexity of multiple streams of data from multiple people, and recognize the subtle contingent social exchanges that take place amongst group members. To tackle this challenge, we propose the Multiparty-Transformer (Multipar- T), a transformer model for multiparty behavior modeling. The core component of our proposed approach is Crossperson Attention, which is specifically designed to detect contingent behavior between pairs of people. We verify the effectiveness of Multipar-T on a publicly available video-based group engagement detection benchmark, where it outperforms state-of-the-art approaches in average F-1 scores by 5.2% and individual class F-1 scores by up to 10.0%. Through qualitative analysis, we show that our Crossperson Attention module is able to discover contingent behaviors.
Keywords:
Machine Learning: ML: Attention models
Computer Vision: CV: Video analysis and understanding   
Humans and AI: HAI: Computer-aided education