Learning Graph-based Residual Aggregation Network for Group Activity Recognition

1 Introduction

Group activity recognition (GAR) has many potential applications including video surveillance, social behavior understanding, and sports video analysis. Compared with individual action recognition (e.g., running or jumping), GAR is a more challenging task (e.g., talking together or queuing up) that lies not only in the recognition of individual behaviors, but also in the exploration of the interactions among different persons [Masato et al., 2011]. Therefore, GAR is more easily explained and makes sense in practice.

Given a video sequence, most of existing methods are executed in the following sequential process. First, individual features are extracted based on a backbone network. Second, the spatiotemporal interactions are explored by the graph neural networks [Wu et al., 2019; Yan et al., 2020] or recurrent neural networks (RNNs) [Wang et al., 2017]. Finally, a holistic feature vector is generated to represent the group activity by typically pooling over personal features. However, most of existing methods are limited to the coarse individual-level features, and the fine-grained relations have not been considered as mentioned in [Qi et al., 2020]. Specifically, the differences between an individual and others are ignored, which are vital to suppress the interference of local similar motions.

∗Corresponding author.

To better illustrate this, the group activities of "right set" and "right pass" are taken as examples in Figure 1. A player who is "setting or passing" in the volleyball game adjusts his pose over time to handle the ball. Meanwhile, some of his teammates also move around him. It is difficult to distinguish them based solely on appearance-level features and their interactions, leading to unsatisfactory recognition performance. Fortunately, although the players of both the "right set" and "right pass" have similar motions, their teammates or opponents may form certain latent patterns. For example, the stance and pose among players will maintain consistent changes in a group activity. Therefore, the differences between an individual and others need to be quantitatively modeled as additional information to enhance the features of a whole activity. To this end, a new module is proposed to exploit the local difference relations among the whole group, namely Local Residual Relation Module (LR2M), which is modeled by measuring the sum of residuals within the group. More concretely, LR2M has the ability to capture the latent differences in participants' spatiotemporal features (e.g., the spatial variations and behavior changes over the whole activity), while suppressing the redundancy of temporal actions (e.g., the same motions over time). Furthermore, LR2M can be easily combined into an existing relation graph framework to form the dual-branch relation reasoning.

Empirically, although all persons may be involved in a group activity, only a few motions or interactions are crucial to inferring the class of group activity. For example, the residual or graph-based relations may contain redundant interactions between outlier persons. PRL in [Hu et al., 2020] has validated that it is vital to model the group-relevant and suppress the irrelevant features for GAR. However, most of existing methods treat multi-level features with equal importance, and integrate them by simple summing operators. Thus, it is necessary to find an effective strategy to integrate the multilevel features. Based on dual-branch relation reasoning, a Weighted Aggregation Strategy (WAS) is proposed to learn the weights of multi-level spatiotemporal features. The basic idea is to use adaptive gates to control the information flows from different branches to the holistic feature vector, while suppressing the irrelevant information across multiple persons. Overall, the main contributions can be summarized as follows.

• Going one step further beyond the existing methods, a novel graph-based residual aggregation network (GRAIN) is proposed to encode the latent differences among all persons and learn the refined features for GAR, which is end-to-end trainable. To the best of our knowledge, GRAIN is the first work to model the differences among all the persons in a whole group for GAR.
• To handle the inter-class similarity of different group activities, a new local residual relation module (LR2M) is
• proposed to model the latent spatiotemporal differences among the whole group.
• To suppress the irrelevant information, a weighted aggregation strategy (WAS) is devised to aggregate multilevel spatiotemporal features.
• Extensive experiments on two publicly available datasets show that GRAIN outperforms the state-of-theart methods.

2 RelatedWork

2.1 Action Recognition

Action recognition aims to detect the individual actions from video sequences or still images [Liu et al., 2021]. To capture the temporal dependencies of visual features, RNNs have been used to model the actions in videos. To handle the spatial configurations of articulated skeletons, a twostream RNN architecture [Wang and Wang, 2017] is proposed to model both temporal dynamics and spatial relations for skeleton-based action recognition. In TEA [Li et al., 2020], both short-range motion excitation and long-range temporal aggregation are designed to capture both short- and longrange temporal evolution. In addition, some graph-based methods have been proposed to model the spatiotemporal information for long-term action recognition [Li et al., 2021a; Zhou et al., 2021]. Different from the focuses of aforementioned methods, our method elegantly explores the different relations in a social group, where multiple persons are involved in modeling their interactions.

2.2 Group Activity Recognition

Initial GAR approaches employ the probabilistic graphical models [Amer and Todorovic, 2015] or AND-OR grammar models [Shu et al., 2015] to predict the group activities after extracting the hand-crafted features. The inter-person relations can also provide some important cues [Qi et al., 2020]. ARG [Wu et al., 2019] firstly employs graph convolution networks (GCNs) to learn the appearance and position relations. HiGCIN [Yan et al., 2020] designs a hierarchical graph-based cross inference network to extract multi-level features from the body-region to group-activity for GAR. Inspired by deep reinforcement learning, a progressive relation learning method is proposed to distill the features of group-relevant actions and interactions for building high-level semantic relation graph [Hu et al., 2020]. Furthermore, transformer [Gavrilyuk et al., 2020; Yuan and Ni, 2021; Li et al., 2021b] is employed to bridge the gap between group activity and visual context information. However, the finegrained relations have still not been elaborately explored by previous methods, such as the differences between an individual and others. These prompt us to develop a novel module to capture the differences, while suppressing the interference of local similar motions.

3 The ProposedMethod

3.1 Pipeline

The overview of GRAIN is illustrated in Figure 2. For the backbone network, ResNet18 [He et al., 2016] or VGG16 [Simonyan and Zisserman, 2015] is employed in the experiments to make a fair comparison with previous methods. The stacked appearance-level features can be denoted as X ∈ RT×N×D , where T , N, D denote the temporal, spatial and feature dimensions, respectively. First, the residual relation module is built to capture the difference-level features ˆX ∈ RT×N×D among all persons of the whole group. Then, the appearance-level features are arranged into the multiple spatiotemporal graphs to encode the interaction-level features among the individuals. After that, the weighted aggregation strategy is introduced to further suppress the irrelevant spatiotemporal features. Finally, following previous methods [Yan et al., 2020; Yuan et al., 2021], the refined features are pooled and fed into a soft-max layer to predict the group activity. The cross-entropy loss function is used to train the model, which can be trained in an end-to-end manner. For the inference of a video clip, the final probability score of group activity is obtained by the output of the soft-max classifier. Details are presented in the following sections.

3.2 Local Residual Relation

Most of existing methods are executed in a similar process, where individual features are fed into a variety of elaborated GCNs for relational reasoning [Wu et al., 2019; Dang et al., 2021]. In fact, when the appearance-level features and their relations are not sufficient, the differences in appearance are crucial for identifying false-positive results, as shown in Figure 1. Although the performance of GAR has been constantly improved, the differences between an individual and others have not been elaborately explored by previous methods. Therefore, how to model the differences is a key component in our framework. The basic idea is to build a new local residual relation module, considering both appearance differences and spatial location constraints of each individual member.

Specifically, the appearance-level features X = {xi}TN i=1 is fed into the residual relation module, where xi ∈ RD . For convenience, the output of residual relation module ˆX is written as ˆX = {ˆxj}TN j=1 , where ˆxj ∈ RD . Then, the appearance difference between xj and other related persons is computed as follows:

where ri(xj) denotes the interaction factor. Intuitively, ri(xj) = 1 if there exists a residual relation between xj and xi, otherwise ri(xj) = 0. It is easy to see that the appearance difference in Eqn. (1) is not differentiable and cannot be trained directly by back-propagation. Inspired by NetVLAD [Arandjeloviˇc et al., 2018], the soft assignment is employed to indicate other persons associated with xj . ri(xj) is defined as

where wj and bj are the learnable weights that project the difference (xj - xi) to a scalar for the j-th person.

However, it should be noted that only a few interactions are crucial to inferring the class of group activity in the real world. Although ˆxj in Eqn. (1) has the ability to capture the residual relation, it may involve some irrelevant persons. As mentioned in ARG [Wu et al., 2019], the relation information in the local scope is more meaningful than the global relation in the modeling of group activities. Based on these facts, a spatial factor δ is introduced to constrain persons who are closer to xj. The final ˆxj in ˆX is formed as

where d(xi, xj) denotes the Euclidean distance between center points (di, dj) of two persons' bounding boxes. The spatial constraint factor δ acts as a hyper-parameter.

3.3 Dual-branch Relation Reasoning

In order to capture the underlying relation for GAR, both appearance features and position information are used to construct the actor relation graph. It is necessary to model two relations due to their different semantic attributes. Formally, given the appearance features {xi, xj} and the position coordinates of two persons, a graph of pairwise appearancelevel relation can be denoted as G = {Gi,j ∈ R1 |i, j =

1, ..., TN}, where G ∈ RTN×TN . The relation value Gi,j is expressed as

where the embedded dot-product and distance mask in [Wu et al., 2019] are used to model the appearance relation value a(xi, xj) and the position relation value s(di, dj), respectively. Intuitively, a soft-max function is used to perform normalization on each graph node. Then, a group of fullyconnected graphs {G1 , ...,Gg , ...,GNg } can be built according to Eqn. (4) with different parameters. After the graphs are built, the one-layer of GCNs is used to perform relational reasoning. The output of relation graphs is written as follows:

where Ng denotes the number of graphs. Wg ∈ Rd×d is the learnable graph-specific weight matrix. Furthermore, LR2M can be easily combined with the relation graph module to form the dual-branch relation reasoning.

3.4 Spatiotemporal Weighted Aggregation

Moreover, previous methods tend to directly fuse all features by using the element-wise sum function, which is also suboptimal to suppress the spatiotemporal redundancy with equal importance. Inspired by SKNet [Li et al., 2019], the basic idea of WAS is to use three gates to selectively aggregate the information flows from different branches.

Given a set of N × T × D-dimensional multi-level features, our goal is to get refined features for GAR. As shown in Figure 4, the features from all branches are first fused by an element-wise summation: F = ∑Nb i=1 Xi , where Nb denotes the number of branches. Then, the channel-wise statistic R is computed by simply using global average pooling and a fully convolution layer along the dimension of channels, which is written as follows:

where WR ∈ R DL ×D is the learned parameters and L is a dimensional shrinkage coeffcient. Further, the channel-wise weights {W i }Nb i=1 can be achieved by fully convolutional layers, followed by the softmax operators:

where wi ∈ RD× DL denotes the learnable parameters. The c-th dimension of refined features ˆ F is computed as

where W i c ∈ R and Xi c ∈ RN×T denote the weight and the specific features of the i-th branch along the c-th dimension of channels, respectively. In our model, Nb = 3. Specifically, {Xi }3i=1 is replaced by the multi-level features X, X′ and ˆX. Finally, a global pooling operation is performed on ˆ F to get the group feature vector F ∈ RD . The proposed model can be trained in an end-to-end manner, where the cross-entropy loss acts as the loss function.

4 Experiments

4.1 Experiment Settings

Datasets. Two popular benchmarks (Volleyball Dataset (VD) [Ibrahim et al., 2016] and Collective Activity Dataset (CAD) [Choi et al., 2009]) are used to evaluate our proposed method, while the players' tracklets provided in [Bagautdinov et al., 2017] are employed to perform feature extraction on the whole clips. In CAD, "walking" and "crossing" are merged as a new class of "moving" following [Yan et al., 2020; Yuan et al., 2021].

Evaluation metrics. Two metrics are used to evaluate our method on these datasets, which are Multi-class Classification Accuracy (MCA) and Mean Per Class Accuracy (MPCA) following previous methods [Yan et al., 2020; Yuan et al., 2021].

Implementation details. Our model is implemented based on Pytorch. Following ARG [Wu et al., 2019], randomly sampling frames from a video clip are selected as the training samples on both two datasets, resulting in T = 3 frames as the input to our model. The same backbone network ResNet18 and VGG16 are used to extract 1024-dimensional appearance-level feature vector for each person with groundtruth bounding box. Ng is set to 16. The ADAM optimizer with different learning rates is used to learn the network parameters. The hyper-parameters for ADAM are set as β1 = 0.9, β2 = 0.999 and ϵ = 10-8 . For the training of 40 epochs on VD, the initial learning rate is set to 1 × 10-4 with the dacay rate is 1/3 every 10 epochs. For the training of 30 epochs on CAD, the learning rates are set to 4×10-5 and 1×10-4 for ResNet18 and VGG16, respectively. The spatial constraint factors δ are set to 0.2, 0.3 of the image width in the training of VD and CAD, empirically. L is set to 16. The batch sizes are set to 2 on both datasets.

4.2 Ablation Studies

Baseline. The person-level features are fed into Actor Relation Graphs (ARG) [Wu et al., 2019], following a global pooling layer to get the feature representation of a group activity. Finally, the softmax classifier is employed to predict the activity score [Yuan et al., 2021].

Variants of LR2M andWAS. (1) LR2M: the appearancelevel features are fed into the LR2Mmodule to capture the local differences among relevant persons (Eqn. 3 ); (2) LR2M w/o SC : a standard LR2Mmodule without local spatial constraint is employed to learn the global differences among all persons (Eqn. 1); (3) LR2M w/DIN : Unlike LR2M with ARG, the relation graph follows DIN [Yuan et al., 2021]; (4) LR2M w/WAS : The proposed weighted aggregation strategy is employed to adaptively select important features along the spatial dimension; (5) LR2M w/TWA: Different from the aggregation direction of LR2M w/ WAS, the frame dimension is used as the aggregation direction to weight all the features; (6) LR2Mw/STWA : this variant involves stacked WAS and TWA, which further investigates the effect of WAS.

As shown in Table 1, it is observed that all models with LR2M outperform the baseline method. MCA and MPCA of LR2M yield 92.74% and 92.99% on the ResNet18 backbone, which perform better than LR2M w/o SC. These reflect that explicitly modeling the local differences brings significant performance improvement. Moreover, the space constraint is more stable to model the local differences. The results also show that only a few persons are crucial to inferring the group activity, which are consistent with the conclusion of previous work [Tang et al., 2019]. Combining two existing relation graphs with LR2M can steadily improve the performance. The superiority and robustness may be attributed to the LR2M's ability to learn additional cues. With WAS on the VGG16 backbone, the performance can be further improved. The results indicate that WAS has the capability to weight spatial-temporal features for group activity recognition. It is noted that the LR2M w/STWA scheme aggregates all features along both spatial and frame dimensions, resulting in the performance dropping dramatically. This observation indicates that the different features weighted along two dimensions will cause confusion for inference process. This may be caused by over-capturing a group of people in a frame while ignoring the information of individuals.

Visualization of LR2M. Figure 5 (a) shows all personspecific interaction across frames. The captured local residual relation and the graph-based relation for a right set activity are shown in Figure 5 (b), (c), which are obtained by parsing two person-specific matrices from the model activation, respectively. Intuitively, the person with ID 7 is setting the ball, who attracts others' attention and builds the interaction between them. Be consistent with existing methods [Yan et al., 2020], the graph-based relation module tends to associate this person with others. Different from this cue, the person with ID 20, who is watching the ball closely, forms another relatively stable relation with relevant people by assigning higher values, as shown in Figure 5 (b). The facts indicate that the relation formed by the local differences is significant in the right set group activity, not just the graph-based relation generated by the person performing setting action.

4.3 Comparison with The State-of-the-arts

For a fair comparison, the RGB video clips are adopted as input for all models and the results with two backbones are reported to demonstrate the effect of our proposed method.

Results on VD. Our method outperforms all of the aforementioned methods with a considerable margin as shown in Table 2. Although the accuracies of AT [Gavrilyuk et al., 2020], TS [Yuan and Ni, 2021] and GFormer [Li et al., 2021b] are over 94% (MCA), their main improvements are owed to the extra optical flow information or the welldesigned transformer module. The results compared with DIN [Yuan et al., 2021] indicate GRAIN is compatible with the spatio-temporal interaction graphs. The confusion matrices on VD are shown in Figure 6.

Results on CAD. Table 3 shows the comparison with different methods on CAD. Among these methods, most of them do not provide MCA and MPCA at the same time. As expected, our method outperforms the state-of-the-art methods. Moreover, GRAIN achieves impressive performance in class "waiting" due to its superiority in capturing the differences between pedestrians along the temporal variations.

5 Conclusion

In this paper, a graph-based residual aggregation network, termed GRAIN, is proposed to capture the local differences among relevant persons for inferring group activities. Specifically, the local residual relation module is employed to generate the difference-level features. The experiments demonstrate GRAIN outperforms the state-of-the-art methods.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (Grant No: 62001400, 61772436 and 61802053), Sichuan Science and Technology Program (Grant No. 2020YJ0207, 2020YJ0037 and 2021YJ0364), Foundation for Department of Transportation of Henan Province, China (2019J-2-2), Grant of Institute of Applied Physics and Computational Mathematics, Beijing (Grant No. HXO2020-118) and China Postdoctoral Science Foundation (Grant No. 2020M683353 and 2021M702713).