I²R-Net: Intra- and Inter-Human Relation Network for Multi-Person Pose Estimation

1 Introduction

2D Multi-person Pose Estimation (MPPE) aims to detect and localize the human keypoints for all persons appearing in a given image. Since human pose provides abundant structural and motion information, MPPE has attracted attentions in wide applications, such as human activity understanding, human-object interaction, avatar animation, etc.

Current MPPE methods can be divided into two fashions, i.e., top-down and bottom-up. Top-down methods [He et al., 2017; Xiao et al., 2018; Fang et al., 2017; Sun et al., 2019; Chen et al., 2018] detect bounding box for each human, then estimate the pose of each human separately. Bottom-up methods [Cao et al., 2019; Newell et al., 2016a; Cheng et al.,

∗Equal Contribution.

†Corresponding Author.

(a)

(b)

Figure 1: Illustrations of Intra- and Inter-Human Relations. (a) the intra-human relation is marked as the green arrow (left-right pairs and two keypoints of the same limb), while the inter-human relation is denoted as the yellow arrow (the same type of keypoints in different individuals).

2020; Papandreou et al., 2018; Jin et al., 2020] localize all joints in the image, and then group them into individuals. Despite having achieved great performance in MPPE, the above methods produce inferior results in crowded scenes, due to overlapping, self-occlusion, and a large variety of appearances and poses. Top-down methods treat each person separately, neglecting the informative cues of mutual interaction among persons. On the other hand, bottom-up methods jointly detect joints of all persons in an image, potentially correlating joints across persons, but the relationships between persons are vague and agnostic. Recently, several pioneers address the above challenges via decoupling poses [Li et al., 2019; Qiu et al., 2020] or fusing multiple possible poses [Wang et al., 2021a] at a single person. However, these methods still do not fully take the multiple-human correlations into account, leaving these important interactions unexplored.

This work investigates how to leverage the correlations of intra-human or cross-human, e.g., persons with similar poses or persons who closely interact with each other, to improve the accuracy of multi-person pose estimation. As shown in Figure 1(a), two people are hugging each other. Their poses show strong intra-human information (e.g., left-right pairs and two keypoints of the same limb) and obvious inter-human relations. In Figure 1(b), these persons are dancing together, indicating that their poses are similar, also providing interhuman cues for more accurate pose estimation, especially in the occluded regions or low-resolution regions.

Based on above observations, we propose a novel twostage Intra- and Inter-Human Relation Networks (I 2R-Net) for MPPE. The first stage targets at learning the dependencies among body parts of a single person. It operates at high resolution and generates the locations of small parts, such as the eyes, wrists, and ankles. The second stage operates multiperson collaboratively at low resolution. It aims to capture interactions between instances, like touching hands, joining arms, and crossing legs. It's worth noting that each instance is represented by a low-resolution feature map rather than merely a vector. By doing this, we can obtain variant responses from regions, since the cross-instance interaction depends on spatial semantic information. Meanwhile, the lowresolution feature map significantly reduces the computation cost of the second stage but is sufficient to capture useful correlation information. We call the first Intra-Human Relation Module, and the second Inter-Human Relation Module. The two stages are sequentially stacked with skip-connection.

This two-stage framework is structure flexible and function specialized for MPPE. The Intra-Human Relation Module can be an arbitrary single-person pose estimation method, aiming to explore part-level patterns for keypoint detection. Moreover, the Inter-Human Relation Module can be implemented by any non-local model, to pay more attention to correlations in semantic features. The two modules are able to work collaboratively to infer multi-person poses, because the first module offers high-quality pose information to the next module, while the second module aids the first to eliminate ambiguities in occluded or less discriminative regions.

We evaluate our method on common benchmarks for crowded scenes including CrowdPose and OCHuman. Experiments have proved that our method surpasses all stateof-the-art approaches by large margins. We achieve 77.4% AP on CrowdPose, outperforming HRFormer-B by 5.0%. We achieve 67.8% AP on OCHuman dataset, outperforming TransPose-H by 5.5%. In addition, we bring improvements on COCO dataset, which means our method can generally work in non-crowded scenes.

In summary, the contributions of our work are as follows:

• We propose a two-stage MPPE framework that not only relates each body part in a single person, but also builds connections among multi-person in the images. The proposed method bridges the gap between pose estimation and human-human interactions.
• The framework is designed to be flexible. The module coping with intra-human relations can be arbitrary pose estimation method. While the module that models interhuman relations is very lightweight.
• Extensive experiments show that, without bells and whistles, our method significantly surpasses state-ofthe-art methods on challenging datasets like CrowdPose, OCHuman, and COCO.

2 RelatedWork

2.1 Multi-Person Pose Estimation

Top-down Framework. Thanks to the great success of object detection, it is intuitive to estimate the pose of each person with the given human bounding box. Thus, lots of work under top-down framework focus on developing single-person estimation without optimizing the off-theshelf detection models. Reviewing the methods in recent years, the major development idea is to improve the feature representation in spatial. [He et al., 2017] directly adds a keypoint detection branch on a CNN-based feature extractor. [Newell et al., 2016b] follows multi-stage fashion to stack modules for dense prediction. Furthermore, applied with intermediate supervision at each stage, the model is driven to perform estimation from coarse to fine. CPN [Chen et al., 2018] employs pyramid structure in spatial to handle parts on different scales. Instead of deepening network with sequential modules, HRNet [Sun et al., 2019] maintains multi-resolution in parallel leading to rich high-resolution representation.

Bottom-up Framework. Bottom-up methods first predict all identity-free body keypoints from an uncropped image, then group these body keypoints into different individuals. Most of them adopt heatmap for keypoint detection, and focus on how to identify and group keypoints to the corresponding person. OpenPose [Cao et al., 2019] proposed the part affinity field to reveal the relationship between two keypoints in the same limb. During the grouping stage, after calculating the line integral score between two candidate keypoints, the pair with the highest score is associated. This approach is denoted as PAF family. Another popular fashion is Associative Embedding, called AE[Newell et al., 2016a; Cheng et al., 2020]. They learn the tag embeddings for each person and group keypoints into individuals by clustering the tag.

Top-down methods estimate poses one by one, without taking the relationship between instances into account. While bottom-up methods detect all poses together, the relationship cues are vague. In contrast, our work pays attention to capturing the hierarchical interactions in the multi-person scenario to promote pose estimation performance. Thus, we provide a novel two-stage framework to focus on modeling part-level and instance-level relationships for MPPE.

Human Pose Estimation in Crowd. The performance of previous work decrease as the number of human increases. Even though top-down methods achieve higher performance than bottom-up methods, they still fail in crowded scenes due to occlusions. Thus, CrowdPose dataset [Li et al., 2019] and OCHuman dataset [Zhang et al., 2019] have been proposed to encourage researchers to study this challenging problem. Existing methods [Li et al., 2019; Qiu et al., 2020] attempt to decompose coupled poses in crowd.

In this paper, however, the proposed Inter-Human Relation Module is designed to conquer this dilemma by leveraging the relationships between instances.

2.2 Vision Transformer

Transformer is an attention-based model, and has been studied in Neural Language Processing in recent years, like BERT [Devlin et al., 2018] and GPT [Radford et al., 2019]. With the effectiveness of Transformer, recently, an increasing number of works [Parmar et al., 2018; Bello et al., 2019; Touvron et al., 2021] use Vision Transformer for computer vision tasks. Several previous works apply Transformer to 2D pose estimation. For example, Transpose [Yang et al., 2021] proposes a pose estimator with attention layers to capture and reveal the spatial dependencies between 2D heatmaps. HRFormer [Yuan et al., 2021] learns high-resolution representations via self-attention schema. However, none of them take human relations into account.

Although recent work [Wang et al., 2021b; Mihai et al., 2021] utilize Transformer to calculate relations, yet they aim to solve 3d motion prediction or reconstruction where the relationships between different persons are intuitive. In contrast, our work is the first to incorporate multi-person correlations in single image 2D Pose Estimation, where the relationship is hard to build and we propose a novel module to solve this. Moreover, instead of using a well-designed framework for only building inter-human relationships, we propose a flexible, lightweight, and general module for this purpose that can be easily integrated into current or future single-person pose estimation frameworks.

3 Approach

3.1 Overview of I 2R-Net

As shown in Figure 2, the framework takes the detected and cropped human image patches in the image as input. In the first stage, Intra-Human Relation Network (Section 3.2) extracts the feature of each human patch separately and preserves relatively high-resolution feature representation. In the second stage, Inter-Human Relation Module (Section 3.3) exchanges information of the per-person features extracted by the first stage, and the per-person features should be downsampled beforehand to reduce computation cost. Finally, the two stages are stacked with a skip-connection to fuse the high-resolution intra-human information and the lowresolution inter-human information.

3.2 Intra-Human RelationModule

As the goal of Intra-Human Relation Module is to learn semantic representations of the human patches and capture the dependencies between body parts, it operates at patches of cropped human images. It can be an arbitrary single-person human pose estimation method. One key factor is the input and output resolutions, which is a trade-off between performance and computational consumption. Following the recent works [Yang et al., 2021; Yuan et al., 2021], we adopt two settings, i.e., 256 × 192 of input and 64 × 48 of output as default, and 384 × 288 of input and 96 × 72 of output for comparison.

3.3 Inter-Human RelationModule

For the task of Multi-Human Pose Estimation, we devise an Inter-Human Relation Module to collaboratively infer the pose of each person via explicitly modeling cross-human correlation.

The Inter-Human Relation Module is lightweight yet effective. We illustrate a vanilla network example in Figure 3. Given an image I with N persons, the Intra-Human Relation Module (e.g., HRNet, TransPose) locates all persons and extracts their features P1, P2, ..., PN from raw patches. These features contain pose information of each person. The InterHuman Relation Module takes these features as input and exchanges pose information among each instance. To fully capture the correlations between different people, we adopt the recently popular Transformer blocks. It is worth noting that, as shown in Figure 3, the Inter-Human relation module not only exchanges information across instances, but also correlates information among parts in each instance. This ensures that even using only an Inter-Human Relation Module (without the Intra- stage, and the input per-person features are extracted by a backbone), it can still achieve state-of-the-art performance in our experiments.

Concretely, suppose the output feature maps of first stage is Pi ∈ Rh×w×d , i ∈ [1, N], where h × w is the feature map resolution, d is the channel dimension, N is the number of persons. We use the max pooling operator with kernel size and stride set to R to operate on Pi and get resulted downsampled feature map Pi ′ ∈ Rh/R×w/R×d . Then we flatten all the features in Pi ′ and concatenate them to get a sequence with a length of L = h/R × w/R × N:

where X ∈ RL×d .

Subsequently, X is fed into M (set to 6 for vanilla network) stacked Transformer blocks, aiming to compute correlations between different instances. The formulation of attention operation is given as follows,

where Wq,Wk,Wv ∈ Rd×d are learnable parameters.

Finally, we use a standard deconvolution layer to upsample the output embeddings X′ to higher resolution and employ a keypoint head to predict K types of keypoint heatmaps ˆH, i.e., ˆH = Head(Upsample(X′ )).

Following the previous methods, we adopt the MSE loss to supervise predicted keypoints heatmaps ˆH with the groundtruth H, i.e., ∑Kk=1 ∥∥∥ ˆHk - Hk ∥∥∥2 . In our framework, the Intra-Human loss LS serves as an intermediate supervision. Thus, after summing up instance-level stage loss LM, the total loss is L = LS +αLM. In our experiments, we set α = 1.

Our proposed Inter-Human Relation Module is general and flexible, and it can be instantiated with different attention methods (e.g., Non-local networks, Transformer Layers) and implementation specifics.

Input of Inter-Human Relation Module. To reduce the computation complexity, we downsample the output feature map of the Intra-Human Relation module with the ratio R and restrict the number of persons to N. Suppose the input feature map size is h × w, the downsampled feature map is h′ ×w′ , then we flatten and concatenate all these elements together. After that, we perform self-attention calculations on all these feature elements (shown as Figure 3). The correlations between poses of different persons are captured. Since N is an important hyper-parameter for the computation complexity. We set a max person number N for each input image differently on different datasets. Specifically, we adopt N = 6 for CrowdPose, N = 3 for OCHuman and N = 4 for COCO, respectively. Experimental results prove that our input setting of the Inter-Human Relation Module achieves a strong performance while introducing little extra computation cost.

Person SelectionMechanism. In fact, the number of people usually varies in images. To this end, we introduce the person selection mechanism. As mentioned above, we set N as the maximum number of person patches.

During the training phase, we use the patch selection mechanism to facilitate the network training. For cases with persons less than N in an input image, we simply calculate correlations between these persons or even a single person. But for parallel computing, we pad noise features to increase the person number to N and mask them out to prevent self-attention calculation in Transformer blocks. When the number exceeds N, we randomly select a person as the target and fill the patches with the N - 1 nearest neighbors.

In the testing stage, we apply the sliding window strategy for the inter-human relation module (for the analysis of the maximum person number N, please see the ablation study). The sliding window divides all persons in the image into multiple N-person groups, and then we apply the inter-human relation module within each group.

4 Experiments

4.1 Datasets and EvaluationMetrics

CrowdPose [Li et al., 2019] is one of the most commonly used datasets in multi-person pose estimation task. The images in this dataset are heavily crowded and challenging. It contains 10K, 2K and 8K images for training set, validation set and testing set, respectively. Following [Cheng et al., 2020; Geng et al., 2021], we train our models on the training set and validation set, and evaluate on the testing set.

OCHuman [Zhang et al., 2019] is a recently proposed benchmark to examine the limitations of MPPE in highly occluded scenarios. It consists of 4731 images, including 2500 images for validation and 2231 images for testing. Following [Qiu et al., 2020], we train models on the validation set and report the performance on the testing set.

COCO [Lin et al., 2014] is one popular benchmark for keypoint detection. Although it contains few crowded scenes, we use it to validate the generation ability of our method for non-crowded scenes. We train the models on the training set and report the results on the validation set.

Evaluation Metrics. We follow the standard evaluation procedure and use OKS-based metrics for pose estimation [Lin et al., 2014]. We report average precision (AP), average recall (AR) with different thresholds. Following [Li et al., 2019], on CrowdPose, we also report APeasy, APmedium and APhard, which refers to the performance under different crowd index.

4.2 Implementation Details

We use Adam optimizer and the cosine annealing learning rate decay from 1e-4 to 1e-5. We also adopt the coordinate decoding strategy in [Zhang et al., 2020] to reduce the quantisation error when decoding from downscaled heatmaps. For the training batch size, we choose 64 for I 2R-Net at 256×192 resolution, and 32 at 384 × 288 resolution. The length of each token channel d is 96. The number of transformer layers M = 6 in the vanilla I 2R-Net, and M = 2 for other models. Each experiment takes 8× 32G-V100 GPUs for training.

4.3 Compare with the State-of-the-art Methods

Results on the CrowPose testing set. As shown in Table 1, thanks to our Inter-human relation module, the I 2RNet (1st stage: TransPose-H) achieves 76.3%AP, which outperforms TransPose-H. Moreover, the I 2R-Net (1st stage: HRFormer-B) achieves 77.4% AP at 256 × 192 resolution, even surpassing the performance of HRNet† at resolution of 384×288. This demonstrates that our method is very suitable for crowded scenes, and performs better than other methods even without bells and whistles.

Results on the OCHuman testing set. On OCHuman dataset, our method also obtains superior performance. The results are shown in Table 3. The vanilla I 2R-Net achieves 64.3% AP on OCHuman test set, which is comparable with state-of-the-art methods. Furthermore, it is clear in Table 3 that, equipped with the Inter-Human Relation Module, HRFormer-B and TransPose-H are significantly improved, reaching 66.5% and 67.8% AP, respectively.

Results on the COCO pose estimation validation set. As shown in Table 2, I 2R-Net achieves 77.3% AP, which is competitive against state-of-the-art methods. The performance gain on COCO is not that large compared with OCHuman and CrowdPose. This is because that COCO contains around 48% single-person images. Hence, our method for multiperson scenarios is not able to give full play. But this result demonstrates that our method also works well in noncrowded scenes.

4.4 Ablation Studies

Effectiveness of Inter-Human Relation Module. On the CrowdPose and OCHuman, I 2R-Net outperforms all existing methods at the same resolution. On COCO datasets, I 2R-Net also achieves competitive performance against state-of-theart methods, especially outperforming on APM and APL. As shown in Table 1, under our two-stage framework, TransPose can significantly surpass HRNet† with Inter-Human Relation Module even at lower resolution. This demonstrates that our Inter-Human Relation Module can bring significant performance improvements.

As is discussed in Section 3.3, the relation modeling capability of inter-human relation module is not limited to crossinstance, but can also relates parts in-instance. This can be evidenced in Table 1, 2 and 3, where the vanilla Inter-Human Relation (without intra- stage, and extract per-person feature using a backbone) achieves comparable performance with state-of-the-art methods.

Instance patch number. An important factor of I 2R-Net is the number of instances fed into the Inter-human relation module. We compare the results of different instance numbers on CrowdPose in Table 4a. We can observe that the AP value tends to be saturated when N is 4 and reaches the highest when N is 6. A higher or lower number of patches will cause a drop in AP value. This is because too few patches may lose information between persons, while too many patches introduce unnecessary cross-human correlations.

Effectiveness of Intermediate-supervision. In our framework, we use MSE loss in the first stage as intermediatesupervision (denoted as IS). As shown in the second and last rows in Table 4b, the intermediate-supervision improves the performance.

Position encoding. We also try to add position encoding into I 2R-Net. As shown in Table 4b, neither using sinusoidal mode nor encoding the global position as a positional encoding can improve the results. One possible reason is that the Intra-Human Relation Module has learned the local correlations between parts of each person, and the global interactions between instances are position-independent.

Person selection mechanism. We compare two strategies for person selection. One is the nearest neighbor strategy, which randomly selects a target person and selects N - 1 closest neighbors; the other is to completely randomly select N persons. As shown in Table 4c, there are no significantly difference between them, which means the multi-person interaction is position-free.

Token size setting. In Table 4d, the setting of 16×12×N achieves a better result. It shows that our method can learn enough inter-human relations at low resolution, but 1×1×N leads to low precision due to the lack of spatial information.

4.5 Qualitative Analysis

In Figure 4, we visualize our results compared with two competitive state-of-the-art methods (TransPose and HRFormer). The proposed method is able to handle the heavily crowded cases and localize the key points correctly. Even though in occluded scenes, we can also infer plausible poses.

By visualizing the attention maps between persons, the performance gain can be justified. As shown in Figure 5, the incorrect localized keypoints are corrected by highly related regions of the other person.

4.6 Model Size and Computation Cost

As shown in Table 2, Inter-Human Relation Module is lightweight yet effective. In the two-stage framework, Patch AP AR

1

2

4

6

8

74.6 77.5

75.7 78.7

76.1 79.0

76.3 79.1

76.2 79.1

(a) Instance patch number.

Strategy AP AR

Nearest 76.0 79.0

Random 76.0 78.9

• (c) Person selection strategy.

Token Size AP

16 ×12 ×N 66.5

32 ×24 ×N 66.4

1×1×N 66.0

(d) Token size setting.

Figure 6: Failure cases. The incorrect regions are marked with white circles.

for instance, HRFormer with Inter-Human Relation Module merely increased the parameters by 0.5M and FLOPS by 0.6G. This demonstrates that our Inter-Human Relation Module cost a few and acceptable extra computations but achieves significant improvement for multi-person estimation.

4.7 Failure Cases

Although our approach outperforms the state-of-the-art methods, we still suffer from several extremely challenging cases. Like previous work, our method may fail in cases where only part of the person is visible (Figure 6(a)) or the inter-human relationships are insignificantly correlated (Figure 6(b)). All these cases with poor relations may lead to failure predictions.

5 Concluding Remarks

This work investigates and evidences that collaboratively estimating poses via modeling multi-person correlation promotes pose estimation. We propose a flexible and efficient framework to integrate intra-human and inter-human information, leading to superior results than state-of-the-art methods. Inspired by the idea of this work, promising future work is to extend the inter-human relation from the spatial dimension to the temporal dimension. Another interesting direction is to explore the performance gain via correlating the humanobject relationships in various vision tasks.

Acknowledgements

This work was supported in part by NSFC (No. 62072382) and Yango Charitable Foundation.