Keypoints as Dynamic Centroids for Unified Human Pose and Segmentation

1 Introduction

Human pose estimation and body segmentation are crucial for human-computer interaction and real-time visual human analysis. The primary objective is to identify individuals and their activities from 2D joint positions and body shapes. The underlying main challenges include handling an unknown number of overlapping, occluded, or entangled individuals and managing the rapidly increasing computational complexity as the number of individuals grows [Han et al., 2025].

* Youngmoon Lee is the corresponding author

Human-to-human interactions further complicate spatial associations due to limb contact and obstructions, necessitating an efficient, scalable, and accurate unified model for human pose and segmentation.

In this paper, we propose KDC, a new centroid-based unified representation for human pose estimation and instancelevel segmentation. It first detects individual keypoints in a bottom-up manner and then employs high-confidence keypoints as dynamic centroids for mask pixels to perform instance-level segmentation. Unlike top-down approaches [Chen et al., 2018; Fang et al., 2017; Huang et al., 2017], KDC detects humans without requiring a box detector or incurring runtime complexity.

KDC is not the first to leverage bottom-up approach [George et al., 2018; Dantone et al., 2013; He et al., 2017; Zhang et al., 2019; Ahmad et al., 2025]. However, the model in [George et al., 2018] employs human poses to refine pixelwise clustering for segmentation, and thus does not perform segmentation well in segmentation tasks. Other models suffer from the computational overhead of a person detector [He et al., 2017], the scalability problem for instance-level segmentation [Zhang et al., 2019], and model complexity [Dantone et al., 2013], making them unsuitable for crowded scenarios and real-time applications. Unlike these models, KDC avoids the computational overhead of a person detector and suffers from neither the degraded segmentation performance nor the scalability problem of pixel-wise clustering.

KDC overcomes these problems using two primary networks: PoseNet, which generates keypoints, and SegNet, which produces segmentation masks using high-confidence keypoints (Fig. 1). PoseNet creates keypoint heatmaps using a keypoint disk representation that estimates the relative displacement between pairs of keypoints, enhancing the precision of long-range, occluded, and proximate keypoints (Fig. 1a). A KeyCentroid is defined for each keypoint heatmap locus offset vectors to the centroid of each keypoint disk, helping KDC identify the precise human keypoint coordinates (Fig. 1b). Additionally, KDC calculates the keypoint confidence score using the precise keypoint coordinates (Fig. 1c), with the final predicted keypoints illustrated in (Fig. 1d).

Meanwhile, SegNet performs pixel-level classification using dynamic high-confident keypoints as MaskCentroids (Fig. 1e). MaskCentroid defines an embedding space that associates pixels with the correct instance (Fig. 1f) and generates high-level semantic maps (Fig. 1g). Leveraging these semantic maps, the system produces instance-level segmentation (Fig. 1h). The PoseSeg module combines high-level features from both PoseNet and SegNet to provide a unified representation of human pose and instance-level segmentation (Fig. 1i).

We evaluated the performance of the KDC using the CrowdPose [Li et al., 2019], OCHuman [Zhang et al., 2019], and COCO [Lin et al., 2014] benchmarks. To the best of our knowledge, KDC is the first real-time model with reliable performance to offer a unified representation of human pose estimation and instance-level segmentation. This paper makes the following contributions.

• The development of KeyCentroid, a novel method that directs keypoint vectors towards the centroid within the keypoint disk. This approach helps identify the precise keypoint coordinates in human pose estimation, thereby enhancing confidence in the results (§3.2).
• The developement of MaskCentroid leverages highconfidence keypoints as dynamic centroids for mask vectors in the embedding space. This approach effectively associates pixels with the correct instance, even during rapid changes in human body movements (§3.3).
• An in-depth evaluation (§4) and ablation studies (§5) demonstrate the effectiveness of the unified representation of human pose and instance-level segmentation.

2 RelatedWork

Human Pose Estimation. Approaches for human pose estimation can be classified as top-down or bottom-up. The topdown approach first runs a human detector and then identifies keypoints. Representative works include HRNet [Cheng et al., 2020], RMPE [Fang et al., 2017], Multiposenet [Kocabas et al., 2018], convolutional pose machine [Wei et al., 2016], CPN [Chen et al., 2018], Mask r-cnn [He et al., 2017], simple baseline [Xiao et al., 2018], CSM-SCARB [Su et al., 2019], RSN [Cai et al., 2020], and Graph-PCNN [Wang et al., 2020a]. The top-down approach explores the human pose in a person detector, thus achieving a satisfactory performance, but it is computationally expensive. The bottom-up approach like DeepCut [Pishchulin et al., 2016] and DeeperCut [Insafutdinov et al., 2016], unlike the top-down counterpart, detects the keypoints in a one-shot manner. It formulates the association between keypoints as an integer linear scheme which takes a longer processing time. Part-affinity field techniques like OpenPose [Cao et al., 2017] and other extensions, such as PersonLab [George et al., 2018], and HGG [Jin et al., 2020] have been developed based on grouping techniques that often fail in crowd. KDC aims to specifically improve hard keypoint detection in crowded and occluded cases by introducing the keypoint heatmaps using keypoint disks and KeyCentroid.

Instance-level Segmentation. Instance-level segmentation is done in either single-stage [Dai et al., 2016; Long et al., 2015; Bolya et al., 2019] or multi-stage [He et al., 2017; Ren et al., 2015]. The single-stage approach generates intermediate and distributed feature maps based on the input image. InstanceFCN [Dai et al., 2016] produces instancesensitive scoring maps and applies the assembly module to the output instance. This approach is based on repooling and other non-trivial computations (e.g., mask voting), which is not suitable for real-time processing. YOLACT [Bolya et al., 2019] runs a set of mask prototypes and uses coefficient masks, but this method is critical to obtain a high-resolution output. The multi-stage approach follows the detect-thensegment paradigm. It first performs box detection, and then pixels are classified to obtain the final mask in the box region. Mask R-CNN [He et al., 2017] is based on multi-stage instance segmentation that extends Faster R-CNN [Ren et al., 2015] by adding a branch for predicting segmentation masks for each Region of Interest. The subsequent work in [Liu et al., 2018] improves the accuracy of Mask R-CNN by enriching the Feature Pyramid Network [Lin et al., 2017]. In contrast, our segmentation pipeline introduces MaskCentroid, a dynamic clustering point that helps cluster the mask pixels to a particular instance under the rapid changes in human-body movements.

Joint Human Pose and Instance-level Segmentation. In the line of multi-task learning paradigm, joint pose estimation and instance-level segmentation have received significant attention in recent years. Mask R-CNN [He et al., 2017] was the first pioneer method, but it suffers from high computational costs due to its top-down nature. PersonLab [George et al., 2018] and PosePlusSeg [Dantone et al., 2013] are closest to KDC. Both of them can be considered as end-to-end joint pose and instance-level segmentation models that use a bottom-up approach. However, there are several major differences that make KDC more effective, scalable, and real-time. First, they rely on static features to detect or group keypoints by using greedy decoding; in contrast, KDC introduces KeyCentroid that calculates the optimal keypoint coordinates, and uses MaskCentroid, a dynamic clustering point for instancelevel segmentation. Second, their segmentation does not perform well on highly entangled instances due to part-induced geometric embedding descriptors. Finally, they involve the complex structure model with a couple of refined networks, making them infeasible for real-time purposes.

3 Technical Approach

3.1 Keypoint Heatmap using Disk Representation

KDC generates keypoint heatmaps using disk representation (KHDR) through PoseNet, forming the foundation for human pose estimation (Fig. 3a). In this phase, individual keypoints are detected and aggregated in the output feature maps. We adopt a residual-based network for a multi-person pose setting to produce keypoint heatmaps-one channel per keypoint-and KeyCentroid, with two channels per keypoint for vertical and horizontal displacement within the keypoint disk.

The keypoint prediction approach is as follows: Let pi represent the keypoint position in the image, where i ∈ {1, . . . ,N} corresponds to the 2D positions of the pixels. A keypoint disk DR(q) = {p : ∥p - q∥ ≤ R} of radius R is focused at point q, centered in the disk. Similarly, qj,k signifies the 2D position of the jth keypoint of the kth person instance, where j ∈ {1, . . . , I} and I is the number of individual keypoints in the image. A binary classification approach is followed for each known keypoint j. Specifically, every predicted keypoint pixel pi is binary classified such that pi = 1 if pi ∈ DR for each person keypoint j; otherwise, pi = 0. Independent dense binary classification tasks are performed for each keypoint, leading to distinct keypoint maps.

During the training process, the heatmap loss is computed using the binary cross-entropy (logistic loss) function defined as:

where N is the total number of pixels, yi is the true binary label for pixel pi, and ˆ yi is the predicted probability that pixel pi belongs to the keypoint. This loss function measures the difference between the predicted probability and the true label, and the average loss across all pixels in the heatmap is used to train the model. Back-propagation is performed throughout the entire image, except for regions that encompass individuals lacking comprehensive keypoint annotations (e.g., crowded and small-scale person segments).

Point-wise Gaussian Optimization. To obtain optimal keypoint coordinates, we apply a Gaussian smoothing technique [Chung, 2020] for each individual keypoint, referred to as point-wise Gaussian optimization. This approach effectively reduces noise while preserving valuable information, producing the keypoint heatmap as:

where G(x, y) is the Gaussian kernel, σ is the standard deviation of the distribution, and x and y represent the 2D keypoint coordinates. We define the σ range from 0.1 to 1 to accommodate variations among keypoints. For high-variance keypoints (HVK) such as the wrist, ankle, elbow, and knee, we set 0.1 ≤ σ < 0.5. Conversely, for low-variance keypoints (LVK) like the nose, shoulder, and hip, we set 0.5 ≤ σ < 1, as depicted in Fig. 3b.

A smaller σ value, close to 0.1, intensifies pixel values of keypoints, proving effective in congested and intricate scenarios. In contrast, a larger σ value, close to 1, yields optimal results in less crowded cases. Our analysis investigates how the σ value impacts system performance in ablation studies (§5.3).

3.2 KeyCentroid

In addition to keypoint heatmaps, our PoseNet, in conjunction with the residual network, introduces KeyCentroid kc for each keypoint as shown in Fig. 2. The objective of KeyCentroid is to improve both the accuracy of keypoint localization and the confidence scores.

For each keypoint pixel pi within the disk DR, the 2D KeyCentroid vector kv = qj,k - pi originates from the pixel position pi and points to the j th keypoint of the kth person instance, as illustrated in Fig. 3c. We generate a vector field within DR by solving a 2D regression problem for the j th keypoint with spatial coordinates (xj, yj), and compute its response on the ground truth feature map F ∗ j as:

where σ2 is the variance related to the disk radius R = 32, used to normalize the KeyCentroid and align its dynamic range with the keypoint heatmap loss.

During training, we penalize the KeyCentroid error using the L1 loss function, which is defined as:

where N is the number of pixels in the disk DR, kv,i is the ground truth KeyCentroid vector for pixel pi, and ˆ kv,i is the predicted KeyCentroid vector. This loss function measures the difference between the predicted and true KeyCentroid vectors, and the average loss across all pixels in the disk is used to train the model.

The error is back-propagated for each pixel pi ∈ DR. We then aggregate the keypoint heatmap and KeyCentroid to determine the optimal keypoint coordinates (xj, yj), which improves the detection of both easily distinguishable and challenging keypoints. Our ablation experiments examine the impact of our uniquely designed KHDR and KeyCentroid on keypoint detection (§5.1).

3.3 MaskCentroid

Instance-level segmentation is a pixel classification problem focused on allocating pixels to the correct instance. We introduce MaskCentroid Ci (a dynamic high-confidence keypoint), as illustrated in Fig. 4a. Our mechanism clusters mask pixels using the defined centroid Ci inside each annotated person, pointing from the image position xi to the centroid Ci of the corresponding instance. At each semantically identified human instance, the pixel embedding e(xi) reflects a local approximation of each mask pixel's absolute location relative to the individual it pertains to, effectively capturing the anticipated 2D structure of the human body.

Consequently, for every pixel, we determine pixel offsets pointing to Ci. Each Ci serves as a high-confidence keypoint that can change with the rapid variation in keypoints, as shown in Fig. 4a. The objective of human-body segmentation is to assign a set of pixels Pi = {m0,m1,m2, . . . ,mi} and its 2D embedding vectors e(mi) into a set of instances I = {N0,N1,N2, . . . ,Nj} to generate a 2D mask for each human instance, as shown in Fig. 4b. Pixels are clustered to their corresponding centroid Ci = 1N ∑mi∈Nj mi. This is achieved by defining a pixel offset vector vi for each known pixel mi, so that the resulting embedding ei = mi +vi points from its respective instance centroid. We penalize pixel offset loss using the L1 loss function during model training:

To cluster the pixels to their centroid, it is important to specify the positions of the instance centroids and assign pixels to a particular instance centroid. We employ a Gaussian function ϕj(ei) for each instance Nj , which converts the distance between a pixel embedding ei = mi + vi and the instance centroid Ci into a probability of belonging to that instance:

DynamicCenter of Attraction. Asignificant innovation has been introduced in SegNet over the state-of-the-art [Dantone et al., 2013], as shown in Figure 2. The previous model relied on a fixed centroid as a parameter to cluster mask pixels, which could lead to inferior results if the centroid is occluded in real-time scenarios. However, we allow the network to learn the optimal center of attraction by introducing the concept of a dynamic centroid. This is achieved by defining the high-confidence keypoint as a learnable parameter.

This approach is especially valuable in scenarios where rapid occlusions occur during real-time operations, allowing the network to dynamically adjust the learned parameter and modify the center of attraction. As a result, the network can influence the location of the center of attraction by altering the embedding positions.

In the inference phase, using keypoints as dynamic centroids for mask pixels effectively addresses complex scenarios where over 70% of the human body is occluded. Our experimental study analyzes the effectiveness of both Static MaskCentroid SMc and Dynamic MaskCentroid DMc in human instance segmentation (§5.2).

Instance-wise Gaussian Optimization. To precisely align the predicted semantic maps, SegNet performs Gaussian smoothing [Chung, 2020] at the instance level, i.e., instancewise Gaussian optimization. We apply instance-wise smoothing to reduce noise while retaining useful information, producing distinct semantic maps. The Gaussian kernel used for smoothing is defined as:

where G(x, y) is the Gaussian kernel, σ is the standard deviation of the distribution, and (m0,m1) represents the pixel's coordinates within the kernel. We maintain σ within the range of 0.1 to 1.

We find that a σ value close to 0.1 yields finer segmentation masks, particularly in scenarios where individuals are overlapped and entangled. Our ablation experiments support this observation and demonstrate the effectiveness of instancewise smoothing (§5.3).

3.4 PoseSegModule

We introduce a new algorithm called PoseSeg, which simultaneously presents human pose estimation and instance segmentation, as illustrated in Fig. 4d. The PoseSeg module leverages high-level features generated by PoseNet and SegNet. Initially, keypoints and their coordinates are stored in a priority queue, facilitating the detection of body instances and the connection of adjacent keypoints. The pose kinematic graph is then followed to accurately estimate the human pose. Additionally, KDC performs instance-level segmentation by clustering pixels around centroids defined for each human instance. Specifically, pixels with a probability exceeding 0.5 are assigned to the corresponding human instances.

4 Evaluation

We evaluate KDC on COCO [Lin et al., 2014], CrowdPose [Li et al., 2019], and OCHuman [Zhang et al., 2019] benchmarks. The model is trained end-to-end using the COCO keypoint and segmentation training set, and ablations are conducted on the COCO val set. ResNet-101 (RN-101) and ResNet-152 (RN-152) [He et al., 2016] are used for training and testing. Hyperparameters for training are: learning rate = 0.1 × e-4 , image size = 401 × 401, batch size = 4, training epochs = 400, and Adam optimizer is employed. Various transformations are applied during model training, such as scale, flip, and rotate operations. Unless otherwise specified, a disk DR's radius is set to be R = 32.

Keypoint Results. Table 1 presents the performance of KDC using the COCO keypoint test set, outperforming the recent single-stage and top-down methods. We also compare our method with bottom-up competitors. KDC with ResNet-152 yields an mAP of 76.1, outperforming existing approaches by a large margin. Specifically, 5% over Qu et al [Qu et al., 2023], 4.9% over DecentNet [Wang et al., 2023], 4.9% over QueryPose [Xiao et al., 2022a], 3.3% over Pose+Seg [Ahmad et al., 2022], and 3.3% over GroupPose [Liu et al., 2023]. Table 2 shows the results on the CrowdPose test set compared to recent single-stage methods, topdown, and bottom-up models. KDC (mAP 74.5) outperforms bottom-up OpenPose [Cao et al., 2017], HrHRNet [Cheng et al., 2020], C.Atten. [Bras´o et al., 2021], and BUCTD [Zhou et al., 2023]. Table 3 shows the results of KDC compared with state-of-the-art models on OCHuman challenging dataset. We assess keypoint accuracy with top competitors LOGO-CAP [Khirodkar et al., 2021], MIPNet [Khirodkar et al., 2021], BUCTD [Zhou et al., 2023], and CID [Khirodkar et al., 2021] both on val and test sets.

Segmentation Results. Table 4 presents instance-level segmentation results using COCO segmentation test sets. KDC delivered a top accuracy of 47.6 mAP and improved the AP by 10.5% over Mask-RCNN [He et al., 2017], 5.9% over PersonLab [George et al., 2018] (multi-scale), and 3.1% over Pose+Seg [Ahmad et al., 2022]. Table 5 presents the segmentation performance on the OCHuman val and test sets. KDC (mAP 58.3), demonstrating a significant improvement of 3.9% and 4.4% over Pose2Seg [Zhang et al., 2019] on the val and test sets, respectively.

Comparing 2D vs. 3D Pose Estimation. We also compare the pose performance with state-of-the-art 3D models CRMH [Golda and others, 2019] and ROMP [Huang et al., 2017] in crowded scenes. We calculate the average precision (AP 0.5 ) between the 2D projection of the 3D pose on the Crowdpose val and test sets shown in Table 6.

Computational Cost. We calculate the computational cost and FPS using an image resolution of 401×401. Fig. 6 shows that KDC has fewer parameters, high FPS, and lower computational complexity compared to the representative models Mask R-CNN [He et al., 2017], PersonLab [George et al., 2018], and Pose+Seg [Ahmad et al., 2022].

5 Ablation Experiments

5.1 KHDR and KeyCentroid

Initially, we evaluate the performance of the proposed KHDR and examine its effectiveness with and without the integration of Kc, as presented in Table 7. Through our ablation study, we observe that the combination of KHDR and Kc is a highly effective approach for human pose estimation, particularly in challenging scenarios and dynamic movement of the human body. Fig. 5 shows the visual performance of keypoint heatmap improved by KeyCenroid. Fig. 7 shows the predicted confidence score of 17 keypoints using the keypoint disk at radius R = 8, 16, and 32.

5.2 Static vs. Dynamic MaskCentroids

We analyze the Static MaskCentroid (SMc) and the Dynamic MaskCentroid (DMc), with the results presented in Fig. 8. The exceptional performance of the proposed DMc approach demonstrates its effectiveness in human body segmentation, particularly in scenarios involving dynamic human body movement. Fig. 5 shows the visual performance of SMc improved by DMc. This capability significantly contributes to advancements in instance-level segmentation.

5.3 Point & Instance-wise Gaussian Optimization

We generated keypoint heatmap utilizing point-wise Gaussian optimization using 0.1 ≤ σ ≤ 1. Fig. 9 summarizes the mAP for different σ with high variation of keypoints (e.g., wrist, ankle, elbow, and knee) and low variation of keypoints (e.g., nose, shoulder, hip).

Finally, we examine the impact of instance-wise Gaussian optimization on the instance segmentation task. We tested the sensitivity of σ ranging from 0.1 to 0.5 on human instance segmentation. Fig. 10 shows the results with different σ values, where low σ provides precise segmentation mask and performs better in crowded cases.

6 Conclusion

This paper considers the challenge of unified human pose estimation and instance-level segmentation, particularly in complex multi-person dynamic movement scenarios. KDC generate keypoint heatmaps by defining keypoint disks and KeyCentroid to determine the optimal 2D keypoint coordinates within the specified keypoint disk. Additionally, MaskCentroid is introduced, representing highly confident keypoint as dynamic centroids to cluster the mask pixels with the correct instance in the embedding space, even under significant occlusion or body movement. The effectiveness of KDC is evaluated on COCO, CrowdPose, and OCHuman benchmarks and proves to be a highly effective approach for unified human pose estimation and instance-level segmentation.

Acknowledgments

This work was partly funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) grant RGPIN-2021-04244, the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant IITP-2025-RS-2020-II201741, RS-202200155885, RS-2024-00423071 funded by the Korea government (MSIT).