Dite-HRNet: Dynamic Lightweight High-Resolution Network for Human Pose Estimation

1 Introduction

Human pose estimation is a task that requires both accuracy and efficiency, especially when it encounters real-time applications on devices with limited resources. In human pose estimation, high-resolution representation is necessary for accuracy, which brings difficulties to achieve high efficiency.

While High-Resolution Network (HRNet) [Sun et al., 2019] achieves remarkable performance in human pose estimation, it has high computational complexity. To make it lightweight, Small HRNet [Wang et al., 2021] reduces the width and depth of the network, but such reduction introduces performance degradation. Lightweight High-Resolution Network (Lite-HRNet) [Yu et al., 2021] shows that lightweight high-resolution network can achieve satisfactory performance by replacing each residual block in Small HRNet with an efficient CNN block. However, it remains unclear whether such static block that is independent of the inputs can provide the optimal trade-off between the network performance and complexity for high-resolution networks, as some operations may have different effects at different locations of the network and with inputs of different sizes [Cui et al., 2021]. The problem is that input-dependent component units may perform more efficiently in high-resolution networks than the input-independent counterparts. Our work focuses on designing dynamic blocks to develop a lightweight high-resolution network with dynamic representations.

∗Contact Author

Recent research [Wang et al., 2018] has demonstrated the benefits of long-range spatial dependency in human pose estimation. The general concept of long-range spatial dependency is the global understanding of spatial information in a large field of view. Previous high-resolution networks [Sun et al., 2019; Cheng et al., 2020; Wang et al., 2021; Yu et al., 2021] mainly rely on the deeply stacked convolution layers in parallel branches that extract multi-scale features to build spatial dependency, but the capacity of lightweight networks can be severely affected by the restricted widths and depths. Hence, another problem is how to enhance lightweight high-resolution networks with more efficient methods for modeling long-range spatial dependency.

To address the above problems, we present a Dynamic lightweight High-Resolution Network (Dite-HRNet). We adopt the architecture similar to HRNet [Sun et al., 2019] and develop two dynamic lightweight blocks to improve the overall efficiency. First, we propose a Dynamic Split Convolution (DSC), which can dynamically extract multi-scale contextual information and optimize the trade-off between its capacity and complexity by two hyper-parameters, and thus is more effective and flexible than the standard convolution. Then, we introduce long-range spatial dependency into the high-resolution network by designing a novel Adaptive Context Modeling (ACM) method that enables the model to learn both local and global patterns of human poses. Finally, we embed DSC and ACM into two dynamic lightweight blocks, which are specially designed for the high-resolution network to make full use of the parallel multi-resolution architecture, as the basic component units of our Dite-HRNet.

The main contributions of this paper can be summarized as follows: (1) We propose a novel lightweight network named Dite-HRNet that can efficiently extract multi-scale contextual information and model long-range spatial dependency for human pose estimation. (2) We design a DSC method to build multi-scale contextual relationship and an ACM method to further strengthen long-range spatial dependency. In addition, we accomplish a joint embedding of DSC and ACM into two dynamic lightweight blocks, which are designed as the basic components of Dite-HRNet. (3) In our experiments, Dite-HRNet achieves superior trade-off between the network performance and complexity on both COCO and MPII human pose estimation datasets.

2 RelatedWork

Lightweight Human Pose Estimation. Recent studies [Wang et al., 2021; Yu et al., 2021] focus on improving the efficiency of huamn pose estimation networks. Small HRNet [Wang et al., 2021] reduces the width and depth of the original large version of HRNet [Sun et al., 2019]. Lite-HRNet [Yu et al., 2021] exploits the potential of lightweight highresolution network by replacing each residual block in the Small HRNet with a conditional channel weighting block, which adopts channel attention mechanism [Hu et al., 2020] by replacing high-cost 1×1 convolutions in shuffle block [Ma et al., 2018] with channel weighting operations.

Efficient CNN Blocks. Efficient CNN blocks have been widely used in many efficient CNN architectures [Howard et al., 2017; Howard et al., 2019; Zhang et al., 2018], which aim to maximize the capacity of models under limited computational cost. MobileNetV2 [Sandler et al., 2018] introduces the inverted residual block, which improves both accuracy and efficiency over traditional bottleneck blocks. Shuffle block in ShuffleNetV2 [Ma et al., 2018] performs convolutions only on half of the channels with a competitive performance, due to the channel split operation that splits the features by channels and the channel shuffle operation that enhances the information exchange across channels. MixNet [Tan and Le, 2019] combines multiple convolution layer into a mixed depth-wise convolution, which is a drop-in replacement of vanilla depth-wise convolution in CNN blocks. Sandglass block [Zhou et al., 2020] flips the structure of the inverted residual block, reducing the information loss without any additional computational cost.

Spatial DependencyModeling. Long-range spatial dependency can be spontaneously captured in the repeated convolution layers, which have better performance with large convolution kernels. However, deeply stacked convolution layers and large convolution kernels are both costly and inflexible to be integrated in lightweight networks. Non-local network [Wang et al., 2018] adopts a self-attention mechanism to model pixel-wise spatial relations in a single layer, which has better flexibility but still high complexity. Global context network [Cao et al., 2019] simplifies the non-local network to a lightweight structure with almost no performance loss.

Dynamic CNN Architectures. There are two main stream methods for dynamic CNN architecture design. One is dynamic network structure, such as PP-LCNet [Cui et al., 2021], which dynamically conducts different operations based on the inputs of different sizes in different positions of the network. And the other is dynamic convolution kernel, such as dynamic convolution [Chen et al., 2020] and CondConv [Yang et al., 2019], which mix different convolution kernels by generating weights through the attention mechanism. In order to exploit more efficient representations, our proposed DiteHRNet embodies its dynamic state not only in the network structure but also in the convolution kernels.

3 Dite-HRNet

As shown in Figure 1, Dite-HRNet is a 4-stage network, consisting of one high-resolution main branch with the highest resolution and three high-to-low resolution branches that are added to the network one by one in parallel at the beginning of each new stage. Each newly added branch has half the resolution and twice the number of channels compared to the previously added branch. As shown in Table 1, among all four stages of Dite-HRNet, the first stage, also regarded as the stem, contains a 3 × 3 strided convolution and a Dynamic Global Context (DGC) block on the main branch. Each subsequent stage consists of a series of cross-resolution modules which are composed of two Dynamic Multi-scale Context (DMC) blocks and a multi-scale fusion layer that exchanges information across all branches. The main branch with the highest resolution maintains a high-resolution representation, which provides the final output of the backbone network for the subsequent pose estimation. For fair comparisons, we presents two instantiations of our network, Dite-HRNet18 and Dite-HRNet-30, which have the network widths and depths corresponding to Lite-HRNet-18 [Yu et al., 2021] and Lite-HRNet-30 [Yu et al., 2021], respectively.

Block Design. As shown in Figure 2, our DMC block and DGC block share similar overall structure, applying the channel split, feature concatenation and channel shuffle operation from ShuffleNetV2 [Ma et al., 2018] to aggregate different features extracted by different layers. One difference between the two blocks is that the DMC block applies a sequence of layers on half of the channels, while the DGC block applies two different sequences of layers on all two groups of channels. The sequence of layers in the DMC block contains one Dense Context Modeling (DCM) operation, one DSC and one Global Context Modeling (GCM). Both the DCM and GCM are the instantiations of the ACMmethod. In the DGC block, one 3 × 3 strided depth-wise convolution, one GCM and one 1 × 1 convolution are performed on one group of channels, while one 3×3 depth-wise convolution, one GCM, one 1×1 convolution and one 3 × 3 strided depth-wise convolution are performed on the other group of channels. Each convolution in the DGC block and the DSC layer generates the convolution kernel via Dynamic Kernel Aggregation (DKA).

Configuration. In order to demonstrate the superior efficiency of our method, we optimize our networks, which are characterized by two hyper-parameters in DSC, to have similar model sizes and computational cost like in Lite-HRNet [Yu et al., 2021]. We configure the hyper-parameters separately on different resolution branches. Our default configuration choice is shown in Table 5 and used for the subsequent comparison experiments.

3.1 Dynamic Split Convolution (DSC)

Split-Concat-Shuffle (SCS) Module. Large convolution kernels can bring broad receptive fields, thereby enhancing the long-range spatial dependency. However, using large kernel sizes on all filters in the network not only leads to a high computational cost, but also fails to capture the local pixelwise information. Therefore, we introduce the SCS module to extract contextual information by multiple kernels of different sizes and integrate them in a single convolution layer.

In the SCS module, channels are first split into multiple groups equally, and depth-wise convolutions with different kernel sizes are applied to each group of channels in parallel. The output of the convolution on each group is formally defined as follows:

where Xi and Yi denote the input and output of the depthwise convolution on the ith group of channels, respectively. DWConv(Ki ×Ki|C/G)(·) is the depth-wise convolution with the kernel size Ki ×Ki and channel dimension C/G, where C denotes the total number of channels among the groups and G denotes the number of groups.

After the depth-wise convolutions, the grouped features are concatenated together. To further integrate the separated information at different scales, the channel shuffle operation [Ma et al., 2018] is used at the bottom of the SCS module.

Note that the SCS module does not expand the width of the network, as it only splits channels into different groups and performs different convolution operations on them in parallel. The computational efficiency is acceptable and can be optimized by the hyper-parameter G.

Dynamic Kernel Aggregation (DKA). To make the SCS module learn rich contextual information even with small convolution kernels, we introduce a DKA operation which strengthens the input-dependency of convolution kernels by dynamically aggregating multiple kernels via kernel-wise attention weights based on the input images.

Standard convolution kernel is defined by a weight matrix w with 4 dimensions which decide the kernel size and input/output channels, respectively. Instead of concatenating the output features of different convolutions, we aggregate the kernel weight matrices {wi} before calculating the convolution results, to dynamically generate different convolution kernels for different inputs. The DKA operation computes attention weights over different convolution kernels, and then applies the element-wise product to the attention weights and the kernel weights. We define the DKA operation as follows:

where ai(X) is the attention weight for the ith convolution kernel, and W(X) is the aggregated weight matrix of N convolution kernels. The input-dependent attention weights a(X) are computed from the input X as follows:

where GAP(·) represents the global average pooling, and FC(·) represents the full-connected layer. Two functions Sigmoid(·) and ReLU(·) are used right after two fullconnected layers for non-linear activation.

Since the DKA operation takes place before the calculation of convolution results, the aggregated kernel performs only one convolution operation on each input feature map without expanding the network width. For a standard convolution with kernel size K×K and input size C×H×W, DKA of N kernels only introduces minor additional computational cost HWC + C 2 /4 + CN/4 from the standard convolution cost HWC 2K2 . The number of aggregated kernels N is another hyper-parameter for the efficiency optimization.

Complementation. The high computational efficiency makes the DKA operation an excellent complementation to the SCS module. Therefore, we combine the above two methods into one single convolution, which is named DSC. With two hyper-parameters G and N from the SCS module and the DKA operation, respectively, it is relatively easier for DSC to optimize the trade-off between the capacity and complexity of the convolution layers.

3.2 Adaptive Context Modeling (ACM)

The ACM method can be abstracted into following three steps: (a) adaptive context pooling, as shown in Figure 3, which creates a shortcut connection with a 1 × 1 channel pooling convolution and a softmax layer to form a context mask, and then applies the mask on the feature maps through a series of resolution-adaptive transformations to obtain spatial contextual features; (b) context shifting, which realigns the spatially related contextual features together by two 1×1 convolutions with non-linear activation; (c) context weighting, which adopts the element-wise weighting operation between input features and shifted context features to model the contextual relationships of the corresponding features. This abstraction of the ACM method can be defined as:

where ACPool(H, W)(X) denotes the adaptive context pooling which pools the input features X to a certain output size H ×W, Shift(·) denotes the context shifting, and Weight(·) denotes the context weighting.

For the high-resolution network, we propose two instantiations of the ACM method, DCM and GCM, which take the advantage of the parallel multi-resolution architecture.

Dense Context Modeling (DCM). We introduce a DCM operation to densely model the spatial contextual relationships of the features from all resolution branches of one stage. At the nth stage, the input features from all n branches are pooled to the lowest resolution Hn ×Wn. Then, all pooled features are concatenated together, so that the context shifting can be densely performed on the parallel contextual features. In the end, the shifted contextual features are upsampled to the corresponding resolutions and distributed back to the corresponding branches for the subsequent context weighting. This instantiation implements the ACM as:

where Cat(·) and Upsamp(·) denote the feature concatenation and upsampling, respectively. Xk represents the input tensor with the kth highest resolution. ̂X represents the pooled tensor from the kth branch. ˜Xk represents the shifted tensor, which is distributed to the kth branch as ˜Xk. Yk represents the corresponding kth output tensor.

Global Context Modeling (GCM). To separately model the global spatial dependency at each resolution, we apply a GCM operation on each branch of the network. It is a instantiation of the ACM when the adaptive context pooling has the output size 1 × 1. The output features of the GCM operation on the kth branch is defined as follows:

The GCM operation captures the spatial relationship of all features with the same resolution in the global aspect containing abundant contextual information, while the DCM operation captures the spatial relationships of all features with different resolutions in a moderate aspect containing more pixelwise information. Meanwhile, both two operations increase the information exchange across features, so that can be better substitutes for the 1 × 1 convolutions in the shuffle block [Ma et al., 2018] than the channel weighting operations in Lite-HRNet [Yu et al., 2021].

4 Experiments

4.1 Implementation Details

Datasets andEvaluationMetrics. TheCOCOdataset [Lin et al., 2014] has images over 200K and 250K person instances, each with a label of 17 keypoints. We train our networks on the train2017 set (contains 57K images and 150K person instances), and evaluate them on the val2017 set (contains 5K images) and test-dev2017 set (contains 20K images) by the Average Precision (AP) and Average Recall (AR) scores based on Object Keypoint Similarity (OKS). To further validate our networks, we also perform experiments on the MPII Human Pose dataset [Andriluka et al., 2014], which contains about 25K images with 40K person instances, and evaluate the accuracy by the head-normalized Probability of Correct Keypoint (PCKh) score.

Training. The presented Dite-HRNet is trained on 8 GeForce RTX 3090 GPUs, with 32 samples per GPU. All parameters are updated by Adam optimizer with a base learning rate 2e-3 . As for the data processing, we expand all human detection boxes to a fixed aspect ratio 4 : 3, and then crop the images with the detection boxes, which are resized to 256×192 or 384×288 for the COCO dataset, and 256×256

for the MPII dataset. All images are used with data augmentations, including random rotations with factor 30, random scales with factor 0.25, and random flippings for both the COCO and MPII datasets. In addition, extra half body transformations are performed for the COCO dataset.

Testing. The two-stage top-down paradigm, which produces the person detection boxes and then predicts the person keypoints, is adopted for our testing process. The person boxes are predicted by the person detectors provided by SimpleBaseline [Xiao et al., 2018] for the COCO dataset, while the standard testing strategy for the MPII dataset uses the provided person boxes. The heatmaps are estimated via 2D Gaussian, and then averaged for the original and flipped images. The highest heatvalue locations in the heatmaps are adjusted by a quarter offset in the direction from the highest response to the second-highest response, to obtain the keypoint locations.

4.2 Results

Results on the COCO val2017 Set. As shown in Table 2, we compare the results of our Dite-HRNet-18 and DiteHRNet-30 with other state-of-the-art pose estimation methods, including both large and small networks. Our networks are trained with two different input sizes 256 × 192 and 384 × 288, achieving better trade-off between the accuracy and complexity than other methods. As compared to other small networks, our Dite-HRNet-30 achieves the highest AP score of 71.5 with the input size 384 × 288. In particular, Dite-HRNet exceeds Small HRNet [Wang et al., 2021] by 10 points of AP with only 40% GFLOPs. As compared to the large pose networks, our Dite-HRNet also achieves comparable or even higher accuracy with far smaller model sizes and lower computational complexity. Benefiting from the abundant contextual information, Dite-HRNet outperforms LiteHRNet [Yu et al., 2021] with the equivalent parameters and GFLOPs, while having the same network depths and input sizes.

Results on the COCO test-dev2017 Set. In Table 3, we report the results of comparisons of our networks and other state-of-the-art methods. Our Dite-HRNet achieves higher efficiency than both large and small networks. As compared to Lite-HRNet-18 [Yu et al., 2021], Dite-HRNet-18 improves AP by 1.5 points and AR by 1.8 points with the equivalent model complexity. Our Dite-HRNet-30 achieves the highest AP of 70.6 among the small networks, outperforming SimpleBaseline [Xiao et al., 2018] while having only 5% parameters and 8% GFLOPs. Despite the accuracy gap with other large networks, our networks are much more competitive in terms of the model size and computational complexity.

Results on the MPII val Set. The results of our networks compared with other lightweight networks are reported in Table 4. Our Dite-HRNet-18 improves PKCh@0.5 score by 0.9 points over Lite-HRNet-18 [Yu et al., 2021] with the equivalent model complexity, has the same score but only half of GFLOPs compared to Lite-HRNet-30 [Yu et al., 2021], and outperforms MobileNetV2 [Sandler et al., 2018], MobileNetV3 [Howard et al., 2019], ShuffleNetV2 [Ma et al., 2018] and Small HRNet [Wang et al., 2021] with much lower parameters and GFLOPs. With the same model size as LiteHRNet-30, our Dite-HRNet-30 achieves the best result of 87.6 PKCh@0.5 among the lightweight networks.

Note that the accuracy improvements of Dite-HRNet-18 over Lite-HRNet-18 are more significant than that of DiteHRNet-30 over Lite-HRNet-30 on all three sets. Moreover, our small version of Dite-HRNet achieves the proximate performance to the large version of Lite-HRNet. Therefore, our proposed methods are more effective for small networks, and also much more efficient than increasing the network depth.

4.3 Ablation Study

Hyper-parameters in DSC. To explore the best configuration of two hyper-parameters G, N in DSC, we conduct a 16-group ablation study on the COCO val2017 set. The results of our Dite-HRNet-18 with different configurations are shown in Table 5. On each branch of Dite-HRNet-18, parameters and GFLOPs increase as N, G increase. However, large N for the upper branches as well as large G for the lower branches have less impact on the model size and computational complexity, and also achieve more improvement on accuracy. It shows a better trade-off between the accuracy and complexity by setting G to 1, 1, 2, 4 and N to 4, 4, 2, 1 for the highest to lowest resolution branches, respectively.

Effectiveness of ACM and DSC. To further verify the effectiveness of our proposed methods, we perform ablation studies on the COCO val2017 and MPII val sets. We first use Lite-HRNet-18 [Yu et al., 2021] as a baseline, to which our ACM and DSC are applied separately or together, and then compare the results with our Dite-HRNet-18. Table 6 shows that our ACM and DSC methods bring considerable accuracy improvements to Lite-HRNet-18 with negligible additional complexity for the overall model. Further, our lightweight blocks helps Dite-HRNet-18 achieve the best results and reduce the model complexity.

5 Conclusion

In this paper, in order to address the problems that previous high-resolution networks were input-independent and lacked long-range information, we presented a Dite-HRNet to dynamically extract feature representations for human pose estimation. Our network achieved impressive efficiency on both COCO and MPII human pose estimation datasets, due to the effectiveness of the DMC and DGC blocks, which performed input-dependent convolutions by embedding DSC and captured long-range information by embedding ACM. The proposed dynamic lightweight blocks can further expand to other networks that have multi-scale representations.

Acknowledgments

This work was partially supported by the National Science Fund for Distinguished Young Scholars of China under Grant No.62125203, the Key Program of the National Natural Science Foundation of China under Grant No.61932013, the National Natural Science Foundation of China under Grant No. 61906099 and No. 61906098.