Exploiting Position Information in Convolutional Kernels for Structural Re-parameterization

1 Introduction

Convolutional neural network (CNN) is widely used in various visual tasks [Krizhevsky et al., 2012; Ren et al., 2015; Toshev and Szegedy, 2014] and produces impressive results, greatly promoting the development of computer vision. How to improve the performance of various visual models becomes a very popular topic [Hu et al., 2018; Ding et al., 2021c; Ding et al., 2019; Wang et al., 2024b; Xiong et al., 2024; Shen et al., 2024; Hao et al., 2024]. As is known to all, the convolutional operator aggregates neighbor information to obtain feature representation. Many prior works [Chen et al., 2017; Xu et al., 2015; Hu et al., 2018]

have found that the position information of features has a significant impact on the aggregation results, especially for finelevel image analysis tasks such as pixel-level semantic segmentation [Fu et al., 2019; Zhong et al., 2020]. For example, spatial-wise attention [Chen et al., 2017] promotes the model by modeling the importance of feature maps. However, few works focus on the role of different positions of convolutional kernels when designing convolution-based networks. While, in this paper, we deeply analyze such a characteristic of the convolutional kernel and improve the performance of CNNs. We find the positional distributions of convolutional kernels are diverse and are influenced by tasks, architectures, and internal layer order as in Fig. 2. In addition, we find that pruning weight on the position with larger magnitude leads to more performance drop as shown in Fig. 3. Therefore, it is appropriate for us to use magnitude to represent the importance of positions. If a kernel position has a large magnitude, then this position is more important compared with the position with a smaller magnitude. Furthermore, we notice that the position distribution of kernel weights is approximately uniform after the beginning random initialization, but rapidly evolves towards dispersion as the training progresses. Based on this observation, we hypothesize that reinforcing the strong and weakening the weak of convolutional kernel weights' positions would help improve the model's performance, and design PBConv to enlarge such gap. Afterward, Fig. 9 will demonstrate that our goal has indeed been achieved.

Motivated by such observation, we propose a novel Position Boosting Convolution (PBConv) to replace the original convolution. A PBConv adaptively learns the importance of kernel positions, promoting the important ones and suppressing the unimportant ones. Our PBConv follows the principle of re-parameterization (rep for short ) [Ding et al., 2019; Ding et al., 2021c; Ding et al., 2021d; Hu et al., 2022; Huang et al., 2022; Ding et al., 2022; Ding et al., 2024; Ding et al., 2021a; Hao et al., 2023a; Hao et al., 2023b; Hao et al., 2023c; Guo et al., 2020; Wang et al., 2024a] mechanism, which trains an over-parameterized model but merge redundant parameters before inference, ultimately retaining only the necessary structure. In this way, these techniques can fully exploit the capacity of the model during training to improve performance without adding any inference overhead. However, existing structural re-parameterization techniques [Ding et al., 2019; Ding et al., 2021c] ignore the internal properties of convolutional kernels and lead to inferior performance. In contrast, as in Fig. 1, PBConv uses multiple concurrent small convolutional kernels and adaptively emphasizes the informative parts of the original kernel. PBConv follows a divide-and-conquer philosophy. Given a convolutional kernel, we regard it as a combination of sub-regions. We tactfully configure several smaller kernels over it. Each of them is responsible for adjacent positions. Such a structure can enable small kernels to learn detailed and discriminative information about the relations between neighbor pixels in the local receptive field. As a result, PBConv strengthens the fine-grained low-level feature extraction, leading to significant improvements. To find the optimal structure of PBConv, we first randomly generate a set of legal candidate that covers the whole region and thus can adaptively learn the importance of each position, and then choose the best candidate according to the dispersion of the position distribution after a quick few iterations training, which is strongly correlated with the final performance as in Fig. 6. Such a heuristic search scheme enables us to adaptively select structures with superior performance compared to those designed by existing re-parameterization methods like [Ding et al., 2019; Ding et al., 2021c]. Besides, benefiting from the linearity as in prior re-parameterization works, our PBConv can trigger the aggregation among kernel weights. After training, PBConv could easily merge the complex structures into the simplest one, and enjoy improvement without any extra inference cost. Better still, our PBConv performs layer-level re-parameterization design and is orthogonal to procedurelevel re-parameterization methods like [Ding et al., 2023; Huang et al., 2022] and architecture-level re-parameterization methods like [Ding et al., 2021d]. Experiments show PB- Conv can further improve such orthogonal methods.

We conduct extensive experiments on four vision tasks, ranging from image-level tasks, and object-level tasks to pixel-level tasks. Experimental results show that PBConv can consistently achieve superior performance compared with existing state-of-the-art methods, improving plenty of architectures on various datasets and tasks. The experiments well demonstrate the superiority of PBConv, indicating a promising application in more practical scenarios.

Overall, we summarize our contributions as follows.

• We discover the positional distributions of convolutional kernel weights are diverse. Weights in different positions are of different importance. Motivated by such observation, we propose to selectively emphasize informative positions and suppress less useful ones.
• We propose PBConv to replace the convolutional layer. PBConv consists of parallel small kernels that aim at the extraction of low-level visual features, showing superiority, especially on pixel-level and object-level tasks.
• We find the dispersion of the convolutional kernel's position distribution is strongly correlated with the final performance, even in very early training. Such findings inspire us to design a heuristic search algorithm to adaptively acquire the optimal PBConv structure with negligible cost. PBConv follows the re-parameterization schema and thus brings no extra inference cost.
• We do extensive experiments on various visual tasks, datasets, and models. PBConv consistently outperforms state-of-the-art layer-level re-parameterization methods, and is verified to be orthogonal to procedure-level and architecture-level re-parameterization method.

2 RelatedWork

2.1 Attention mechanism

Attention [Chen et al., 2017; Hu et al., 2018; Chen et al., 2020; Xu et al., 2015] is a popular technique that calculates an importance score for the features and influences the update of corresponding parameters. The importance score can be spatial-wise [Xu et al., 2015], channel-wise [Hu et al., 2018] or other mixed metrics [Chen et al., 2017]. Intuitively, attention can derive informative parts by weighting the importance of input features. In contrast, our work aims to emphasize the positional importance of convolutional kernels, instead of features.

2.2 Re-parameterization mechanism

[Ding et al., 2021c; Ding et al., 2019; Guo et al., 2020; Ding et al., 2021d; Ding et al., 2021b; Huang et al., 2022; Hu et al., 2022; Ding et al., 2022; Ding et al., 2024] belong to re-parameterization methods. These works train a large over-parameterized network and then convert the complex structure to a simple one in inference time to improve performance without increasing inference cost. [Ding et al., 2021d; Ding et al., 2021b; Ding et al., 2022; Ding et al., 2024] design a whole architecture with strong performance, while [Ding et al., 2019; Ding et al., 2021c] and our PBConv aim to design a generic convolution block suitable for various models to replace the ordinary convolution to improve the performance. Besides, [Huang et al., 2022; Hu et al., 2022; Ding et al., 2023] introduce procedure-level design during training of previous re-parameterization structure [Ding et al., 2021c; Ding et al., 2021d]. In Experiments section, we will show our PBConv is a better generic block than previous layer-level re-parameterization methods [Ding et al., 2019; Ding et al., 2021c], and PBConv is orthogonal to procedurelevel re-parameterization methods and architecture-level reparameterization methods and can lead to further improvement if combined together.

3 ACloser Look at Positional Distribution of Convolutional Kernels

Before demonstrating our method, we start by deeply analyzing the property of the positional distribution of the weights of convolutional kernels.

We first define the importance of a position in the convolutional kernel. Motivated by prior works [Han et al., 2015; Li et al., 2016; Ding et al., 2019], we use the average magnitude of the kernel weights to describe such property. Formally, for a convolutional layer with output channel D, input channel C and kernel size K × K, we study the positional distribution of W ∈ RD×C×K×K by calculating:

where |W| computes the element-wise magnitude of weight matrix W. Then we can get an average magnitude matrix M ∈ RK×K with entries ∈ [0, 1].

We choose some widely used models and analyze the positional importance distributions. As in Figs. 2a and 2b, there are significant differences in the distribution of kernel positions between different layers of the same model. We also averageM of every 3×3 convolutional layer of the model to obtain an overall inspection. Seen from Figs. 2c to 2e, different positions exhibit distinct importance properties for different models. For example, the largest region in ResNet-18 [He et al., 2016] is the bottom right 2×2 square. In VGG-16 [Simonyan and Zisserman, 2014], it is the central cross while 4 corners' weights are the smallest. However, in U-Net [Ronneberger et al., 2015] there is quite an opposite pattern to that in VGG-16, i.e., the largest weights are located in 4 corners and center.

To investigate how such property influences performance, we prune each position of 3×3 convolutional layers of VGG16 on CIFAR-10 [Krizhevsky et al., 2009], respectively, by masking corresponding weights to zeros. The relationship between performance drop and position magnitude is shown in Fig. 3. Pruning the position with a higher magnitude will generally cause more damage. Therefore, it is reasonable to use magnitude to represent positional importance. Positions with higher magnitudes are more important.

Furthermore, we visualize the variation of the positional distribution of 3×3 convolutional kernels during the training process of ResNet-50 on ImageNet in Fig. 4. We use different colors for the nine positions in the 3×3 kernel. The distribution is roughly uniform at the beginning since the model is randomly initialized. But it rapidly disperses as the training goes on, i.e. the gap between important positions and unimportant positions enlarges consistently.

According to the observations above, we can conclude that the positional importance distribution in the convolutional kernel presents a large variety. It can show different features due to different visual tasks, different networks, and even different layers in the same network. The positional distribution is approximately uniform at the beginning, but rapidly disperses in the training. As the training goes on, important positions tend to be more important, while unimportant positions tend to be more unimportant.

4 Position Boost Convolution

4.1 Preliminary

Suppose a convolutional layer with weights W ∈ RD×C×K×K and bias b ∈ RD takes feature map F ∈ RC×H×W as input, and its output is denoted by O ∈ RD×H′×W ′ . We use B ∈ RD×H′×W ′ to represent replicas of b among the extra two dimensions. Formally,

where ⊛ denotes convolution operation, and + denotes element-wise addition. From the perspective of sliding window, the output at (h, w) on d-th output channel is given by

where Sh,w ∈ RC×K×K denotes the sliding window in F corresponding to Od,h,w. The relationship between Sh,w and F depends on the convolutional layer.

Fuse convolutional layer and BN. In modern CNNs, there is usually a normalization layer following a convolutional layer. Batch Normalization (BN) [Ioffe and Szegedy, 2015] is one of the most popular modules, which would continuously update its running mean µ, running variance σ2 , scaling factor γ and bias β in training. These values are fixed after training, and thus BN turns into a linear transformation in inference and we could fuse it into the convolutional layer. Particularly, for a conv-BN sequence, the output ˆO ∈ RD×H′×W ′ at (h, w) on d-th output channel is given by

Therefore, we can equivalently transform a conv-BN sequence to a single convolutional layer by Eqs. (3) and (4). New ˆW and ˆb for any filter d is given by

Similar with B, we use ˆB ∈ RD×H′×W ′ to denote replicas of ˆb. Eventually, the conv-BN sequence is equivalently converted to a single convolutional layer.

4.2 Full coverage with parallel small kernels

For a convolutional kernel, we use multiple parallel small convolutional kernels to boost its positional distribution.

Positional embedding for the convolutional kernel. For a convolutional layer with weights W, we aim to emphasize its informative parts and suppress less useful parts by learning an embedding vector for each position. We use Ei,j ∈ RD×C to denote the embedding vector of W:,:,i,j, and ˜ W to denote enhanced weights, which is the element-wise sum of the original weights and embedding vectors.

By replacing W with ˜ W, we could enhance the weights on more important parts to help aggregate local information. Finally, we acquire significant improvement without increasing any computation cost. We employ a multi-branch structure to learn such embedding vector during training and add it to the original kernel weights via the re-parameterization mechanism.

For a K × K convolutional layer, ∀1 ≤ h ≤ K, 1 ≤ w ≤ K, 1 ≤ kh ≤K-h+1 and 1 ≤ kw ≤K-w+1, it is feasible to adaptively learn positional embeddings for a rectangular area denoted by top left corner (h, w) and bottom right corner (h+kh-1, w+kw-1) via a concurrent convolutional branch with proper configuration. The key is making the new sliding window become a part of the original one and thus the fusion is possible due to additive and associative properties. The kernel size of the new branch should be (kh, kw), and other settings should be the same as those of the original one, except for the padding. Note that new padding values might be negative, which means cropping. Subsequently, we can sum up their weights and biases in advance, instead of forwarding all branches and then summing up their outputs.

Furthermore, it is feasible to merge concurrent small convolutional branches into different regions of the original kernel. We use W 0 , b0 (B0 ) to denote weights and biases of the original convolutional layer, andW l , bl (Bl ) to denote those of l-th extra branch. Suppose there are n extra branches. After training, we can integrate all the branches into a single one with parameters ˜ W and ˜B due to linear property as follows. First, ∀ 0 ≤ l ≤ n, convert l-th conv-BN branch to a single convolutional layer with parameters ˆW l and ˆbl ( ˆBl ) via Eq. (5). Then ∀1 ≤ i ≤ K, 1 ≤ j ≤ K we have

where Ωl = {(x, y) | hl ≤ x ≤ hl + klh - 1 and wl ≤ y ≤ wl + klw - 1}. The target positional embedding E can be easily derived: Ei,j = ∑nl=1 I l (i, j). Eventually,

An important rule for boosting positions is "full coverage", i.e. every region of the original kernel should be covered by at least one small kernel. Full coverage ensures PBConv can adaptively learn the embedding of every position and enhance or suppress corresponding weights. Formally, ∀1 ≤ i ≤ K, 1 ≤ j ≤ K, ∃1 ≤ l ≤ n, s.t. I l (i, j) ̸= 0.

4.3 Heuristic search for optimal structure

In the previous section, we have demonstrated how to use multiple concurrent small convolutional kernels to enhance the positional distribution of the original large kernel. For a K×K convolutional kernel, each kernel smaller than K×K with proper padding can enhance a specific region of the original kernel. Due to the extremely large number of combinations of small kernels, it is not feasible to train and compare the final performance for each combination to get the best. We thus design a fast heuristic search algorithm to quickly select the optimal structure that can maximize the model's performance from a set of randomly generated candidates.

First, we introduce a random candidate generation algorithm. For a K×K kernel, there are K2-1 types of small kernels in different sizes,e.g. 1×1, 1×2, 1×3, 2×1, 2×2, 2×3, 3×1, 3×2 are all the legal components for a 3×3 kernel. For simplicity, we randomly select t ∈ {1, 2, 3} types of kernels for candidate generation, i.e. the possibility that each candidate has 1/2/3 types of small kernels is 33%/33%/33%. We aim to cover all the regions of the original kernel with such t types of chosen kernels. We continue adding small kernels over the original kernel, until all the regions are covered.

Every time a new small kernel is added, we set its coverage area based on the following three rules. The priority of these rules decreases from the first to the last. That is to say, only when the preceding rules provide multiple areas of equal priority will we resort to the subsequent rules for priority.

1. Coverage rule: the kernel should be placed onto areas that contain more uncovered positions.
2. Center rule: the kernel should be placed across the central position of the original kernel, e.g., since we consistently observe extremely large importance in the center position as shown in Fig. 2.
3. Alternative rule: the kernel should be placed from left to right, from top to bottom if the priority can not be determined by former rules.

We give an example in Fig. 5 for better clarification.

Now we have designed a complete mechanism to randomly generate some candidates. Next, we use the aforementioned algorithm to randomly generate c candidates, where c is set to 10 in practice. Motivated by the observation that the positional distribution of convolutional kernel rapidly disperses at the very beginning of training as in Fig. 4, we further study the relationship between standard deviation of kernel positions and model performance. In Fig. 6, the standard deviation of the positional distribution after few training iterations is strongly correlated to the final performance. Therefore, we can leverage such a heuristic search scheme to quickly select the best candidate from massive randomly generated candidates. The overall design of PBConv is in Fig. 1. The heuristic search process is very efficient and fast, typically accounting for only 1% to 10% of the total training time.

5 Experiments

5.1 Datasets and Competitors

We do evaluation on CIFAR [Krizhevsky et al., 2009] and ImageNet [Deng et al., 2009] classification, Cityscapes [Cordts et al., 2016] segmentation, GoPro [Nah et al., 2017] deblurring and COCO [Lin et al., 2014] detection.

The layer-level reparameterization methods ACB [Ding et al., 2019] and DBB [Ding et al., 2021c] are the most related competitors. We first build a baseline and then replace its conv-BN sequence with ACB/DBB/PBConv, and train all models with identical configurations for a fair comparison. Notably, we also combine PBConv with newest architecturelevel [Ding et al., 2021d] and procedure-level [Huang et al., 2022; Ding et al., 2023] re-parameterization methods, and earn consistent improvements.

5.2 Image Classification

We first evaluate PBConv on image classification. Results are shown in Section 5.1. On CIFAR-10, PBConv improves VGG-16 by 0.66%. On the large scale ImageNet dataset, PBConv improves MobileNetv2 by 0.66%, ResNet-18 by 1.33%, ResNet-50 by 0.64% and ResNet-101 by 1.13%, significantly outperforming competitors like ACB and DBB.

Furthermore, we combine PBConv with state-of-the-art procedure-level and architecture-level re-parameterization methods. Results on CIFAR are in Section 5.2. Clearly, PBConv is orthogonal to these re-parameterization algorithms and they could be combined for further improvement.

To explore how PBConv helps CNN aggregate feature representations, we use Grad-CAM [Selvaraju et al., 2017] to show the attention maps produced by ResNet-18 and PBConv-ResNet-18. In Fig. 7, the attention maps given by PBConv more precisely locate the target objects and keep the background out of attention. No matter what size and shape the target objects are, the attention given by PBConv is better confined to the semantic regions and thus more integral objects are identified in more precise locations.

5.3 Semantic Segmentation

Next, we experiment on pixel-level tasks. We start with highlevel semantic segmentation. Models are trained with ImageNet pre-trained backbone weights on Cityscapes.

On various segmentation architectures with various backbone models, our PBConv consistently outperforms all competitors by a significant margin. Seen from Section 5.2, when using ResNet-18 as backbone, PBConv improves PSPNet by

1.50% mIoU, which is 0.29/0.67% higher than ACB/DBB, Deeplabv3 by 0.88% mIoU, which is 0.68/0.78% higher than ACB/DBB, and Deeplabv3+ by 0.57% mIoU, which is 0.49/0.63% higher than ACB/DBB. When using lightweight MobileNetv2 as the backbone, PBConv still outperforms all competitors by improving PSPNet by 0.59% mIoU, which is 0.49/0.31% higher than ACB/DBB. The advantages are still kept when we use a heavy ResNet-50 as the backbone.

5.4 Image Deblurring

We then evaluate PBConv on image deblurring, where normalization is not in usual deblurring models [Nah et al., 2017; Cho et al., 2021]. So is ACB/DBB/PBConv here.

As in Section 5.2, PBConv improves 1.05, 0.55, and 0.59 PSNR for UNet, DeepDeblur, and MIMO-UNet respectively.

5.5 Object Detection

For object detection, PBConv improves mAP of [Duan et al., 2019] by 1.20% on COCO, far better than others.

5.6 Analysis

Ablation studies In Section 5.2, we study the effect of each separate rule for PBConv generation on VGG-16 on CIFAR. Besides, we find the number of branches and parameters don't show significant influence in PBConv. PBConv uses from 85% to 130% of the parameters of DBB for different models, but consistently outperforms DBB.

Why PBConv is effective? We do dimension reduction by the t-SNE algorithm for the feature maps of all stages of a ResNet-18 and PBConv-ResNet-18. In Fig. 8, for low-level features (stages 1 and 2), PBConv shows clear superiority against baseline, aggregating the features with the same labels while decoupling those with different labels. For highlevel features (stages 3 and 4), the gap is marginal, relatively. Therefore, the effectiveness of PBConv mainly comes from the strong low-level feature extraction ability, e.g., the ability to catch edges and textures as seen from Fig. 7. As a result, PBConv is naturally more suitable for fine-grained tasks.

Does PBConv truly enhance the positional distribution? The equivalent kernel after training for ResNet-18 is in Fig. 9. Both layers 3-1 and 5-2 obey our assumption that PBConv indeed enlarges the gap between informative parts with larger magnitude and uninformative parts with smaller magnitude. The position distribution before final fusion is in Fig. 10. A position's value in a branch is negatively correlated with the number of times the position is covered and positively correlated with the original value at the corresponding position.

6 Conclusion

Motivated by the observation of positional distributions of convolutional kernels, we propose PBConv, an architectureneutral block, to improve CNN. By learning positional embedding vectors for kernel weights, we selectively emphasize the informative parts and suppress less useful ones. We integrate small kernels in PBConv to better extract fine-grained low-level features. To find an optimal structure, we propose a heuristic search algorithm based on the correlation analysis between kernel positional distribution and performance. Extensive experiments demonstrate the effectiveness of PBConv across various visual tasks, datasets, and architectures. Moreover, PBConv brings no extra cost in inference for it could be equivalently converted to a simplest convolutional layer after training. Through controlled experiments, we show the key of PBConv is the mechanism to enhance positional weight distributions by learning embedding vectors adaptively. Such a paradigm shows fairly strong low-level feature extraction capacity. Furthermore, our PBConv is a layer-level re-parameterization design, and thus is orthogonal to architecture-level and procedure-level re-parameterization methods, i.e. the model performance could be further improved if we combine PBConv with them together.

Acknowledgements

This work was supported by Pioneer and Leading Goose R&D Program of Zhejiang (No. 2024C01142), Beijing Natural Science Foundation (No. L22302) and National Natural Science Foundation of China (Nos. 62271281, 62441235, 62021002).