Online Hybrid Lightweight Representations Learning: Its Application to Visual Tracking

1 Introduction

Recent studies on deep neural representation learning [He et al., 2016; Szegedy et al., 2015] have achieved remarkable progress in various computer vision tasks. However, since their success depends heavily on high-performance GPUs and large memory, it is still challenging to deploy trained models to real-world applications in resource-hungry environments, e.g., visual surveillance, robot navigation, and embedded systems. Such drawbacks are aggravated in the problems handling videos, which critically demand efficient representation learning methodologies for real-time processing.

The mainstream approaches for reducing computational cost in neural networks are categorized into two groups: lowbit quantization and channel pruning. Low-bit quantization methods [Jung et al., 2019; Bulat and Tzimiropoulos, 2019] attempt to reduce the computational burden of deep neural networks by using low-precision weights and activations. Although these approaches are successful under simple offline learning scenarios for the tasks in the image domain, e.g., image classification [Russakovsky et al., 2015], they still suffer from the challenges in streaming data including online adaptation and fine-grained prediction with limited precision. The other paradigm of model compression, channel pruning focuses on reducing the number of channels in each layer of deep neural networks [He et al., 2017; Gao et al., 2018; Kang and Han, 2020]. Recently, a couple of channel pruning methods [Jung et al., 2020; Che et al., 2018] have shown competitive performance to the baseline models with full parameters on online tasks for streaming data. However, their compression efficiency concerning the size and inference cost remains inferior to the low-bit quantization methods.

By observing the complementary characteristics of the two methods, we propose a novel framework to learn hybrid representations for streaming data based on an ensemble of lowbit quantization and compact full-precision networks. In the fusion of the hybrid networks, the low-bit quantized network delivers primary information of data efficiently while the lightweight channel-pruned network augments missing details in the representation and facilitates online model update to handle temporal variations of data. We evaluate the proposed algorithm on online visual tracking, which requires online model updates and highly accurate bounding box alignments in real-time streaming data. Our algorithm is unique in the sense that the output model is lightweight and suitable for systems with resource constraints. Moreover, we empirically verify that our hybrid representation model is applicable to various types of tracking approaches [Jung et al., 2018; Li et al., 2019; Chen et al., 2021]. Our contributions are three-folded as follows:

• We propose a hybrid representations learning framework based on low-bit quantization and channel pruning by learning an efficient low-bit-width primary network and a precise residual network, respectively.
• We successfully deal with the trade-off between two popular model compression techniques and facilitate online adaptation of the hybrid network on streaming data by incorporating a residual architecture.
• Our learning algorithm achieves competitive performance in terms of accuracy and efficiency and, in particular, illustrates competency in the resource-aware environment.

2 RelatedWorks

2.1 Low-bit Quantization

Recent studies on low-bit quantization [Jung et al., 2019; Park and Yoo, 2020] have made successful progress in image classification. The research related to binary quantization [Hubara et al., 2016; Bulat and Tzimiropoulos, 2019] shows the potential of low-precision optimization, achieving reasonable accuracy only with binarized activations and parameters. Other approaches based on the 4- or 5-bit quantization [Jung et al., 2019; Park and Yoo, 2020] propose more practical solutions, accomplishing lower or negligible accuracy loss while enabling efficient fixed-point computation.

On the other hand, there exists another research trend that attempts to boost the performance of quantized models with assistance from full precision networks [Zhuang et al., 2020]. Specifically, they train the quantization networks simultaneously with parameter-shared full precision networks to allow the relevant gradient flow for better accuracy. However, the full precision network is abandoned for inference, which loses the full potency of the hybrid structure. We judiciously design the framework that utilizes the benefit of both fullprecision and quantized networks for visual tracking.

To mitigate the instability of training on network quantization, [Qin et al., 2020] learns binary neural networks with an additional regularizer that maximizes the entropy of the distribution of quantized model parameters. On the other hand, [Yu et al., 2020] proposes a new regularizer to let the distribution of quantized model parameters mimic the uniform distribution. Although those methods achieve competitive performance in offline problems such as image classification, their accuracy is degraded substantially during online updates, making the application to visual tracking not straightforward. This paper introduces the newly designed normalization method considering the quantization interval to maintain the feature quality during online model updates.

2.2 Channel Pruning

Another prominent deep neural network lightening method, channel pruning, aims to reduce the number of channels in model parameters. The pioneering work based on the LASSO regularization [He et al., 2017] enforces the sparsity of the model while maintaining the accuracy in the image classification task. [Gao et al., 2018] has proposed the feature boosting method by dynamically pruning channels at run-time using the channel saliency predictor. Besides, [Jung et al., 2020] adopts the conventional channel pruning method [He et al., 2017] to online object tracking, showing considerable performance gain. While the power of channel pruned models is valid for both online and offline learning, their compression rates remain inferior to low-bit quantization.

Our hybrid representation learning algorithm aims to improve accuracy and efficiency by combining complementary representations from a low-bit quantization network and a lightweight full precision network. While maintaining primary information in the quantization network obtained from rich offline pretraining, the full precision network compressed by channel pruning allows straightforward the online model update as well as accurate real-valued representation learning.

3 Hybrid Representation Learning

3.1 Overview

Our goal is to enhance the quality of representations using an ensemble of two lightweight deep neural networks obtained from quantization and channel pruning. The quantized network is responsible for feature encoding of the primary timeinvariant information while the channel-pruned model covers high-precision time-varing residual information.

Given an input image A0, the standard convolutional neural network (CNN) learns a feature extractor fconv(·; ·) and estimates a feature map Aconv as follows:

where Wconv = {W1, ...,WNl } is a collection of convolutional filters throughout the Nl layers. In the ith intermediate layer, we apply a convolution with auxiliary operations and derive a feature map, which is given by

where σ(·) denotes a non-linear activation function and ∗ indicates the convolution operator. Contrary to the conventional CNNs, our framework utilizes two networks in parallel, a quantized network fq (·; ·) and a channel-pruned network f p (·; ·), which generate outputs respectively as

Then, the proposed hybrid representation, Ah , is given by the sum of two outputs as follows:

We can train prediction networks of arbitrary tasks followed by the feature extraction. Note that the hybrid representation is also utilized in the prediction networks; channel pruning is applied to all layers of the prediction networks while low-bit quantization is partially applied to the activation quantization.

3.2 Low-Bit Quantization of Primary Network

To maximize the benefit of the reduced precision network, we quantize both model parameters and activations in all Nl layers. We adopt the quantization method proposed in [Jung et al., 2019], and introduce the additional regularization loss terms to the quantized data for fast and stable training.

Weight Quantization

The weight quantization function QlW (·) : wl → wq l is composed of two steps. First, the weights are linearly transformed to [-1, 1] by the learnable parameters clW and dlW , which define the center and the width of the quantization interval, respectively. Only the weight values within the interval [clW - dlW , clW + dlW ] are quantized while the rests are clamped. The formal definition of the first step is given by

where sign(·) ∈ {-1, 1} means the sign of an input, αlW = 0.5/dlW , and βlW = -0.5clW /dlW +0.5. Second, the normalized weights are discretized as

where round(·) denotes the element-wise rounding and Nb is the bit-width, leading to (2Nb - 1) quantization levels.

Activation Quantization

Similarly, the activation quantization function QlA(·) : al → aql has transformation and round steps. First, the activations al ∈ Al in the lth layer are linearly transformed to the values in [0, 1] as follows:

where αlA = 0.5/dlA and βlA = -0.5clA/dlA +0.5. The quantization interval is defined by the learnable parameters for its center (clA) and width (dlA). We assume that the activations are non-negative due to the ReLU function. The final quantized output Aql is obtained by applying the rounding function to the individual elements in ˆ Al ,

Operation of Quantization Network

The output from each convolutional layer is now given by Al+1 = σ(QlA(Al) ∗ Ql+1 W (Wl+1)), where the standard convolution operation is replaced by a hardware-friendly fixedpoint operation for additional acceleration. By applying the quantization to the whole network, the representation of the final convolutional layer is denoted by Aq ≡ fq (A0;Wq ), where fq (·; ·) is the feature extraction network via the quantization of both weights and activations.

Training Quantization Networks with Normalization

We now present our training scheme for robust and stable quantization. In particular, we discuss a novel regularization loss for quantization, Lquant, which is divided into weight and activation normalization losses as

Weight normalization. Clipping model parameters using the learned interval [clW -dlW , clW +dlW ] leads to error in the quantized representations and makes the optimization procedure unstable. It turns out that the final accuracy is sensitive to the initialization of {clW , dlW }, especially with a lower bitwidth. To alleviate the limitation, we propose a simple but effective regularizer, LqW, enforcing a majority of parameters to be located within a certain range, which is given by

where avg(·) and std(·) denote mean and standard deviation of all entries in their input matrices, respectively, | · | returns an absolute value matrix of the same size to its input, and (µ0, σ0) are hyperparameters for desirable mean and standard deviation values. The proposed regularizer encourages the transformed model parameters to be in [0, 1] and reduces the error resulting from the clamping process. Note that it allows the quantized network to identify more appropriate parameters {clW , dlW } and undergo more stable optimization.

Activation normalization. Clipping activations outside of the interval [clA -dlA, clA +dlA] incurs critical information loss in practice. To mitigate this issue, we define the following regularization loss termLqA:

where Abn = {Abn i }Nl i=1 is a set of activations after batch normalization and [·]+ denotes the ReLU function. Note that σi and µi are determined by combination of current quantization interval parameters (clA, dlA). Assuming that the activation Abn i follows a Gaussian distribution, the regularization loss enforces the activations larger than the mean of the Gaussian and lower than avg(Abn i ) + 2 · std(Abn i ) to be within the active range. Statistically speaking, it implies that almost 96% of positive values are trained to be located in the active range.

3.3 Channel Pruning of Residual Network

We propose a stochastic channel pruning algorithm using a Gumbel-softmax trick to generate discrete mask samples from the model with a continuous output space. Our algorithm learns a set of channel selection probability vectors, {b1, ..., bNl }, in each layer, where bl indicates which channel is likely to be sampled in the lth layer. By employing the Gumbel-softmax trick introduced in [Jang et al., 2017], bl generates discrete channel selection maskMl as follows:

where gi and g′i represent random noise samples from the Gumbel distribution, and τ denotes a temperature. We apply the straight-through Gumbel estimator for gradient approximation in backpropagation similarly to [Jang et al., 2017]. In the forward pass of training, we apply the pruning mask Ml to the output activation Apl in the lth layer, which is given by

where ⊙ denotes an channel-wise multiplication. Then, the representation of the final convolution layer is obtained by applying the channel pruning network fp (·; ·), as follows: Ap = fp (A0;Wp ). Note that a channel with a selection probability lower than the threshold is removed at testing. For training, the channel selection probabilities {bl}Nl l=1 are learned by minimizing the following loss term

Regression where the number of active channels is penalized.

classification

4 Hybrid Representation for Online Tracking

We apply the proposed hybrid representation learning method to online visual tracking, which aims to sequentially estimate the location of a target object in a streamed video given the ground-truth bounding box at the first frame. This task is appropriate for testing the proposed algorithm since visual tracking requires efficient online target appearance modeling and accurate bounding box regression.

The overall framework with online hybrid representation learning for visual tracking is illustrated in Figure 1, which presents two pathways of backpropagation for offline pretraining and online model updates. In the pretraining stage, our framework trains the parameters of both the quantized network and the channel-pruned network. On the other hand, during the online model update in test time, we freeze the parameters of the quantized network to maintain static coarse information while allowing the lightweight full precision network to adapt to target appearance changes. We apply our algorithm on top of three baseline trackers, RT-MDNet [Jung et al., 2018], SiamRPN++ [Li et al., 2019], and TransT [Chen et al., 2021]. We pretrain each tracker with a loss term

where Lbaseline denotes a loss term proposed from baseline tracker, and {λ1, λ2} are the balancing hyperparmeters.

Hybrid RT-MDNet. RT-MDNet first generates a feature map from a whole image and then predicts candidate windows sampled from the feature map. Hybrid RT-MDNet applies our hybrid representation network to the feature extraction network while the prediction network employs the channel-pruned network only. We perform online updates with the channel-pruned network.

Hybrid SiamRPN++. SiamRPN++ deals with a tracking algorithm via the cross-correlation between the target representation and the candidate feature map using Siamese networks. The predictor network, which is composed of a binary classifier and a bounding box regressor, estimates the confidence score of each adjusted candidate location based on the cross-correlation results. To formulate Hybrid SiamRPN++, we apply our hybrid representation network to the Siamese backbone and adopt channel-pruned network in the predictor.

Hybrid TransT. TransT is a variant of SiamRPN++ that applies the Transformer encoder-decoder networks [Vaswani et al., 2017] to the feature correlation. Similarly to Hybrid SiamRPN++, we incorporate the hybrid network into the backbone and employ the channel-pruned network in other parts. Additionally, for all self-attention in the Transformer, we perform the activation quantization to reduce the computation burden in the dot-product operations.

5 Experiments

5.1 Preparation for Training

To pretrain the hybrid trackers, we employ the same training datasets as the baseline networks [Jung et al., 2018; Li et al., 2019; Chen et al., 2021]. We adopt the pretrained model of each baseline tracker for initialization and fine-tune them using the proposed loss function. The learned hybrid models are tested on the standard visual tracking benchmarks, OTB2015 [Wu et al., 2015], UAV123 [Mueller et al., 2016], LaSOT [Fan et al., 2019], VOT2016 [Kristan et al., 2016], and VOT2018 [Kristan et al., 2018], for comparisons with recently proposed competitors. We evaluate algorithms using Precision rate (Prec) and Success rate (Succ) in OTB2015, Expected Average of Overlap (EAO) in VOT2016/2018, and Normalized Precision (Norm. Pr) in LaSOT.

5.2 Implementation Details

We employ the same learning rates as the baselines but reduce the rate of the quantized network to 1/10 of the original value in Hybrid SiamRPN++. As in [Jung et al., 2019], we use the full precision in the first convolution layer in Hybrid SiamRPN++ and Hybrid TransT.

For Hybrid RT-MDNet, we train the model with batch size 48 and run 120K iterations while optimizing model parameters and auxiliary parameters using SGD and Adam, respectively. For Hybrid SiamRPN++, training is performed on 8 GPUs for 100K iterations with batch size 22, and the same optimization methods are employed as Hybrid RT-MDNet.

Hybrid TransT is trained for 400 epochs (1K iterations per epoch) using the AdamW optimizer with batch size 40, where the learning rate is decayed by a factor of 10 after 250 epochs.

Two variants of SiamRPN++ are tested, HybSiamRPN++ and OnHybSiamRPN++, which correspond to the models based on offline and online hybrid representation learning, respectively. Specifically, during the online update stage, OnHybSiamRPN++ fine-tunes the last linear layer in the classification branch stemmed from the first cross-correlation block every 10 frames for 15 iterations using the data collected from the latest 20 frames. The sample features are collected after the target prediction and annotated by the same labeling strategy used in the pretraining stage.

Our algorithm is implemented in PyTorch with Intel I76850k and NVIDIA Titan Xp Pascal GPU for RT-MDNet and Quadro RTX4000 for SiamRPN++ and TransT.

5.3 Effects on Acceleration

Following [Baskin et al., 2018], we measure the number of bit-wise operations (BOPs) for unit convolution to estimate the performance benefit of the low-precision computation. For each layer with {C in ,C out} input and output channels based on K × K kernels, BOPs is given by

where ba and bw are the precisions of input activations and parameters, respectively. Table 1 shows the layer-wise analysis of the effect of our optimization method.

The proposed hybrid representation learning approach reduces the computation overhead by maintaining a small fraction of parameters that are critical to accuracy. Specifically, we observe more channels pruned in mid-layers (conv2-3 and ℓ2-4) than the parameters in head layers (conv1 and ℓ1) and the prediction layers (fc, RPN, att, and pred), which tend to be intact for the stability of training. Consequently, the proposed model with hybrid representations only requires a fraction of BOPs (0.26 to 0.42 as presented in Table 1) compared to the original network.

5.4 Effect of Hybrid Representation

We analyze the impact of each component in the proposed hybrid representation learning. Note that we use HybRT- MDNet pretrained in VOT-OTB dataset for the ablative study.

Benefit of each component. Table 2 illustrates the benefit of each component in our hybrid representation. The quantization with normalization (Q+N) (10) achieves better accuracy than the model by a simple quantization (Q), which implies the effectiveness of our normalization strategies for stable quantization. In addition, the hybrid representation model (Q+N+P) trained with both (10) and (15) outperforms the Q+N model, demonstrating that the proposed hybrid representation learning improves performance effectively. Finally, our final models present competitive accuracy to the baseline models while reducing the computation cost significantly.

Combinations of quantization and pruning. Table 3 illustrates the performance resulting from several combinations of quantized and pruned networks, which include the proposed hybrid network (PQ), the ensemble of pruned networks (PP), and the ensemble of quantized networks (QQ). We perform this experiment on the RT-MDNet baseline. The numbers in the table denote the relative computational cost in terms of BOPs between the model corresponding to each row and the RT-MDNet baseline. According to the results, the proposed hybrid representation presents a competitive performance in terms of both accuracy and efficiency, whereas the ensemble of quantized networks and pruned networks suffer from performance degradation in accuracy and computational cost, respectively.

Impact of pruning rates. Table 4 analyzes the computational cost by varying channel pruning ratios for our hybrid model. Note that our model based on the hybrid representation maintains competitive accuracy in both measures, precision and success rates, even after pruning almost 80% of channels.

Impact of quantization bit-width. Table 5 presents the experimental results by varying the bit-width for quantization in our hybrid model. The results show that the success rate drops sharply with less than 4-bit quantization while the precision rate is hardly affected by the bit-width variation.

5.5 Comparison onMultiple Benchmarks

Table 6 compares the performance of our hybrid networks with conventional models in visual tracking on the standard benchmarks, where SiamRPN++ and SiamRPN++ (Rep) indicate the official model and its reproduced version employed in our experiment, respectively. The proposed hybrid networks such as HybRT-MDNet, HybSiamRPN++, OnHybSiamRPN++, and HybTransT achieve competitive performance to the baseline models while significantly reducing the computational cost (0.29 to 0.42 in RC). Besides, the modified Hybrid SiamRPN++ with online update (OnHybSiamRPN++) improves tracking accuracy compared to the baseline SiamRPN++, which implies the effectiveness of our online hybrid representation learning for streaming data. Our model with the hybrid representation based on Transformer, HybTransT, also accomplishes outstanding accuracy with the significantly reduced computational burden.

6 Conclusion

We presented a novel framework of online hybrid representation learning applicable to streaming data by an ensemble of an efficient low-bit quantized network and a precise lightweight full-precision network. Adopting a residual architecture, the online hybrid representation successfully complements the two model compression approaches. We further improved the performance of the networks based on the hybrid representation through effective normalization of weights and activations for low-bit quantization. The proposed algorithm was incorporated into online visual tracking tasks and verified that the proposed hybrid representation makes competitive accuracy to baseline models despite lowbit quantization and channel pruning.

Acknowledgements

This work was supported in part by Samsung Electronics Co., Ltd. (IO210917-08957-01), and by the IITP grants [2021-0-01343, Interdisciplinary Program in AI (SNU); 2021-0-02068, AI Innovation Hub] and the NRF grant [2021M3A9E4080782, 2022R1A2C3012210] funded by the Korean government (MSIT).