Efficient Screen Content Image Compression via Superpixel-based Content Aggregation and Dynamic Feature Fusion

1 Introduction

With the surge in screen-content applications like online conferences and webcasting, efficient SCI compression is increasingly needed. Unlike natural images, SCIs contain large uniform regions, repetitive patterns, and limited color palettes, challenging traditional compression algorithms such as JPEG [Wallace, 1992] and JPEG2000 [Rabbani and Joshi, 2002]. The HEVC [Sullivan et al., 2012] extension for screen content coding (HEVC-SCC)[Liu et al., 2015] introduced innovative coding tools to address these challenges, later incorporated into the VVC standard [Bross et al., 2021]. Meanwhile, The advent of deep neural networks has revolutionized visual signal compression and processing. End-to-end networks have made significant progress in natural image compression [Ball´e et al., 2016; Ball´e et al., 2018; Minnen et al., 2018; Minnen and Singh, 2020; Cheng et al., 2020;

∗Corresponding author.

Zou et al., 2022; Liu et al., 2023], even surpassing the latest VVC standards. However, when applied to SCIs, the rate-distortion (RD) performance tends to deteriorate, as evidenced by our comparative experiments illustrated in Figs. 1, 7, and 9. Therefore, in this work, we focus on LIC based SCI compression.

To cope with the unique characteristics of SCIs, content dependent redundancy exploration and energy compaction methods are required. There are some end-to-end natural image coding networks exploring this approach by using window-based self-attention mechanisms. However, they can only capture the correlations among a small local region, which degrades their capabilities in using the non-local similarities. Meanwhile, the SCIs usually have many repetitive patterns and the pixels in a local region may be very similar due to the piece-wise constant properties. Therefore, efficient non-local and local redundancy exploration is essential for SCIs. A straightforward solution for exploring non-local correlations is to employ global self-attention. However, this approach introduces heavy computational costs.

Based on the observations above, we introduce a superpixel based content aggregation strategy, motivated by the HEVCSCC intra-block copy technique. Unlike the hard copy mode of intra-blocks, our method aggregates repetitive and local information through a soft copy approach (pixel aggregation). Similar pixels in a local region are first grouped into a superpixel to represent regional information, which significantly reduces the number of image primitives. Correspondingly, we propose a superpixel based feature aggregation mechanism. In this way, pixels within the superpixel can utilize global correlations. We then apply the proposed algorithm to enhance both the feature extraction network and the entropy model. The contributions of this paper can be summarized as follows:

1. We propose a Learned Image Compression (LIC) framework for efficiently compressing SCIs. A Superpixelbased Content Aggregation Block (SCAB) is proposed to aggregate regional information and explore non-local correlations via super-pixel based transformer. This mechanism enables our network to dynamically utilize non-local redundancy with small computing costs.
2. We introduce a Dynamic Feature Fusion (DFF) module into the channel-wise context entropy model. The DFF module dynamically combines decoded slices with side information based on their global correlation, which is realized by gating coefficients derived from super-pixel aggregation result. In this way, our entropy model can adaptively adjust the probability distribution of undecoded channels based on the dynamic fusion result.
3. Experiments demonstrate that our method outperforms six benchmarking LIC methods on three SCI datasets (i.e., SCID, CCT, and SIQAD) with different resolutions. In addition, our method is the first network that can be comparable with the advanced VVC-SCC standard.

2 RelatedWork

2.1 Traditional Screen Content Image Compression

Popular image/video coding standards like JPEG [Wallace, 1992], JPEG2000 [Rabbani and Joshi, 2002], HEVC [Sullivan et al., 2012], and VVC [Bross et al., 2021] are designed mainly for camera-captured images and videos. To address the unique features of screen content, research has built upon these standards for screen content coding. The HEVCSCC [Liu et al., 2015] extension, for instance, includes tools like intra block copy (IBC) [Xu et al., 2016], adaptive color transform, and palette mode, enhancing the codec's adaptability to SCIs and improving compression efficiency. These tools, with modifications, are integrated into the VVC standard.

2.2 Learning-Based Image Compression

LIC has seen significant advancements. Ball´e et al. [Ball´e et al., 2016] introduced an end-to-end CNN-based model, later enhanced with a VAE architecture and hyper-priors [Ball´e et al., 2018]. They also improved the entropy model using a local context model [Minnen et al., 2018]. Guo et al. [Guo et al., 2021] proposed a causal context model, while He et al. [He et al., 2021] and Minnen et al. [Minnen and Singh, 2020] designed models for parallel computation. Various convolutional neural networks have been explored to boost image compression, such as the residual network by Cheng et al. [Cheng et al., 2020] and the INNs by Xie et al. [Xie et al., 2021]. However, these models struggle with SCIs due to the local receptive fields of convolutions. Recent LIC solutions have explored the Transformer architecture to capture long-range dependencies. Attention mechanisms have been integrated to enhance the transformation and entropy models of learned LICs. Several studies [Zou et al., 2022; Zhu et al., 2022; Liu et al., 2023] have attempted to establish a Swin-Transformer-based LIC model.

LIC for Screen Content

LIC's application for SCIs is under-explored. Mitrica et al. [Mitrica et al., 2019] proposed a semantic compression scheme for airplane cockpit screen content. Tang et al. [Tang et al., 2022] extended this with a text semantic-aware scheme (TSA-SCC) for ultra low bitrate settings. A learned image codec integrating transform skip was also proposed to enhance SCI compression [Wang et al., 2022], but lacks comparison with VVC and public dataset validation. A multi-task learning architecture was presented for SCI encoding performance improvement [Zamanshoar Heris and Baji´c, 2023], but it increases model complexity. This paper addresses these challenges by leveraging repetitive patterns in SCIs. We propose a super-pixel attention-based end-to-end encoding compression scheme, tailored for SCIs, outperforming the conventional encoding scheme VVC on multiple public datasets.

3 Method

3.1 ProblemFormulation

Figure 2 presents our proposed network architecture. We adopt the VAE structure, a classic architecture in the deep image compression framework. It forms the foundation of both our primary and hyper encoder-decoder pairs. The primary encoder-decoder pair processes the raw input image x and the reconstructed image ˆx1 , while the hyper encoder-decoder pair models the spatial dependencies among the elements of the latent representation y. The entire framework can be summarized as follows:

where the analysis function ga with parameters ϕ transforms x into y, which is then quantized by Q into ˆ y. We adopt a channel-wise entropy model [Minnen and Singh, 2020; Zou et al., 2022; Liu et al., 2023] that computes Φ = (µ, σ) for each channel of y, as shown in Equation 2 and Figure 2. The entropy model partitions y into s evenly slices y0, y1, ..., ys-1 and enhances the encoding efficiency by exploiting the dependencies among the slices. This can be expressed as:

where the hyper-prior analysis ha with parameters ϕh obtains side information z to capture the spatial dependencies among the elements of y. A factorized density model ψ is used to encode quantized ˆ z as pˆ z|ψ(ˆ z|ψ) follow the prior work [Ball´e et al., 2018; Minnen et al., 2018], as shown in Equation 3. Then, the hyper-prior synthesizer hs, with parameters θh, decodes ˆ z to produce two latent features, Fmean and Fscale, which are subsequently fed into each slice network Ci. Each slice yi is processed sequentially to generate ˆ y. During this operation, the decoded slices y<i = {y0, y1, ..., yi-2, yi-1} and the current slice yi are input to the slice network Ci to estimate the distribution parameters Φi = (µi, σi), which are then used to generate bit-streams. As such, we can posit pˆ y|ˆ z(ˆ y|ˆ z) ∼ N(µ, σ2 ). Furthermore, the residual r is employed to mitigate the quantization errors (y - ˆ y) introduced by quantization. Finally, the synthesizer gs with parameters θ reconstructs ˆx from ˆ y. Figure 5 provides a clear illustration of the detailed process of this channel-wise entropy model. The loss function is defined as follows:

1 The encoder is commonly referred to as the analysis transform, playing the same role as prediction and transform in the traditional coding process.

where the trade-off between the rate and distortion in our compression approach is modulated by a Lagrange multiplier λ. The average number of bits needed to encode the input data is represented by R, and distortion, denoted by D, can be quantified using metrics such as MSE or MS-SSIM. Our objective is to minimize the loss function defined as Equation 3.

3.2 Content-Adaptive Transform via Stacked SCABs

We implement the content adaptive analysis transform by stacking Superpixel-Context Adaptive Blocks (SCABs), as depicted in Figure 2. Each SCAB unit comprises a superpixel transformer layer and a parallel convolutional residual layer.

Motivation

Traditional coding algorithms such as HEVC and VVC exhibit outstanding performance. These algorithms combine intra prediction and transform coding, effectively leveraging reconstructed neighboring regions to adaptively capture the dynamic characteristics of the content, thereby efficiently handling the complexity and variability of the image. The process can be simply formulated as:

where ga and gs denote the DCT [Ahmed et al., 1974] and its inverse IDCT, respectively. Nrec represents the reconstructed neighbors. Each mode m has a specific Wm to weight the reconstructed Nrec to form the prediction I pm. A special case occurs when the IBC mode (specifically designed for screen content in HEVC-SCC and VVC-SCC) is applied: Wm becomes a one-hot mask, copying the same block as shown in Figure 3b. The content-adaptive weighting of reconstructed neighbors allows us to exploit screen content redundancy. Many CNN-based compression networks [Ball´e et al., 2016; Ball´e et al., 2018; Ball´e et al., 2020] may not support this feature due to fixed weights after training. Some studies [Cheng et al., 2020; Zou et al., 2022; Liu et al., 2023] use window attention mechanisms for more dynamic and adaptive learning of neighborhood content, but these mechanisms have a limited dynamic range and integrate local features based on each pixel. A better method is to use larger pixel sets to perceive features, as shown in Figure 3b, where IBC performs content-awareness based on blocks. The symbol "s" finds its most similar block through block matching. Therefore, if the pixels representing "s" can form a superpixel, we can better perceive the surrounding information. Based on this, we propose a superpixel-based content aggregation for screen content compression.

Superpixel Based Aggregation Module

As shown in Figure 2, the Superpixel-based Content Aggregation (SCA) module consists of three processes: superpixel sampling, content aggregation in the superpixel space, and pixel resampling. Specifically, we first aggregate pixels into superpixels by using a superpixel sampling algorithm, then perform attention models in the superpixel space to handle global dependencies, and finally map the super pixels back to visual pixels by using a pixel resampling algorithm.

Superpixel Sampling: In the Superpixel Sampling process, we use the soft k-means based superpixel algorithm from SSN [Jampani et al., 2018] to cluster the features F ∈ RB×N×C (where N = H × W is the number of pixels) into m super pixels S ∈ RB×m×C . Each pixel feature Fi ∈ RB×1×C is assigned to one super pixel, and the assignment map is denoted byM ∈ RN×m. Formally, superpixel aggregation follows an Expectation-Maximization-like process, consisting of a total of T iterations. We initialize the super pixels S0 by averaging the features in regular grid regions. The grid size is h×w, and the number of super pixels is m = Hh × Ww . Then we iteratively update the super pixels and the association with the following two steps:

1. Pixel-superpixel association: The pixel-superpixel association at iteration t is computed as:

where i denotes the pixels, j denotes the superpixels. d is the channel number. It is noteworthy that the superpixel aggregation calculates the association between each pixel and only its 9 surrounding superpixels (as shown in Figure 4), ensuring the locality of superpixels, which also renders the computation and memory more efficient [Huang et al., 2022].

2. Superpixels cluster centers updates: The superpixels are updated as the weighted sum of pixel features, defined as:

where Z t j = ∑i Mtij is the normalization factor. After T iterations, the final superpixel clusters ST and the associated mappingMT are obtained.

Content Aggregation in the Superpixel Space: Given that superpixels serve as compact representations of visual content, we employ attention mechanisms to emphasize their significance, enabling us to prioritize global contextual dependencies over local features. Specifically, we apply the standard self-attention technique to the sampled superpixels S ∈ Rm×C , which are defined as follows:

where q(·), k(·), v(·) are linear functions, respectively. We omit the multi-head mechanism for clarity.

Pixel Resampling: After aggregating information in the superpixel space, the pixels composing the superpixels also receive global information flow. Subsequently, we map them back to pixels using the association mapMt and incorporate them into the original input F.

3.3 Superpixel Based Aggregation Block

We introduce an innovative transformer-style block that significantly enhances the screen image encoding process. This block, inspired by but diverging from the design in [Huang et al., 2022], consists of three core components: Convolutional Position Embedding (CPE), Superpixel based Content Aggregation (SCA), and Convolutional Feed-Forward-Network (ConvFFN). These components are intricately designed to optimize the transformation of model features, Ftrans, in the following manner:

where the CPE utilizes a 3×3 depth-wise convolution to provide positional information. The SCA is tailored to screen images, leveraging their characteristic structures to improve global context representation by efficiently capturing longrange dependencies. The ConvFFN, comprising two 1 × 1 convolutions, one 3×3 depth-wise convolution, and a GELU non-linear function, is specifically optimized for strengthening local feature representations in screen images.

Recognizing the importance of both global and local information in screen image encoding, we complement our transformer with a parallel convolution residual layer, uniquely designed for capturing fine-grained local details often present in screen content. This dual approach allows our model to effectively balance between global and local feature extraction. The entire process, incorporating this local learning aspect, is formulated as:

The complete architecture of our Screen Content Aggregation Block (SCAB) is depicted in Figure 2, illustrating how it uniquely addresses the challenges of screen image compression.

3.4 Advanced Entropy Coding with Channel-wise Context Modeling and Dynamic Feature Fusion

The most popular context modeling is proposed by Minnen [Minnen et al., 2018], but it comes with increased decoding complexity. To address this, [Minnen and Singh, 2020] proposes a context model along the channel dimension, improving decoding efficiency. The latent tensor y is divided into N slices along the channel dimension with entropy parameters for each slice conditioned on the previously decoded ones. However, this approach lacks dynamic integration of side information from each slice and its decoded counterpart for entropy model parameter estimation, an aspect that could be further improved for more efficient screen content image encoding.

To address this issue, we introduce a dynamic feature fusion mechanism (DFF) combined with superpixel aggregation. As illustrated in Figure 5, we first compress the concatenated features using a 1x1 convolution. A gating mechanism then dynamically determines weights to enhance the entropy parameter estimation process by adapting to the decoded slices and side information. Simultaneously, the embedded Superpixel Context Aggregation (SCA) module allows the learning process to incorporate global information. Finally, the learned weights are multiplied back into the input branch, followed by another convolution to map the channels back to their original size.

To validate that our proposed DFF introduces minimal deviation, we follow the approach by Xie et al. [Xie et al., 2021]. This involves analyzing pixel-wise differences between the compressed ˆ y and the original latent y, and measuring the information loss during compression by evaluating the deviation between ˆ y and y. The mean absolute pixel deviation ϵ is defined as:

We further introduce a scaling factor γ to normalize the deviation, accounting for different pixel value ranges in different models:

Figure 6 illustrates the scaled deviation map of y for 'siqad16' and 'siqad20' from the SIQAD dataset, using models both with and without DFF. The result indicates that the model with DFF incurs less information loss, thus producing higher-quality decompressed images.

4 Experimental Evaluation

4.1 Experimental Setup

Training Settings

We select the SCI2K [Shen et al., 2022] as our training dataset, where image samples are randomly cropped into 256 × 256 × 3 patches. The network is optimized with the Adam optimizer, a batch size of 8, and trained on a single RTX 3090 GPU for 5M steps. The initial learning rate is 10-4 , reduced to 10-5 after 4.5M steps.

Model Settings

Our network is implemented using the open-source CompressAI PyTorch library [B´egaint et al., 2020]. Six models are trained from scratch to match different bitrates by adjusting λ. The number of channels, M, for the latent variable y is set to 320, and for z it's 192. These settings balance computing cost and model performance. The number of slices s of the entropy model is set to 5. Additional hyperparameters within the entropy model align with the settings described in [Minnen and Singh, 2020]. For the distortion measurement using MSE loss, λ2 is chosen from the set {0.00013, 0.00025, 0.00067, 0.0018, 0.0035, 0.0067, 0.013}. They were chosen based on preliminary experiments that yield good balance between compression rate and quality. For the MS-SSIM loss, λ is chosen from the set {2.4, 4.58, 8.73, 16.64, 31.73, 60.5}.

Testing

During testing, we use three common datasets for SCI quality estimation: CCT 3 , SCID 4 , and SIQAD 5 . These representative SCI datasets are also used in the SCI coding quality assessment research community[Min et al., 2021]. The CCT dataset contains samples with a resolution of 915 × 1627, the SCID dataset provides images with a resolution of 720×1280, and the SIQAD dataset contains 24 screen images with 2k spatial resolution approximately. We use both PSNR and MS-SSIM to quantify the quality of decoded images, and bpp to measure the compressed bitrate.

2 In the implementation, the actual loss function is given by λ × 2552 ×D+R, which is specifically tailored for optimization in the context of Mean Squared Error (MSE). When optimizing with the Multi-Scale Structural Similarity Measure (MS-SSIM), as suggested in [B´egaint et al., 2020], the actual loss function is modified to λ×

(1 - D) + R. 3 https://sites.google.com/site/minxiongkuo/uca 4 http://smartviplab.org/pubilcations/SCID.html 5 http://smartviplab.org/pubilcations/SCI GSS.html

4.2 Evaluation on Rate-Distortion Performance

Anchor and Comparative Methods

We compare our model with several notable LIC solutions, such as Minnen et al. (2018) [Minnen et al., 2018], Cheng et al. (2020) [Cheng et al., 2020], Zou et al. (2022) [Zou et al., 2022], and Liu et al. (2023) [Liu et al., 2023]. We use VVC Intra, with its reference software VTM-20.0, as the anchor for calculating BD-rate gains. We also present results of VVC-SCC Intra with screen content coding options. RD curves are plotted in Figures 7 and BD-rate gains against the VVC anchor are in Table 1. For fairness, all compared methods are finetuned on the SCI2K dataset for 300 epochs based on their best pretrained models.

Quantitative Performance

As for the results on SIQAD dataset, our proposed method outperforms all other LIC solutions for the distortion measured by both PSNR and MS-SSIM, as shown in Figure 7.

We convert MS-SSIM to -10 log10(1 - MS-SSIM) to ease the comparison. Remarkably, our proposed method surpasses the state-of-the-art H.266 screen content coding tool, VVCSCC, especially at lower bit rates (below 0.2 bpp). Our model achieves this by effectively prioritizing the most visually significant information for compression, thereby retaining higher image quality even at lower bitrates. For a quantitative evaluation, we employ BD-rate derived from PSNRBPP curves as the metric. Remarkably, our method exceeds VVC (VTM-20.0) by 17.00%, 35.94%, and 51.04% in BDrate on the CCT, SCID, and SIQAD datasets, respectively. These results are summarized in Table 1. More comparisons are reported in supplementary.

Qualitative Visualization

We perform a meticulous qualitative analysis by visually comparing the results generated by our proposed model with those generated by CNN-based models [Minnen et al., 2018; Cheng et al., 2020] and Transformer-based models [Zou et al., 2022; Liu et al., 2023], using images from the SIQAD dataset, specifically the 'siqad17'. Figure 1 illustrates the results of this comparative analysis. We focus on two distinct local regions to assess detail reconstruction. Our method outperformed others in the upper local region of 'siqad17', generating fewer artifacts, especially in the reconstruction of the characters "O" and "V", and enhancing the clarity of background elements like the drainage pipe. For more visual results, see the supplementary material.

4.3 Ablation Studies

SCAB Module

We demonstrate the effectiveness of our proposed SCAB through an ablation study, comparing our model with an alternative using Swin Transformer blocks [Liu et al., 2021]. Unlike Swin Transformer blocks that focus on local attention, SCAB calculates global attention over superpixels for more holistic information capture. The study results, presented in Figure 8, show that our model with SCAB blocks significantly improves efficiency and performance over the "SwinT" model that employs Swin Transformer blocks.

DFF Module

To highlight the efficacy of the DFF Module within our proposed entropy model, we conducted ablation studies and present the results in Figure 6 and Figure 8(b). After removing the DFF module, the RD performance is degraded, as shown in Figure 8(b). Furthermore, the visualization of residual errors in Figure 6 also shows that the probabilities estimated using the DFF module are more accurate, resulting in smaller latent feature errors. These results confirm the crucial role of the DFF module in improving the performance of the entropy model.

4.4 Complexity and Efficiency Analysis

We evaluate the computational complexity and decoding quality of various image compression methods [Minnen et al., 2018; Cheng et al., 2020; Zou et al., 2022; Liu et al., 2023], including our proposed method, based on the SIQAD dataset. Figure 9 visualizes the trade-off between the model complexity (parameters and inference time) and the compression performance (BD-rate) for various models. Our proposed method achieves an impressive balance by maintaining faster inference times and a relatively smaller model size while also reaching lower BD-rates. For detailed information, see the supplementary material.

5 Conclusion

In this work, we propose a new end-to-end network for screen content image compression. Specifically, a superpixelbased content aggregation block is proposed to aggregate local regional information through superpixel grouping and the global redundancy is explored via super-pixel based transformer. Additionally, we also introduce a dynamic feature fusion mechanism to enhance the channel-wise context entropy model. The DFF module adaptively adjusts the probability distribution of undecoded channels, optimizing information utilization during decoding. Comprehensive experiments on three datasets validate the superiority of our method in terms of both RD performance and efficiency.

Acknowledgements

This work was supported in part by National Natural Science Foundation of China under Grant 62231018 and Grant 62072331.