Universal Video Style Transfer via Crystallization, Separation, and Blending

1 Introduction

Style transfer aims to render a content target with the desired style of a given style reference. Recently, the seminal work of [Gatys et al., 2016] showed that the Grammatrix (i.e., feature correlation) of features extracted from a Deep Convolutional Neural Network (DCNN) could represent the style patterns well. Since then, significant efforts have been made to improve the performance in various aspects, including efficiency [Johnson et al., 2016], quality [Wang et al., 2021; Chen et al., 2021b], generality [Huang and Belongie, 2017; Li et al., 2017], and diversity [Ulyanov et al., 2017; Wang et al., 2020b], etc. However, existing style transfer approaches are mostly designed for still images, while the stylization for dynamic videos is still an open challenge.

∗Corresponding author.

Compared with image style transfer, video style transfer is a much more challenging task. The method should consider both the temporal consistency and stylization quality of the stylized videos. Existing image style transfer methods, while achieving satisfactory results for still images, often result in significant jitters and flickers when applied to videos due to the lack of in-depth consideration into video temporal stability. To overcome this problem, [Ruder et al., 2016] introduced optical flow prediction to [Gatys et al., 2016] to maintain the video-frame consistency. However, it takes too much time to stylize a single frame. For fast video style transfer, [Chen et al., 2017] designed a feed-forward optical flow and mask estimation network, and [Gao et al., 2020] proposed a multi-style framework that estimates the light flow and uses temporal constraint. These methods benefit from optical flow prediction to maintain temporal stability but rely highly on estimation accuracy. What's more, works like [Wang et al., 2020a] have justified that optical flow not only suppresses the stylization quality but also may bring obvious video jelly effects in the results.

Recently, [Li et al., 2019] proposed a stable linear transformation that is suitable for frame-based video style transfer. Later, [Wang et al., 2020a] introduced a novel interpretation of temporal consistency without optical flow estimation and achieved consistent zero-shot video style transfer. Based on it, [Deng et al., 2021] proposed multi-channel correlation for arbitrary video style transfer. These approaches achieved the state-of-the-art video style transfer. However, their stylization quality is not satisfactory, and some slight brushstroke jitters still exist. Therefore, how to maintain the temporal consistency of videos while achieving high-quality arbitrary style transfer is still a hard nut to crack.

In order to avoid using the optical flow method and complete high-quality arbitrary style transfer while maintaining the video continuity, in this paper, we propose a novel framebased network for universal video style transfer. Our method, termed CSBNet, consists of three key modules: 1) the Crystallization (Cr) Module that uses singular value decomposition to generate several orthogonal matrices called crystal nuclei, representing hierarchical stability-aware content and style components, from raw VGG features; 2) the Separation (Sp) Module that separates these crystal nuclei to generate the stability-enhanced content and style features; 3) the Blending (Bd) Module to cross-blend these stabilityenhanced content and style features, producing more stable and higher-quality stylized videos. At the same time, we also introduce a new pair of component enhancement losses to improve network performance. Extensive qualitative and quantitative experiments demonstrate that our method can effectively suppress the generation of local artifacts and inherently maintain the smoothness of input videos while successfully keeping the content structures and style details, significantly outperforming the state-of-the-art (SOTA) models.

The main contributions of our method are:

• A novel frame-based network CSBNet consisting of a Crystallization (Cr) Module, a Separation (Sp) Module, and a Blending (Bd) Module, to achieve consistent universal video style transfer with higher stylization quality.
• A new pair of component enhancement losses to improve the performance of our network, helping produce more stable and higher-quality results.
• Extensive qualitative and quantitative experiments to demonstrate the effectiveness and superiority of our method in both image and video style transfer.

2 RelatedWork

Image Style Transfer. [Gatys et al., 2016] firstly introduced DCNN to artistic style transfer. In their method, the style of an image is represented by the Gram matrix (i.e., feature correlation) of the intermediate features extracted from DCNN. While achieving visually stunning results, the iterative procedure is time-consuming, and the output is sensitive to input noise. Later, [Li and Wand, 2016] improved the stability by introducing Markov Random Fields (MRFs) and proposed a patch-based method. To achieve real-time transfer, [Johnson et al., 2016] introduced a feed-forward approach by training feed-forward networks. However, the trained networks are fixed for pre-defined styles. To achieve arbitrary style transfer, AdaIN [Huang and Belongie, 2017] aligned the feature maps of content images with style ones by normalizing means and variances. Also, WCT [Li et al., 2017] orthogonalized the content features and matched the covariance matrix of the style features. However, outputs generated by AdaIN and WCT failed to represent the detailed textures sufficiently and unexpectedly brought twisted style patterns at some local areas more or less. Further, Linear [Li et al., 2019] learned a linear transformation to better preserve content affinity during style transfer. SANet [Park and Lee, 2019] employed a style-attention module to better capture local style patterns. AAMS [Yao et al., 2019] introduced self-attention methods to improve output quality. AdaAttN [Liu et al., 2021] proposed a novel attention and normalization module for arbitrary style transfer. IECAST [Chen et al., 2021a] incorporated the adversarial nets and contrastive losses to improve SANet.

Despite the recent progress in image style transfer, few of them considered the temporal consistency, resulting in obvious flickering effects when applied to videos. Moreover, there's still improvement room for existing style transfer approaches to better blend content structures with style patterns to generate higher-quality stylized results.

Video Style Transfer. Current video style transfer methods can be roughly divided into two categories: 1) methods based on optical flow prediction and 2) non-optical flow prediction methods with single frame constraints.

In order to maintain video-frame consistency, [Ruder et al., 2016] introduced optical flow prediction of the previous frame to guide the generation of the current time. However, it needs several minutes to generate a single output frame, which is unsuitable for large-scale applications. Consequently, [Chen et al., 2017] designed a feed-forward network for fast video style transfer. [Gao et al., 2020] introduced ConvLSTMs to stabilize results and FlowNet(s) to simulate optical-flow prediction. These methods benefit from optical flow prediction to maintain temporal consistency but rely highly on estimation accuracy. In addition, optical-flow guidance also limits the quality of stylization [Wang et al., 2020a].

Recently, Linear [Li et al., 2019] learned a stable linear transformation that is suitable for frame-based video style transfer. ReReVST [Wang et al., 2020a] introduced a novel interpretation of temporal consistency without optical flow estimation and achieved consistent zero-shot video style transfer via relaxation and regularization. MCCNet [Deng et al., 2021] proposed multi-channel correlation for arbitrary video style transfer. These methods achieved the state-of-theart video style transfer. However, their stylization quality is unsatisfactory, and some slight brushstroke jitters still exist.

To avoid using the optical flow estimation while obtaining high-quality stylization and maintaining the video continuity, we propose a novel frame-based universal video style transfer approach via crystallization, separation, and blending. The stability-aware content and style components of the content and style images are first crystallized and separated. Then, the stability-enhanced content feature of the content image and the style feature of the style image are cross-blended, thus producing temporally more stable and higher-quality results.

3 ProposedMethod

3.1 Network Structure

We first depict the main structure of our network from the high level, which is based on the encoder-decoder model. As shown in Fig. 1, CSBNet uses a pre-trained VGG-19 network [Simonyan and Zisserman, 2014] as the encoder and the symmetric network as the decoder. Given a content image Ic and a style image Is, we first generate their corresponding VGG features Fc = VGG(Ic) and Fs = VGG(Is).

Then, the VGG features Fc and Fs are fed into the Crystallization (Cr) Module (Sec. 3.2) to generate several stabilityaware orthogonal matrices A1,A2, ...,AN, namely crystal nuclei in our work:

Next, the Separation (Sp) Module (Sec. 3.2) completes the separation of these crystal nuclei according to the empirically determined cursors Kc and Ks (which control the degree of separation), generating the stability-aware content component F content and style component F style :

In the following step, the style component F style c of Fc and the content component F content s of Fs will be removed, and we only use the content component F content c of Fc and style component F style s of Fs to generate the stability-enhanced content feature F enhance c and style feature F enhance s . They are cross-blended via the Blending (Bd) Module (Sec. 3.3):

Finally, the blended feature Fcs is decoded to obtain the stylized output Ics:

3.2 Crystallization & SeparationModule

After obtaining the VGG features Fc and Fs, existing neural style transfer approaches always use these raw features directly to produce the stylized feature Fcs. However, we argue that for raw VGG features, the content components and style components are mixed. And directly using the mixed features will easily generate unnatural and flickering effects where the style components of the content images remain, or the content components of the style patterns appear in the form of ghosting artifacts. To generate these stability-aware content and style components and complete the separation of them, we propose two key modules in our CSBNet, i.e., the Crystallization (Cr) Module and Separation (Sp) Module.

The structures of Cr and Sp modules are shown in Fig. 3, where these two modules are closely related.

Crystallization (Cr) Module. Given a raw VGG feature F ∈ {Fc, Fs} ∈ RC×H×W (where C is the number of channels, H and W are height and width, respectively), we first flatten it to ˆ F ∈ RC×HW and center ˆ F by subtracting its channel-wise mean.

Then we perform orthogonal projection on ˆ F via singular value decomposition (SVD):

where N = min(C,HW), σi is singular value, ui ∈ RC×1 and vi ∈ RHW×1 are the ith column of U and V , respectively. We define Ai = σiuivT i ∈ RC×HW and a matrix set Φ = {A1, A2, ..., AN}. According to the properties of SVD, for every Ai, Aj (i ̸= j) in Φ, we have:

which means Ai and Aj are orthogonal. In experiments, we find these orthogonal matrices can represent different content and style components hierarchically (see later "Separation (Sp) Module"). These matrices are stability-aware, e.g., if we separate and remove the style component of the content feature and the content component of the style feature, we can fundamentally enhance the transfer stability. Therefore, we design the Sp Module for this purpose. For easy identification, we call these orthogonal matrices crystal nuclei.

Separation (Sp) Module. This module is designed to separate the above orthogonal crystal nuclei into the content component F content and the style component F style :

where K is an empirically determined cursor to decide the degree of separation between the content and style components. Θ is the unflatten operation to reshape the feature size to C × H × W .

To demonstrate the motivation and rationality of this separation, we visualize the pixel-level content components and style components in Fig. 2. As can be observed, with the increase of K, the content component (top row) filters out more style information and preserve more pure structural details. In contrast, the style component (bottom row) adds more detailed style information and filters out certain content information. In this way, we can further separate the content and style information to obtain more pure content and style features, which may be helpful to improve the stylization quality and transfer stability.

Specifically, we set K = Kc to separate the crystal nuclei of the content feature Fc, and K = Ks to separate the crystal nuclei of the style feature Fs. Thus we can easily control the content-style separation for the content image and style image via adjusting Kc and Ks, respectively.

After separation, the style component F style c of Fc and the content component F content s of Fs are removed, and we only use the content component F content c of Fc and the style component F style s of Fs to generate the stability-enhanced content feature F enhance c and style feature F enhance s :

where f1(·), f2(·), g(·), and h(·) are 1×1 learnable depthwise convolution layers. We use depthwise convolution here in order to maintain the hierarchical independence of each channel. Besides, depthwise convolution also offers less computational complexity. As shown in Fig. 3, a double-pole switch is used to generate F enhance c and F enhance s based on whether the input feature is from the content or style image.

3.3 BlendingModule

We further design a novel Blending (Bd) Module to blend the stability-enhanced content feature F enhance c with the style feature F enhance s more effectively. Our Bd module is based on MCCNet [Deng et al., 2021], which considers the multichannel correlation to achieve more stable feature alignment:

where f i/k ∈ R1×HW is the ith /kth channel of flattened feature F ∈ { ˆ Fc, ˆ Fs, ˆ Fcs}, C is the number of channels and wk represents the weight of the k-th style channel learned by a fully connected layer. While it has been verified that this operation can improve the stability and temporal consistency of video style transfer, we found it is easy to bring bar-like artifacts to the generated outputs (e.g., the 1st and 5th rows in Fig. 5). It is because the MCC in Eq. (9) is sub-optimal, which only learns partial information of the Gram matrix.

For explanation, let's start with the Gram matrix of Fs:

It can be easily derived that the weighted item ||f k s ||2 in Eq. (9) equals f k s (f k s )T , which is the kth diagonal element of the Gram matrix in Eq. (10). This indicates that MCCNet conducts style transfer based on only C diagonal elements of Gram matrix, while C × C - C non-diagonal elements, which also represent the correlation between channels, are abandoned. Thus it may lead to degraded transfer results.

Our proposed Bd Module overcomes this problem through better capturing the associations between different style channels. As shown in Fig. 4, we first normalize and tansform the enhanced content feature F enhance c and the style feature F enhance s as follows:

where J (·) and K(·) are 1 × 1 learnable convolution layers, and Norm(·) is channel-wise mean-variance normalization.

Then for ˜ Fs, we calculate its Gram matrix Gs = Gram(˜ Fs) and weight each row gis of Gs:

where wi ∈ R1×C is the ith row of W ∈ RC×C , a parameterlearnable matrix. Based on this, we further obtain the fusion component ˜ Fcs. The ith column of ˜ Fcs is as follows:

where ˜ f i ∈ R1×H×W is the ith channel of ˜ F ∈ {˜ Fcs, ˜ Fc}. The final fusion feature Fcs can be obtained as:

where R(·) is a 1 × 1 learnable convolution layer.

3.4 Loss Function

We train our CSBNet using the following total loss function consisting of a peceptual loss Lpercp, our proposed component enhancement loss Lcomp, and a smooth loss Lsmooth:

Perceptual Loss. To achieve style transfer, we use a pretrained VGG-19 to extract content and style features and compute the content and style perceptual losses similar to AdaIN [Huang and Belongie, 2017]:

where λcontent and λstyle are hyper-parameters to balance the content and style performance.

Specifically, the content perceptual loss is used to minimize the content differences between the generated images Ics and content images Ic:

where φi represents the ith layer in VGG-19, here we use Relu4 1 and Relu5 1 to calculate Lcontent.

The style perceptual loss minimizes the style differences between the generated images Ics and style images Is:

where µ(·) and σ(·) denote the channel-wise mean and standard deviation, respectively. And we use Relu1 1, Relu2 1, Relu3 1, and Relu4 1 to calculate Lstyle.

Component Enhancement Loss. In order to enhance the quality of content-style component separation in the network, we design a novel pair of component enhancement losses, consisting of a content componet loss Lc-comp and a style component loss Ls-comp:

where λc-comp and λs-comp are hyper-parameters.

We hope to minimize the content and style differences of corresponding components. Thus, we define the content component loss as the content differences between the stabilityenhanced content features F enhance c and the generated features Fcs:

Similarly, the style component loss is used to minimize the style differences between the stability-enhanced style features F enhance s and the generated features Fcs:

With these component enhancement losses, our network can better learn the stability-enhanced content and style information, thus effectively suppressing the generation of artifacts and improving the stylization quality as well as stability (see ablation studies in later Sec. 4.4).

Smooth Loss. Similar to MCCNet [Deng et al., 2021], we also adopt a total variation loss Ltv [Johnson et al., 2016] and an illumination loss Lillum to smooth the results and improve the robustness to complex light conditions in input videos.

where λtv and λillum are hyper-parameters. The illumination loss Lillum is defined as:

where I noise cs is the results generated with noisy content image Ic + θ (θ is random Gaussian noise) and style image Is.

4 Experimental Results

4.1 Implementation Details

The hyper-parameters in our model are set to λcontent = 3, λstyle = 10, λc-comp = 3, λs-comp = 1, λtv = 0.0001, and λillum = 3000. The separation cursors are empirically set to Kc = 4, Ks = -10 (here '-' represents the inverted index of array). We choose MS-COCO [Lin et al., 2014] as content image set and WikiArt [Phillips and Mackintosh, 2011] as style image set. During the training stage, the input images are resized to 512×512 pixels and randomly cropped to 256× 256 pixels. Batch size is chosen as 4 for 160,000 iterations.

4.2 Qualitative Comparisons

Eight SOTA universal image and video style transfer methods are selected for comparison, including AdaIN [Huang and Belongie, 2017], WCT [Li et al., 2017], Linear [Li et al., 2019], AAMS [Yao et al., 2019], MCCNet [Deng et al., 2021], AdaAttN [Liu et al., 2021], IECAST [Chen et al., 2021a], and ReReVST [Wang et al., 2020a].

Image Style Transfer. The comparisons of image stylizations are shown in Fig. 5. The over-simplified alignment of AdaIN easily leads to style artifacts and content distortion (e.g. 2nd and 5th rows). Due to global parameter transformation, WCT seriously damages the content structures in the results. Linear can better preserve content affinity, but the degree of stylization is low (e.g. 1st and 2nd rows). AAMS, MCCNet, AdaAttN, and IECAST are essentially based on the attention mechanism. However, point-like artifacts appear in AAMS (e.g. 1st, 3rd, and 5th rows), and bar-like artifacts appear in MCCNet (e.g. 1st and 5th rows). AdaAttN preserves the content structures well but still suffers from noticeable artifacts (e.g. 3rd and 5th rows) and poor transfer of style color distributions (e.g. 1st and 3rd rows). IECAST is prone to produce eye-like artifacts (e.g. 2nd and 3rd rows). ReReVST is designed for video style transfer, which often produces less stylized results for image stylizations.

In contrast to these SOTA approaches, the results of our CSBNet exhibit higher stylization quality. The active separation of content and style components by the Cr and Sp Modules successfully suppresses the appearance of style artifacts and better maintains the content structures. Moreover, the better utilization of style information by the Bd Module leads to a higher degree of stylization in the results.

Video Style Transfer. The comparisons of video stylizations are shown in Fig. 6. We present heatmaps to show the average differences between adjacent frames. It can be seen that the difference of our results is closest to that of the input frames, obviously outperforming other methods in terms of temporal stability and consistency. This is due to the fact that our CSBNet can effectively maintain the content structures of the video frames and suppress the generation of random artifacts causing frame-flicker.

4.3 Quantitative Comparisons

Stylization Quality. We leverage the content and style errors Ec and Es of [Gatys et al., 2016] to quantitatively evaluate the stylization quality. As shown in the top two rows of Tab. 1, CSBNet enjoys the lowest style error Es among 9 methods, while content error Ec is only slightly higher than Linear [Li et al., 2019], ranking the second. Metrics Ec and Es show that our method can effectively improve stylization quality while better preserving content structures, leading to comprehensively more satisfying transfer results.

Video Consistency. We adopt LPIPS (Learned Perceptual Image Patch Similarity) [Zhang et al., 2018] and D∗ [Deng et al., 2021] to quantify the stability and consistency of video style transfer. Given a video clip with N frames, we compute the average perceptual distances between adjacent frames via LPIPS, and the average pixel distances via D∗ , i.e., D∗ = 1 N-1 ∑N-2 t=0 ||Ft+1-Ft||2. The results shown in the middle two rows of Tab. 1 further reflect the superiority of our CSBNet on video style transfer, which is consistent with the qualitative results in Fig. 6.

Efficiency. As shown in the last row of Tab. 1, CSBNet achieves comparable speed with SOTA methods, only slightly behind AdaIN, Linear, MCCNet, and IECAST. However, we believe it is worthwhile to sacrifice slight run time speed for higher stylization quality and improvement of consistency.

4.4 Ablation Studies

Crystallization and Separation (Cr-Sp) Module. Fig. 7 shows the results of video style transfer with and without CrSp module for Fc and Fs, respectively. It can be found that (b) without Cr-Sp for Fc, the unstable strip and dot artifacts are generated in the results; (c) without Cr-Sp for Fs, there will be a loss of content details (e.g., the feather and eyes of the bird) and the blue color of the content image leak into the result (e.g., the eyes of the bird). In contrast, with Cr-Sp for both Fc and Fs, our method generates smoother and more satisfying results with better-preserved content structures and

(a) Effects of Kc (Ks = -10)

(b) Effects of Ks (Kc = 4)

Table 2: Effects of different settings of Kc and Ks.

fewer local artifacts. This observation is consistent with the quantitative scores shown in column (e), which validates that our Cr-Sp module indeed helps improve the stylization quality and temporal stability.

Component Enhancement Loss. A pair of new losses is introduced in our framework for enhancing transfer results, and thus we further conduct ablation experiments on it. From Fig. 8, we can observe that without the proposed component enhancement loss, the edge of the generated result is prone to produce grid-like artifacts (e.g., the edges of tree and the squirrel's ear and back), which not only degrades the stylization quality, but also reduces the transfer stability. Best viewed in conjunction with the quantitative scores below.

Setting of Kc and Ks. We also conduct ablation experiments to demonstrate the effects of different settings of our separation cursors Kc and Ks. We can observe from Tab. 2 that increasing the values of |Kc| and |Ks| within a certain range not only helps to reduce the content and style errors Ec and Es, but also increases the temporal stability of output videos (lower D∗ and LPIPS scores). It is because increasing the value of |Kc| can filter out the style components of content images, thus helping improve the stylization of the results. Meanwhile, increasing the value of |Ks| can filter out the content components of style images, thus helping preserve the integrity of the content structures and improving the videos' fluency. However, too high values of |Kc| and |Ks| will reduce the stylized quality and stability of the output videos.

5 Conclusion

In this work, we propose a novel frame-based network, termed CSBNet, for universal video style transfer. Three innovative modules, i.e., the Crystallization (Cr) Module, Separation (Sp) Module, and Blending (Bd) Module, and a new pair of component enhancement losses are introduced to improve video consistency and achieve higher-quality stylization. Extensive qualitative and quantitative experiments are conducted to demonstrate the effectiveness and superiority of our CSBNet against state-of-the-art algorithms.