Video Frame Interpolation with Densely Queried Bilateral Correlation

1 Introduction

Video Frame Interpolation (VFI) aims to synthesize nonexistent intermediate frames between existent frames. VFI is used for a wide range of applications such as slowmotion generation [Jiang et al., 2018], video compression [Wu et al., 2018], video restoration [Haris et al., 2020; Kim et al., 2020; Xiang et al., 2020], novel view synthesis [Zhou et al., 2016; Kalantari et al., 2016], animation production [Siyao et al., 2021; Chen and Zwicker, 2022; Shen et al., 2022] and video prediction [Wu et al., 2022]. Nowadays, the flow-based paradigm dominates the methodology of VFI owing to its superior performance. However, the absence of the reference frame makes it challenging to estimate the intermediate optical flow fields or motion fields used to warp the neighboring frames backward. [Bao et al., 2019; Jiang et al., 2018; Liu et al., 2019; Niklaus and Liu, 2018; Xue et al., 2019] approximate the intermediate motion fields by linearly combining the optical flow fields between the two input frames. [Reda et al., 2022; Huang et al., 2022; Kong et al., 2022] directly estimate the motion fields using Convolutional Neural Networks (CNN) in a coarse-tofine manner. [Park et al., 2020; Park et al., 2021; Lu et al., 2022] explicitly model correlations between two input frames, which appears as a strong prior in the VFI task and helps to produce more accurate motion fields.

∗Corresponding author.

Correlation modeling is often conducted locally in a coarse-to-fine manner coupled with the motion fields estimation as in [Park et al., 2020; Park et al., 2021; Lu et al., 2022]. The top half ("The common") of Figure 1 illustrates this idea. At each resolution level, for every location in the motion fields, two local windows on I0 and I1 are picked to model correlations between them. In correlation modeling, similarities of patches in two local windows (Each local window contains 3 × 3 patches in the top half of Figure 1.) are computed. These similarity values play an important role in motion fields revisions performed by neural networks. After being revised, the motion fields are up-sampled, and correlation modeling is conducted again at a higher resolution. This process is repeated across several resolution levels.

However, such a paradigm suffers from the "receptive field dependency" problem, which will be explained in detail next. As shown in the top half of Figure 1 , correlation modeling is conducted within two local windows, and the locations of the two local windows are determined by motion vectors. Considering the local windows are relatively small at higher resolution, if the motion vectors estimated at low resolution are inaccurate, the local windows at higher resolution may not cover the butterfly at all. In that case, the correlation modeling at higher resolution does not work as expected because the local windows are not placed in the intended areas. In other words, the receptive field of correlation modeling depends on the motion fields estimated beforehand. So, in the common approach shown in the top half of Figure 1, we expect that the motion fields estimated at lower resolution are reasonably accurate to prevent misleading the correlation modeling at higher resolution. However, it is not fair to expect the motion fields estimated at low resolution to be "reasonably accurate", especially for small objects. The reasons are as follows: At low resolution, patches used for computing similarities are relatively large, and small objects may only occupy a small fraction of the patch. Thus the enormous background may dominate the similarity computing regarding patches that contain small objects. Besides, suppose a large patch on one frame includes two or more small objects, and these small objects are separated across different patches on the other frame. In that case, the correlation modeling and motion estimation at low resolution may face ambiguity.

Low-resolution correlation modeling is expected to cope with fast-moving objects but not small objects, and highresolution correlation modeling is expected to cope with small objects but not fast-moving objects. Superficially, they are complementary. But considering the aforementioned receptive field dependency problem, if low-resolution correlation modeling fails to capture small and fast-moving objects (which may easily happen), high-resolution correlation modeling may also miss that object. The receptive field dependency problem makes the common approach poor at coping with small and fast-moving objects.

In this work, we propose the Densely Queried Bilateral Correlation (DQBC) to better model correlations between two frames and produce more accurate motion fields. We conduct correlation modeling differently compared with the common approach. The bottom half of Figure 1 illustrates our paradigm. Instead of modeling correlations iteratively from very low resolution, we model correlations all at once at a single and relatively high resolution. Within the local windows, DQBC is calculated asymmetrically: When computing similarities between patches, query patches keep their resolution and key patches are down-sampled several times for larger receptive fields. DQBC gets rid of the receptive field dependency problem and helps generate more accurate motion fields, especially for small and fast-moving objects. To refine and up-sample the motion fields to full size, we further devise a Motion Refinement Module (MRM). MRM refines and up-samples the input motion fields with the help of context features. Compared with [Lu et al., 2022] that uses heavy transformers to synthesize the intermediate frame after motion estimation, we devise a relatively lightweight CNN called SynthNet to synthesize the final result.

Our contributions are summarized as follows:

• We propose the Densely Queried Bilateral Correlation (DQBC) to better model correlations for the VFI task. DQBC gets rid of the receptive field dependency problem and thus is more friendly to small and fast-moving objects.
• With the DQBC, our VFI framework achieves state-ofthe-art performance on various VFI benchmarks. In our VFI framework, we devise a Motion Refinement Module (MRM) to refine and up-sample the motion fields and a CNN-based SynthNet to synthesize the interpolated frame.

2 RelatedWork

VFI is a long-standing task, and various approaches have been proposed.

Kernel-based approaches such as [Niklaus et al., 2017a; Niklaus et al., 2017b; Cheng and Chen, 2020; Cheng and Chen, 2021; Lee et al., 2020] estimate spatially-adaptive convolutional kernels for each pixel to be synthesized. [Meyer et al., 2015; Meyer et al., 2018] represent motions in the phase shifts of individual pixels. [Choi et al., 2020; Choi et al., 2021] distribute spatial information into the channel dimension and perform channel attention for pixel-level frame synthesis.

Besides these, the flow-based approaches enjoy a long history and have been prevailing. In recent flow-based VFI works, [Niklaus and Liu, 2018] proposes to warp the context features using the motion fields for better interpolation quality. [Jiang et al., 2018] proposes a network for variable-length multi-frame video interpolation. [Bao et al., 2019] exploits the additional depth information to better handle occlusions in the sampling process. [Peleg et al., 2019; Sim et al., 2021] explore high-resolution video interpolation. [Tulyakov et al., 2021] explores the event-based approach. [Park et al., 2020] proposes to model correlations between two frames explicitly via cost volume and is improved by [Park et al., 2021] who further models the asymmetric motions. [Huang et al., 2022; Kong et al., 2022] aim at efficient VFI algorithms and yet have shown compelling performance. [Lu et al., 2022] proposes a transformer-based synthesis network to blend two warped frames better and hallucinate details.

Our work follows the flow-based paradigm and gives focuses on correlation modeling. Among the approaches mentioned above, [Park et al., 2020; Park et al., 2021; Lu et al., 2022] explicitly model the correlations between two frames. [Park et al., 2020] computes similarities between any two centrosymmetric patch pairs between two local windows.

[Park et al., 2021] further models the correlation between the intermediate anchor frame produced by [Park et al., 2020] and two input frames. [Lu et al., 2022] adopts the IFNet [Huang et al., 2022] without explicit correlation modeling as the base motion estimator and further refines the motion fields with explicit correlation modeling. It computes similarities between the center patch of one local window and all patches of the other. Despite the difference in the modeling details, they all model correlations locally in a coarse-to-fine manner, as illustrated in the top half of Figure 1.

Our work takes two frames as inputs. There are also VFI works that take four input frames and are beyond the scope of this paper. We refer interested readers to [Xu et al., 2019; Liu et al., 2020; Chi et al., 2020; Shi et al., 2022; Danier et al., 2022; Yu et al., 2022] for more details.

3 The ProposedMethod

3.1 Overview

Given 2 adjacent frames I0 and I1, we synthesize the interpolated frame ˆ It(t = 0.5) as:

In Equation (1), Mt-→τ ∈ RH×W×2 is the estimated optical flow field from ˆ It to Iτ where τ ∈ {0, 1}. They are referred to as motion fields. O ∈ [0, 1]H×W is the occlusion map and R ∈ RH×W×3 is the image residual. The ←- w (x, y)

operation is to spatially sample the input tensor x with the vector field y.

As shown in Figure 2, our VFI pipeline can be roughly divided into three stages: (1) Correlation Modeling, (2) Motion Estimation, and (3) Frame Synthesis. In stage (1), the Densely Queried Bilateral Correlation (DQBC) is extracted from two input frames. In stage (2), DQBC helps to generate preliminary motion fields that are further refined and upsampled by the Motion Refinement Module (MRM). MRM also produces an initial occlusion map. In stage (3), the SynthNet predicts the image residual and the occlusion map residual. Then the interpolated frame is synthesized following Equation (1).

3.2 Densely Queried Bilateral Correlation

The proposed DQBC explicitly encodes the comprehensive correlation information needed for motion estimation.

As shown in Figure 3(a), DQBC extraction consists of three parts: (1) Unilateral Correlation Gathering, (2) Correlation Enhancement (via "Enh(·)"), (3) Spatial Alignment (via "Dist(·)"), which can be summarized as follows:

where [·, ·] denotes concatenation along channels.

Unilateral Correlation Gathering

As shown in Figure 3(a), we take one frame as the query frame and the other as the key frame to gather unilateral correlations. Firstly, a feature extractor f(·) extracts highdimensional visual features F0, F1 from I0, I1. F1 is kept as a query set and won't ever be down-sampled. Thus the queries are spatially dense. F0 is down-sampled (L - 1) times by factor 1/2 to form the key sets of different resolution. For every location (xq, yq) on F1 and each resolution level l ∈ {0, 1, . . . , L - 1}, we calculate the embedding vector UniCorrl1-→0 (xq, yq) whose ith element is:

where ⟨·, ·⟩ is the inner product operation and F l0 is spatially down-sampled F0 by factor 1/2l and (xil , yi l ) is the ith location in the local window LWl defined as:

where rl is the radius of this local window.

Then the embedding vectors from all resolution levels at (xq, yq) are concatenated together to form the unilateral correlation embedding for (xq, yq):

Note that UniCorr0-→1 can be acquired in a similar way.

Correlation Enhancement

In Figure 3(a), the correlation enhancement block Enh(·) performs denoising and enhancement on unilateral correlation embeddings. Due to occlusions and 3D rotations, some objects may only appear on one of the two input frames. Thus the vanilla unilateral correlation embeddings may contain noise. The Enh(·) consists of two convolutional layers and a skip connection.

Spatial Alignment

The enhanced unilateral correlation embeddings are not spatially aligned with the intermediate frame. In Figure 3(a), the feature distributing operation Dist(·) performs spatial alignment by shifting feature maps.

We first define the displacement vector for each feature map of UniCorr. According to Equation (3), the ith feature map of UniCorrl1-→0 encodes the similarities between query locations on I1 and key locations on I0. We define the displacement vector as the displacement from the query location to the key location. Thus according to Equation (3), the displacement vector vi of the ith feature map of UniCorrl1-→0 is just (xil , yi l ). Equation (5) implies that every feature map of UniCorr1-→0 also has its displacement vector.

The Dist(·) function in Equation (2) shifts each feature map of the input embeddings by a fraction of its displacement vector. Dist 0-→t(·) shifts each feature map of its input by tv and Dist 1-→t(·) shifts each feature map of its input by (1 - t)v where v is the displacement vector of the feature map.

Figure 3(b) illustrates the intuition of Dist(·). A butterfly appears at P0, P12 , P1 on I0, I1 2 , I1 respectively. Following Equation (3) and Equation (5), we first gather the unilateral correlation embeddings from I0 to I1, denoted as Corr0-→1. By definition, the value at (x, y) on the cth feature map of Corr0-→1 (denoted as Corr0-→1,c) is the similarity between (x, y) on I0 and (x, y)+vc on I1 where vc is the displacement vector of this feature map. If P0 + vc is reasonably close to P1, Corr0-→1,c will have a relatively high activation value at P0. In Dist0-→ 1 2 (·), we displace this high activation value at P0 to be near P12 by shifting the entire feature map by 12 vc.

Now we have described every part of the DQBC extraction process summarized in Equation (2). The DQBC is 1/8 in spatial resolution of I0 and I1 since the feature extractor f(·) contains three convolutional layers with stride 2.

3.3 Motion Estimation

As shown in the middle part of Figure 2, in the motion estimation stage, the Motion Generation Module (MGM) generates preliminary motion fields facilitated by DQBC and then the Motion Refinement Module (MRM) refines and up-samples the motion fields to full size.

Motion Generation

With the extracted DQBC, MGM generates the preliminary motion fields as:

In Equation (6), [·, ·] denotes concatenation along channels. c(·) is a CNN that extracts context information from two input frames. m(·) is a Multi-Layer Perceptron (MLP) with one hidden layer to perform dimensionality reduction on DQBC. g(·) is a CNN to generate the motion fields.

The preliminary motion fields are 1/8 in spatial resolution of I0 and I1, just the same as DQBC.

Motion Refinement

MRM up-samples the motion fields to full size and compensates more details with the warped context features.

Context features are extracted by ContextNet which contains three down-sampling convolutional blocks. It takes Iτ as input and produces the context feature C iτ (i ∈ {0, 1, 2}) in 1/23-i spatial resolution of Iτ .

MRM consists of three UpBlocks each of which upsamples the input motion fields by factor 2. The first UpBlock takes the motion fields produced by Equation (6) and context features C 0 0 , C 0 1 as inputs and produces the up-sampled motion fields ˜M 1 t-→0 , ˜M 1 t-→1 and a hidden feature H1 as:

The second and third UpBlock are similar to the first UpBlock but also take the hidden feature produced from the previous UpBlock as input:

Figure 4 shows the structure of the (i + 1)th UpBlock (i ∈ {0, 1, 2}). Hi is surrounded by parentheses since there is no H0 for the first UpBlock. ←- C iτ denotes the warped context feature ←- w(C iτ , ˜M i t-→τ ). A convolutional head outputs residual motion fields ∆M i and the up-sampling parameter w. The input motion fields are added with ∆M i and then up-sampled with the parameter w. We up-sample the motion fields by taking the convex combination of neighboring 3×3 pixels on the input motion fields. Softmax is performed on w to get the convex combination weights that sum up to 1.

The occlusion map O is estimated from the last hidden feature H3 using a convolutional head.

3.4 Frame Synthesis

We synthesize the interpolated frame following Equation (1). As shown in Figure 2, a SynthNet is devised to estimate a residual occlusion map ∆O and a residual image R for better interpolation quality. The residual image helps to compensate for details and to eliminate artifacts.

[𝑈𝑈𝑖𝑖 , 𝐷𝐷𝑖𝑖 , 𝐶𝐶଴𝑖𝑖 , 𝐶𝐶ଵ𝑖𝑖 ]

Figure 5: Structure of SynthNet. The Residual Block (RB) consists of 2 convolutional layers and a skip connection.

Our SynthNet is a modified U-Net as shown in Figure 5. The down-sampling path consists of three ConvDown blocks each of which has two convolutional layers and the first one has stride 2. The up-sampling path consists of three SynthUp blocks that take the warped context features shared by the MRM as auxiliary inputs and fuse them with hidden features.

3.5 Training

Loss Functions

Our model is trained end-to-end with three loss terms: reconstruction loss Lrec, teacher reconstruction loss Ltea rec and distillation loss Ldistill. The overall loss L is:

Lrec is the L1 loss between the estimated intermediate frame ˆ It and the ground truth I gt t :

Following [Huang et al., 2022], we adopt a privileged teacher to better supervise the learning of motion estimation. Specifically, during the training process an auxiliary IFBlock [Huang et al., 2022] which is accessible to I gt t takes our estimated motion fields Mt-→0,Mt-→1 as preliminary motion fields and produces the revised ones M tea t-→0 , M tea t-→1 . The teacher reconstruction loss is defined as:

where O is the occlusion map produced by our model.

The distillation loss is defined as the weighted sum of L2 losses between M tea t-→0 , M tea t-→1 and all the in-process motion fields that our model produces during the motion generation and refinement process.

Datasets

We use the training set of the Vimeo90K dataset proposed by [Xue et al., 2019] as our training set. The training dataset contains 51,312 triplets with a resolution of 448 × 256.

4 Experiments

4.1 Implementation Details

Network Achitecture

The down-sampling levels of key sets when extracting DQBC and the radii (rl in Equation (4)) of local windows of each resolution level typically determine the preferred motion range that the network copes with. We use three levels of resolution. The radii for each level are 6, 5, and 4, respectively.

Training Details

Our network is optimized by AdamW [Loshchilov and Hutter, 2017] with weight decay 10-4 on 256×256 patches. The patches are randomly cropped from the training set and are randomly flipped, rotated and reversed. We use a total batch size of 64 distributed over 4 Tesla V100 GPUs for 510 epochs. The training process lasts for about three days. The memory consumption for each GPU is about 18 GB. The learning rate gradually decays from 2 × 10-4 to 2 × 10-6 using cosine annealing during the training process.

4.2 Evaluation Datasets

Vimeo90K. The evaluation set of Vimeo90K dataset [Xue et al., 2019] contains 3,782 triplets with resolution 448×256.

UCF101. The UCF101 dataset was originally proposed by [Soomro et al., 2012] and contains human action videos of resolution 256 × 256. [Liu et al., 2017] selected 379 triplets for VFI evaluation.

Middlebury. Proposed by [Baker et al., 2011], the Middlebury benchmark provides two sets: Other and Evaluation. We adopt the Other set which has ground truth intermediate frames and contains 12 triplets for VFI evaluation.

SNU-FILM. Proposed with [Choi et al., 2020], the SNUFILM benchmark contains four splits of different difficulty for VFI: Easy, Medium, Hard and Extreme. The difficulty is categorized by motion amplitude. Each split contains 310 triplets of resolution up to 1280 × 720.

4.3 Comparisons with the State-of-the-Art

We quantitatively compared our model with some other competitive state-of-the-art VFI models, including ToFlow [Xue et al., 2019], SepConv [Niklaus et al., 2017b], CyclicGen [Liu et al., 2019], DAIN [Bao et al., 2019], CAIN [Choi et al., 2020], AdaCoF [Lee et al., 2020], BMBC [Park et al., 2020], RIFE [Huang et al., 2022], ABME [Park et al., 2021] and VFIformer [Lu et al., 2022]. Results are presented in Table 1.

Besides the accuracy metrics (IE/PSNR/SSIM), we also provide Table 1 with numbers of model parameters, MACs and inference time for some methods. MACs are calculated on 256 × 256 patches using thop 1 . The inference time is recorded as the average inference time per sample on the Vimeo90K evaluation set and is tested on the same GTX1080-Ti GPU.

Considering that our base model ("Ours" in Table 1) runs much faster than VFIformer both theoretically (indicated by MACs) and practically (indicated by inference time), we use test augmentation to boost the performance at the price of more computation. We combine temporal reversal and 90◦ spatial rotation to augment the test frame pair to 4 groups and directly average the 4 outcomes to get the final interpolation result. The augmented version is denoted as "Ours-Aug" in Table 1.

Compared with ABME, our base model achieves +0.19 dB improvement and runs in less than 1/4 inference time on the Vimeo90K evaluation set. Compared with VFIformer, our augmented model achieves +0.07 dB improvement and runs in less than 1/3 inference time on the Vimeo90K evaluation set.

As illustrated by Figure 6, we visualized the motion fields estimated by VFIformer, ABME and our base model. It can

1 https://pypi.org/project/thop/

𝑀𝑀଴.5→଴

𝑀𝑀଴.5→ଵ

Figure 6: Visualizations of motion fields that are estimated by VFIformer, ABME and DQBC (Ours). Colors represent the directions of motion vectors in motion fields.

be seen that our model is more capable of coping with small and fast-moving objects (the waving arm and fist in this case). The waving arm and fist are totally missed in the motion estimation of VFIformer, leading to a poor interpolation result. ABME succeeds in capturing the motion of the waving arm but misses the fist which is smaller and moves faster.

We also present some visual comparisons of the interpolated results as shown in Figure 7. Both the quantitative comparisons and interpolation visualizations show that our approach is pretty competitive.

4.4 Ablation Study

For the ablation experiments below, we report the results on Vimeo90K evaluation set at 300 training epochs.

Ablations for DQBC

To verify the effectiveness of DQBC, we compared our model with the baseline model in which we adopt the bilateral motion estimation module of [Park et al., 2021] as the motion generation module. The motion refinement module and SynthNet stay the same. Basically, the baseline model adopts the paradigm illustrated by the top half of Figure 1. Table 2 shows that DQBC outperforms the baseline by a notable margin. Besides, it can be seen that the correlation enhancement block and the feature distributing operation also contribute to performance promotion. These results verified the effectiveness of the proposed DQBC.

Ablations for Motion Refinement Module

The Motion Refinement Module (MRM) refines motion fields by producing residuals and up-samples the motion fields by estimating convex combination weights. We compared our approach with the naive bilinear way, where we first bilinearly up-sample the coarse motion fields and then estimate residuals to refine the up-sampled motion fields. We also explored the effectiveness of the refinement function of MRM by removing the residual estimation from it. Table 3 shows that our approach is better than the naive bilinear approach. The "convex" approach only works well along with the residual estimation, indicating that it is necessary for MRM to have the ability to revise the motion fields rather than merely up-sample them.

5 Conclusion

We have proposed the Densely Queried Bilateral Correlation (DQBC) for better modeling correlations between two input frames in the VFI scenario. DQBC gets rid of the receptive field dependency problem in common approaches and thus is more friendly to small and fast-moving objects. A Motion Refinement Module (MRM) has also been devised to upsample the motion fields to full size and refine them with context information. We have verified the effectiveness of DQBC and MRM via ablation studies. Both quantitative experiments and interpolation visualizations show that our approach has reached and outperformed the state-of-the-art level on various VFI benchmarks.

Acknowledgements

This work is supported by the Fundamental Research Funds for the Central Universities, No.40020214380099.