Dual Enhancement in ODI Super-Resolution: Adapting Convolution and Upsampling to Projection Distortion

1 Introduction

Omnidirectional images (ODIs), as a novel visual form of virtual reality (VR) [Zhang et al., 2023], are anticipated to become integral components of future social environments, enriching and enhancing the ways in which we share experiences. Consequently, ODIs are being introduced rapidly into people's lives. As their name suggests, ODIs offer a 360◦ × 180◦ field of view, and the expansive visual range naturally demands high resolution. The level of detail and sharpness in ODIs directly influences the realism of user experiences. However, due to storage and transmission constraints, omnidirectional images often lack the necessary resolution to provide users with a truly immersive experience. Super-resolution (SR) technology transforms lowresolution (LR) images into high-resolution (HR) ones. As a long-standing computer vision problem, single-image superresolution (SISR) has attracted extensive attention from researchers [Anwar et al., 2021]. In recent years, several SISR studies based on convolutional neural networks (CNN) have achieved remarkable results [Senmao et al., 2023; Wang et al., 2021]. These studies utilize CNNs to extract features from images and subsequently employ the extracted features for upsampling. CNNs use convolution kernels to sense images through sliding windows, allowing them to capture local patterns. However, these works primarily target twodimensional (2D) planar images, neglecting the global distortion information present in omnidirectional images (ODIs). For ODIs, equirectangular projection (ERP) from sphere to plane is widely employed for convenient storage and transportation. Regrettably, the distortion caused by ERP results in unevenly distributed features within ODIs. Fig. 1 illustrates the correlation between distortion degree and pixel position in ODI projection. Notably, stretch distortion is particularly severe in non-horizontal regions, often leading to unsatisfactory feature extraction results for ODIs. This phenomenon raises the question: How can we achieve super-resolution in the presence of distortion and improve CNNs to effectively extract features fromODIs?

∗Corresponding author.

For ODI super-resolution (SR), traditional CNN-based feature extraction strategies have proven ill-suited for ODIs in practice. The issue arises from the fact that both existing CNN filters and image data are "flat", making it challenging for these filters to achieve the right balance with ODIs that exhibit varying pixel densities and texture complexities. Recently, [Deng et al., 2021] proposed a multilevel CNN network that allocates additional computational resources to the central regions of ODIs. Their approach modified existing SR networks but did not fundamentally enhance the sensitivity of CNNs to ODI distortion. Conversely, most existing ODI SR research works [Ozcinar et al., 2019; Su and Grauman, 2017; Sun et al., 2021] tend to focus on the local resolution of ODIs. To mitigate the effects of distortion, these methods process tangent planes surrounding the sphere repeatedly. While this boosts accuracy, it also significantly heightens computational costs. These methods are heavily dependent on hardware performance, which poses a considerable burden for portable devices like head-mounted displays and cell phones.

In an effort to strike a balance between accuracy and cost, numerous CNN-based SR methods [Dai et al., 2019; Liu et al., 2020; Mei et al., 2020; Zhang et al., 2021] have introduced attention mechanisms. [Dai et al., 2019] proposed a deep second-order attention network to capture spatial context information over long distances. [Liu et al., 2020] incorporated spatial attention into the residual network to enhance model efficiency. Drawing on recent neuroscience research, [Zhang et al., 2021] proposed CRAN, which utilizes semantic knowledge as a factor influencing network attention. However, these attention mechanisms appear to be ineffective in addressing the distortion caused by projection. This is primarily because these methods focus on the comprehensive utilization of global information without considering distortion shifts. These challenges limit the representational power of CNNs, and as a result, the development of an optimal single ODI SR network remains an ongoing endeavor.

Motivated by the observations and analyses discussed above, we propose a novel method named Omnidirectional Distortion Attention Super-Resolution Networks (ODASRN). Contrary to existing strategies, the core idea of our ODA-SRN is the dual enhancement of two fundamental components in SR, convolution and upsampling, rather than designing intricately structured integrated network models. Our innovation resides in considering the global and local differences of ERP during the convolution process, and introducing pixel-level offsets in the upsampling phase to accommodate distortion. This focus on enhancing fundamental components, rather than simply stacking larger models, enables our ODA-SRN to efficiently conserve computational and memory resources.

Our main contributions can be summarized as follows:

• We have introduced novel convolution and upsampling techniques for ODIs. These components have been integrated into our ODA-SRN, achieving superior SR re-

sults.

• We propose extracting the potential representation of distortion information to obtain a set of routing weights. By leveraging the relationship between routing weights and segment positions, we enhance the distortion sensitivity of CNN.
• We introduce a Jacobian matrix of spatially transformed elements and mathematically adjust the sampling window of the upsampling layer. The reusable Jacobian matrix improves the efficiency of our ODA-SRN.

2 RelatedWork

Deep CNN for Super Resolution. Over the past decade, SR has been one of the most extensively studied inverse problems [Chen et al., 2022]. The combination of SR and CNN technology has led to significant progress in reconstruction error and practical applications [Chen et al., 2021; Ma et al., 2020; Senmao et al., 2023]. Since [Dong et al., 2014] first proposed SRCNN, numerous SR works based on deep CNN have been introduced. These works have made improvements in various aspects, such as increasing network depth to utilize more parameters, incorporating residual attention block mechanisms to enhance performance accuracy, and modifying the loss function [Li et al., 2022]. All of these works have achieved remarkable results in SR for 2D images. However, a common characteristic of CNN-based SISR methods is that their "flat" convolutional layers are inefficient for ODIs in feature extraction. Specifically, due to the uneven pixel density and varying texture complexity of ODIs, it is challenging to optimize a convolutional kernel that can account for the effects of all degrees of distortion.

Super Resolution for VR. With the rise of virtual reality (VR) technology, omnidirectional super-resolution has emerged as a challenging task for the research community. In recent years, viewport-based methods have been proposed as a promising approach for enhancing large-sized ODIs [Li et al., 2020]. [Su and Kristen, 2017] utilized different convolution kernels to extract features along various latitudes on ODIs; however, this approach required training a large number of convolution kernels. Inspired by the success of the Transformer architecture, [Shen et al., 2022] proposed PanoFormer, a deep neural network capable of estimating depth in ODIs. SphereSR [Yoon et al., 2022] incorporated RGB prediction into feature extraction and learned the relationships between different ODI formats to achieve more flexible SR. However, using Transformers for ERP-format ODIs poses challenges: their global attention may overlook local nuances critical for SR, and their computational demands can be burdensome given the lage-size nature of ODIs.

Techniques for Projection Distortion. Projection distortion in ODIs leads to distinct regions showing varied distortion degrees. In the realm of efficient adaptability, parameterized convolutions have emerged as a promising technique [Brandon Yang, 2020]. Unlike traditional convolutional layers that utilize static filters, parameterized convolutions change their filters based on the input. However, current attention mechanisms are not guided by distortion information, which limits their ability to address the regional differences in ODIs caused by projection.

3 ProposedMethod

Overview. We designed a concise network architecture, illustrated in Fig. 2. It demonstrates how a low-resolution ODI input (ILR ∈ Rc×h×w ) generates a high-resolution ODI (ISR ∈ Rc×sh×sw ), where s is a multiple of the superresolution, c represents the number of channels, and h and w denote the ODI size.

3.1 Network Architecture

ODA-SRN mainly consists of two parts: the feature extraction part and the upsampling recovery part. First, shallow features F0 are extracted from the input LR ODI:

where fconv represents a 3 × 3 convolution operation, and F0 is used as input to DABGs (Distortion Attention Block Groups). Furthermore, each DABG contains several DAB modules. We adopted a residual-in-residual (RIR) structure as the backbone of our network. In this way, we aim to superimpose low-frequency and high-frequency feature information, producing an SR image with more texture details. The i-th DAB group is represented as follows:

The operation ⊕ can be expressed element-wise, where ∗ represents dot multiplication, and WSSC is the weight associated with the DAB group output. f kDAB represents the k-th distortion attention block operation. Distortion information Md contains all ODI distortion information Dx,y. The mathematical representation of the ERP distortion information is as follows:

H represents the height of the ODI, and we denote 1-Dx,y as the distortion degree of pixel (x, y). Note that the higher the Dx,y value, the less the distortion. Eq. (3) is derived from the JVET standard D0040 [Sun et al., 2016], which defines evaluation metrics for omnidirectional images. After the data stream passes through the DABGs, the deep information of the ODI is extracted and provided to the subsequent module for upsampling and reconstruction.

Eq. (4) shows the upsampling and reconstruction part. Similar to Eq. (2), WLSC represents the weight associated with the deep feature Fn. Eq. (4) indicates how, after fusion, the features are sent to the distortion offset upsampling layer fUP . Finally, through the reconstruction process fREC, the upsampled features are converted to SR ODI, which can be upscaled by s × (s is an integer, e.g., 2, 3, . . .).

3.2 Enhanced Distortion Attention Block (DAB) Methodology

The main challenge of feature extraction is the distortion caused by ERP. Previous CNN-based feature extraction methods treat the features of all LR regions equally and cannot sense the distortions in a targeted way. As shown in Fig. 3, we designed a Distortion Attention Block (DAB) mixed with general convolution layers and a distortion attention layer to learn features in a "total-local" manner. The DA layer is responsible for piecewise learning according to the distortion degree. Meanwhile, the general convolutional layer appears alternately to supplement the global association information.

Distortion Attention layer. We attempt to incorporate the global distortion information into the convolution process. As shown in Fig. 3, the input features and distortion degree & scale information are fed into the Distortion Mask (DM) module. DM cuts the feature maps into k segments (Section 4 provides a more comprehensive analysis and guidance on the choice of this parameter). The calculation of the segmentation position p follows the equal distortion principle, which means that the difference between the maximum and the minimum distortion degree of each segment are equal. After segmenting transversely k - 1 times, we represent the set of segments as {G1,G2, . . . ,Gk}. As depicted in Fig. 3, this set of segments is introduced to a network to generate a two-dimensional routing weights matrix. The learning process of the routing weights matrix W ∈ Rk×d from our network is defined as:

where σ can be a Softmax function, BN signifies batch normalization, ReLU represents the Rectified Linear Unit activation function, and Pool indicates pooling. For each segment Gi, we extract a weight vector wi from the routing matrix W. Each element wij in wi quantifies the influence of the j-th expert (a conditionally applied convolutional kernel) on the i-th segment. To normalize weights, we employ a softmax operation over each expert pertaining to segment Gi:

Given this weight normalization, we proceed to combine these weights with the routing matrix W through a Hadamard product. This results in a tailored distortion-aware filter Pi. Through the procedures delineated above, we've refined traditional convolution layers. A standout feature of the DA Layer is its adaptability: it can adjust its convolutional filters based on the nuances of the incoming feature maps. Importantly, it ensures efficiency, circumventing the computational overhead associated with training a larger number of filters thus marrying efficiency with model precision.

A comparative analysis was conducted against the feature extraction attention block incorporated in RCAN [Zhang et al., 2018]. As shown in Fig. 5, with the deepening of the network, the traditional feature extraction block has poor performance for ODIs affected by distortion, and the features in the severely distorted regions appear to fade (Feature vanishing in the polar region.). We found that traditional feature extraction blocks cannot find a balance between regions affected by different distortion degrees. In contrast, our DABs can still efficiently extract features from different regions affected by distortion at the same time in deep networks.

Distortion attention Block (ours)

Figure 5: The feature maps output of block 3, 6 and 9. Comparison of Distortion Attention Block and baseline. (Zoom-in for best view)

3.3 Distortion Offset Upsampling (DOU) Layer

The upsampling layer is a pivotal component, leveraging deep information to reconstruct larger-sized feature maps. For ODIs, the increased sampling density at the sphere's poles engenders information redundancy in these polar regions. We conceptualize the SR process for ODIs as a three-tiered progression: from the sphere to a low-definition plane, and to a high-definition plane, as illustrated in Fig. 4(a). Our analysis seeks to determine the tile-to-pixel area ratio before and after projection, termed as the zoom factor δ. This metric is instrumental in gauging the influence of planar pixels on spherical visual perceptions. With ERP mapping a point (X, Y, Z) on the sphere to (xLR, yLR) on the plane, we delve into its reverse process as delineated below:

where (u, v) represents the normalized (xLR, yLR), the derivation of Eq. (5) is omitted for brevity. Based on linear algebra, we calculate the Jacobian J(u, v) of the space projection using the following approach:

The Jacobian J(u, v) can be used to obtain the mapping of any 2D vector in 3D space. We use (u, v) and (∆u,∆v) to represent any pixel in the plane. Through Eq. (6), we can obtain the four vectors corresponding to the tile in 3D space. We approximate the tile area as the sum of two triangle areas, as shown in Fig. 4(a). We use the cross product of two vectors to compute the areas of A1 and A2. It should be noted that the zoom factor δu,v and the Jacobian are reusable, making this approach computationally efficient and cost-friendly. In the process of converting low-resolution pixels to high-resolution pixels, we calculate the relative distance between LR pixels and HR pixels. The value of the HR pixel (xHR, yHR) is unknown, but we can obtain the position of the LR pixel (xLR, yLR) corresponding to the HR pixel. Further, we can determine their correlation distances R(xHR) and R(yHR). As shown in Fig. 4(a), where ⌊⌋ indicates integer rounding down. R(yHR) are calculated in the same way. The correlation distance represents the bias relationship between HR pixels and LR pixels. Crucially, excessive reconstruction of redundant information could affect the ODI SR process. To overcome this, we propose DOU for correcting the sampling window. Pixels near the polar regions tend to perceive information near the center of the ODIs, which helps better understand the surrounding valuable information. Specifically, we add an offset θoffset = γ/δu,v to the sampling window, where γ is a coefficient. The purpose of this design is to add more sampling bias in areas with smaller zoom factors. As shown in Fig. 4(b), a convolution layer is used to generate output features of (C × sH × sW) size. R(xHR), R(yHR), and δu,v are fed into a bottleneck network structure consisting of two FC layers to obtain the weights, which are combined with the experts to generate an adaptive filter of size (C × k × k). Finally, the output is a (C × 1 × 1) size feature. This step is repeated xHR×yHR times during the upsampling of an ODI.

Loss function. For ODA-SRN, we design a loss function similar to the L1 norm:

where σ represents the weight parameters of our network. The training goal is to minimize L(σ). ISR and IHR stand for the generated high-resolution ODI and the original highresolution ODI, respectively. δ is the zoom factor, applied to each pixel, which is used to guide the backpropagation under the influence of distortion.

4 Experiment

4.1 Datasets andMetrics

Dataset. ODI SR is currently underexplored, and few works focus on this new challenge. Consequently, there are only a few datasets available for evaluation. We selected 2600 high-quality ODIs from the 360 Panorama Database [Xiao et al., 2012] and ODI-SR Database [Deng et al., 2021]. All selected ODIs are in ERP format and have a resolution of 2048×1024. To enhance the robustness of the dataset, the selected ODIs include indoor scenes, outdoor scenes, natural scenes, and cityscapes. After collecting the HR ODI dataset, we generated LR ODIs by processing HR ODIs with bicubic downsampling. The LR-HR pairs were randomly divided into 2000 for training, 300 for testing, and 300 for validation.

Metrics. To address both the plane and the sphere, we used four metrics to evaluate SR performance. In our experiments, on one hand, the widely used metrics PSNR and SSIM were employed to evaluate the performance of solutions in viewport format. On the other hand, the Weighted to Spherically Uniform PSNR (WS-PSNR) and Weighted to Spherically Uniform SSIM (WS-SSIM) [Sun et al., 2016] were used as metrics specifically designed to evaluate the performance of methods in ERP format.

4.2 Implementation Details

For the model setup, the final version of ODA-SRN consists of 5 DABGs, with each DABG containing 2 DABs. The number of experts in the DA layer is set to 3, Coefficent γ is set to 0.1. Our model was trained using the ADAM optimizer [Kingma and Ba, 2015] with β1 = 0.9, β2 = 0.999, and ε = 10-8 . The learning rate was set to 10-4 , decaying by half every 50 epochs. The weight initialization of the model follows [He et al., 2015]. We used PyTorch [Paszke et al., 2017] to implement our model. During training, random crops were not used to preprocess ODIs in order to retain complete distortion information.

4.3 Comparison with State-of-the-Art Methods

In our experiments, we compared our ODA-SRN with other SOTA methods. To ensure consistent conditions across various models, each was deployed within its dedicated Docker container. This methodology was adopted to mitigate the inconsistencies stemming from divergent code implementations.

Quantitative Analysis. Table 1 shows the ODA-SRN consistently outperforms its contemporaries in a wide spectrum of evaluations. This overarching superiority of our proposed method becomes particularly pronounced when diving deeper into the results of the viewport format, a pivotal metric that simulates what a user would witness through their HeadMounted Display (HMD). Such an unequivocal lead testifies to the meticulous engineering of the ODA-SRN, tailored to replicate and enhance the user's visual experience. An underpinning idea of our model is the segmental processing of the ODI. By partitioning the ODI regionally, each segment is processed through individualized Distortion-aware filter. This strategy addresses a perennial challenge: mediating the inherent tug-of-war between varying distortion regions during the feature extraction phase. By localizing the processing, the ODA-SRN can give specialized attention to each region's idiosyncratic distortions. The results, as illuminated in the Table 1, attest to the efficacy of this approach.

Qualitative Analysis. Fig. 6 visually presents the results of the ×4 super-resolution performances by various algorithms. Distinct regions from the image, central and polar, are highlighted using red and green boxes for clearer examination. On the other hand, methods such as EDSR, 360-SS, and OSRT, despite their sharp outputs, occasionally exhibit discrepancies in their outline representations. On the left of Fig. 7, the transformation of an image before and after ERP projection is showcased. Through VR equipment, users experience a spherical field of view. However, when this image undergoes ERP projection onto a two-dimensional plane, perceptual distortions arise. For instance, the skyscrapers that span across both the central and polar regions of the image exhibit window outlines that, while straight in the original threedimensional context, become curved upon projection.

The right portion of Fig. 7 offers a comparative analysis of the results generated by various algorithms. Traditional

2D-oriented methods, like EDSR and RCAN, tend to restore features by treating the ERP-projected image as a regular 2D grid. This method, however, fails to accommodate the curvatures introduced by the ERP projection. As a result, when their outputs are translated back to a spherical perspective, the previously straight window contours appear misshapen and curved. While methods explicitly designed for ODIs, such as 360SS and LAU-Net+, make commendable efforts to tackle the anomalies caused by ERP projection, they still face challenges, particularly in extreme polar regions. These areas, depicted by the side of the blue building and the apex of the white structure in Fig. 7, are marked by results that are overly smoothed and lacking in contour fidelity. This limitation stems from the inherent sparsity of feature information in these polar regions, making the precise restoration of details a formidable task. ODA-SRN is designed to address the scarcity of features in the polar regions, the distortion-offset upsampling layer of ODA-SRN adopts a unique approach. By leveraging a receptive field offset mechanism, it pulls richer feature details from more central areas. This allows ODASRN to supplement the sparse features in the extremities and bring a marked improvement in restoring details. The result is a more accurate representation of even subtle elements, such as the high-rise window outlines, when they are projected back to the spherical viewpoint.

Anti-distortion Analysis. ERP projection significantly stretches polar regions in images, where inaccuracies can magnify during spherical conversion. We illustrate this with a comparison against various SR algorithms. Fig.8 shows that vents in ERP images are notably stretched. Our method effectively reduces this distortion, with ODA-SRN outperforming other tested methods in anti-distortion capability.

Model Efficiency Analysis. We have included Table 2. This table provides a comprehensive performance comparison of our method with other algorithms, covering various aspects such as FLOPs, WS-PSNR, network parameters, and runtime. By presenting this multidimensional comparison, we aim to offer a thorough evaluation of the effectiveness and efficiency of our approach compared to existing methods. ODA-SRN strikes a balance between performance and efficiency, delivering the highest WS-PSNR using fewer parameters, reduced FLOPs, and enhanced speed.

4.4 Ablation Study

Ablation Study on Weighted Loss Function. This study examines the zoom factor δ in loss calculation, which reflects a pixel's projection scale from planar to spherical views. Table 3 demonstrates the superiority of the δ-weighted loss function over conventional L1 loss in SR.

Ablation Study on Distortion Attention Layer (DA). In the absence of the DA layer. Considering the unique patterns in ODIs in ERP format, Table 3 shows that integrating the DA layer significantly enhances WS-PSNR and WS-SSIM. Our DA layer, focusing on specific ODI features, improves structural similarity in SR outcomes. This is due to ODASRN's adaptive feature extraction for different ODI regions.

Ablation Study on Distortion Offset Upsampling (DOU). Data in Table 3 emphasizes the effectiveness of DOU in ODI super-resolution. Standard upsampling doesn't adequately address ODI specifics, leading to the creation of the DOU layer. This layer computes θoffset to amend inherent distortions, allowing polar-proximate pixels to utilize central ODI information and enhance output quality. Compared to the pixel shuffle baseline, DOU consistently achieves superior WS-PSNR and WS-SSIM results.

Effect of Segments Number. We propose DAB for extracting features of ODI. Unlike typical CNNs, DAB adopts a segmented strategy to handle regions with varying degrees of distortion. We attempted to change the number of segments in DAB to observe the impact of different segment numbers on network performance. Table 4 shows the experimental results. With the increase in the number of segments, the performance of the network improves. However, the improvement in FLOPs is linear, which means that more segments do not always yield better results. We need to choose an appropriate number of segments to achieve a balance between performance and cost. When the number of segments increases to more than 5, the model becomes difficult to converge and performance decreases. Therefore, considering multiple factors, we finally chose 5 as the number of segments in ODA-SRN.

5 Conclusion

We addressed the SR challenges of ODIs. Our solution was not rooted in proposing a completely new network. Rather, our main contribution centers on the innovative redesign of two plug-and-play modules: the Distortion Attention Block and the Distortion Offset Upsampling Layer. These modules, crafted to counter the unique challenges posed by ODIs, not only enhance super-resolution performance but also offer the flexibility of being integrated into various existing network architectures. Comprehensive experiments highlight that our modular approach excels in metrics such as PSNR, SSIM, WS-PSNR, and WS-SSIM, and also stands out in terms of model efficiency.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (NSFC) under grant No. 62225105,

62394323 and 62301070. Han Xiao wishes to acknowledge support from the Postdoctoral Science Foundation of China under grant No. 2022M720518. G.-M. Muntean wishes to acknowledge the Science Foundation Ireland's support via grant nos. 21/FFP-P/10244 (FRADIS) and 12/RC/2289 P2 (Insight).