Self-Supervised Vision for Climate Downscaling

1 Introduction

The increase in global temperatures is resulting in unprecedented changes in Earth's weather and climate patterns, characterized by an increase in the frequency of extreme weather events [Seneviratne et al., 2021] as well as other changes like melting ice caps, rising sea levels, lowering crop yields, and impacts on oceans and the fishing industry [H.-O. P¨ortner, 2022]. Th climate change impact has critical consequences for Earth's ecology and human societies. Future climate projections are largely driven by a suite of computer-based physics models that simulate the Earth's climate system, called ESMs. These physics models are fundamentally important tools for climate research and the design of future climate policies and strategies. High-resolution (HR) climate data offer detailed predictions at a regional level and can help plan local mitigation strategies. However, finer-scale models are constrained by the enormous computational resources required to run them. Hence, simulations are generally run at coarser scales, and HR data on a finer scale is obtained by downscaling. We limit the scope of this work to statistical downscaling.

∗ Corresponding author

Climate downscaling resembles the super-resolution (SR) task in computer vision. Supervised SR models assume that paired or unpaired low-resolution (LR) and HR data are available. Moreover, ground truth data are assumed to be accessible in sufficient quantity for the desired model fit and generalizability. These models further assume that the test data distribution is similar to training. Under these assumptions, several advanced machine learning-driven downscaling models have been proposed that are fully or partially supervised [Groenke et al., 2020; Vandal et al., 2017; Park et al., 2022].

However, access to ground truth data is not guaranteed in real-world climate research. In addition, the inherent stochastic nature of climate simulations makes it infeasible to obtain LR-HR data pairs. Minor changes in different simulation runs can lead to vastly different outcomes, a phenomenon also known as the butterfly effect [Palmer et al., 2014]. Time scale brings another challenge, as climate simulations can span thousands or even millions of years, making it computationally infeasible to generate sufficient HR simulation data. For instance, an HR simulation with the Community Earth System Model (CESM) uses 250K core hours per simulated year, generating approximately one terabyte of data per compute day [Small et al., 2014].

This research presents an interdisciplinary collaboration between AI and climate physics, introducing a deep learning based climate downscaling model that does not require HR ground truth data for training. Our model uses selfsupervised learning to train an instance-specific model on a single data point and adapts to its data distribution at runtime. We designed three components to reflect underlying weather variables and improve transferability. First is self-supervised pretraining, which enables knowledge transfer across the input data instances. Second is channel segregation, which enhances the learning representations of different weather variables that have vastly different characteristics (as shown in the precipitation and temperature in Figure 1. Third is topoclimatic attention, which forces the model to learn intricate inter-channel dependencies. The latter two components help extract improved features and understanding of physical relationship between climate variables.

The model is evaluated with multiple scaling factors by downscaling the surface temperature (TS) and total precipitation (PRECT) data obtained from CESM (v. 1.2.2) and utilizing the gradient of topography (dPHIS) as an additional helping variable. The evaluation results illustrate that our model achieves performance superiority ranging from 8% to 47% over existing baselines and greatly reduces the overall runtime computational complexity against that of self-supervised model baseline. In addition to quantitative performance gain, the qualitative analysis section demonstrates the enhanced downscaling quality of the model output. The model code and data is available at: https://github.com/k-s-b/climate sd

Without reliance on ground truth HR data, and reduced computational complexity, our downscaling method has great potential for climate downscaling research. Our work contributes towards the ongoing efforts to enable (1) "temporal downscaling" by running coarser models with higher temporal resolution, (2) larger ensemble runs which are necessary for understanding climate variability, (3) democratizing climate research and allowing organizations with limited access to computational power to run HR simulation models, (4) downscaling already conducted simulations and increasing the resolution without rerunning the model, and (5) promoting a greener climate with a lower energy budget.

As research plays a crucial role in understanding, predicting, and developing mitigation and adaptation strategies for climate change, this work contributes to several aspects of the United Nations Sustainable Development Goals (UN SDGs), with SDG 13 (climate action) in particular. By increasing downscaling quality and reducing computational complexity, our work enables a wider adoption of climate data, empowering informed decisions by policymakers and stakeholders. Our contributions to democratizing climate predictions also align with SDG 9 (industry, innovation, and infrastructure). Further, the interdisciplinary research presented in this work is conducted with an active collaboration of computer science (AI) and climate researchers, which resonates well with SDG 17 (partnerships for the goals).

2 RelatedWork

Single Image Super-Resolution. This task aims to enhance image resolution, typically using convolutional neural networks (CNN) and generative adversarial networks (GAN) [Dong et al., 2014; Ledig et al., 2017; Kim et al., 2016; Lim et al., 2017; Haris et al., 2018; Zhang et al., 2018c; Ledig et al., 2017; Zhang et al., 2018b; Lu et al., 2022]. These methods involve training CNN models or vision transformers to directly map the LR images to HR images. While most methods rely on supervised learning with predefined blur kernels, some tackle unpaired or blind SR, addressing the absence of direct LR-HR pairs [Wang et al., 2021; Ji et al., 2020; Cornill`ere et al., 2019; Zhang et al., 2018a; Assaf Shocher, 2018].

Deep Learning for Climate Downscaling. Deep learning has been widely explored for for climate downscaling by applying SR techniques to various climate features. Previous research utilized deep neural networks such as CNN [Vandal et al., 2017; Ducournau and Fablet, 2016; Kumar et al., 2021; Ba˜no-Medina et al., 2020; Vandal et al., 2019; Stengel et al., 2020; Misra et al., 2018]. Those works demonstrate the potential of deep learning methods over traditional approaches. Recent approaches point out that applying SR methods directly to climate data presents challenges due to the distinct nature of climate data compared to images. A branch of work has addressed this by incorporating climate-specific attributes or physics into deep learning models, yielding promising outcomes [Chou et al., 2021; Rampal et al., 2022].

3 Problem Setting

Problem: Given anLRclimate data instance 𝑋 andno availability of paired HR data, the goal is to obtain a HR version 𝑋↑ of 𝑋 such that visual quality of the downscaled data is improved in terms of the root mean squared error (RMSE), while minimizing the inference time.

Climate simulation data is represented in grids, where the height and width of the data represent the number of grid points over a spatial area. The larger the number of grid points for a given area, the higher the resolution of the data and the better the representation of regional climatic conditions. If a LR data instance for a given area has dimensions of R 𝐶×𝑊×𝐻 , we downscale it toR 𝐶×𝑊 ′ ×𝐻 ′ such that𝑊 ′ = 𝑠×𝑊 and ℎ ′ = 𝑠 × 𝐻, where 𝑠 is the scaling factor (𝑠 > 1), and 𝐶 denotes the number of variables in the data. TS and PRECT are critical variables that determine weather and climate conditions and impact climate adaptation and mitigation policies. These variables are depicted in Figure 1. Thus, we consider 𝐶 = 3 with temperature (TS), total precipitation (PRECT), and topography gradient (dPHIS) as the input weather variables. We only downscale TS and PRECT; dPHIS remains constant throughout the simulation years. Nonetheless, dPHIS carries valuable information to predict the two target variables; for example, temperature decreases with height. Moreover, using dPHIS instead of PHIS reduces opposing biases on the windward and leeward sides of mountains. Our approach is flexible, agnostic to the type and number of input weather variables, and can downscale multiple variables at once.

3.1 Data

The data utilized in this work is obtained from CESM 1.2.2 [Hurrell et al., 2013; Small et al., 2014], which logs the daily means of present-day climate conditions for 140 simulation years on a spatial resolution of approximately 0.25 ◦ in the atmosphere component and 0.10 ◦ in the ocean components [Chu et al., 2020]. The last 20 (corresponding to 7,300 data points) years of data are used for this work, which are considered from the periods that have received an equilibrium state.

4 Model

Our model is a self-supervised climate downscaling model that operates without any HR ground truth dataset. Figure 3 depicts the core architecture of the model.

4.1 Information Redundancy in Climate Data

Natural images have been shown to possess recurring patches of information across their original images and their coarser versions [Glasner et al., 2009]. The internal information of an image is known to have stronger predictive power than the information from external databases [Zontak and Irani, 2011]. Inspired by previous SRworks [Assaf Shocher, 2018], we analyze information redundancy in the raw climate data. Figure 2 presents the statistics of this analysis from our dataset. It can be seen that climate data have information redundancy for both the TS and PRECT channels across multiple scales of the input data. This forms the basis for a SR model that learns a generalized mapping from the downgraded LR data point to its higher resolution.

4.2 Training Process

Self-supervised training using pseudo data pair. The main research question we address in this work is downscaling LR climate data when HR data is not available. Similar to [Assaf Shocher, 2018], we generate the "pseudo" LR-HR pair by degrading the original HR data (𝑋 𝐻𝑅 ). We regard this degraded data as the LR climate data (𝑋) to be downscaled, and apply degradation to generate its downgraded version 𝑋↓𝑑 by a factor 𝑑 (𝑑 < 1). The data pair 𝑋↓𝑑 - 𝑋 now becomes a pseudo-LR (𝑋↓𝑑) and pseudo-HR (𝑋) pair and is used as training data of an instance-specific CNN model 𝑓𝜃 . The downscaled data 𝑋↑𝑠 is obtained by applying the trained model on 𝑋, i.e., 𝑋↑𝑠= 𝑓𝜃 (𝑋). Here, 𝑠 indicates the target downscaling factor and 𝑑 is the downgrading factor, with 𝑑=1/𝑠 typically.

Before passing as an input to the model, 𝑋 and 𝑋↓𝑑 are rescaled to the target downscaled size. The optimization objective of the model becomes minimizing element-wise loss such as mean squared error (MSE) or mean absolute error (MAE) between 𝑓𝜃 (𝑋 ↓𝑑) and 𝑋. The original HR data is only used for quantitative analysis of the final output and is never used for model training.

4.3 Self-supervised Pretraining

Training on just one data point poses two critical challenges: long convergence times and limited knowledge utilization from the rest of the data. Deep learning models trained on larger datasets for one task can be adapted to similar downstream tasks, known as transfer learning [Finn et al., 2017; Sun et al., 2019; Soh et al., 2020]. Inspired by this, we perform a self-supervised pretraining step where a single model learns a mapping for the entire downgraded input LR data (𝑋 ↓𝑑) and the input LR (𝑋) data itself. Exposing the model to the entire input data enables knowledge transfer across the dataset. Additionally, we finetune the pretrained model on randomly sampled data points augmented by uniform random noise, optimizing it for diverse scenarios and enhancing its generalization. This two-step approach combines pretraining with regularized data-specific adaptation, resulting in a faster, more knowledgeable, and ultimately more robust model. The effect of this component on performance and runtime is detailed in Section 5. The final downscaling task is realized by loading this pretrained model and optimizing it for a single data instance on the go (individual 𝑋↓𝑑 - 𝑋 pairs).

4.4 Channel Segregation

Figure 1 presents a snapshot of the TS and PRECT channels and highlights their characteristic differences. The TS channel has a smoother gradient and spans the entire spatial extent. In contrast, PRECT is highly non-linear, chaotic, localized and has a high degree of variance within localized regions. This characteristic contrasts with natural images, where different channels, albeit with different values, represent measurements of the same snapshot. The standard convolutional neural network (CNN) models operate under this assumption since a weighting filter convolves over the entire input volume to output a unified representation as another volume.

The model may be forced to simultaneously optimize for low-frequency TS features and high-frequency PRECT features at the same spatial location. This perplexity is unwanted and can lead to the model producing undesirable artifacts in downscaled output. However, CNN should adapt to a particular weather variable's low- and high-level characteristics and leverage them for the SR task. A mechanism is needed to handle each channel's vastly diverse properties effectively. We enable it by constraining the CNN model to learn separate weights for each input weather variable. Our model consists of 𝐶 different pipelines with several feature maps (evenly divided among the input channels) running in parallel, which are stacked at the last layer and optimized simultaneously. If 𝐶 ′ represents the number of channels in the intermediate feature map, then each channel is convolved by a filter of size 𝐶 ′ 𝐶 where 𝐶 ′ = 𝐼 ×𝐶 and 𝐼 is an integer. This operation can be represented as:

where 𝑁 is the batch size, 𝐶 the number of weather variables, 𝐶 ′ the number of output channels, and [·] represents the stacking operation. This architecture can learn features for each channel by reducing the undesired artifacts in the model output.

4.5 Topoclimatic Attention

Separating the filters for individual channels can improve model performance but has the drawback of missing out on inter-channel dependencies. This is particularly true for weather data, where intricate relationships between temperature, precipitation, and topography can exist. To account for such relations, we propose a topoclimatic attention mechanism that learns the inter-channel dependencies and enforces them by reweighting low, intermediate, and high-level feature maps in the main model pipeline. We define low to highlevel feature maps as the intermediate output of the model layers from the first through the last. The attention mechanism learns weights A𝑙 (𝑙 ∈ 𝐿, where 𝐿 are the model layers) that encompass these dependencies with the following steps.

Convolution over volume. If M ′ 𝑙 ∈ R 𝐶 ′ 𝑙 ×𝑊×𝐻 is the feature map at the layer 𝑙, then A𝑙 ∈ R 𝐶 ′ 𝑙 ×𝑊×𝐻 is the attention weight at layer 𝑙, computed by a convolution over the entire input volume. A separate set of weight filters are trained for computing 𝐴𝑙 . Spanning the entire volume layer-wise enables learning of dependencies between the weather variables at a given level 𝑙 of the feature map.

Generation of attention weights. At given level 𝑙 the attention weights 𝐴𝑙 can be represented as:

where Conv 𝑙 is a 2D covolutional operator at layer 𝑙, 𝜙 and 𝜎 are LeakyReLU and Sigmoid activation functions respectively. If 𝐴 ′ 𝑙 represents the raw output of the convolution operation at layer 𝑙 (i.e. without the Sigmoid activation) then 𝐴𝑙 (𝑙 > 1) is computed by passing 𝐴 ′ 𝑙-1 through Conv 𝑙 . At 𝑙 = 1, the 𝐴𝑙 is computed over the input. To enable this operation to learn complex non-linear correlations, the convolved output is transformed by the LeakyReLU non-linear activation function. Finally, the entire output is normalized in the range [0, 1] by passing it through the sigmoid activation function.

Feature map reweighting. The intermediate feature map M ′ 𝑙 is updated to M𝑙 by attention weight 𝐴𝑙 as:

where ⊙ is the Hadamard product, and M ′ 𝑙,M𝑙, A𝑙 ∈ R 𝐶 ′ 𝑙 ×𝑊×𝐻 . The attention mechanism reweights the feature maps to highlight the importance of inter-variable dependencies and the regions of the highest importance.

5 Experiments and Results

The model training and evaluation are performed with the dataset described in Section 3.1. The results are reported over 2x, 3x, and 4x factors, representing downscaling from resolutions of 50km, 75km, and 100km, respectively, to 25km. We use a scale factor of 2 when generating pseudo-LR and HR pairs in the pretraining phase. The model performance is measured by the RMSE between the downscaled model output and the ground truth HR data. Additional analyses are presented on the project's GitHub page.

Evaluation. The model evaluation is performed with the entire CESM dataset (7,300 data points of size 213×321×3), as the ground truth data is not exposed to the model at any point during the training. The results are reported over 2x, 3x, and 4x scaling factors, representing downscaling from resolutions of 50km, 75km, and 100km, respectively, to 25km. For all tables, the RMSE units are in 1e-08 for PRECT. Smaller values are better for RMSE.

5.1 Performance Evaluation

Table 1 shows the model performance over 2x, 3x, and 4x downscaling factors for the entire dataset of 7,300 data points.

Linear and bicubic interpolation are commonly used baseline methods in climate downscaling research. We also compare the model performance against that of strong baselines such as self-supervised pretraining (PT), which is a generalized SRResNet model without any finetuning with all (𝑋 ↓𝑑, 𝑋) pairs. Additionally, we evaluate the model against a more sophisticated baseline (see 'PT+FT' in the table) where the pretrained model is finetuned with individual data points but does not contain channel segregation and topoclimatic attention components.

For the 2x scaling factor, our model outperforms all baselines, with a 47.50% (TS) and 31.86% (PRECT) reduction in the RMSE over that of bicubic interpolation and a reduction of 26.52% (TS) and 8.32% (PRECT) over PT. The overall runtime is reduced greatly, as discussed in Section 5.2. Except for the 2x scaling factor of PRECT, the full model consistently outperforms the baselines for the 3x and 4x scaling factors.

We finally compare our model performance against GINE [Park et al., 2022], a supervised model for climate downscaling. We train this model in a supervised fashion by using 5% and 10% of the HR ground truth data (𝑋-𝑋 𝐻𝑅 pairs). As this model uses HR data for training, we compare its performance against 1,000 randomly chosen data points from the held out set. The results in Table 2 illustrate that our model outperforms the baseline, and even GINE model trained with some HR ground truth data, although it performs better for PRECT when using 10% labels. Base refers to a model such as [Assaf Shocher, 2018], with architecture as in Fig. 3, but without the proposed components. Given the limited compute and runtime constraints, we do not include vision transformer based baselines, although they should be explored for the future versions of this work.

The TS and PRECT channels do not show the same patterns of performance improvement because the two climate variables have distinct dynamics (as shown in Figure 1). The improvement in one channel results in a minor performance trade-off in the other channel.

5.2 Runtime

As climate simulations usually cover many time steps, the downscaling method should have a low overall runtime for practical use. We assess the overall computation times for a model trained on a single data instance from scratch (Base), a pretrained and finetuned model without channel segregation and topoclimatic attention components (PT+FT), and a full model with these components (Ours). The base model requires thousands of epochs for convergence. Due to the computational bottleneck induced by the base model, this particular assessment is only performed with randomly selected (seeded) 1,000 data points (only the analyses presented in Tables 2, 3, and 5 is performed over (the same set of) randomly chosen 1,000 datapoints, whereas other results are presented for the full dataset). A single NVIDIA Tesla A100 GPU is used for experiments. The results demonstrate that our model substantially reduces overall runtime while reducing the RMSE values.

5.3 Ablation Study

We remove each of the proposed components, i.e., channel segregation, topoclimatic attention, and finetuning on augmented data in the pretraining and finetuning on individual data points (represented by (-) FT), and measure the model performance for all data points. Table 4 confirms that every model component is essential for performance. Removing channel segregation results in some performance gain for PRECT over the full model. At the same time, it also causes a performance drop in the TS channel. This implies that after an initial reduction in the RMSE, the TS and PRECT performance exhibit orthogonality to some extent. Nonetheless, the performance is better than the most commonly used baselines.

The proposed components improve the quality of the downscaled output substantially. The zoomed-in regions in the Figure 4, we demonstrate that the channel segregation component effectively eliminates the undesirable artifacts present in the unsegregated model, resulting in output that closely resembles the ground truth.

5.4 Model Robustness

Non-ideal blur kernel. We test themodel's ability to downscale climate data when the blur kernel is non-ideal (e.g., not bicubic). To realize this kernel, we inject freshly generated uniform random noise into 𝑋 and 𝑋↓𝑑, independently for each data point. A noisy setting implies that the relationship between 𝑋 and 𝑋↓𝑑 can be different from 𝑋↓𝑑 and 𝑋 𝐻𝑅 . This step is also included in the pretraining and instance-wise finetuning phases. The results are shown in Table 5. Our model is robust against non-ideal blur settings and still outperforms the baselines by a substantial margin.

5.5 Supervised Learning

As most existing downscaling models are supervised, we extend our investigation to ascertain whether the integration of channel segregation and topoclimatic attention components can improve performance in scenarios where supervised climate downscaling is feasible. We present preliminary results for this analysis on the CMIP6-ERA5 (LR-HR, respectively) data pairs. The LR data in these experiments is not a degraded version of the HR data but a separate dataset that emulates the real data. While CMIP6 is a collection of various coupled climate simulation models, ERA5 is the reanalysis data, that combines observational data and numerical predictions to create a continuous record of the past weather. For these datasets as well, we downscale TS and PRECT, while HR topography (PHIS) is the third variable that aids the model learning.

To reduce computational complexity, we regrid ERA5 data with CDO tool to 1 ◦ from the original resolution of 0.25 ◦ . Similarly, CMIP6 data is regridded from 1 ◦ to 3.75 ◦ . Thus, the LR CMIP6 data is downscaled with a factor of 3.75x (48x96→128x256). Both datasets are reconciled for spatial grid and temporal scopes, and the final dataset covers daily averages from the year 1979 to 2014. Moreover, to limit the extraneous perplexity, we also clip the polar regions from the data and utilize data in the region of 70 ◦ North and South of the equator. We use SRResNet as the baseline and the backbone network for all experiments [Ledig et al., 2017]. The model was trained on 𝑋 - 𝑋 𝐻𝑅 pairs, where 𝑋 ∈ 𝐶𝑀𝐼𝑃6 and 𝑋 𝐻𝑅 ∈ 𝐸𝑅𝐴5. The data is split into train, validation, and test data as 7:1:2. The model was trained for 50 epochs, and the model checkpoint with minimum validation error was chosen for evaluation. Table 6 shows that using channel segregation and topoclimatic attention improves performance in both the channels. This preliminary analysis highlights that the proposed components are effective in general for deep learning based climate downscaling models.

6 Discussion

We now compare our model with the bicubic interpolation baseline (standard approach in climate downscaling) and discuss its strengths and weaknesses in different domains within the region of interest. The metrics used for this analysis are the RMSE and mean bias (Figure 6). We look at the error distribution within domains such as oceans, land, and coastal areas (Figure 5). Analyses are performed for 2x scaling factor.

The two variables under consideration show considerable differences in their spatiotemporal characteristics. As the noisier variable, the PRECT has a larger error distribution spread than TS. Our model's error distribution (purple plots) shows improvement in all error metrics. The values corresponding to the 5 𝑡ℎ , 25 𝑡ℎ , 75 𝑡ℎ , and 95 𝑡ℎ percentiles are lower in our model than in the bicubic interpolation, extending the confidence in our model to perform better downscaling.

Even the maximum and minimum (above and below whiskers, 95 𝑡ℎ and 5 𝑡ℎ percentile) values of errors are smaller in our model (Figure 5 a,b). We emphasize that our inferences are valid for the entire region and separately in the different domains (ocean, land, and coast) within it. Both the bicubic interpolation baseline and our model demonstrate lower performance in TS over the coastal region (Figure 5a). PRECT also shows similar behavior, albeit with smaller differences between different domains (Figure 5b).

Figure 6 shows the spatial distribution of RMSE and mean bias in North America. We find that TS exhibits relatively large RMSE values over the mountainous regions. The errors are also high along the warm western boundary current along the east coast (Figure 6a). The former might stem from the model's inability to resolve mountainous topography well, while the latter is due to the unresolved small-scale eddies formed along the boundaries of the current system. Our model underestimates the TS over the Pacific and Atlantic Oceans (Figure 6b). The RMSE spatial distribution for PRECT indicates higher values at lower latitudes and along the continent's eastern coast (see Figure 6c). The errors are introduced on either side of the elevated terrain (Figure 6d). Therefore, error hot spots include coastlines, mountainous terrains, and eddy-rich oceanic regions. The above suggests that our model consistently outperforms baseline interpolation across all standard error distribution metrics.

7 Conclusion

This work proposed a deep learning-driven climate downscaling model that does not require HR ground truth data for training. Our model incorporates three components into the standard CNN model to make it adaptable to climate data: self-supervised pretraining that enables knowledge transfer from the input data and improves runtimes, channel segregation that enables learning better representations for individual weather variables, and topoclimatic attention that enforces the learning of the intricate relationship between the weather variables and topography.

Our work opens up possibilities such as effective downscaling of historical data, running larger climate ensembles and democratizing climate research with a reduced need for computational power. In the future, we would like to better understand trade-offs between weather variables of different natures, incorporating temporal dimension, extending the analysis in the supervised setting and newer methodologies like vision transformers, diffusion models, and improving the explainability of the deep learning downscaling model.

Acknowledgments

The authors would like to thank Axel Timmermann for his insightful feedback and Sun-Seon Lee for sharing the Highresolution CESM Climate simulation data. The work was supported by the Institute for Basic Sciences (IBS), South Korea, under IBS-R029-C2 (Singh, Shidqi, Park, and Cha) and IBS-R028-D1 (Nellikkattil and Zeller). This work was also partly supported by the National Research Foundation of Korea grant RS-2022-00165347 (Shidqi, Park, and Cha).