Weather Foundation Model Enhanced Decentralized Photovoltaic Power Forecasting Through Spatio-temporal Knowledge Distillation

1 Introduction

According to [Masson et al., 2024], over 1.6 TW of PV systems were operational globally by 2024, producing 2,136 TWh of electricity, which accounts for 8.3% of global electricity demand. These PV systems reduced 0.92 gigatons of CO2 emissions, equivalent to 2.5% of global energy-related emissions. Accounting for more than 40% of the PV systems, the decentralized PV systems play an important role in global renewable energy infrastructure.

A decentralized PV system refers to a solar power setup where electricity generation is distributed across multiple locations rather than being concentrated in a single, large-scale facility [Zisos et al., 2024]. Decentralized PV systems produce electricity near where it is consumed, reducing the need for long-distance transmission and minimizing energy losses associated with transporting electricity [Kim and Bae, 2017]. SPPF is one of the most fundamental applications for decentralized PV systems since SPPF performs as a prepositive task for a number of PV-oriented applications (e.g., electric grid management, solar intermittency migration, power trading, etc.) [Mansoor et al., 2023]. SPPF involves predicting the power generation of PV systems on various time horizons [Antonanzas et al., 2016]. Traditionally, elaborating individual SPPF models for each PV is required in decentralized PV systems, which is labor-intensive for SPPF developers and thus not scalable [Gaboitaolelwe et al., 2023]. A general SPPF model is imperative for a decentralized PV system.

As highly related to weather, existing studies of SPPF place an emphasis on addressing the challenge of analyzing the impact of weather variation on PV power generation [Ahmed et al., 2019]. Traditional SPPF approaches obtain the weather information from the numerical weather prediction (NWP) models, which are a set of mathematical models of the atmosphere and oceans that predict the weather based on current weather conditions [Travieso-Gonz´alez et al., 2024]. However, using NWP models faces two limitations: 1) the requirements for extremely high computational resources, which can limit the accessibility and affordability of NWP models for SPPF developers, and 2) the errors and uncertainties of NWP models can propagate and amplify in the forecast, affecting the accuracy and reliability of SPPF. Recently, weather foundation models (WFMs) show potentials to break these bottlenecks since WFMs can provide highly accurate global weather forecasts up to 14 day ahead with less computation resource [Lam et al., 2023].

Developed along with advanced deep learning techniques and enormous global climate datasets, the WFMs increasingly raise attentions and may be considerable for SPPF tasks [Hamann et al., 2024]. Figure 1 shows a real-world decentralized PV system in California, USA. The weather variation changes significantly across this region, where the distance between the two PVs furthest apart is 930km. The WFMs generally operate at 0.25° resolution (28km x 28km at the equator) and thus are suitable for providing regional weather information for decentralized PV systems.

However, efficiently utilizing the WFMs to develop SPPF

models is still challenging. There are three possible paradigms of adapting WFMs to SPPF models: sequential modeling [Ahmed et al., 2022], fine-tuning [Ding et al., 2023], and knowledge distillation. The sequential modeling and fine-tuning incorporate the complete body of WFMs and output a large model with a redundant structure. SPPF models developed in these two ways have two main limitations: 1) The ceiling of these SPPF models is rather lower since, intuitively, there is non-indigenous weather knowledge irrelevant to SPPF of specific PVs reserved in the output models; and 2) Both the training and inference phases of these SPPF models induce high computational overhead which escalates the cost of PV sectors. Taking GraphCast as a reference, the training of GraphCast takes four weeks on 32 Cloud TPU v4 devices and the inference of GraphCast takes one minute on a single Cloud TPU v4 device [Lam et al., 2023]. Knowledge distillation, a novel deep learning paradigm proposed by [Hinton, 2015], is feasible to address these problems.

Knowledge distillation learns a smaller refined model to mimic the performance of a large comprehensive model. The SPPF models distilled from WFMs can benefit from the following aspects: 1) the model architecture can be flexibly elaborated to fit the characteristics of SPPF; 2) the training is efficient since WFMs can provide informative weather-related knowledge; and 3) the output SPPF models are much more compact and thus affordable for the PV sectors and residential users. The main challenge of distilling WFMs to SPPF is that a model is required not only to inherit weather-related knowledge from WFMs but also to reserve generalizability for SPPF of specific PVs.

To address this challenge, in this paper, we proposed spatio-temporal knowledge distillation (STKD) fromWFMs to SPPF. It outputs STKD-PV models for SPPF, which takes the regional weather data and PV power data as input and forecasts six hour to day-head PV power generation. We evaluated the STKD-PV models over six real-world datasets distributed worldwide. The results show that STKD-PV models outperforms than a set of five SOTA time-series models and models developed by fine-tuning the WFMs. The SKTD-PV models can achieve an average MAE of 0.0497 over the six datasets. We summarized our contribution as:

1. We formulated the problem of SPPF for decentralized PV systems. We explored to utilize the WFMs to enhance the decentralized SPPF by knowledge distillation. To the best of our knowledge, this is the first study to perform knowledge distillation from WFMs for SPPF.
2. To address the challenges of developing SPPF models from WFMs, we proposed the STKD-PV, which outputs transformer-based models for SPPF by spatio-temporal knowledge distillation from WFMs and fine-tuning with PV power data of the decentralized PV system;
3. We evaluated STKD-PV on six real-world solar power datasets. The results show that STKD-PV can accurately forecast six hour to a day ahead PV power generation and outperform the SOTA time-series models.

2 RelatedWork 2.1 SPPFModels

Traditional approaches of SPPF obtain the weather information from the physical models, e.g., the NWP models. SPPF models are developed with regional meteorological information integrated by using the forecasts of NWP models [Verbois et al., 2022; Mayer et al., 2023].

Recently, there has been an increasing number of deep learning models for SPPF that incorporate weather information with power-related PV system data. [Lim et al., 2022] proposes a CNN-LSTM hybrid model for SPPF where the CNN classifies the weather condition and the LSTM learns power generation patterns. [L´opez Santos et al., 2022] develops a temporal fusion transformer for day-ahead SPPF. [Yan et al., 2021] performs short-term SPPF by using a frequencydomain decomposition-based CNN. These studies are limited since models are developed for individual PVs and have insufficient generalizability within decentralized PV systems.

2.2 Weather FoundationModels

As a significant advancement in the use of AI for meteorology, WFMs are trained with vast global weather data and offer the potential for faster and more accurate weather predictions, which can be crucial for planning and responding to weather-related events [Chen et al., 2023]. WFM learn the intricate relationships between weather and geography, capturing meteorological interactions and spatial dependencies between grids [Mukkavilli et al., 2023]. There is a flourish of well-developed WFMs such as: GraphCast by Google DeepMind [Lam et al., 2023], Pangu-Weather by Huawei [Bi et al., 2023], FourCastNet by NVIDIA [Kurth et al., 2023], and ClimaX and Aurora by Microsoft [Bodnar et al., 2024; Nguyen et al., 2023]. WFMs provides opportunities for SPPF models to learn the impact of meteorological variation on specific sites of PV systems.

2.3 Knowledge Distillation

Formally popularized by [Hinton, 2015], knowledge distillation is transferring knowledge from a large complex model (referred to as the "teacher") to a smaller model (known as the "student"). It is successfully applied in numerous domains such as object detection [Li et al., 2022; Zhang et al., 2023], large language models [Guo et al., 2025], computer vision [Beyer et al., 2022], etc. There also are existing efforts to transfer weather-related knowledge from a weather model by using knowledge distillation. [He et al., 2024] distilling physical knowledge from a differential equation network to a spatio-temporal transformer network. [Tang et al., 2023] develops a spatio-temporal graph knowledge distillation to enhance regional weather prediction. Though incorporate modules to distill spatio-temporal knowledge from pre-trained weather models, these studies still show limitations: 1) the teacher models are regional weather models, pretrained with non-global weather data, and can hardly integrate the global meteorogeographical patterns; and 2) the student models are still weather prediction models, rather than SPPF models. It is still unknown if these distilled models performs well when adapting them to SPPF tasks. In this paper, we try to address these problems by using WFMs as teacher model and develop strategies to reserve the generalizability for SPPF during knowledge distillation.

3 Problem Statement

SPPF for decentralized PV systems can be viewed as a multivariate time-series prediction task within a target region where the decentralized PV system is located. The target region can be represented by a geographical grid, which is a two-dimensional graph G = (V, E). V is a node-set that denotes the geographical grid points, and E is an edge set that represents the geographical proximity among different grid points. The weather data within the target region is denoted as Xw ∈ RT×N×dw , where N is the number of geographical grid points, T is the length of time-series, and dw is the dimension of weather variables. The power data collected from the decentralized PV system is denoted as Xp ∈ RT×M×dp , where M is the number of terminal PV within the decentralized PV system and dp is the dimension of power variables of a single terminal PV. The problem of SPPF for decentralized PV system can thus be represented as follows:

Given a target region G, H steps of historical weather conditions Xw(t-H+1):t , and historical power generation of a decentralized PV system Xp(t-H+1):t , for a terminal PV pm within the decentralized PV system, the SPPF aims to learn a function F(·) to predict Q steps of future power generation:

where Xw(t-H+1):t ∈ RH×N×dw , Xp(t-H+1):t ∈ RH×M×dp , and Xpm (t+1):(t+Q) ∈ RQ×1×1 .

4 Methodology

The proposed STKD-PV architecture consists of three main components: 1) spatio-temporal transformer as the backbone of decentralized SPPF; 2) knowledge distillation from WFMs to the backbone model; and 3) fine-tuning the distilled model with power data of decentralized PV systems to obtain SPPF models for terminal PVs.

4.1 Spatio-temporal Transformer

The decentralized SPPF requires a backbone model to: 1) incorporate the knowledge of spatio-temporal variation of regional weather fromWFMs effectively; and 2) reserve generalizibility for the SPPF of specific terminal PVs within the decentralized PV system. We accordingly elaborated the STKD-PV model, which consists of two parts: 1) STKDRW for learning regional weather and 2) an adapter head for adapting to SPPF. Here, we focus on the STKD-RW and leave the adapter head in Section 4.3.

STKD-RW is a stack of multiple transformer blocks with spatial and temporal modules starting with an embedding layer. Specifically, the input historical weather condition Xw is first encoded to obtain the representation E ∈ RH×N×d by an embedding layer, where d is the dimension of the feature representation. To encode the temporal information, we add the positional encoding to the output embeddings as follows:

where PE ∈ RH×N×d , pos is the positional index of the time step, i = 1, ..., d is the index of feature dimension, and 1 ∈ RN is a vector of all ones. The obtained Z(0) is fed into the spatio-temporal modules for further manipulation.

Spatial Attention Module To extract the spatial dependency of regional weather conditions among the grid points within the target region, we introduced a spatial attention module. The regional geographical grid Gregion can be split from the global graph Gglobal. A geographical adjacency matrix Gregion can be derived from Gregion, where the connected edges are set as 1 and others as 0.

We use the multi-head attention mechanism to capture the spatial diversity of regional weather, where h heads of attention are calculated and concatenated as features of the next block. Denote the input of l-th block as Z(l) ∈ RH×N×d , the query, key, and value matrices of an attention head can be computed as:

where WQ, WK, WV ∈ RN×d/h are the linear projection matrices for query, key, and value. The spatial attention score of i-th head can be calculated as:

where ⊙ denotes the element-wise multiplication operator. The results of multi-head attention can be concatenated accordingly as follows:

where W(l) is a projection matrix of the multi-head attention at l-th block. To stabilize the training, we adopt residual connection and layer normalization after each block of multihead spatial attention.

Temporal ConvolutionModule Given the output of the l-th spatial attention, for capturing the temporal dynamics of the regional weather conditions, we use a standard 1D convolution layer with the length of K to force on processing along the temporal dimension, as expressed in the following:

Following the regular operation in the Transformer structure, we also apply layer normalization after residual connections at the end of the temporal convolution layer. The output is finally fed to a feed-forward layer to introduce non-linearity and further transform the learned representation:

4.2 Knowledge Distillation fromWFMs

In the knowledge distillation, the WFMs perform as teacher networks, and the STKD-RWmodels perform as student networks. The output of WFMs is firstly pre-processed to align the forecasting horizon with SPPF tasks. We use hybrid distillation strategy to train the STKD-RWmodels.

Pre-processing of WFM output The inputs of a WFM is the latest previous observed global weather conditions Xw(t-1):t , and the outputs is the predicted global weather condition of the next time step Xwt:(t+1) . There is a mismatch of prediction horizons between the WFMs and their student networks. To handle this, we introduced a weather data composition by taking auto-regressive prediction with WFMs and concatenating the continuous time steps of predicted weather conditions:

where ˆXwt+i is the i-th time step of future weather conditions predicted by the WFM and ˆXw(t+1):(t+Q) is the composited future weather conditions over Q time steps. Note that the the output of WFMs is still a global weather condition, and the regional weather condition for knowledge distillation can be easily split from the global weather condition.

Hybrid Distillation Strategy To retain the generalizability of STKD-RW models during the knowledge distillation for the downstream SPPF tasks, we introduced the masked selfsupervised distillation, and data-free and data-driven distillation. These distillation strategies can be flexibly combined and conducted together.

Masked Self-supervised Distillation We follow the masked self-supervised learning to enhance the generalizability of our model in the spatial dimension [Hou et al., 2022]. During the knowledge distillation training, the geographical adjacency matrix Gregion is randomly masked with a predefined masking threshold. With such masking, weather information of partial grid points is missing, so the trained STKD-RW models are enforced to infer the weather condition of a certain grid point by gathering information from its adjacent points as far as possible. This enhances the generalizability of STKD-RWmodels over the target region.

Data-free and Data-driven Distillation To enhance the generalizability in the temporal dimension, we combined data-free and data-driven distillation. Denoting the prediction results of the student networks as ˆYstudent ∈ RQ×N×d , the distillation loss is defined by calculating the distance between ˆYstudent and the output of regional weather conditions Yteacher from the WFMs:

For data-free distillation, the input data of WFMs is randomly generated from a pre-defined Gaussian distribution. The WFM takes the input and outputs Yteacher as labels of future weather conditions. The student network also takes the input and outputs predicted weather condition ˆYstudent and is trained by back propagation of the distillation loss Ld.

For data-driven distillation, the training data is the true data, where the ground truth future weather condition Y is used as labels. The loss in this process is comprised of two parts: the distillation loss Ld and prediction loss Lp:

where α is the weight to balance the distillation loss and prediction loss. By default, we conduct data-free and data-driven distillation sequentially.

4.3 Fine-tuning for SPPF

For a single terminal PV within a decentralized PV system, the power data of the target decentralized PV system Xp ∈ RT×M×dp is used to fine-tune the distilled STKD-RW models into STKD-PV models. Xp is combined with the outputs of the STKD-RW models and passed through an additional adapter head. The adapter head is built with a similar structure to the temporal convolution module for further capturing the time-series dynamics of PV power data:

During the fine-tuning process, the parameters of STKD-RW are frozen, and only the parameters of the adapter head are updated with the mean squared error loss.

5 Evaluation

5.1 EvaluationMethodology

Datasets Two parts of time-series data are required in our evaluation: weather data and PV power data. For the weather data, we follow the WFMs and adopt ERA5 [Hersbach et al., 2020], which is a climate reanalysis dataset covering the period 1950 to the present. ERA5 is being developed through the Copernicus Climate Change Service (C3S) 1 . ERA5 covers the Earth on a 0.25 degree latitude-longitude grid and provides hourly estimates of a large number of atmospheric, land and oceanic climate variables. For the PV power data, we select five public datasets 2 and collect a private dataset for decentralized PV systems, the specifications of which are shown in Table 1. The regions of these datasets cover the world, which can verify the performance of the proposed STKD-PV on a global scale and thus ensure the scalability of STKD-PV.

Weather Foundation Models We select four impactive WFMs as the knowledge source of weather conditions for SPPF in the experiments: GraphCast by Google DeepMind, Pangu-Weather by Huawei, FourCastNet by NVIDIA, and Aurora by Microsoft. All these four WFMs can provide highresolution, accurate global weather forecasts. We conduct the knowledge distillation for STKD-PV by using the four WFMs as teacher models individually.

Baseline models To conduct a comparison study for the proposed STKD-PV, considering that the solar power data and weather condition data are both time-series data in essential, we select five SOTA time-series models that are widely practiced in real-world applications as our baseline models:

• TimesNet [Wu et al., 2022]: It is a convolution-based network that focuses on discovering the periodicity within the input time-series data. It utilizes a Fourier

1 https://climate.copernicus.eu/

2 https://www.tudelft.nl/ewi/over-de-faculteit/afdelingen/

electrical-sustainable-energy/photovoltaic-materials-and-devices/ dutch-pv-portal/pv-power-databases

Transform to transform the input 1D time-series data into 2D tensors so that the convolutional kernels can detect the inter- and intra-period variation of the input.

• Autoformer [Wu et al., 2021]: It renovates the Transformer as a deep decomposition architecture, which can progressively decompose the trend and seasonal components during the time-series analysis process.
• STGCN [Yu et al., 2017]: It is a spatio-temporal graph convolutional network for time-series prediction tasks by using complete convolutional structures.
• TCGAN [Huang and Deng, 2023]: It is a convolutional GAN for time-series data, which learns by playing an adversarial game between two CNNs (i.e., a generator and a discriminator) in the absence of label information.
• TimeGAN [Yoon et al., 2019]: It is a GAN for generating realistic time-series data that combines the flexibility of the unsupervised paradigm with the control afforded by supervised training.

All these models are trained with regional weather data and PV power data. Besides, we also consider fine-tuning the WFMs with PV power data as our baselines. Specifically, an adapter head is added after a WFM to integrate the PV power data and further training for SPPF. During the fine-tuning, the parameters of the WFMs are frozen, and the adapter head's parameters are updated. We extend the four aforementioned WFMs with adapter heads as the backbone for fine-tuning, which are denoted as GraphCast-PV, Pangu-PV, FourCastNet-PV, and Aurora-PV, respectively.

Metrics To measure the performance of developed SPPF models, we use the mean absolute error (MAE), root mean squared error (RMSE), and the coefficient of variation of root mean squared error (CV-RMSE) as our metrics (lower is better), which are all widely adopted metrics in the energy domain [Granderson et al., 2016].

Configuration We split each dataset into non-overlapping subsets by 7:1:2 partitions along with the time dimension as our train/validate/test set. Each data set is normalized prior to training. The frequency of time-series is synchronized to 15 min. By default, the input sequence length and output sequence length of models are set to 24 time step (6 hours). For our method, STKD-PV is trained with an Adam optimizer with maximum 200 epochs. The number of spatiotemporal blocks of STKD-PV models is set to three. The balance weight between distillation loss and prediction loss is set to 0.4. The masking rate of masked self-supervised distillation is set to 0.15. The dimension of the embedding layer is set to 256. The results are concluded with 10 trials using different random seeds.

5.2 Results

Overall performance Table 2 shows that STKD-PV models outperform the baseline models on all six datasets. STKDPV can achieve best performance on all three metrics reported. This indicates that STKD-PV can accurately perform SPPF worldwide. The MAEs of STKD-PV are 0.0584, 0.0336, 0.0406, 0.0578, 0.0453, and 0.0607 on the six dataset, respectively. We emphasis the optimal values of MAE, RMSE, and CV-RMSE with bold font in Table 2.

We have observations as follows: 1) Compared to SOTA time-series models, the STKD-PV models perform better since they not only consider the spatio-temporal features of regional weather but also learn high-level global weather knowledge fromWFMs. Specifically, STKD-PVmodels improve 65.04% of MAE, 57.46% of RMSE, and 44.39% of CV-RMSE, respectively, as compared to the SOTA timeseries models; 2) Compared to models directly fine-tuned from WFMs which redundantly reserve information irrelevant to weather of the target regions, the STKD-PV models refine the regional weather information and remove this irrelevant information. Specifically, STKD-PV models improves 20.98% of MAE, 20.77% of RMSE, 25.10% of CVRMSE respectively as compared to the models by fine-tuning WFMs; 3) Within the STKD-PV models, their performance differ from each other. The models distilled from Aurora outperform models distilled from other WFMs. This is because compared to GraphCast, Pangu, and FourCastNet, Aurora is a more general system that can learn from many diverse datasets and adapt to various prediction tasks. Specifically, STKD-PV distilled from Aurora improves 29.86% of MAE, 39.08% of RMSE, 18.41% of CV-RMSE as compared to those distilled from other WFMs.

Internal factors We analyze three internal factors which may influence the performance of STKD-PV models: 1) masked self-supervised distillation, 2) data-free and datadrive distillation, and 3) spatio-temporal modules. Since STKD-PV distilled from Aurora performs best, we use it as the default setting in the following evaluation.

Masked Self-supervisedDistillation Figure 3 shows the results of STKD-PV with and without masked self-supervised distillation. Compared to STKD-PV without masked selfsupervised distillation, STKD-PV performs better on both MAE and RMSE over a six to 24-hour forecasting horizon. Specifically, STKD-PV models have an average improvement of 13.76% on MAE and 23.24% on RMSE. This reveals that masked self-supervised distillation can successfully improve the generalizability of STKD-PV models and thus achieve better performance.

Data-free and Data-Driven Distillation To evaluate the strategy of data-free and data-driven distillation, we developed STKD-PV models with three strategies: data-free distillation only, data-driven distillation only, and combining datafree and data-driven distillation together. The size of the training data of data-free distillation stays the same as that of the ground truth training data. Figure 4 shows the comparison results. The combined strategy outperforms the individual data-free and data-driven strategy. Compared to data-free distillation, combined strategy can have an average improvement of 19.84% on MAE and 17.27% on RMSE for STKDPV models. Compared to data-driven distillation, it also improves 9.44% on MAE and 10.94% on RMSE averagely. This indicates that both data-free and data-driven distillation strategy are effective for enhancing the STKD-PV models.

Spatio-temporal Modules To evaluate effectiveness of designed spatio-temporal modules, we developed three variants of STKD-PV: STKD-PV-rS: STKD-PV with spatial attention module replaced by a dense layer; STKD-PV-rT:

STKD-PV with temporal convolution module replaced by a dense layer; and STKD-PV-rST: STKD-PV with both spatio-temporal modules replaced by dense layers. Figure 5 shows that STKD-PV outperforms the three developed varieties. Specifically, STKD-PV lower 20.82%,8.25% and 34.31% of MAE as compared to STKD-PV-rS, STKD-PVrT, and STKD-PV-rST. STKD-PV also lower 20.43%, and 11.70%,29.70% of RMSE correspondingly. This demonstrates that both the elaborated spatial attention module and temporal convolution module are crucial for learning the spatio-temporal variation of the regional weather and thus can improve the performance of STKD models.

Hyper-parameter Analysis we conduct a hyper-parameter analysis on three vital hyper-parameters of STKD-PV: 1) the number of blocks of spatio-temporal modules; 2) the balance weight α between the distillation loss and prediction loss; and 3) the mask rate on the geographical grid during the masked self-supervised distillation.

Figure 6 shows the variation of MAE to the hyperparameters. For the number of spatio-temporal blocks, we can observe that models with three spatio-temporal blocks perform best. We can also see that a smaller number of blocks may lead to insufficient distillation, and a larger number of blocks may lead to a redundant model structure. For the balance weight α, we can observe that the optimal value of α is 0.4. Larger α leads to overdependent on distilling from the WFM and smaller α results in insufficient distillation. For the mask rate during distillation, we can observe that the optimal value is 0.15. A larger mask rate will make the distillation tough, and a smaller mask rate may lower the generalizability of STKD-PV models.

6 Conclusion

In this paper, to adapt the knowledge of WFMs for SPPF, we proposed the STKD-PV model which consists of spatiotemporal transformers. STKD-PV model is first trained by distilling knowledge from WFMs and then fine-tuned with PV power data. To enhance the generalizability of the STKD-PV model, we introduce a hybrid knowledge distillation strategy that includes masked self-supervised distillation, as well as data-free and data-driven distillation approaches. Experimental results demonstrate that the STKD-PV model achieves significant performance improvements, surpassing baseline models.

Acknowledgements

The authors greatly thank assistance of Qicong Fu and Kerui Wang. Dan Wang's work is supported in part by RGC GRF 15200321, 15201322, 15230624, ITC ITF-ITS/056/22MX, ITS/052/23MX, and PolyU 1-CDKK, G-SAC8. What's more, the authors are indebted to the anonymous reviewers for their constructive comments