Monitoring Vegetation From Space at Extremely Fine Resolutions via Coarsely-Supervised Smooth U-Net

1 Introduction

Monitoring crop growth is vital for preventing food insecurity and supporting humanitarian efforts to address the UN Sustainable Development Goal 2 ("Zero Hunger") [Peng et al., 2020]. 1 This is particularly crucial due to the impacts of climate change on agriculture [Ortiz-Bobea et al., 2021]. Thus, there have been many attempts to observe vegetation progress from space via remote sensing. A common approach is to use vegetation indices that can be computed from satellite imagery, such as NDVI. However, these indices are only indicative of vegetation "greenness", and cannot directly measure photosynthetic activity [Li and Xiao, 2019]. On the other hand, Solar-Induced Chlorophyll Fluoresence (SIF) holds the potential to directly monitor plant productivity, as it has mechanistic linkages to plant photosynthesis, and provides more accurate and timely information about plant growth [Kira and Sun, 2020]. Satellite-based SolarInduced Chlorophyll Fluorescence (SIF) can outperform traditional vegetation indices for crop yield prediction [Peng et al., 2020], as well as for monitoring the effects of drought and heat stress on vegetation productivity [Song et al., 2020].

∗Contact Author

1 Appendices can be found at http://joshuafan.github.io/files/SIF supp.pdf

However, despite recent progress, SIF continues to be a very difficult signal to measure from space [Duveiller et al., 2020]. Most satellite SIF measurements are noisy, and are thus only available at a coarse spatial resolution, such as 35km [Yu et al., 2019]. Within a large region of many square kilometers, SIF can vary dramatically, depending on crop type, management practices, and genetic varieties. For example, the soybean fields in a particular region may be growing very well, while the corn fields may be experiencing stress. Yet existing satellite measurements cannot resolve this important variation [Kira and Sun, 2020].

There have been several attempts to predict SIF at finer spatial resolutions, given coarse-resolution SIF measurements and additional fine-resolution variables [Yu et al., 2019; Duveiller et al., 2020; Wen et al., 2020]. This task is sometimes known as "statistical downscaling." These works train a supervised machine learning model at a coarse scale, to map from auxiliary input variables (averaged across an entire coarse tile) to the SIF of that tile. Then this trained model is applied at a fine scale, using the fine-resolution input variables to predict fine-resolution SIF. Since the input variables are averaged across a large region during training, information about fine-grained spatial variation is lost. Thus, these works only attempt to resolve SIF down to a 0.05-degree (≈ 5 km) resolution. They do not attempt to resolve SIF at superfine resolutions such as 30 meters; our results show that these methods are inaccurate at such fine resolutions.

This problem poses a challenging coarsely-supervised regression task. We want to learn a model that maps from fineresolution input features to fine-resolution SIF, yet the only SIF labels available at training time are at a much coarser resolution. We note that this task (inferring a fine-resolution map of a variable, given only coarse-resolution observations and other fine-resolution auxiliary data) is not exclusive to SIF, but is applicable to many problems in diverse fields, such as improving the resolution of climate model outputs [Wood et al., 2004], soil moisture [Peng et al., 2017], and malaria risk [Sturrock et al., 2014].

To address this issue, we propose a novel technique, Coarsely-Supervised Smooth U-Net (CS-SUNet), which can produce SIF predictions at a 30-meter resolution, even though it is only trained with SIF labels at a 3km resolution (100x coarser). CS-SUNet takes in a fine-resolution input image, and outputs a SIF prediction for each pixel; the average of the pixel predictions is trained to match the true coarse SIF for the entire 3km image. The U-Net architecture is good at producing localized predictions, as the prediction for a pixel primarily depends on nearby pixels' features [Wang et al., 2020]. We further enhance the architecture to make the localization more precise and reduce instability. Still, the coarseresolution SIF labels are an extremely weak form of supervision, and do not provide enough constraints for the pixel predictions. However, we show that CS-SUNet can still learn to predict accurately through regularization techniques inspired by prior knowledge.

Our contributions. (1) We propose CS-SUNet, a novel approach to coarsely-supervised regression problems, which builds on the expressive power and localization abilities of UNet. To our knowledge, this type of approach has never been applied to resolving fine-resolution SIF or similar "coarselysupervised regression" problems with spatial structure. (2) We find that naively training a U-Net can result in a unique form of overfitting (by outputting extreme pixel predictions that happen to average to the right value). Thus, to regularize the model, we design a smoothness loss that encourages pixels with similar features to map to similar SIF. We also use early stopping based on a small fine-resolution validation set. (3) We evaluate our method on real-world remote sensing and SIF data. CS-SUNet predicts SIF at fine resolutions more accurately than all baselines, reducing RMSE by almost 10%.

2 RelatedWork

SIF Downscaling: Mechanistic Approaches. There have been several attempts to downscale coarse-resolution SIF measurements. One line of work uses a mechanistic Light Use Efficiency modeling approach to improve the spatial resolution of the GOME-2 satellite's SIF measurements from 0.5-degree to 0.05-degree [Duveiller et al., 2020]. Hand-crafted theoretical models are fitted to establish semiempirical relationships between SIF and explanatory variables (vegetation indices, evapotranspiration, and temperature). These methods impose strong assumptions on the nature of these relationships, and assume that they hold across all land cover types.

SIF Downscaling: Data-Driven Approaches. There have also been attempts to use completely data-driven statistical downscaling methods. These studies use standard machine learning algorithms, such as Artificial Neural Network [Yu et al., 2019] or regression trees [Li and Xiao, 2019], to learn an empirical relationship between averaged coarse-resolution input features (e.g. reflectance) and coarse-resolution SIF. This relationship is then applied on fine-resolution reflectance to estimate fine-resolution SIF. These approaches work well at a relatively coarse resolution (such as downscaling from 0.5 to 0.05 degrees), but become less accurate at finer resolutions, since coarse-resolution SIF labels used in training typically represent the SIF of a large region with multiple land cover types. However, we are interested in predicting SIF for individual fields of a single crop type. Because reflectance data for the entire large tile is averaged together, the model cannot observe variation between crop types within a tile during training, or learn crop-specific patterns. Recently, [Kira and Sun, 2020] addressed this "SIF unmixing" problem by estimating crop-specfic SIF for corn and soybean portions of a larger region. They produce a training dataset of artificial "mixed pixels", which were generated by mixing pure-corn and pure-soy tiles together in random proportions. Then they train a neural network to estimate how much of the SIF was from the corn areas vs. the soybean areas. However, this approach requires that pure tiles for each crop type exist, which is not the case for most crops other than corn and soybean.

Statistical Downscaling in Other Fields. The "statistical downscaling" approach has also been applied to numerous problems in geoscience and other fields, including downscaling soil moisture [Peng et al., 2017] and other remotely sensed variables. As previously mentioned, these methods discard information about spatial variability within the large regions while training. Bias-corrected spatial disaggregation [Wood et al., 2004] is another method to downscale projections from climate models, but it requires extensive groundtruth fine-resolution labels. Hierarchical Bayesian models have also been used for downscaling, such as for species distributions and malaria risk [Sturrock et al., 2014]. However, they model fine-resolution probabilities as a simple linear function of the inputs, which limits the expressiveness of the learned functions. Some works have proposed methods based on Gaussian Processes [Tanaka et al., 2019], but they are computationally intractable on large datasets (such as our setting where we want to predict for millions of pixels).

Multiple-InstanceLearning. In themachine learning literature, there are some prior works on multiple-instance learning, where each training label is over a group of examples. While most of these works focus on classification, a few papers consider regression tasks (aggregate output regression). [Musicant et al., 2007] presented adaptations of supervised learning algorithms to this setting. [Kotzias et al., 2015] studied this in the context of fine-grained sentiment analysis of online reviews, and proposed adding a smoothness loss term, which is similar to the idea in our paper. [Zhang et al., 2020] proposed probabilistic models for this setting, but the method they derive for regression tasks is just a simple L2 loss between the predicted and true aggregated outputs, similar to the coarse loss term in our approach. Unlike these works, our approach utilizes the spatial structure in our data via U-Net.

Supervised Single-Image Super-Resolution. Many papers have studied single-image super-resolution using convolutional neural networks (CNNs), where the goal is to learn a function mapping directly from a coarse-resolution image to the fine-resolution version. For example, [Liu et al., 2020]

use a novel fusion CNN architecture to downscale climate model variables to finer resolution. However, these methods require abundant fine-resolution data for training, which is not available for SIF. By contrast, CS-SUNet does not use fine-resolution labels at training time, so it is usable in settings where fine-resolution labels are scarce.

Image Segmentation. CS-SUNet is inspired by techniques originally developed for image segmentation, specifically the U-Net architecture [Ronneberger et al., 2015]. The U-Net produces a prediction for each pixel in the original image, based on the contents of that pixel and neighboring pixels. The U-Net is typically used for classification, but CS-SUNet is adapted for regression.

Weakly-Supervised Pixel Predictions. While image segmentation tasks are usually trained using pixel-level labels, some works use weaker forms of supervision (e.g. imagelevel labels). Recently, [Wang et al., 2020] showed that the U-Net can accurately segment remote sensing imagery (e.g. classifying each pixel as cropland or not), even if it is only trained with image-level labels. This is largely due to the UNet's impressive localization abilities, which we harness in our approach. [Wang and Yang, 2018] show that CNNs can provide localization abilities for regression tasks also. They train a CNN to predict diabetic retinopathy from a retina image, and propose a method, Regression Activation Mapping, to determine which regions of the image have the greatest influence on the final prediction. By contrast, our paper explicitly predicts the label (SIF) for every pixel in the image.

3 Datasets

We summarize the datasets used in this paper. More details (including citations) can be found in Appendices E and F.

3.1 Input Features

Landsat Surface Reflectance. Satellite imagery is available from Landsat at a fine 30m resolution, every 16 days. For each pixel, there are 7 bands (channels). We zero out pixels with missing data and create a mask to mark this.

FLDAS land data. Temperature, rainfall, and surface radiation data are available from FLDAS (Famine Early Warning Systems Network (FEWS NET) Land Data Assimilation System), at a 10km resolution. We linearly interpolate these variables to a 30m resolution to match the other variables.

Land cover data. Land cover data is available from the Cropland Data Layer at 30m resolution. Each pixel is labelled with a land cover type. For each land cover type that makes up more than 1% of the dataset, we create a binary mask of which pixels are covered by that type.

3.2 SIF Labels

To evaluate model performance, we need some groundtruth fine-resolution SIF data. The Chlorophyll Fluorescence Imaging Spectrometer (CFIS) provides SIF observations at fine resolutions for extremely limited areas. We aggregate the CFIS observations to a 30m resolution, matching the Landsat grid. Then, we produce datasets at different resolutions: {30, 90, 150, 300, 600} meters for evaluation, and 3km for training (to simulate the realistic setting in which only coarseresolution SIF observations are available).

OCO-2 [Sun et al., 2017] provides additional SIF labels at a coarse spatial resolution, which we can also use for coarseresolution training. The original footprints cover areas of around 1.3 × 2.25 km at nadir. We grid these to a 3km resolution (on the same grid as our CFIS 3km dataset). To correct between the differing wavelengths of the different instruments, we multiply OCO-2 SIF values by 1.11 to match CFIS. (This scaling factor was empirically determined by fitting a model on OCO-2, evaluating it on CFIS, and determining what scaling factor produced the optimal fit.)

Since SIF observations are noisy, we filter our dataset to only include pixels and tiles with enough observations (so that noise can be reduced through averaging). We also remove tiles with many missing features. See Appendix G for details.

3.3 Study Region and Time Range

All tiles in our dataset correspond to a coarse-resolution CFIS or OCO-2 SIF label that lies within the Midwest US, from 38 to 48.7 degrees N and 108 to 82 degrees W. The region is plotted in Figure 1. As shown in the figure, even coarseresolution SIF datasets are sparse geographically, so machine learning is needed to fill in the gaps and predict at a higher resolution. We extract input feature data (reflectance, crop cover maps, etc.) from the same time periods as the CFIS SIF observations: June 15-29, 2016 and August 1-16, 2016. We also use OCO-2 observations from those time periods.

3.4 Data Preprocessing

We use a total of 22 input features: 7 Landsat reflectance bands (plus 1 binary mask indicating missing Landsat data), 3 FLDAS variables (temperature, rainfall, surface radiation), and 11 land cover binary masks. All features were resampled to a fine resolution of 30 meters. For each coarse (3km) SIF label, we extracted the corresponding tile of feature data. These tiles are 3-dimensional tensors, of size 22 (features)×100 × 100 (pixels). For the continuous variables, we standardized them to have mean 0 and variance 1, and clipped them to [-3, 3] standard deviations from the mean.

We randomly assign each tile to one of 3 sets: train (60%), validation (20%), and test (20%); we ensure that CFIS and OCO-2 tiles that overlap end up in the same set. The model is trained on coarse-resolution (tile-level) SIF labels from the train tiles; there are a total of 712 CFIS and 1390 OCO-2 tiles in the train set. Then, we select hyperparameters based on NRMSE on the fine-resolution CFIS validation set.

4 Methods

We propose CS-SUNet, a novel approach for coarselysupervised regression tasks. CS-SUNet utilizes the localization abilities of the U-Net architecture, adapts it to our coarsely-supervised regression setting, and employs regularization techniques informed by prior knowledge, such as a smoothness loss and early stopping.

4.1 U-Net Architecture

CS-SUNet makes use of the U-Net architecture [Ronneberger et al., 2015], which was originally designed for segmenting biomedical images using relatively little training data. The U-Net is a fully convolutional neural network which can take in input images of any size. It consists of a contracting path which aggregates information from local regions (left side), and an expanding path which combines broader large-area information with precise local characteristics (right side). The final output is a map, with a prediction for each pixel.

In our implementation, we used 2 upsampling and 2 downsampling blocks, with {64, 128} hidden units. For half of the convolutional layers, we decreased the filter size from 3 to 1 to ensure that each pixel's prediction only depends on nearby pixels' features. The output is a single scalar per pixel. We removed the batch normalization layers as they caused training to be unstable; they do not appear to be a good fit for regression tasks like ours that depend on the absolute magnitude of the inputs (see Appendix G). Further model details can be found in Appendix J.

4.2 Coarsely Supervised Training

CS-SUNet is trained in a coarsely-supervised fashion. Our training set consists of pairs {(Xi, yi)}Ni=1 : Xi ∈ RC×H×W is a large tile "image" with C channels, height H, and width W, and yi ∈ R is the ground-truth average (coarse) SIF for tile i.

We train a U-Net model fθ that takes in large tile Xi and predicts SIF for each pixel, Zi ∈ RH×W . Let z(p) i ∈ R be the predicted SIF of pixel p. During training, we do not have pixel-level SIF labels. Instead, we train the model by optimizing a combination of two loss functions: (1) a coarse loss that encourages the average predicted SIF of the entire tile to equal the true SIF of the tile, and (2) a smoothness loss that encourages pixel pairs with similar input features to map to similar SIF predictions.

The coarse loss is defined as the Mean Squared Error between the true and predicted coarse tile SIFs; this is similar to [Zhang et al., 2020]. We average predictions over P, the set of valid fine pixels that contributed to the tile's coarse SIF:

However, achieving a low coarse loss does not guarantee that the fine-resolution predictions will be accurate. It is possible for the U-Net to output extreme pixel predictions that still produce the correct coarse-tile average. We find that this occurs frequently in practice (see Appendix B). To reduce this unique form of overfitting, we introduce a smoothness loss [Kotzias et al., 2015], which encourages pixels with similar input features to map to similar SIFs. To compute this loss, for each batch we randomly sample a set of pixels S; in our experiments we sample 1000 pixels per batch. For each pair of pixels in S, we compute a similarity score between their input feature vectors (x(j) , x(k) ), and multiply that by the squared difference between their predicted SIFs (z(j) , z(k) ):

This places a large penalty if two pixels with high input similarity have very different SIF, as this represents overfitting. The similarity function can be customized based on one's prior knowledge of the problem. We set the similarity to 0 if pixels j and k are of different land cover types. Otherwise, if they are of the same land cover type, we define their input similarity as

Here x(j) ref is the reflectance feature vector for pixel j. Note that if τ is positive, the similarities are bounded between 0 and 1. τ is a hyperparameter that controls how quickly the similarities decay towards 0; we set this to 0.5. Appendix C discusses the impact of the smoothness loss, and Appendix D motivates the design of this loss.

The final loss is a weighted combination of the coarse loss and the smoothness loss. For a batch,

where λ is a hyperparameter that balances the loss terms. We optimize this loss via backpropagation. Figure 2 depicts the training process.

4.3 Early Stopping

CS-SUNet does not look at any fine-resolution SIF data during training. However, during validation, CS-SUNet does peek at the model's performance on the fine-resolution validation set to select which epoch's model to use. Like the smoothness loss, early stopping encourages "input smoothness" (pixels with similar input features should map to similar SIF) [Rosca et al., 2020], which is crucial for regularizing our model. The fine-resolution data is only used to determine how much regularization is necessary, so it may not need to be very similar to the evaluation setting. If we do not have any fine-resolution data at all, it is difficult to determine the optimal amount of regularization, but our smoothness loss can still prevent overfitting on later epochs (see Appendix C).

5 Experiments

We evaluate our pixel-level SIF predictions against airborne measurements from CFIS, which are available at a 30-meter resolution in limited regions. To reduce measurement noise, we only evaluate on pixels for which CFIS had at least 30 observations and where the SIF value was at least 0.1. We use Normalized Root Mean Squared Error (NRMSE) and R2 as evaluation metrics. (NRMSE is Root Mean Squared Error, normalized by the average SIF of the training dataset.)

5.1 Training Details

The CS-SUNet approach is compared against existing "statistical downscaling" baselines that have been used in papers such as [Yu et al., 2019; Li and Xiao, 2019]. These involve averaging each input feature (channel) over all valid pixels in the tile, and training a model to predict SIF from the feature averages. We consider 4 types of models: Ridge Regression, Gradient Boosting Regressor, Random Forest, and a fullyconnected artificial neural network (ANN).

We also include a few novel baselines. The "Pixel NN" approach involves feeding each pixel through the same neural network, averaging all pixels' predictions across a tile, and then training using coarse loss (tile-level supervision); this substitutes the U-Net architecture with an per-pixel architecture where each pixel's prediction is only influenced by that pixel's features. The "vanilla U-Net" approach involves training a U-Net with only the coarse loss, but without regularization (early stopping or smoothness loss). Finally, "U-Net fine supervision" trains a U-Net using fine-resolution labels, and represents an upper limit on our model performance if we were actually able to train on fine-resolution labels.

We train all deep models with the AdamW optimizer [Loshchilov and Hutter, 2018]. For each method, a gridsearch was performed over hyperparameter values (see Appendix I for details); we selected the hyperparameters that performed best on the fine-resolution validation set. For CSSUNet, we used a batch size of 128, learning rate 3e-4, weight decay 1e-4, τ = 0.5, and λ = 0.5. For methods that involve randomness, we took the average of 3 runs using different random seeds. For tiles where the coarse-resolution SIF label is observed during training, we also include a trivial "predict coarse" baseline where we predict the SIF of every pixel to be the same as the SIF of the entire tile.

5.2 Results

Table 1 presents the results for fine pixels in train tiles (i.e. the coarse-resolution SIF of the entire tile was known during training, but not the fine-resolution SIF of the individual pixels). The model is tasked with interpolating the fineresolution SIF from the coarse-resolution labels. Since we know the coarse-resolution SIF of these tiles, we multiply each tile's pixel predictions by a per-tile constant, such that the average of the pixel predictions exactly equals the true coarse SIF. Table 2 presents results for fine pixels where not even coarse-resolution SIF was seen during training; it tests the model's ability to generalize to "gaps" where SIF observations were not directly available. (Note that the relatively lower R2 in Table 2 is because there is less total variation in these tiles; the absolute error magnitudes (NRMSE) are similar to Table 1.)

According to both metrics, CS-SUNet outperforms standard "statistical downscaling" methods in both settings, reducing NRMSE by 9-10% and increasing R2 by 13-33% (relative) over the best baseline, Ridge Regression. This improvement occurs across multiple land cover types and resolutions (see Appendix A). CS-SUNet outperforms the "Pixel NN" architecture, which shows that the prediction accuracy for a pixel can be improved if you look at neighboring pixels' features, not just that pixel's features. Notably, the "Vanilla U-Net" actually does poorly - simply applying a U-Net out of the box does not work, due to the "overfitting to coarse labels" problem described earlier. In order for the U-Net to work, it needs to be highly regularized, which we achieve in this paper by early stopping and a novel smoothness loss.

While an NRMSE of 18-19% is not perfect, note that even a model trained with fine-resolution labels can only achieve an NRMSE of 16% on the test tiles, so our performance is not far off. In any case, increasing the resolution of SIF measurements by 100x is an extremely challenging task, and our method provides significant improvements.

Qualitative evaluation. Figure 3 presents an example of the predictions outputted by various measurements. All methods provide useful results; even the basic Ridge Regression method is able to identify fields with higher and lower SIF. However, the error maps on the top row show that CS-SUNet produces more reliable predictions, as there are fewer dark blue or red pixels (which indicate large errors). In addition, CS-SUNet takes the spatial context of pixels into consideration, so the predictions outputted by CS-SUNet are much more smooth and consistent across local regions.

Limitations. The evaluation of our models is limited by the fact that there is significant noise in both SIF measurements and Landsat reflectance (and there may be a few days' gap between them). Our approach for SIF prediction requires Landsat reflectance data, yet Landsat data is unavailable or noisy in some places/times due to cloud cover. This could be alleviated in the future as more satellites are deployed.

6 Conclusion

We present Coarsely-Supervised Smooth U-Net (CS-SUNet), which is capable of predicting SIF at a very fine resolution (30m), even when only coarse-resolution (3km) SIF measurements are available. Due to its localization properties, CSSUNet is able to figure out which farms within a large tile had higher and lower SIF. CS-SUNet harnesses the expressive power of deep convolutional networks, yet avoids overfitting due to its smoothness loss and early stopping.

Although CS-SUNet's predictions are not perfect, they clearly outperform existing state-of-the-art methods, and can provide valuable vegetation information. Even noisy fineresolution SIF estimates may facilitate improvements in crop yield prediction and monitoring, as SIF contains detailed information about plant photosynthesis that is not captured in vegetation indices such as NDVI [Peng et al., 2020], which are simple combinations of a few spectral bands. Moreover, in addition to SIF, CS-SUNet could also potentially be applied to make fine-resolution predictions of any numerical variable that is only available at coarse resolution, if there exists additional fine-resolution data that is correlated with the variable of interest. Such applications could include predicting soil moisture, precipitation, disease prevalence, and species distributions at fine resolutions.

Acknowledgements

This research was supported by NSF award CCF-1522054 (Expeditions in Computing) and the NSF NRT Digital Plant Science Fellowship (1922551). We thank Junwen Bai for proofreading and providing valuable suggestions.