AG-STELLA: Spatio-Temporal Learning for Water-related Agricultural Land Use Activity Mapping with AlphaEarth

1 Introduction

Agricultural land use activity refers to the temporal transitions of farmland among cropland, pasture, and fallow states, reflecting farmer responses to climate variability [Norton et al., 2021], water availability [Tortajada et al., 2017], and economic conditions [ ˙Zarczy´nski et al., 2023]. In water stressed regions, these transitions increasingly emerge as adaptive responses to water scarcity and drought [Wang, 2023], serving as early indicators of irrigation constraints, groundwater depletion, and agricultural risk [Satapathy et al., 2024]. Moreover, evidence from highly crop-intensive regions shows that short-term fallowing is commonly used to reduce groundwater withdrawals during drought [Chen et al., 2023; Eslamifar et al., 2024], while longer-term adaptation often involves transitions from irrigated cropland to lower-intensity pasture or grass-based systems under persistent water stress [Wang, 2023]. These transition reflects both water conservation responses [Mirzaei and Garcia, 2024] and structural shifts in agricultural production systems [Greiner et al., 2021]. Accurate and timely mapping of land use activity is therefore essential for crop yield forecasting, drought response, supply-chain planning, and long-term sustainability assessments-particularly as fields increasingly rotate between productive and low-intensity states within short time horizons [Liu et al., 2025; Song et al., 2022]. For instance, drought relief decisions often require evidence of agricultural hardship beyond reduced water supply; quantifying short-term transitions from cropland to fallow or pasture can directly support faster and equitable drought response [WA State Legislature, 2018]. By enabling transparent assessment of agricultural stress, adaptive land management, land use activity mapping informs water allocation, safeguards ecosystem services, and supports rural livelihoods, thereby directly aligning agricultural land use activity mapping with the United Nations Sustainable Development Goals SDG 2 (Zero Hunger) and SDG 8 (Decent Work and Economic Growth) [WWAP, 2019].

Agricultural land use activity mapping faces several technical challenges. (i) Spectral ambiguity (as shown in Figure 1): Static satellite imagery suffers from severe spectral ambiguity, as fallow land, dormant pasture, and harvested cropland often appear visually indistinguishable [Morais et al., 2024]. Classification algorithms routinely conflate these two vital states due to low spectral disparity across most bands [Bak et al., 2025; Lei et al., 2024]. Existing methods rely heavily on vegetation indices such as Normalized Difference Vegetation Index (NDVI), yet vegetation estimates are highly sensitive to data-quality issues, such as cloud occlusion, snow cover, atmospheric effects, and sensor noise [Hoque et al., 2025]. Moreover, in regions with high crop diversity, overlapping crop temporal and spectral signatures amplify the ambiguity (e.g., weeds in a fallowed field vs fallowed of another crop) [Noorazar et al., 2025]. (ii) Temporal sparsity: Temporal modeling is further complicated by irregular satellite revisit schedules and missing observations, which produce incomplete and noisy multi-month time series and obscure key temporal cues such as green-up when plants start to grow, and senescence when plants get mature and stop growing or start yellowing. (iii) Context dependence: Beyond temporal challenges, land use decisions are strongly context-dependent: visually similar patterns may arise from fundamentally different drivers, such as water scarcity-induced fallowing in arid regions versus soil-restoration practices in water-sufficient areas. (iv) Limited annotated data with severe class imbalances: Supervised learning approaches additionally face high annotation costs, data scarcity, and severe class imbalances, particularly for a minority activity class such as fallow land (e.g., only 6.9% for UT) and pasture land (e.g., 7.8% for CO). While recent foundation models offer strong representations, they remain insufficient for agricultural land use activity mapping. (v) Limitations of existing models. State-of-the-art models like Prithvi [Li et al., 2023], though pretrained on large-scale remote sensing data and robust to noise and missing observations, lacks explicit agronomic reasoning needed to distinguish fallow land from pasture. Recent geospatial embeddings such as AlphaEarth [Brown et al., 2025] introduce valuable spatial and geolocational context. However, these representations are typically provided at an annual temporal scale which limits their ability to capture fine-grained intra-seasonal management dynamics such as crop rotation, green-up, and senescence. As a result, existing methods continue to conflate visually similar yet agronomically and societally distinct land use states.

Contributions. To address the challenges, we propose AGSTELLA, a knowledge guided spatio-temporal model for agricultural land use activity mapping. Our main contributions are as follows:

• End-to-end foundation-model-driven spatio-temporal model. We introduce AG-STELLA, an end-to-end training model that jointly leverages Prithvi for multi-month spatio-temporal representation learning and AlphaEarth for geospatial context encoding. By coupling Prithvibased spatio-temporal transformer representations with AlphaEarth embedding conditioned feature modulation and land use aware segmentation, AG-STELLA learns the representation that encodes intra-seasonal land use activity by
• emphasizing temporal changes in feature activations rather than static spectral intensity.
• Transition-aware temporal representation. We propose a Temporal Transition Latent Space that captures intraseasonal dynamics from pretrained transformer tokens, guided by vegetation index variation to emphasize land use activity rather than static appearance.
• Knowledge guidance through geo-spatial context and hydroclimatic constraints. We incorporate geospatial context using AlphaEarth embedding to condition the model prediction and introduce hydroclimatic consistency constraints by imposing a class-wise ET-precipitation correlation constraint to enable context-aware predictions that generalize across heterogeneous environmental and climatic regimes.
• Land Use-Aware Spatial Decoder. We introduce a land use-aware spatial gating module that conditions the decoder on coarse land use priors to modulate spatial feature responses. By restricting feature activation to land use consistent regions, the model suppresses background interference and prioritizes land use activity relevant regions, resulting in more precise land use delineation across heterogeneous agricultural landscapes.
• Extensive evaluation in state-specific and cross-state setting. We conduct a comprehensive cross-state and in-state evaluation across Colorado, Washington, and Utah to assess generalization under diverse agricultural and cropping regimes. We evaluate across three U.S. states (Colorado, Washington, and Utah) and demonstrate consistent performance gains over state-of-the-art baselines and improved transferability across the U.S. states. AG-STELLA outperforms Prithvi and AlphaEarth with up to 16.0% higher macro accuracy and 27.0% higher F1-scores. Beyond performance metrics, we analyze crop-specific sources of systematic land use mapping errors across regions.

2 RelatedWork

Remote Sensing for Mapping Agricultural Land and Infrastructure. Deep learning has been widely applied in agricultural remote sensing for crop classification, field boundary detection, and irrigation mapping [Weiss et al., 2020; Paolini et al., 2022]. Convolutional Neural Network (CNN)-based models have been used for invasive species classification [Hung et al., 2014], mixed-crop segmentation [Mortensen et al., 2016], and weed detection [Di Cicco et al., 2017], while attention-based architectures and channelwise feature selection improve performance in complex landscapes [Cheng et al., 2021]. However, both CNNs and Vision Transformers (ViTs) struggle to distinguish irrigation and land use classes with strong spectral similarity, and multistream fusion of spectral indices (NDVI, Normalized Difference Water Index (NDWI), and Normalized Difference Turbidity Index (NDTI)) improves robustness but still requires large labeled datasets for reliable generalization [Touvron et al., 2021].

Monitoring Agricultural Assets and Land using Remote Sensing. Remote sensing is widely used to analyze agricultural dynamics and how climate, water, and socioeconomic factors influence land-use decisions [Norton et al., 2021]. Deep time-series models on Landsat, MODIS, and Sentinel enable large-scale detection of fallowed and abandoned cropland [Liu et al., 2025; Koley and Chockalingam, 2025], while vegetation indices and learning-based models support pasture identification and condition monitoring [Christofi et al., 2025; Rosanna et al., 2025]. Yet, single-modality cues struggle under spectral ambiguity between fallow and pasture.

Semantic Segmentation Models. CNN-based u-net architectures, such as ResNet and DeepLabV3+, are widely used for crop, field, and irrigation mapping at high spatial resolution [Safarov et al., 2022; Raei et al., 2022]. RNN (e.g., CNN+LSTM and 3DCNN) and attention-based (e.g., TsViT) sequence modeling has been used for crop and agricultural asset detection [Garnot et al., 2020]. Transformer based models such as, ViT, Swin, and masked auto encoder (MAE) architectures improve multi-temporal land use and crop segmentation [Khan et al., 2023; Chen and Li, 2025]. Recently, foundation models, such as Prithvi and geospatial embeddings like AlphaEarth, further introduce largescale environmental contexts for agricultural analyses, including fallowing land segmentation [Brown et al., 2025; Li et al., 2023]. (more details in Appendix Section).

3 Methodology

3.1 ProblemFormulation

We formulate agricultural land use activity mapping from remote sensing as spatio-temporal semantic segmentation. Given multi-modal observations collected over the growing season, our objective is to predict the probability of each pixel belonging to one of four land use classes: fallow, pasture, cropland, or none. For each sample, the input consists of: (i) a spatio-temporal satellite image X ∈ RT×H×W×C , where W, H, C, and T denotes satellite image width, height, channel, and selected time window; (ii) a spatial environmental context embedding from AlphaEarth A ∈ RH×W×DA , where DA is the dimension of the embedding space; and (iv) a land use mask L ∈ {0, 1, 2, 3}H×W , indicating coarse land categories such as crop/cultivated land, developed, grassland, and forest land areas. The goal is to learn a segmentation model,

Let Pi,j denote the predicted class-probability vector at pixel (i, j) in the image P. Final labels are obtained via ˆ Yi,j = arg maxk P (k) i,j .

Given ground-truth masks Y ∈ {0, 1, 2, 3}H×W , the model parameters θ are learned by minimizing loss L (defined in the next section):

3.2 ProposedModel: AG-STELLA

We propose AG-STELLA (shown in Figure 2), an agronomic knowledge-guided spatio-temporal segmentation model for agricultural land use activity mapping. The model comprises adaptive temporal window selection, environmental time-series conditioning, transformer-based spatio-temporal encoding, change-aware feature aggregation, land-aware decoding, and knowledge-guided training objectives.

Prithvi MAE Temporal Tokenizer. Given a time series image X ∈ RT×H×W×C , we encode it using an MAEpretrained Prithvi (300M) Vision Transformer. The encoder partitions X into patches of size (16 × 16) and maps them to a token sequence of length L = T · Hp · Wp, where Hp = H 16 ,Wp = W16 with embedding dimension D, followed by stacked self-attention layers that jointly model spatial context and temporal dependencies. The output is a token representation, Zt ∈ RL×D , together with intermediate token embeddings {Z (ℓ) t } extracted from selected layers, providing hierarchical spatio-temporal representations. As the Prithvi encoder is pretrained via masked autoencoding on large-scale satellite time series with temporal positional embeddings, it yields noise-robust spatial tokens that preserve temporal ordering across monthly observations, i.e.,

Adaptive Temporal Window Selection. We adaptively select a contiguous four-month satellite image window by maximizing phenological change across the growing season. Given available months M = {1, . . . , T}, we first compute vegetation index (NDVI) maps Vt ∈ RH×W from the satellite imagery after filtering cloud-contaminated observations. We then compute temporal change, et, as follows:

To select the optimal window, we maximize the temporal changes within the selected window:

This mitigates the effects of cloud contamination and missing acquisitions while preserving the contiguous temporal window that captures phenological transitions, which are essential for land use activity detection. Note that we fix the temporal window length to four months following the Prithvi encoder's fixed-length temporal clip design.

Temporal Feature Fusion (TFF). We reshape Z into a spatio-temporal token grid: Z ′ ∈ RT×(HpWp)×D . To explicitly capture phenological transitions, we compute absolute temporal differences between consecutive frames:

The resulting Temporal Transition Latent Space (Ftemp) encodes the magnitude of temporal variation at each spatial location. This emphasizes regions undergoing strong temporal change while suppressing temporally stable background areas (shown in 3). This latent representation provides a compact and discriminative summary of land use activity dynamics.

AlphaEarth and Land Use Prior-Conditioned Decoder. We design an AlphaEarth (geospatial context-aware) and land use-aware decoder that integrates temporal dynamics, geospatial context, and coarse land use priors to produce pixel-level land use predictions. The decoder progressively upsamples the temporal transition latent map while incorporating multi-scale contextual features, AlphaEarth conditioning, and land use-based spatial regularization.

Hierarchical Spatial Context. To preserve spatial structure and boundary information, we derive context-focused skip features from intermediate encoder representations Z (ℓ) , extracted from layers ℓ ∈ {2, 5, 8, 11}. After temporal aggregation across the T-month horizon, the resulting skip features are reshaped into spatial maps:

The change-focused latent representation (Ftemp) is used as the decoder bottleneck, while the skip features {S(ℓ) } are fused at corresponding decoding stages via feature concatenation:

where D(·) denotes a decoder update at scale k. This skip connection facilitates gradient flow across network depths and retains high-resolution spatial cues during decoding.

AlphaEarth Conditioning (AEC). Given a decoder feature map x(12) ∈ Rd×H×W , we incorporate geospatial context

(AlphaEarth embedding) via feature-wise linear modulation:

where ϕalpha(·) denotes a learnable convolutional projection. The conditioned feature map is then computed as

where ⊙ denotes elementwise multiplication. This conditioning provides spatially varying feature modulation informed by local geospatial context and ensures consistent interpretation of temporal transition signals.

Land Use-Aware Spatial Gating (LUASG). To enforce coarse spatial priors, we utilize land use mask LH×W (normalized to M ∈ [0, 1]H×W ). Learnable soft gates modulate both contextual features and AlphaEarth conditioned feature (xalpha) representations as

where ϕland(·) denotes a channel-wise projection. This mechanism suppresses activations in non-agricultural regions while preserving smooth spatial transitions.

Segmentation Head. The final decoder features are mapped to per-pixel class logits via a lightweight prediction head composed of a 1×1 convolution, and converted to class probabilities through a softmax operation:

where ϕhead denotes 1 × 1 convolution. This formulation yields spatially consistent pixel-level predictions that integrate temporal transition cues and contextual priors.

3.3 Knowledge-Guided Training Objectives

We train the model using a composite objective that combines standard segmentation losses with two knowledge-guided regularization terms derived from hydroclimatic constraints and temporal transition dynamics:

The pixel-wise cross-entropy loss is defined as:

where P (k) i,j denotes the predicted probability of class k at pixel (i, j) and Y (k) i,j is the ground-truth label.

Furthermore, to improve semantic segmentation under class imbalance, we apply Dice loss:

Here, P (k) i,j and Y (k) i,j denote the predicted probability and ground-truth indicator for class k at pixel (i, j), respectively, and ϵ is a small constant added for numerical stability.

Hydroclimatic Consistency Loss. Agricultural land use decisions, particularly fallowing, are influenced by hydroclimatic conditions. To encode this prior, we construct a spatially dense ET-precipitation correlation map. Let {Et(u, v)}6t=1 and {Rt(u, v)}6t=1 denote evapotranspiration and precipitation time series at coarse resolution. For each coarse pixel (u, v), we compute the Pearson correlation:

where Cov(·) and Var(·) are computed over t = 1, . . . , 6.

The correlation map is normalized via a sigmoid transformation and upsampled to image resolution (α ∈ [0, 1]H×W ). Then, using soft class probabilities P k i,j , we compute classwise expected correlation values:

We enforce the hydroclimatic constraint that fallowed regions exhibit higher ET-precipitation correlation than actively vegetated classes by minimizing:

where m is a margin. We set m = 0.2, based on empirical separation observed in the training set between classconditional hydroclimatic statistics.

Temporal Transition Latent Space Loss. To guide the Temporal Transition Latent Space to encode meaningful seasonal dynamics, we provide auxiliary supervision derived from vegetation index variations. Given NDVI maps {Vt}Tt=1 , we compute the average temporal transition magnitude:

The transition map ∆V is downsampled to ∆V ′ ∈ R(HpWp) to match the spatial resolution of Ftemp. We then minimize the following consistency loss:

where Norm(·) denotes per-sample normalization.

4 Experimentation and Results

4.1 Dataset Details

We construct a multi-source agricultural land use activity dataset across three U.S. states: Utah (UT), Colorado (CO), and Washington (WA). land use labels are obtained from state-specific authoritative sources, including the Utah WaterRelated Land Use (WRLU) dataset (2023) 1 , the Washington State Department of Agriculture Agricultural Land Use dataset 2 , and the Colorado Division of Water Resources GIS dataset 3 . The study period spans 2018-2020 for WA, 2020 for CO, and 2024 for UT. Original land use categories are unified into three activity classes (fallow, pasture, and cropland) for consistency across states.

We collect monthly Sentinel-2 surface reflectance imagery (April to September) via Google Earth Engine 4 for all study years (applying cloud filtering of 50% or lower). Following standard remote sensing procedures, images are segmented into non-overlapping patches of size 224×224 pixels [Hoque et al., 2025]. We retain only patches overlapping labeled agricultural regions, using land-cover masks derived from state data and the National Land Cover Database 5 . Auxiliary climate variables, including evapotranspiration (ET; 1km resolution) 6 and precipitation (800m resolution) 7 , are spatially aligned to satellite pixels using nearest-neighbor matching. All satellite imagery, labels, and auxiliary variables are coregistered to generate aligned multi-modal training samples. The constructed dataset contains 3129 samples for CO, 4366 for UT, and 13861 for WA, with strong class imbalances; for example, UT and CO contain only 6.9% and 9.7% fallowed land (detailed in Appendix).

1 https://dwre-utahdnr.opendata.arcgis.com/pages/wrlu-data

2 https://agr.wa.gov/departments/land-and-water/ natural-resources/agricultural-landuse

3 https://dwr.colorado.gov/services/data-information/gis

4 https://earthengine.google.com/

5 https://www.mrlc.gov/

6 https://earlywarning.usgs.gov/fews/product/460/

7 https://prism.oregonstate.edu/

4.2 Experimentation Setting

We split each state's dataset into 70% for training, 15% for validation, and 15% for testing. We follow a spatially disjoint [Mandal et al., 2025] partitioning strategy to avoid spatial leakage and to ensure that training, validation, and test samples originate from non-overlapping geographic regions.

Model implementation was conducted using PyTorch and executed on an NVIDIA A40 GPU. We initialize the spatiotemporal encoder with a pretrained Prithvi MAE model and fine-tune only the last six transformer layers, while keeping earlier layers frozen to balance adaptation and overfitting.

We optimize hyperparameters via grid search over learning rates {10-4 , 5 × 10-5 , 10-5 }, batch sizes {8, 16}, and loss weighting coefficients λDice, λhyd, λtemp ∈ {0.1, 0.2, 0.5, 0.7}. The optimal configuration is selected based on macro-F1 performance on the validation set (More details in Appendix).

4.3 Result Analysis

Comparison with Baseline Models. We employ statespecific training and evaluate the models independently for each state to capture regional agricultural heterogeneity. Our model is compared against 12 fully finetuned baselines, including vision-only models (ResNet, ViT, Swin, DeepLabV3+, CLIP), temporal vision models (TSViT, VideoMAE, UNet3D, ConvLSTM), and Earthobservation foundation models (Prithvi, AlphaEarth+MLP, AlphaEarth+CNN). All baselines are trained on identical splits using the same supervision and evaluation protocol to ensure a controlled comparison.

Across all states, our model achieves the highest macro accuracy (mAcc) and class-wise F1-score (shown in Table 1), with the most pronounced improvements observed for fallow and pasture (the two most challenging and underrepresented agricultural land use activity classes). Compared to the strongest vision and temporal baselines, our approach improves macro accuracy from 64.3% to 70.0% in Utah (+8.9% relative), from 68.9% to 74.3% in Colorado (+7.8%), and from 74.1% to 85.4% in Washington (+15.3%). These consistent improvements in minorityclass performance (including substantial increases in Fallow and Pasture F1-score) indicates improved discrimination of land use activity beyond static appearance or coarse temporal trends (shown in Figure 3). Note that performance for fallow and pasture remains comparatively lower in Utah and Colorado due to strong spectral similarity with active cropland and low prevalence (severe class imbalance in the dataset). We further analyze performance against foundation models. Relative to Prithvi, our method improves macro accuracy by +11.0% (UT), +10.9% (CO), and +15.3% (WA), with notable gains in Fallow and Pasture F1-score across all regions. Compared to AlphaEarth-based models, our approach achieves additional improvements of 8-16% in macro accuracy, with especially large gains for pasture, including an increase from 58.6% to 74.4% (+27.0%) in Washington. These results suggest that while foundation models provide strong generic representations, they remain limited in resolving fallow-pasture ambiguity. By explicitly modeling land use activity, our approach delivers more balanced and region-robust performance, particularly for fallow and pasture (more details in Appendix).

Transfer Learning for Cross-State Agricultural Land Use Activity Mapping. In the cross-state setting, models are trained on data pooled from all three states. Prior studies have shown that strong state-level heterogeneity in agricultural practices often favors state-specific training for optimal performance [Mandal et al., 2025]. We therefore evaluate whether cross-state training can transfer effectively and achieve performance comparable to state-specific models. In Table 3, the baseline denotes the best state-specific model and serves as an approximate upper bound. Compared to its own state-specific performance (Table 1), AG-STELLA exhibits a markedly smaller fallow performance drop (-3.4% UT, -4.7% CO, -2.4% WA) compared to other models. In particular, models based on Prithvi and AlphaEarth suffer fallow degradation (e.g. Prithvi: 41.4% UT, 40.5% CO, 75.2%WA; AlphaEarth+CNN: 37.3% UT, 36.6% CO, 74.0%WA). In contrast, AG-STELLA achieves F1-score of 43.4% (UT), 44.4% (CO), and 82.3% (WA) for fallow land (outperforming the comparing models by +4.8%, +9.6%, and +9.6%, respectively). This demonstrates AG-STELLA's ability to transfer knowledge across the states.

Error Analysis (Confusing Crop Types). To characterize dominant failure modes, we analyze crop types most frequently misclassified as Fallow or Pasture. For each state, we identify the crop types along with the proportion of pixels predicted as these classes (See Appendix). Across states, wheatrelated crops (e.g., winter wheat and spring grain) consistently appear among the top fallow confusions, particularly in Colorado and Washington (potentially reflecting similar spectral and temporal signatures during dormant or post-harvest periods). In contrast, grass and hay dominated crops (e.g., bluegrass and grass hay) dominate the top pasture misclassifications (likely because their sustained herbaceous cover, limited intra-seasonal spectral variation, and management practices closely resemble actively grazed pasture systems)(see details in Appendix).

5 Ablation Studies

We conduct an ablation study to quantify the contribution of each architectural and loss component in AG-STELLA. Starting from the full model, we systemically remove one component at a time while keeping all other settings fixed. As shown in Table 2, each component contributes positively to performance, with the full model achieving the highest macro F1score. In particular, removing TTLSP or TFF leads to the largest performance degradation which highlights the importance of explicitly modeling temporal land use transitions and multi-temporal feature fusion. Furthermore, ablating AEC, LUASG, or HCL consistently reduces performance, indicating that AlphaEarth conditioning, land use-aware spatial gating, and hydroclimatic consistency jointly improve robustness and generalization.

6 Conclusion

In this work, we present AG-STELLA, a knowledge-guided, end-to-end trainable spatio-temporal model for agricultural land use activity mapping. Our model integrates temporal vegetation dynamics, AlphaEarth geospatial context, and hydroclimatic constraints. Experiments across three U.S. states show consistent gains over strong spatio-temporal baselines and demonstrates the effectiveness of combining pretrained foundation models with domain-informed inductive biases. Beyond accuracy, AG-STELLA aligns predictions with hydro-climatic constraints which enables scalable agricultural monitoring in water-stressed regions. Moreover, AGSTELLA supports practical applications, including drought response, water allocation, and groundwater management through monitoring seasonal land-use transitions. It can also aid agricultural policy verification and analysis of fallowing behavior impacting regional crop production and economic planning.

Despite these promising results, we assume a single dominant agricultural land use activity within a growing season, which does not capture cross-season or within-year transitions such as mid-season fallowing. Pasture land remains inherently ambiguous, as it can be managed under both irrigated and non-irrigated regimes that lead to overlapping temporal and hydro-climatic signatures. Additionally, hydroclimatic constraints are derived from coarse-resolution ET and precipitation data, which may not fully reflect field-scale management variability. Therefore, future work will focus on cross-season and multi-year modeling to capture land use transitions across planting cycles. Incorporating irrigation intensity signals, higher-resolution hydro-climatic data, and probabilistic or multi-label formulations, along with scaling via weak supervision and active learning, offers a promising path toward large-scale operational agricultural monitoring.

Acknowledgments

This material is based upon work supported by the AI Research Institutes program supported by NSF and USDANIFA under the AI Institute: Agricultural AI for Transforming Workforce and Decision Support (AgAID) award No. 2021-67021-35344. This work was partially supported by University of Virginia Strategic Investment Fund award number SIF160.