Unified Sequence Modeling for Remote Sensing: A Parameter-Efficient Foundation Model via Prompt-Driven Granularity Alignment

1 Introduction

Comprehensive interpretation of Earth observation imagery demands a duality of capabilities. A practical system must provide high-level semantic understanding and low-level geometric localization with fine spatial precision. However, the remote sensing community has long developed these capabilities along separate trajectories. High-level recognition [Yu et al., 2020; Zhang et al., 2022; Zheng et al., 2023] and reasoning [Lobry et al., 2020; Chappuis et al., 2022; Wang et al., 2024; Wang and Ghamisi, 2024] are typically addressed by task-specific classification [Guo et al., 2022; Duan et al., 2025; Liu et al., 2025] and multimodal understanding models, while dense localization relies on dedicated segmentation backbones and losses [Wang et al., 2022a; Zhang et al., 2024; Sun et al., 2024]. This task heterogeneity structurally decouples semantic understanding and geometric precision, leading to fragmented pipelines with taskdependent heads, objectives, and post-processing rules.

Vision-language models have shown strong generalization through language supervision and instruction-style prompting [Radford et al., 2021; Li et al., 2023; Dai et al., 2023; Liu et al., 2023a; Zhu et al., 2023]. In remote sensing, largescale VLMs improve multimodal understanding and offer a promising route toward unified perception [Kuckreja et al., 2024; Hu et al., 2025; Zhan et al., 2025]. Yet current progress often follows scaling-law driven expansion to billions of parameters, while localization is commonly treated as continuous regression that mismatches discrete token generation in language modeling. These trends motivate a more efficient, sequence-centric formulation for studying semanticgeometric synergy in unified remote sensing perception. Figure 1 contrasts fragmented pipelines with a unified interface.

This paper studies a unified formulation for remote sensing perception and reasoning under a single generate-thenparse workflow. We focus on three aspects. First, we examine whether heterogeneous objectives can be reduced to one unified decoding interface without task-specific heads. Second, we evaluate whether a compact model with 0.23B parameters can remain competitive on language-centric benchmarks while still supporting dense geometric prediction. Third, we study whether multitask joint training provides positive transfer across task families under matched budgets. This matters in remote sensing because classification, captioning, VQA, and localization are often built and maintained as separate pipelines, which increases implementation overhead. A single prompt interface can reduce that fragmentation by sharing one backbone and one decoding stack across multiple outputs. More broadly, the goal is not to replace every dedicated specialist, but to provide a compact shared representation in which language and geometry can be learned together and compared under the same interface.

To address these aspects, we propose RS-Florence, a prompt-driven framework that reformulates heterogeneous tasks into a single sequence-to-sequence language modeling problem. The key idea is to introduce discrete geometric tokens that quantize spatial coordinates into a location vocabulary. The model generates a single interleaved sequence in which semantic tokens and geometric tokens share the same decoding space, and task outputs are obtained by deterministic parsing. We instantiate RS-Florence with a compact vision-language encoder-decoder backbone that combines a hierarchical vision tower based on DaViT with a BART-style encoder-decoder [Ding et al., 2022; Lewis et al., 2020].

We emphasize compact modeling and empirical cross-task effects. Our evaluation reports task metrics together with parameter counts, enabling transparent accuracy-efficiency comparisons against lightweight task-specific models and large remote sensing VLM baselines [Yu et al., 2020; Wang et al., 2022a; Kuckreja et al., 2024; Hu et al., 2025; Zhan et al., 2025]. Despite its compact size, RS-Florence achieves competitive performance across multiple task families. Our experiments suggest a positive effect from multitask training. Joint training improves over single-task fine-tuning under matched budgets, indicating that semantic objectives can provide useful signals for geometric localization.

Our main contributions are:

• We propose RS-Florence, which reformulates heterogeneous remote sensing tasks into a single prompt-driven sequence-to-sequence language modeling problem via discrete geometric tokens, bridging semantic reasoning and geometric localization within a shared decoding space.
• We evaluate a compact 0.23B model against taskspecific systems and larger remote sensing VLMs, providing an accuracy-efficiency comparison for unified remote sensing modeling.
• We analyze multitask joint training in remote sensing, showing improvements over single-task baselines under

matched budgets and indicating useful interaction between semantic and geometric objectives.

2 RelatedWork

2.1 FromTask-Specific Specialists to Unified Generalists

Remote sensing vision has been dominated by task-specific specialist models that achieve strong performance within narrow problem definitions. Compact CNN and Transformer architectures remain highly competitive for scene classification on established benchmarks [Xia et al., 2016; Cheng et al., 2017; Yang and Newsam, 2010], and modern segmentation backbones deliver accurate dense prediction under fully supervised protocols [Wang et al., 2021; Wang et al., 2022a; Zhang et al., 2024; Sun et al., 2024]. These specialist designs have advanced efficiency and accuracy for their respective tasks, and they form the backbone of many operational RS pipelines [Yu et al., 2020; Zhang et al., 2022; Zheng et al., 2023; Guo et al., 2022; Duan et al., 2025].

Despite their success, specialist models commonly suffer from architectural silos. Their interfaces are tightly coupled to task-specific heads, losses, and output representations, which hinders knowledge transfer between semantic reasoning and geometric localization. As a result, semantic understanding learned for recognition is not naturally reusable for dense delineation, and geometric precision optimized for segmentation does not translate into language-grounded interaction. This structural decoupling limits holistic understanding of remote sensing scenes, where a system is expected to jointly explain scene semantics and localize objects or regions with fine geometry.

In general computer vision, unified perception frameworks have begun to reduce these silos by exposing a shared interface across heterogeneous tasks [Wang et al., 2022b]. However, remote sensing scenes introduce unique challenges such as extreme scale variation, dense instance packing, and strong orientation effects, which make direct transfer of unified CV systems non-trivial. Our work follows the unified perception direction while emphasizing a sequence-based unification tailored to remote sensing, so that semantic and geometric predictions can be produced within one coherent decoding space. Related RS efforts toward language-aligned dense prediction have explored open-vocabulary or training-free settings [Li et al., 2025; Ye et al., 2025], but they typically remain focused on specific regimes or rely on heavy pretrained components rather than a compact unified interface.

2.2 RS-VLMs: Scaling Versus Efficiency and Modality Alignment

Vision-language models (VLMs) have demonstrated that language supervision and instruction-style prompting can unify multiple vision capabilities behind a single interface [Radford et al., 2021; Li et al., 2023; Dai et al., 2023; Liu et al., 2023a; Zhu et al., 2023]. Motivated by this progress, recent remote sensing vision-language models adapt general VLMs or LLMs to remote sensing imagery and show strong gains on captioning and VQA [Kuckreja et al., 2024; Hu et al., 2025;

Zhan et al., 2025; Chappuis et al., 2022; Wang et al., 2024]. These systems largely follow a scaling-law trajectory, where broader capability is pursued by increasing model capacity and relying on large backbones to absorb domain shifts.

A key limitation of this trend is that parameter scaling alone does not resolve the modality mismatch between language generation and geometric localization. Many VLMs treat localization through continuous regression or add external detection components, which results in an implicit twostream design where discrete token generation governs language while geometry is handled by a separate prediction mechanism. This decoupling weakens modality alignment and complicates the goal of a truly unified inference interface. In addition, remote sensing applications often face strict latency and compute constraints, which makes compact modeling practically important.

RS-Florencetargets these issues by prioritizing modality alignment and compact modeling. By introducing discrete geometric tokens, we align spatial signals with language tokens in a shared discrete decoding space, enabling unified generation and deterministic parsing for both semantic and geometric outputs. This design complements parameter-efficient adaptation techniques that reduce training cost [Wang and Ghamisi, 2024], while differing in that our primary objective is a unified sequence-centric interface rather than improving a single downstream task.

2.3 Vision as Sequence Modeling

Our formulation is rooted in the vision-as-sequence paradigm, which casts visual perception as conditional language modeling and unifies structured prediction through sequence generation [Chen et al., 2022; Wang et al., 2022b; Xiao et al., 2024]. Pix2Seq demonstrates that object detection can be modeled as token prediction with structured decoding, while later unified systems extend sequence modeling to broader task families by generating text and structured outputs in a shared vocabulary [Chen et al., 2022; Wang et al., 2022b; Xiao et al., 2024]. These advances provide a principled foundation for unifying heterogeneous tasks under one generate-then-parse workflow.

Adapting this paradigm to remote sensing is not straightforward. Remote sensing scenes exhibit extreme scale variation, dense small instances, and pronounced orientation effects, which amplify sensitivity to coordinate representation, sequence length, and parsing validity. A direct application of natural-image sequence modeling may therefore degrade geometric fidelity or lead to brittle decoding under RS distributions. RS-Florence addresses these challenges by coupling prompt-driven task conditioning with a coordinate quantization strategy, so that geometric localization can be expressed as discrete tokens and optimized jointly with semantic generation. This makes unified sequence modeling practical for remote sensing without reverting to task-specific heads.

RS-Florence extends a lightweight, sequence-based unified paradigm to remote sensing by adapting the Florence-2 sequence-modeling interface to this domain.

3 Method

3.1 Overview

As illustrated in Fig. 2, RS-Florence is a unified promptconditioned sequence modeling framework for remote sensing perception. It contains a DaViT visual tower and a BARTstyle encoder-decoder language tower [Ding et al., 2022; Lewis et al., 2020]. The visual tower extracts hierarchical features V = Eimg(x) with multiple spatial resolutions, which is essential under large scale variance. The language tower projects visual tokens into its hidden space, concatenates them with prompt tokens, and decodes an output sequence conditioned on the multimodal context.

RS-Florence follows unified general-vision models such as Florence-2 [Xiao et al., 2024], but remote sensing introduces heterogeneous supervision and severe task imbalance. We address these issues by unifying the decoding space for language and geometry, and by adopting multitask optimization with temperature-controlled task sampling. Under the same interface, it supports captioning, instruction-style visual question answering, localization, and segmentation. We further instantiate two remote sensing tasks within this interface. Scene classification is implemented by generating category names with <CLS>, and instance segmentation is implemented by composing <OVD> with <R2S>.

3.2 ProblemFormulation: Perception as Language Modeling

We formulate heterogeneous remote sensing perception tasks as conditional sequence generation. Given an input image x and a prompt p, RS-Florence produces an output sequence

where Fθ denotes the decoded sequence produced from the model distribution under a fixed decoding strategy. The output sequence y = {y1, . . . , yL} is generated from a unified vocabulary V that includes natural language tokens and discrete geometry tokens. We model the posterior probability autoregressively:

We write the prompt as a pair

where t is a task token and s is an optional text condition. When a task does not require extra text input, s is empty. The final prediction is obtained by a task-dependent parser

which maps the generated sequence to either a plain-text output or a structured output. This generate-then-parse interface follows Florence-2 and enables a single backbone to serve multiple tasks through prompts [Xiao et al., 2024; Chen et al., 2022].

3.3 Discrete Geometric Quantization

Unifying semantic generation and geometric prediction is hindered by the incompatibility between continuous coordinate regression and discrete token prediction. RS-Florence resolves this mismatch by quantizing spatial coordinates into a finite set of location tokens, so that localization is learned as token classification under the same autoregressive decoding space.

Following Florence-2 [Xiao et al., 2024], we use 1000 discrete location tokens indexed by k ∈ {0, . . . , 999} as a balance between spatial granularity and vocabulary size. For a normalized coordinate u ∈ [0, 1], the quantization operator is defined as

where clip(·, 0, 1) truncates a value into [0, 1]. Given an image with width W and height H, a pixel coordinate (x, y) is normalized as ux = x/W and uy = y/H, and then mapped to a location index by applying Eq. 5. We denote the k-th location token by τk.

This quantization yields a unified serialization for different geometric structures. A bounding box b = (x1, y1, x2, y2) is serialized as a four-token location sequence

followed by its category token. For polygon-based segmentation, each vertex is represented by a pair of location tokens, and the polygon is serialized by concatenating all vertex pairs in order. During inference, the task-specific parser Gt deterministically converts the generated location-token stream into boxes or polygons in the image coordinate system.

3.4 Multi-Task Optimization

RS-Florence is optimized with a unified autoregressive objective over the output tokens. For each training triple (x, p, y), we minimize the negative log-likelihood

For multi-task training over a task set T , the overall objective is

where λt controls the relative importance of each task. We adopt a temperature-controlled task sampling scheme based on dataset sizes:

which reduces the dominance of large datasets when mixing heterogeneous tasks.

3.5 Task Instantiations

Remote-Sensing Scene Classification

Classification is instantiated as category-name generation with the task token <CLS>:

The output ycls contains a single class label, and the final prediction is ˆ ocls = G<CLS>(ycls) through plain-text parsing. To keep the behavior consistent with prompt-based activation [Xiao et al., 2024], we map <CLS> to the instruction: "What is the remote sensing scene category?"

Algorithm 1 Inference for classification and instance segmentation in RS-Florence.

Require: Image x; task type τ; optional query text q

• 1: if τ = CLASSIFICATION then 2: y ← GENERATE(Fθ, x, ⟨<CLS>,∅⟩) 3: return TRIMTEXT(y) 4: else if τ = INSTANCESEG then 5: ydet ← GENERATE(Fθ, x, ⟨<OVD>, q⟩) 6: B ← PARSEBOXES(ydet) 7: for each b ∈ B do 8: pb ← ⟨<R2S>, ENCODEBOX(b)⟩ 9: yb ← GENERATE(Fθ, x, pb) 10: mb ← PARSEMASK(yb) 11: end for 12: return {(b,mb)}b∈B 13: end if

Instance Segmentation via Open-Vocabulary Detection and Region Segmentation

Instance segmentation is instantiated by composing two prompted skills under the same interface. We first predict instance boxes using open-vocabulary detection with <OVD>:

where q is the query text and B is the parsed set of bounding boxes. For each b ∈ B, we then generate a regionconditioned mask using <R2S>:

The final instance set is obtained by aggregating {(b,mb)} over all predicted regions. This box-then-mask decomposition follows region-based instance segmentation designs and benefits from open-vocabulary detection for open-set generalization [He et al., 2017; Li et al., 2022; Minderer et al., 2022; Liu et al., 2023b].

Algorithm 1 summarizes inference for the two newly introduced tasks under the unified prompt-to-sequence-tostructure interface. Here, GENERATE is equivalent to calling Fθ(x, ⟨t, s⟩) with the corresponding task token t and text condition s.

4 Experiments and Empirical Analysis

4.1 Experimental Setup

We evaluate RS-Florence as a unified remote-sensing model on four task families: visual question answering, image captioning, instance segmentation, and scene classification. All tasks are triggered by textual prompts and solved by sequence generation followed by deterministic parsing, so one decoding pipeline is shared across heterogeneous objectives.

Benchmarks and protocols. Our main evaluation suite covers seven benchmarks spanning semantic abstraction and geometric precision. For RS-VQA, we use RSVQA-LR and RSIVQA and report average accuracy in percentage, following common practice [Lobry et al., 2020]. For captioning, we use RSICD and NWPU-Captions and report BLEU-4, METEOR, ROUGE-L, and CIDEr. For instance segmentation, we evaluate on iSAID and report mAP and Recall@100 under the GZSRI protocol [Zheng et al., 2021; He et al., 2023; Huang et al., 2024]. For scene classification, we report Top-1 accuracy on AID, NWPU-RESISC45, and UC Merced [Xia et al., 2016; Cheng et al., 2017; Yang and Newsam, 2010]. We additionally include WHU-RS19 only for representation analysis in Fig. 3, and we exclude it from quantitative tables.

Implementation details. We initialize RS-Florence from Florence-2-base with 0.23B parameters and perform instruction tuning with Discrete Geometric Tokens. Unless otherwise stated, all tasks are trained jointly to evaluate multitask synergy under a unified interface. We train for 10 epochs with batch size 10 on 8× A100 GPUs, using AdamW with learning rate 1e-6 and a constant schedule. We decode with beam size 3 and a maximum generation length of 1024 tokens. Structured outputs are validated and converted to predictions by deterministic parsers with strict checks, and ill-formed sequences are discarded. This standardization keeps training and inference consistent across task families and avoids taskspecific heuristics.

4.2 Comparison with State of the Art

Tables 1-4 summarize the main results on captioning, RS-VQA, instance segmentation, and scene classification, together with parameter counts to reflect the accuracyefficiency trade-off.

Performance against task-specific pipelines. Across task families, RS-Florence remains competitive with specialist systems optimized for a single output format. For captioning, it achieves CIDEr 2.850 on RSICD and 2.013 on NWPUCaptions with balanced BLEU-4, METEOR, and ROUGE-L in Table 1. For RS-VQA, it reaches 88.60 on RSVQA-LR and 83.20 on RSIVQA in Table 2, which is close to strong RS-VQA baselines under matched protocols. For iSAID, it obtains 10.65 mAP and 45.22 Recall@100 under GZSRI in Table 3, improving mAP over FC-CLIP while keeping recall in a comparable range. For scene classification, it achieves 96.92 on AID, 94.20 on NWPU-RESISC45, and 98.52 on UC Merced in Table 4, showing that the same generate-then-parse interface supports recognition without a dedicated classifier head.

Efficiency against large RS foundation models. RSFlorence uses 0.23B parameters, which corresponds to about 1.77% of a 13B model and about 3.29% of a 7B model. Despite the scale gap, it sustains competitive performance on language-conditioned tasks and supports dense prediction through the same sequence representation. These comparisons suggest that discrete geometric vocabularies and prompt-driven decoding can provide strong domain alignment without relying on parameter scaling.

4.3 Analysis of Multitask Synergy

We study whether heterogeneous objectives interfere or cooperate under a shared backbone through a controlled comparison between single-task fine-tuning and multitask joint training under matched parameter count and training budget. The generate-then-parse interface is unchanged, and only the task token and minimal text condition vary across tasks. Optimization follows Eq. 8 with temperature-controlled task sampling in Eq. 9, which balances gradients from imbalanced datasets and stabilizes cross-task updates. For segmentation, we use the same iSAID protocol as the main results and report GZSRI mAP, consistent with Table 3.

Table 5 shows consistent gains from joint training across all task families. VQA Avg Acc increases from 87.90 to 88.60, captioning CIDEr increases from 2.561 to 2.850, iSAID mAP increases from 9.75 to 10.65, and classification Top-1 accuracy increases from 95.40 to 96.92. This pattern indicates that, in our setting, multitask supervision provides complementary signals rather than pervasive negative transfer, and it improves both semantic discrimination and coordinatesensitive decoding under the same backbone capacity.

We further interpret the cross-task effect through a semantic-to-geometric lens as an empirical association. Under joint training, language-conditioned objectives co-exist with improved geometry-centric decoding, evidenced by the iSAID mAP gain in Table 5. Fig. 4 localizes typical false positives and misses, and over-detections concentrate in crowded layouts and texture-confusing backgrounds, while omissions often occur on small instances and thin structures. Together, Table 5 and Fig. 4 support the view that multitask supervision regularizes shared representations and can improve geometric precision without changing the decoding interface.

4.4 Qualitative Visualization and Error Analysis

We provide qualitative visualization to characterize representation changes induced by joint training and the stability of token-based dense decoding. This qualitative evidence supports that joint training improves cross-task consistency and reduces brittle decoding under domain-specific clutter in realworld imagery. These analyses complement the quantitative comparisons by exposing how the unified generate-then-parse interface behaves in challenging remote-sensing scenes.

t-SNE representation analysis. Fig. 3 visualizes scenelevel embeddings from the Florence visual tower for the multitask model and its single-task counterpart under the same backbone setting. We extract one embedding per image and apply t-SNE on AID, NWPU-RESISC45, UC Merced, and WHU-RS19. Across datasets, the multitask model forms tighter clusters with less category mixing, suggesting improved feature robustness and sharper class boundaries. Notably, confusion is reduced among fine-grained categories with similar textures, which is consistent with the stronger recognition results observed under joint training.

Instance segmentation error decomposition and stability. Fig. 4 shows qualitative iSAID results with explicit error decomposition. Over-detection concentrates in crowded layouts and texture-similar backgrounds, while missed regions are more common for small instances, thin structures, and boundary-adjacent pixels. These patterns indicate that density and scale variation remain the primary sources of ambiguity. Meanwhile, the decoded outputs remain structured and spatially coherent in dense scenarios, and the deterministic mapping from discrete tokens to boundaries supports stable delineation and faithful error localization. The difference maps make it easier to distinguish boundary drift from true semantic confusion, which guides future improvements.

5 Conclusion

In this work, we introduce RS-Florence, a compact unified model that casts heterogeneous remote sensing tasks as prompt-driven sequence generation with Discrete Geometric Tokens. By sharing one decoding space for language and geometry, it supports classification, visual question answering, captioning, and instance segmentation without task-specific heads. Experiments across multiple benchmarks show the practicality of this formulation and suggest benefits from multitask joint training under matched capacity. Limitations remain on crowded scenes, small instances, prompt design, and coordinate discretization, leaving clear directions for future refinement within the same unified interface.

Acknowledgments

This research was funded by the National Natural Science Foundation of China under Grant 62376207; in part by the Xidian University Specially Funded Project for Interdisciplinary Exploration, China (No. TZJH2024045), in part by the Innovation Fund of Xidian University (No. YJSJ26022), and in part by the Fundamental Research Funds for the Central Universities, China.