SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision

1 Introduction

Spatial intelligence refers to the ability to understand, interpret and reason about the spatial relationships and properties of objects and scenes. This ability is fundamental for realworld tasks ranging from autonomous driving [Zhou et al., 2025; Tang et al., 2026] to robotic manipulation [Song et al., 2025], where understanding and interacting with the 3D environments is essential. Despite significant progress in spatial reasoning and scene understanding, even the most advanced multimodal large language models (MLLMs) face challenges in maintaining cross-view consistency and reasoning about occluded objects. These limitations highlight a fundamental weakness of MLLMs-the reliance on 2D visual-textual data and autoregressive training paradigms-which fails to yield robust and consistent internal 3D spatial representations. This absence of intrinsic 3D spatial awareness is a major bottleneck toward genuine spatial intelligence.

∗Corresponding authors

To address this limitation, one major line of research focuses on providing MLLMs with external spatial priors to compensate for their insufficient intrinsic representations, as illustrated in Figure 1 (a). Among these, prompt-based methods rely on external models or tools to construct prompts enriched with spatial priors, [Yang et al., 2025b; Li et al., 2025d; Qi et al., 2025; Zhu et al., 2025b; Liu et al., 2025]. However, such methods introduce substantial inference overhead and are sensitive to errors from external models. A more direct alternative is to explicitly encode 3D data such as point clouds and depth maps, and project them to the languagealigned space of large language models [Chen et al., 2024;

Li et al., 2025b; Wang et al., 2025b; Zhu et al., 2025a]. However, these methods typically depend on highly customized encoding and alignment modules and will introduce both training and inference complexity. Moreover, the strong 2D pretraining bias of MLLMs poses an additional obstacle to the explicit integration of 3D data.

Instead of emphasizing the construction and utilization of external spatial representations, we argue that MLLMs should internalize spatial awareness to form an internal spatial mental model [Johnson-Laird, 1980; Yin et al., 2025]. Imaging in a practical embodied AI system where acquiring 3D data online is often costly and challenging with strict real-time constraints, MLLMs must rely on their own spatial awareness to construct internal representations of the environment, while maintaining a lightweight architecture to enable fast inference. To this end, recent works attempt to inject spatial awareness into MLLMs by distilling features from 3D vision foundation models (VFMs) [Huang et al., 2025; Chen et al., 2025b]. However, the representation learned via distillation is inherently hard to interpret, limiting a deeper understanding of the internal spatial mental modeling mechanism [Johnson-Laird, 1983] within MLLMs.

Inspired by the probing techniques [El Banani et al., 2024; Chen et al., 2025a], we conduct a pilot study to investigate the quality of spatial representations across different MLLMs under two paradigms: pure autoregressive text supervision and feature distillation. The probing results, as illustrated in Figure 2, reveal the following observations: (1) The quality of an MLLM's intrinsic spatial representation is positively correlated with its level of spatial intelligence. (2) Introducing 3Daware supervision improves the quality of spatial representations. These findings validate the effectiveness of 3D-aware supervision in enhancing the spatial intelligence of MLLMs. However, the performance gap between the depth probing results and the ground-truth further exposes the limitations of the distillation paradigm. This is because distilling features from 3D VFMs constitutes a coarse-grained alignment process that lacks explicit and structured spatial constraints. In addition, aligning feature dimensions during distillation inevitably incurs information loss in the target representations, further weakening the robustness of the supervision signal.

These insights raise a central question: can we identify a more intuitive and fine-grained form of 3D-aware supervision? In response to this question, we propose SpatialSV, a framework that internalizes interpretable and robust 3D spatial awareness in MLLMs via task-oriented visual supervision, as illustrated in Figure 1 (b). Specifically, we perform 2D-to-3D lifting of MLLMs' visual features and align them with explicit spatial representations such as depth maps and point clouds, so as to unlock their intrinsic spatial intelligence. Inspired by 3D VFMs which leverage overlapping yet complementary downstream 3D tasks to learn geometrically consistent representations, we incorporate a set of complementary 3D tasks, including depth estimation, point-cloud reconstruction, and ray-map prediction. Technically, we extract multi-layer hidden visual features from the MLLM and employ projection layers together with decoupled DPT modules to perform multi-task prediction. By combining the coarse-grained guidance of feature distillation with fine-grained, task-oriented constraints, SpatialSV facilitates robust intrinsic spatial representation learning of MLLMs. Extensive experiments across multiple MLLMs and benchmarks demonstrate the superior performance of SpatialSV in enhancing spatial intelligence.

Additionally, the 3D lifting results derived from MLLMs serve as an intuitive and interpretable manifestation of models' internal spatial representations. Through both quantitative and qualitative analyses, we investigate a distinct correlation between 3D lifting results and intra-model spatial intelligence as well as sample difficulties, highlighting the inherent interpretability of SpatialSV. Furthermore, we apply SpatialSV to a semi-supervised setting where 50% of the samples have no textual annotations and achieve performance comparable to full text supervision. This is particularly important for scenarios where 3D text annotations are scarce. In summary, our contributions include:

• We propose SpatialSV, a novel framework that internalizes robust 3D spatial awareness in MLLMs through task-oriented visual supervision, with the 3D lifting results serving as an intuitive and interpretable proxy for intra-model spatial representations.
• We introduce a probe-based representation analysis framework for MLLMs, revealing the limitations of existing supervision paradigms and offering insights into improving supervision signals.
• Extensive experiments demonstrate SpatialSV's strong cross-model and cross-dataset generalization in enhancing the spatial intelligence of MLLMs, and highlight its inherent interpretabiliy, as well as the potential for exploiting unlabeled visual data.

2 Method

2.1 Preliminaries

A typical MLLM consists of a visual encoder Ev and a decoder-only large language model (LLM) F. In the context of multi-view spatial understanding, the multimodal input comprises N multi-view images I = {Ii}Ni=1 along with a language instruction. The visual encoder transforms the images into visual features v0 = Ev(I) ∈ RNv×d aligned with the LLM's input space, where Nv is the number of visual tokens, and d is the feature dimension. The LLM then performs cross-modal feature interaction between the visual features and instruction tokens, producing L layers of multimodal representations {vi , xi }Li=1 , where vi and xi denote the i-th layer of visual and textual features, respectively.

During the fine-tuning stage, the LLM applies causal modeling to the final-layer multimodal features and autoregressively predicts the next text token. The model is optimized by minimizing the standard cross-entropy loss:

where pθ is the probability distribution conditioned on the previous multimodal context, and T is the length of the text token sequence. Notably, only the textual features are explicitly supervised under this training paradigm.

Following prior work [Huang et al., 2025], we inject spatial awareness into the MLLM by explicitly supervising its visual features. Specifically, we introduce a 3D vision foundation model F3D to extract target features that encodes geometric priors, and apply 2D average pooling to match the size of MLLM's visual features: v3D = fpool(F3D(I)) ∈ RNv×d3D , where d3D is the dimension of target features. In addition, we project MLLM's last-layer visual features into the target feature space to ensure dimension compatibility: vproj = ϕ(vL) ∈ RNv×d3D . Finally, the model is optimized by maximizing the cosine similarity between vproj and v3D in a distillation paradigm:

where S(·, ·) denotes cosine similarity. However, this distillation supervision suffers from inherent limitations in the interpretability of representations. In contrast, an intuitive analysis of MLLMs' internal representation quality and its correlation with spatial intelligence is crucial for understanding the spatial mental modeling mechanism within MLLMs and uncovering their capability boundaries.

2.2 Probe-based Spatial Representation Analysis

To acquire deeper insights into the intrinsic spatial intelligence of MLLMs, we investigate the correlation between representation quality and intelligence level under different supervision paradigms. Inspired by prior work [El Banani et al., 2024; Chen et al., 2025a] which investigate the 3D-awareness within visual models, we introduce a probe-based representation analysis framework for MLLMs.

Specifically, we evaluate multiple MLLMs, including Qwen2.5-VL [Bai et al., 2025], LLaVA-NeXT-Video [Zhang et al., 2024], InternVL3 [Zhu et al., 2025c], and LLaVAOneVision-1.5 [Li et al., 2024a], on the multi-view spatial understanding benchmark MindCube-Tiny [Yin et al., 2025]. For each MLLM, we consider three variants: (i) an untuned version, (ii) a version fine-tuned solely with autoregressive text supervision, and (iii) a version that additionally incorporates visual distillation supervision beyond text supervision.

To quantify spatial intelligence, we compute each model's question-answering accuracy on MindCube-Tiny. To access the quality of intrinsic representations, we attach a DPT [Ranftl et al., 2021] module on top of MLLM's visual features for depth estimation. During this process, all MLLM parameters are frozen and only the DPT module is trainable. The depth map annotations for multi-view images in MindCube are obtained using DepthAnything-v3 [Lin et al., 2025], and the DPT head is trained on the training split. After training, we evaluate the depth estimation quality on MindCube-Tiny using the RMSE metric, which serves as a proxy for the quality of MLLM's internal spatial representations.

Figure 2 presents both quantitative and qualitative results, from which we draw two key observations: (1) for a given MLLM, introducing visual distillation supervision substantially improves both intrinsic representation quality and spatial intelligence; (2) as the quality of intrinsic representations improves, the model's spatial intelligence consistently increases. These findings strongly validate the effectiveness of 3D-aware visual supervision in enhancing the spatial intelligence of MLLMs. However, from the depth visualizations in Figure 2, we can observe a noticeable performance gap between the probing depth maps obtained under the distillation paradigm and the ground-truths. Specifically, the probing results appear overly blurred and suffer from significant loss of geometric details. This can be attributed to the inherent limitations of feature distillation supervision: (1) distillation at the feature level is inherently coarse-grained and lacks explicit, structured geometric constraints; (2) feature dimension alignment (e.g., via 2D average pooling) inevitably leads to information loss in the target representations. Collectively, these insights motivate a central question: can we identify a fine-grained and intuitive form of 3D-aware visual supervision that can not only enhance but also interpret the intrinsic spatial intelligence of MLLMs?

2.3 SpatialSV

To overcome the inherent limitations of feature distillation supervision, we propose SpatialSV, a framework that internalizes interpretable and robust 3D spatial awareness in MLLMs via task-oriented visual supervision, as depicted in Figure 3.

3D VFMs [Wang et al., 2025a; Lin et al., 2025] facilitate robust and geometrically consistent representation learning via overlapping yet complementary 3D downstream tasks, spanning point cloud reconstruction, camera pose estimation, depth estimation, and point tracking. Motivated by these work, we select three types of explicit spatial representations that are highly compatible with MLLM's visual feature maps, namely depth maps, ray maps and point cloud maps. These representations encode per-pixel spatial semantics, e.g., depth, camera pose and 3D coordinates, and are therefore more intuitive, structured and fine-grained than abstract feature maps. To obtain the ground-truth data, we leverage 3D VFMs to estimate multi-view depth maps and camera parameters, which are further transformed into ray maps and point clouds (see Supp.A.1 for detailed computations). We align MLLM's visual features with these 3D task outputs to improve the quality of internal spatial representations.

Concretely, we employ a set of two-layer projectors Φ3D to lift MLLM's multi-layer visual features into a shared 3D feature space. Among the hidden visual features of an MLLM, higher layers capture richer cross-modal semantics, whereas lower layers preserve more low-level visual information. Accordingly, we select visual features from four different layers to capture complementary spatial and semantic representations, with the set of selected layer indexes as {li}4i=1 , and the projector set Φ3D = {ϕi}4i=1 . The projectors map the selected layer-wise features into multi-task shared 3D features:

where vi3D ∈ RNv×di3D is the i-th selected layer of 3D-lifted features, and di3D is the corresponding dimension. Subsequently, we introduce task-decoupled DPT modules FDPT [Ranftl et al., 2021] which receive the lifted features as input and independently predict per-pixel depth, camera pose, and 3D coordinates:

where t ∈ {depth, ray, pointcloud} denotes the task type; mt3D is the 3D outputs of task t; and ct ∈ RNv denotes the confidence of mt3D . Specifically, mdep. 3D ∈ RNv×1 corresponds to per-pixel depth values; mray 3D ∈ RNv×6 denotes per-pixel ray representations, where the first three channels encode the ray origin and the last three channels encode the ray direction; and mpoi. 3D ∈ RNv×3 represents perpixel 3D spatial coordinates. Following [Wang et al., 2025a; Lin et al., 2025], we then compute task-specific losses between the MLLM's 3D-lifted predictions and the corresponding ground-truth annotations:

where yt3D is the ground-truth of task t; I{·} denotes the indicator function; Lconf is the confidence loss; Lgrad is the gradient loss applied to the depth residual, penalizing blurred, misplaced, or spurious depth transitions; and Lnorm is the surface normal loss that enforce the alignment between the local surface geometry of predicted points with the ground-truth. See Supp.A.2 for the detailed loss computation procedure.

Consequently, the overall training objective integrates an autoregressive text loss, a coarse-grained feature distillation loss, and fine-grained 3D-aware task-oriented losses:

This approach combines coarse-grained guidance with finegrained geometric constraints, which promotes robust intrinsic spatial representation learning within MLLMs. Moreover, during inference, the 3D-lifted predictions derived from the MLLM serve as an intuitive proxy to assess representation quality and delineate the model's capability boundaries, as discussed in Sec. 3.3.

3 Experiments

3.1 Experimental Settings

Datasets. For training, we use 10k training samples from MindCube [Yin et al., 2025] to fine-tune the model. For evaluation, we focus on two benchmarks, MindCube-Tiny [Yin et al., 2025] and VSI-Bench [Yang et al., 2025a], both of which target spatial understanding from limited ego-centric views. We evaluate models on MindCube-Tiny and the multiplechoice split of VSI-Bench.

Evaluation Metrics. We use Accuracy to evaluate the overall performance on multiple-choice questions, computed by exact matching between model predictions and ground-truths.

For the 3D-lifted depth estimation task, we follow [El Banani et al., 2024] to report RMSE as the evaluation metric.

Implementation Details. We adopt multiple MLLMs as the baselines for SpatialSV, including Qwen2.5-VL [Bai et al., 2025], LLaVA-OneVision-1.5 [Li et al., 2024a], LLaVANeXT-Video [Zhang et al., 2024], and InternVL3 [Zhu et al., 2025c]. For all models, we fine-tune the vision-language projector, the large language model, all 2D-to-3D projectors, and the task-decoupled DPT modules, while freezing the visual encoder. All models are optimized using AdamW with a batch size of 16 and a warm-up ratio of 0.03. We use DepthAnything-v3 [Lin et al., 2025] by default to obtain the ground-truth supervision signals. More detailed hyperparameter settings of all models are provided in Supp.B.2. All experiments are conducted on 8 NVIDIA H800 GPUs.

3.2 Comparison with Baselines

Comparison with Other Supervision Paradigms. We evaluate our approach against the baselines, the pure-text supervision variants, and the feature distillation variants on 6 MLLMs spanning 4 model families (more detailed results present in Supp.C.1). As shown in Table 1, SpatialSV consistently achieves the best performance, demonstrating its strong cross-model generalization capability. Specifically, on MindCube-Tiny, SpatialSV yield overall gains ranging from 3.42% to 12.66% over the pure-text variants, and from 2.97% to 7.81% over the distillation variants. On VSI-Bench, we observe an overall improvement by 4.36% to 6.79% compared with the distillation variants. Notably, SpatialSV achieves an average improvement of 10.0% over the pure-text variants on the relative distance task, while on the more challenging route planning task, the improvement reaches 32.6%. This indicates that SpatialSV effectively enhances the intrinsic spatial representations for 3D scenes within MLLMs, which is critical for estimating spatial attributes such as distance and direction. Moreover, it is worth noting that no VSI-Bench-related sample is incorporated in training data; nevertheless, SpatialSV still achieves significant improvements, highlighting its remarkable crossdataset generalization ability.

Comparison on Multiple Benchmarks. We further evaluate our approach on several additional datasets, including 6 spatial understanding benchmarks, namely Ego3D-Bench [Gholami et al., 2025], Spatial457 [Wang et al., 2025c], ViewSpatial-Bench [Li et al., 2025a], 3DSR-Bench [Ma et al., 2025], SP-Bench [Li et al., 2025c], TopViewRS [Li et al., 2024b], and 2 general benchmarks, CVBench [Zhu et al., 2025d] and MMBench [Liu et al., 2024]. Detailed information about these datasets is provided in Supp.B.1. As shown in Table 2, our method consistently outperforms baseline models and pure-text variants across all benchmarks, indicating that SpatialSV enhances spatial intelligence while also preserving the general understanding capability. This further underscores SpatialSV's strong cross-dataset generalization.

3.3 Interpretability Analysis

In this section, we investigate the inherent interpretability of SpatialSV-based 3D lifting results. We conduct both quantitative and qualitative analyses to reveal their correlations with intra-model spatial intelligence and model-specific sample preferences.

Correlation with Intra-Model Spatial Intelligence. We use the depth estimation performance to quantify the quality of 3D lifting results, and VQA performance to measure MLLMs' intrinsic spatial intelligence, evaluating 8 MLLMs on MindCube-Tiny. In Figure 4, we observe a strong correlation between the quality of 3D lifting results and intramodel spatial intelligence, which is aligned with the findings in Sec.2.2. Qualitatively, Figure 6 shows that MLLMs exhibit varying representation capability under the same spatial context. In the first case, the depth maps and point clouds derived from LLaVA-OneVision-8B lack necessary details of the key object small table, leading to its deviation from the correct option involving this object. In the second case, the failure of Qwen2.5-VL-7B can be partly attributed to its ignorance of chairs, which could help establish cross-view correspondences. In contrast, models that answer correctly can represent these critical objects in their 3D lifting results. These strongly validate the effectiveness of SpatialSV-based 3D lifting results as a proxy of intra-model spatial representations. Correlation with Model-Specific Sample Preferences. For each MLLM, we partition the samples in MindCube-Tiny into four bins according to the quality of estimated depth maps. From Bin1 to Bin4, the samples correspond to progressively high-quality 3D lifting results. In Figure 5, QA accuracy exhibit a clear upward trend across all models, suggesting that samples with unfaithful 3D lifting results tend to be more challenging for the models. As shown in Figure 6, Qwen2.5VL-7B succeeds in the first sample-accurately representing the spatial scene and providing the correct answer-but fails in the second sample in both aspects. This further highlights the correlation between SpatialSV-based 3D lifting results and model-specific sample preferences.

3.4 Ablation Study

Different 3D-Aware Visual Supervision Signals. As reported in Table 3, we examine different 3D-aware visual supervision signals, including 3D VFM features, depth maps, point maps and ray maps. Each supervision signal yields performance gains to varying degrees, with depth maps contributing the most, likely due to their intuitive spatial semantics and relatively low alignment difficulty. When all taskoriented supervision signals are jointly applied, the model significantly outperforms any single-task supervision, highlighting the effectiveness of overlapping yet complementary multi-task supervision for learning robust spatial representations. Furthermore, incorporating VFM features brings additional gains, suggesting that coarse-grained feature alignment and fine-grained task-oriented spatial constraints can mutually guide and reinforce each other.

Different Supervised Visual Feature Layers and Numbers of 2D-to-3D Projectors. As shown in Table 4, we explore different configurations of visual feature layers and projector numbers on Qwen2.5-VL-3B, which contains 37 visual feature layers (e.g., L = 37). Regarding the selection of visual layers, choosing {li}4i=1 = {4, 12, 20, 28} yields the best performance, achieving a 4.18% improvement over supervising only the final visual layer. This is because overly deep layers tend to lack low-level visual details crucial for 3D reconstruction, thereby hindering the learning of 3D spatial representations. In contrast, combining shallow visual details with deep cross-modal semantics which are complementary proves feasible. For projector numbers, employing layer-wise decoupled projectors leads to a 2.64% performance gain compared to using a single shared projector. This can be attributed to the semantic heterogeneity across different visual layers, which necessitates distinct 2D-to-3D mapping functions.

Semi-Supervised Learning with SpatialSV. Given the scarcity of high-quality 3D vision-language data, we explore directly exploiting unlabeled visual data. Since task-oriented visual supervision is independent of text autoregressive supervision, SpatialSV is naturally suited to semi-supervised learning. Specifically, we split the training data into two nonoverlapping subsets and consider 4 configurations: (i) applying pure-text supervision (Ltext) to 50% data; (ii) applying Ltext and SpatialSV (Lspatial) to 50% data; (iii) applying both losses to 50% data and only Lspatial to the remaining unlabeled 50%; (iv) applying Ltext to 100% data. From Table 5, We observe that the semi-supervised setting in (iii) achieves performance close to full text supervision (53.9 vs. 55.3), while yielding a remarkable 14.19% improvement over setting (i). These results highly validate the potential of SpatialSV for effectively exploiting unlabeled raw visual data. Notably, SpatialSV can directly leverage off-the-shelf 3D VFMs to generate 3D task annotations as semi-supervised signals, which is much less costly and easier to obtain than constructing spatial question-answering datasets.

4 Conclusion

We propose SpatialSV, a novel framework that internalizes robust and interpretable 3D spatial awareness into MLLMs through task-oriented visual supervision. SpatialSV performs 2D-to-3D lifting of MLLMs' multi-layer visual features to align with explicit 3D representations, with the 3D-lifting results serving as intuitive and interpretable proxy of models' internal representations. Extensive experiments across multiple models and benchmarks demonstrate SpatialSV's effectiveness in enhancing and interpreting spatial intelligence. Semi-supervised learning results further validate SpatialSV's potential to leverage unlabeled visual data for scalable spatial representation learning.

Acknowledgments

This work is supported in part of the National Natural Science Foundation of China (Grant 62373387) and Shenzhen Fundamental Research Program (Grant No. JCYJ20240813151301003).