SpecBridge: Spectral Structure Alignment and Transitive Bridging for 3D–2D–Text Pre-Training

1 Introduction

3D understanding tasks [Jain et al., 2025; Zha et al., 2025] aim to decipher geometric shapes and semantic characteristics within physical environments, serving as a critical foundation for autonomous driving [Yang et al., 2025a], virtual reality [Jiang et al., 2024], and embodied agents [Yang et al., 2025b]. Currently, the field is expanding beyond traditional closed-set classification toward open-vocabulary learning. Previous methods [Xue et al., 2024; Tang et al., 2025] primarily leverage the generalization capabilities of 2D Vision-Language Models (e.g., CLIP [Radford et al., 2021]) via large-scale 3D-2D-Text triplet pre-training to project 3D representations into a pre-aligned feature space. However, these methods insufficiently explore the extensive knowledge priors of CLIP to resolve the intrinsic structural discrepancies and semantic heterogeneities between modalities. Therefore, the lack of deep cross-modal alignment limits the performance of open-vocabulary 3D understanding tasks.

* Corresponding author.

The main challenge faced by existing 3D understanding methods is the inherent modality asymmetry between sparse, partial 2D observations and holistic 3D geometries, as shown on the left of Figure 1(a). Existing methods [Xue et al., 2023; Liu et al., 2023] commonly enforce rigid global alignment over one-to-one 2D-3D pairs, which induces optimization ambiguity and hinders the model from learning viewpointinvariant representations. Although some advancements [Gao et al., 2024] attempt to mitigate this through many-to-one joint learning, they remain constrained by linear aggregation in Euclidean feature space, essentially performing superficial appearance matching. Such learned representations remain sensitive to viewpoint changes and local noise, resulting in inaccuracy when capturing the geometric essence. Our motivation is to achieve robust, viewpoint-invariant geometric alignment by incorporating cross-modal correlations with CLIPguided observation inference in the intrinsic eigen-spectral space, as illustrated on the left side of Figure 1(b).

The second challenge is that the profound semantic chasm between discrete 3D coordinates and abstract natural language exacerbates the complexity of modal interaction, as shown in Figure 1(a) right. Most works [Lee et al., 2025; Zhai et al., 2025] commonly lean toward expanding the scale of training data rather than refining the structural interaction mechanism. However, the supervised information obtained solely from coarse global concatenation is still poor, as it fails to extract fine-grained 2D-Text correlations inherent in CLIP. When the model is trained with such imprecise interaction patterns, the semantic mapping between 3D and language remains suboptimal. Inspired by the transitive nature of visionlanguage priors, this paper further utilizes a semantic propagation chain to mitigate the chasm between modalities, as illustrated on the right side of Figure 1(b).

To effectively tackle the above-mentioned challenges, the main contributions of the proposed SpecBridge framework are summarized as follows:

(i) Spectral Eigen-Modal Alignment (SEMA) is designed to introduce cross-modal alignment in the intrinsic eigenspectral space. By combining Laplacian eigendecomposition with CLIP-guided observation inference, SEMA constructs low-frequency geometric harmonics to ensure structural consistency between 3D geometries and sparse 2D observations while effectively decoupling extrinsic viewpoint noise. (ii) Transitive Spectral-Semantic Alignment (TSSA) establishes a 3D→2D→Text semantic propagation chain to mitigate the semantic chasm. By exploiting the spectral affinity between 3D and 2D tokens, TSSA distills fine-grained CLIP priors into 3D representations through transitive soft supervision, achieving robust knowledge transfer throughout the pre-training process. (iii) Extensive experiments demonstrate that the proposed method can effectively integrate spectral geometric priors and linguistic knowledge into 3D models. SpecBridge achieves state-of-the-art performance on zero-shot classification and retrieval tasks, while providing interpretable alignment through fine-grained visualizations.

2 RelatedWork

2.1 Open-vocabulary 3DUnderstanding

Existing 3D understanding tasks focus on evolving toward open-vocabulary representation learning by aligning 3D shapes with 2D vision and language descriptions. As open-vocabulary representation learning is data-hungry, previous approaches including ULIP-2 [Xue et al., 2024] and GPT4Point [Qi et al., 2024b] employ automated annotation engines to build large-scale 3D-2D-Text datasets for establishing a diverse data foundation. By comparison, methods including LLaVA-3D [Zhu et al., 2025] and ShapeLLM [Qi et al., 2024a] are dedicated to representation learning paradigms, which capitalize on pre-trained 2D visionlanguage priors to empower large language models with 3D understanding capacity. However, these learned representations still suffer from the challenge of a significant gap with respect to the inherent complexity of 3D geometries. Recent methods such as Text4Point [Huang et al., 2024] and LidarCLIP [Hess et al., 2024] capitalize on the visual similarities between 2D and 3D modalities, utilizing 2D images as an intermediary to mitigate the difficulty of cross-modal learning. Nevertheless, these aforementioned methods rely on coarse global alignment and thus fail to preserve fine-grained 3D geometric structural information. To overcome these challenges, this paper proposes SpecBridge, which establishes cross-modal correlations in the intrinsic eigen-spectral space for more robust fine-grained cross-modal alignment.

2.2 Homogeneous Modalities Alignment

Homogeneous modality alignment aims to establish correlations between data with different modalities but similar information structures, such as the relationship between 2D and 3D modalities. Methods such as NCLR [Zhang et al., 2025b] and MM-Point [Yu and Song, 2024] enforce the distillation of dense semantic priors from 2D foundation models into 3D encoders via knowledge distillation. In contrast, other approaches including Plain-PCQA [Chai et al., 2024] and MMPCQA+ [Zhang et al., 2023] are inspired by the similar structural information across 2D and 3D modalities, and leverage sequences of 2D views to facilitate texture-aware feature extraction. Specialized tasks in I2PNet [Wang et al., 2025] and HandDiff [Cheng et al., 2024] further explore dense pointto-pixel correlations. However, Euclidean alignment remains susceptible to viewpoint variations and neglects the intrinsic manifold topology. Inspired by spectral analysis, SEMA is developed to capture viewpoint-invariant geometric harmonics in the intrinsic eigen-spectral space, thus ensuring structural consistency beyond superficial appearance matching.

2.3 Heterogeneous Modalities Alignment

Heterogeneous modality alignment attempts to associate data with similar semantics but vastly different structures, such as discrete 3D coordinates and abstract natural language. Recent works focus on mitigating the semantic chasm by integrating 3D encoders into Large Language Models. Typical methods, including PointLLM-V2 [Xu et al., 2025] and MeshLLM [Fang et al., 2025], fuse hierarchical 3D tokens with linguistic embeddings to enable context-aware reasoning. To circumvent the scarcity of paired data, GreenPLM [Tang et al., 2025] leverages massive text data to compensate for limited 3D samples. However, Beacon3D [Huang et al., 2025] reveals that direct 3D-vision-language alignment provides limited meaningful grounding coherence due to the significant modality chasm. While ImOV3D [Yang et al., 2024] has explored an auxiliary strategy that employs 2D modalities as an intermediary, it lacks a structured semantic propagation mechanism, making it difficult to achieve accurate mapping of fine-grained semantics between 3D and Text modalities. To address these issues, TSSA reformulates the 2D modality as a structural intermediary and establishes a 3D→2D→Text propagation chain, effectively avoiding the limitations of direct heterogeneous mapping.

3 Method

3.1 Framework Overview

This section designs a SpecBridge framework to unify geometric structures and linguistic semantics within a spectral representation space, as presented in Figure 2. The overall workflow of the proposed method consists of two main modules: spectral eigen-modal alignment (SEMA) and transitive spectral-semantic alignment (TSSA). To address the challenge of structural asymmetry, SEMA (Figure 2A) performs Laplacian eigendecomposition on modality-specific adjacency matrices A2D and A3D to extract geometric harmonics U ∈ {U2D,U3D}. Feature tokens are subsequently projected into the spectral domain by ̂ F = U ⊤F , where F3D are modulated by viewpoint-aware observation weights derived from CLIP-based matching. By aligning the resulting spectral Gram matrices G2D and G3D via the SEMA loss, the proposed method captures intrinsic structures and transcends superficial appearance matching. Furthermore, TSSA (Figure 2B) is developed to mitigate the semantic chasm by establishing a 3D→2D→Text propagation chain. A correspondence matrix S2D→3D is constructed to correlate heterogeneous modalities, enabling the generation of transitive semantic prototype as pseudo-label P pseudo 3D→Text for token-level alignment with prediction P pred 3D→Text . Optimization by the TSSA loss ensures that fine-grained semantics are effectively distilled into 3D geometries through the 2D visual intermediary.

3.2 Spectral Eigen-Modal Alignment

Euclidean domain limitations and viewpoint sensitivity are transcended by SEMA through the construction of crossmodal correlations in the intrinsic eigen-spectral domain. Rather than matching superficial features, SEMA separates extrinsic noise by capturing inherent geometric signatures through three processes: (i) intra-modal structural correlation modeling, (ii) intrinsic geometric harmonic decomposition, and (iii) spectral coherence synchronization. The detailed process of SEMA is illustrated in Figure 2A.

Intra-modal Structural Correlation Modeling. To capture intrinsic relational dependencies, modality-specific topology graphs are constructed to represent the structural distribution of feature tokens. Given 2D visual tokens F2D = {f 2D i }Mi=1 ∈ RM×d and 3D geometric tokens F3D = {f 3D j }Nj=1 ∈ RN×d , adjacency matrices A2D ∈ RM×M and A3D ∈ RN×N are computed. For modality m ∈ {2D, 3D} with cardinality nm, the entries of Am are defined as:

where i, j ∈ {1, . . . , nm} and σ indicates the graph bandwidth determined by a self-tuning scheme [Zelnik-Manor and Perona, 2004] based on k-NN distances. By modeling the adjacency in the feature space, the semantic and geometric relationships between discrete tokens are effectively encoded into a symmetric correlation graph. This formulation ensures that subsequent spectral analysis reflects the underlying structural consistency of the feature space.

Intrinsic Geometric Harmonic Decomposition. To extract the fundamental structural components from the constructed topologies, a spectral analysis is performed on the adjacency matrices. For adjacency matrix Am, a diagonal degree matrix Dm ∈ Rnm×nm is defined to quantify the connectivity density of each token:

To ensure spectral stability and scale-invariance when aligning M 2D modality patches with N 3D modality tokens, the symmetric normalized Laplacian Lm ∈ Rnm×nm is instantiated as core structural descriptors:

where I indicates an identity matrix. The intrinsic geometric properties of each modality-specific manifold are characterized through the eigendecomposition of Laplacian matrix:

where Um = [um1 , um2 , . . . , umK ] ∈ Rnm×K denotes the first K eigenvectors, termed geometric harmonics, and Λm = diag(λm1 , λm2 , . . . , λmK ) is a diagonal matrix of corresponding eigenvalues. Truncating the spectrum to K components allows SEMA to filter high-frequency noise and provide robust spectral bases for subsequent cross-modal synchronization.

Spectral Coherence Synchronization. This process establishes structural alignment by projecting spatial tokens into intrinsic eigen-spectral domain. For the 2D modality, spectral coefficients are obtained by Graph Fourier Transform ̂ F2D = U ⊤ 2D F2D. Due to the modality asymmetry, not all 3D point clouds can be effectively observed from the corresponding 2D image viewpoint. A series of observation prompts "A photo of an object observed from the [Orientation]" are introduced for 2D image observation inference based on CLIP priors. Notably, since CLIP focuses more on category descriptions rather than viewpoint information, a lightweight linear projection layer is pre-learned to filter out the viewpointrelated information in text representations. More details are provided in the supplementary materials. Furthermore, based on observation inference, weight adjustment is performed on the tokens corresponding to 3D point cloud clusters under the current 2D observation view with decay from the visual center to the unobservable region. The 3D spectral coefficients ̂ F3D integrate this spatial weighting into the spectral projection:

where ω = [w1, . . . , wN ]⊤ represents the observation weight vector, δi denotes the distance of the i-th cluster from the visual center, ¯ δ indicates the average distance, and η defines the minimum weight threshold to preserve essential geometric information even in unobservable regions. To characterize patterns independently of cardinality, a normalized harmonic descriptor Gm ∈ RK×K is formulated by computing the Frobenius-normalized spectral Gram matrix:

where Gm encodes the correlations of all distinct frequency modes [Yi et al., 2017] and reveals the intrinsic structural vibration profile of the modality. The SEMA loss is subsequently defined as the squared Frobenius distance between the normalized spectral Gram matrices of the 3D and 2D modalities:

Minimizing LSEMA enforces structural coherence between the 3D latent representations of the intrinsic 2D visual manifold.

3.3 Transitive Spectral-Semantic Alignment

Existing methods remain plagued by the challenge of the semantic gap between 3D geometry and abstract language description. However, the fine-grained semantic connections between 3D and 2D, as well as between 2D and text, which are respectively realized by SEMA and CLIP, provide an opportunity to overcome this challenge. TSSA thus leverages the 2D modality as an intermediary pipeline to construct a 2D-3D-Text semantic propagation chain, minimizing this chasm. This transitive alignment is executed in two phases: (i) spectral transitive semantic prototype synthesis, and (ii) transitive semantic consistency optimization.

Spectral Transitive Semantic Prototype Synthesis. To implement an efficient semantic propagation mechanism, it is critical to define high-quality 3D-Text semantic prototypes as learning objectives. The fine-grained 2D-Text correlation can be directly derived from the CLIP-prior-based cross-modal token correlation S2D→Text = F2DF ⊤ Text , where FText ∈ RC×d denotes the text tokens. However, since there is no pre-aligned prior between the 2D and 3D encoders, to circumvent potential viewpoint noise in homogeneous modalities, the fine-grained 2D-3D correlation S2D→3D is characterized by cross-modal geometric harmonics:

The synthesized semantic prototypes, denoted as P pseudo 3D→Text ∈ RN×C , are computed by projecting S2D→Text and S2D→3D token-wise affinities through the spectral topology followed by:

Through the intermediary role of the 2D modality, P pseudo 3D→Text achieves fine-grained soft matching between heterogeneous modalities. This process ensures that the resulting prototypes inherit the rich, zero-shot priors of the foundation model while remaining strictly constrained by the intrinsic manifold structure of the 3D data, resulting in a robust, structurally-consistent supervisory signal.

Transitive Semantic Consistency Optimization. This process distills knowledge within the synthesized prototypes into 3D encoder through consistency-driven optimization. The predictive 3D-Text matching P pred 3D→Text ∈ RN×C is derived by computing the normalized similarity between the 3D geometric features F3D and the text embeddings FText of the linguistic tokens:

To inherit the dense semantic priors from the foundation model, the TSSA loss LTSSA is defined as the KullbackLeibler (KL) divergence between P pred 3D→Text and P pseudo 3D→T ext :

LTSSA projects 3D geometric features into linguistic space of CLIP. By aligning in fine-grained soft distribution, the correlation between heterogeneous modal tokens is revealed, enabling the 3D encoder to achieve robust open-vocabulary generalization through cross-modal spectral alignment.

4 Experiments

4.1 Experiment Setting

Datasets. SpecBridge is pre-trained on a consolidated ensemble of four large-scale 3D datasets including Objaverse [Deitke et al., 2023], ShapeNet [Chang et al., 2015], 3DFUTURE [Fu et al., 2021], and ABO [Collins et al., 2022], following the OpenShape protocol to ensure the learning of diverse geometric representations of 875,665 shapes. Performance evaluation utilizes two downstream benchmarks characterized by distinct data properties. ModelNet40 [Wu et al., 2015] comprises 12,311 synthetic CAD models, and Objaverse-LVIS [Deitke et al., 2023] is a long-tailed distributed subset of Objaverse including 1,156 semantic categories.

Implementation Details. SpecBridge is implemented using the PyTorch environment and trained on four NVIDIA RTX A6000 GPUs for 300 epochs. The framework utilizes Point-BERT [Yu et al., 2022] as the 3D backbone encoder to extract geometric features. To ensure stable multi-modal semantic priors throughout the optimization process, a frozen OpenCLIP ViT-bigG-14 [Cherti et al., 2023] model is employed as the vision-language foundation. Optimization is conducted via the AdamW [Loshchilov and Hutter, 2019] optimizer with a global batch size of 1280, incorporating a linear warmup phase during the initial 20 epochs to reach a peak learning rate of 5×10-3 , followed by a cosine decay schedule to 3 × 10-3 .

4.2 Ablation Study

Ablation on Harmonic Order K. The harmonic order K determines the granularity of geometric features within the intrinsic feature spectral domain, with the experiment result shown in Figure 3. Due to the truncated spectral bases fail to encapsulate the fundamental low-frequency harmonics of complex 3D manifolds, lower values of K perform suboptimal accuracy. Conversely, increasing K beyond 32 leads to stagnant or even degraded performance, indicating that high-frequency eigenfunctions introduce extraneous noise and make it difficult to maintain stable internal topology. Therefore, K = 32 is selected as the optimal tradeoff, which ensures sufficient structural details while preserving robustness against local perturbations and viewpoint variations.

Effect of Different Modules. The synergistic effect between the SEMA and TSSA modules in SpecBridge is further verified, with the experimental results presented in Table 1. For 3D-2D visual learning, the complete SEMA module in line 3 significantly outperforms the baseline in line 1 and the incomplete learning setup without CLIP-guided observation inference in line 2, especially on the 2D→3D retrieval task. This indicates that SEMA effectively explores CLIP priors to disentangle extrinsic observation noise and achieve more stable cross-modal structural correlations. In terms of 3DText learning, the spectral semantic propagation chain setup in line 5 maintains a substantial performance margin over the direct token-wise semantic propagation strategy in line 4 on the Text→3D retrieval task. This demonstrates that geometric harmonics preserve more effective structural consistency during the distillation of CLIP semantic priors. In line 6, the complete SpecBridge integrating both SEMA and TSSA achieves excellent performance across all benchmarks, which illustrates that SEMA and TSSA can synergize to enable superior 3D-2D-Text representation learning.

4.3 Main Results

Zero-shot Classification. Table 2 presents the performance comparison results with existing methods. Pioneering frameworks like OpenShape rely on rigid global alignment that hinders structural understanding for 3D shapes. While methods such as ULIP-2 and Uni3D leverage large-scale data expansion, they are still constrained by the performance bottleneck of Euclidean appearance matching. SpecBridge overcomes these limitations by performing alignment learning on low-frequency geometric harmonics to capture the intrinsic geometric essence of 3D shapes. Under the Ensemble (no LVIS) training setup, it achieves Top-1 accuracy on ModelNet that is 4.9%, 6.2%, and 3.4% higher than that of OpenShape, ULIP-2, and Uni3D, respectively. Furthermore, methods like MixCon3D and CLIP-GS focus on information quantity stacking. Although they incorporate multi-view and texture information to optimize 3D representation, they neglect the detailed mapping between such information and 3D geometry. SpecBridge introduces CLIP-prior-based observation inference to guide the model in learning viewpoint-invariant

3D representations. Ultimately, SpecBridge attains state-ofthe-art performance with 90.6% and 56.6% Top-1 accuracy on ModelNet40 and Objaverse-LVIS, respectively, surpassing the sub-optimal PointCache method that relies on test-time optimization, validating the superiority of intrinsic spectral alignment.

Cross-modal Retrieval. Table 3 presents cross-modal retrieval results on Objaverse-LVIS. Because cross-modal retrieval tasks require model to learn high-quality cross-modal mappings, they are far more challenging than classification tasks. Methods such as OpenShape and Uni3D, which rely on data expansion, are clearly unable to capture the finegrained alignment between complex 3D geometries and standard vision-language priors. While MixCon3D and CLIPGS mitigate the modal discrepancy in visual learning by incorporating multi-view and texture information, they are still constrained by the semantic chasm in 3D-Text cross-retrieval tasks. SpecBridge leverages SEMA and TSSA to effectively overcome the modality discrepancies in 3D-2D and 3D-Text learning, respectively, thus achieving state-of-the-art performance. Notably, SpecBridge achieves significant performance improvements of 12.9% and 14.8% on the Text→3D and 3D→Text tasks, respectively.

2D-3D Structural Correspondence Visualization. The visualization in Figure 4 shows patch-to-cluster matching, where point clouds are split into 128 colored clusters and each ViT patch is assigned to its nearest 3D token. OpenShape produces coarse maps because its global alignment lacks part-level supervision. MixCon3D uses many-to-one 2D-3D learning with multi-view images, and roughly separates the statue's upper body (purple) and lower body (green). In contrast, SpecBridge integrates CLIP-guided observation inference into SEMA, learning viewpoint-invariant 3D representations and yielding finer part segmentation, such as the left-hand dagger (red) and pedestal (yellow).

Linguistic Robustness of 3D-Text Alignment. Figure 5 evaluates semantic alignment robustness under increasing textual complexity. While MixCon3D exhibits noticeable performance decay, evidenced by decreasing cosine similarity and rising FD as word counts increase, SpecBridge maintains a remarkably stable and superior alignment profile. Specifically, SpecBridge yields higher similarity and lower FD across the entire linguistic spectrum. The nearzero slopes in the linear fits for SpecBridge substantiate its resilience to information density. Such robustness is facilitated by the TSSA module, which utilizes the 2D modality to propagate dense CLIP priors across the foundational bridge. Consequently, SpecBridge effectively surmounts the semantic chasm, ensuring consistent 3D text grounding for complex descriptive scenarios.

5 Conclusion

This paper presents SpecBridge, a framework synergizing spectral graph theory with transitive semantic learning for 3D-2D-Text pre-training, which consists of two core modules: SEMA and TSSA. SEMA explores CLIP priors to guide 3D-2D modalities in establishing viewpoint-invariant associations within the eigen-spectral domain. TSSA constructs a 3D→2D→Text semantic propagation chain, which serves to mitigate the semantic chasm between 3D geometries and abstract textual descriptions. Benchmark evaluations substantiate the superiority of this approach in surmounting modality discrepancies. Furthermore, given the demand of open vocabulary 3D understanding for complex scenes, as well as the superior geometric representation capability of the recently emerging 3D Gaussian Splatting over point clouds, our future research will focus on two key directions: extending objectlevel understanding tasks to complex indoor and outdoor scenarios, and developing a unified 3D representation learning paradigm compatible with both point clouds and 3D Gaussian Splatting.