G3PT: Unleash the Power of Autoregressive Modeling in 3D Generation via Cross-Scale Querying Transformer

1 Introduction

In recent years, the field of 3D shape generation has experienced significant advancements. One notable approach is the use of Large Reconstruction Models (LRMs) [Hong et al., 2023; Tochilkin et al., 2024], which convert images into 3D shapes through a pipeline that employs transformers [Vaswani et al., 2017] to create and optimize implicit 3D representations with multi-view image supervision. Another line of approach extends 2D diffusion models [Rombach et al., 2022] into the 3D domain, aiming to combine multi-view images into consistent 3D shapes using techniques such as sparse view reconstruction [Li et al., 2023] and score distillation sampling [Poole et al., 2022]. However, these methods heavily depend on the fidelity of the multi-view images and often struggle to generate high-quality meshes, particularly in capturing intricate geometric details. To mitigate this issue, a newer paradigm [Zhang et al., 2023a] leverages 3D variational auto-encoders to compress high-resolution point clouds into a compact latent space before performing diffusion to directly generate 3D shapes. Despite its potential, this approach is limited by the lengthy training time and the lack of a scaling strategy, which constrain its effectiveness and scalability.

In parallel, the emergence of autoregressive (AR) Large Language Models [Brown et al., 2020] and multimodal AR models [Liu et al., 2023] has opened a new era in artificial intelligence. These models demonstrate exceptional scalability, versatility, as well as generalization and multimodal capabilities. At the core of these AR models is the tokenizer [Esser et al., 2021], which transforms diverse data into discrete tokens, enabling the model to employ self-supervised generative learning for next-token prediction.

AR models have also made notable advancements in visual generation, leveraging their sequential processing capabilities to construct images as raster-scan token grids [Yu et al., 2023]. However, given the unordered and unstructured nature of 3D data, extending next-token prediciton to 3D generation tasks remains a hurdle. For example, MeshGPT [Siddiqui et al., 2024] and its successors tokenize serialized mesh data with a GNN-based encoder [Zhou et al., 2018] and rely on manually defined sequence ordering [Wu et al., 2024b]. Despite the promising results on shapes with simple topology, they generally fail to represent complex geometry. On the other hand, recent attempts that encapsulate 3D shapes as structured 3D volumes [Cheng et al., 2023; Xiang et al., 2024] or 2D triplanes [Wu et al., 2024a] also struggle to learn effective feature representations from unordered 3D data, due to the lack of compression expressivity, robustness, and scalability in representing high-quality geometry.

Remarkably, we argue that 3D data inherently exhibits level-of-detail characteristics, with a natural sequential relationship across different scales - a concept well established in 3D rendering [Lindstrom et al., 1996] and reconstruction [Zhang et al., 2021]. In light of this insight, we introduce G3PT - a scalable, coarse-to-fine 3D native generative model that effectively maps unordered point-based 3D data into discrete tokens at various levels of detail, creating a sequential relationship ideally suited for autoregressive modeling. Unlike the recent Visual AutoRegressive [Tian et al., 2024a] (VAR) model, which also use "next-scale prediction" but rely on average pooling and bilinear interpolation (operations poorly suited for unordered data), G3PT employs Cross-scale Querying Transformer (CQT), which uses crossattention and learnable queries to connect tokens across different scales, to effectively enable global integration of information without imposing a specific token order.

The training of G3PT comprises two stages, namely Crossscale Vector Quantization (CVQ) and Cross-scale AutoRegressive modeling (CAR). In CVQ, we employ a transformerbased tokenizer to encode high-resolution point clouds into latent tokens and decode them into 3D occupancy grids through querying points [Zhang et al., 2023a]. During this process, CQT is utilized to decompose the latent feature into discrete tokens at various scales, yielding a level-of-detail 3D representation for residual vector quantization [Tian et al., 2024a] (Figure 1 (a)). With the cross-scale quantized tokens, the CAR process of G3PT begins at the coarsest scale with only one token, and the transformer predicts the next-scale token map conditioned on all previous ones by reusing the CQT for dimension alignment (Figure 1 (b)). This CAR approach provides G3PT with a versatile 3D generation pipeline, which seamlessly supports diverse conditional modalities, including image-based and text-based inputs. Extensive experiments show that G3PT not only surpasses previous LRMs and diffusion-based 3D generation methods in terms of generation quality, but also, for the first time in 3D generation, reveals distinct scaling-law behaviors.

In summary, the key contributions of this work are:

• The introduction of the first cross-scale autoregressive modeling framework for generating unordered data, offering insights to AR models on the 3D generation task.
• The development of a Cross-scale Querying Transformer (CQT) that tokenizes 3D data into discrete tokens at varying scales, enabling sequential coarse-to-fine AR modeling.
• Demonstration through extensive experiments that G3PT sets a new state-of-the-art in 3D content creation, outperforming previous LRMs and diffusionbased methods.

2 RelatedWork

Large reconstruction models. Extensive large-scale 3D datasets [Deitke et al., 2023; Deitke et al., 2024] have enabled the development of LRMs [Hong et al., 2023; Tochilkin et al., 2024], which utilize transformers to map image tokens to triplanes with multi-view supervision. Instant3D [Li et al., 2023] and MeshLRM [Wei et al., 2024] extend LRM from single-view to sparse multi-view inputs by integrating a multi-view diffusion model. Methods like InstantMesh [Xu et al., 2024a] and CRM [Wang et al., 2024] incorporate Flexicubes [Shen et al., 2023] for direct mesh optimization. To improve rendering efficiency, LGM [Tang et al., 2024] and GRM [Xu et al., 2024b] replace NeRF with 3D Gaussians [Kerbl et al., 2023]. However, these approaches often prioritize minimizing rendering loss over explicit mesh generation, resulting in noisy and coarse geometry.

3D native generative models. Generating 3D content with direct 3D supervision offers a more efficient approach, yet training 3D generative models directly on 3D data poses significant challenges due to high memory and computational demands. Recent methods, such as MeshGPT [Siddiqui et al., 2024], Shap-E [Jun and Nichol, 2023], and others [Zhang et al., 2023a; Zhao et al., 2024; Li et al., 2024; Zhang et al., 2024], compress 3D shapes into a compact latent space before performing diffusion or autoregressive processes. While MeshGPT shows promise, its performance is limited by the mesh tokenizer. Direct3D [Wu et al., 2024a] and LAM3D [Cui et al., 2024] further enhance generation quality by introducing explicit 3D triplane representations. Make-A-Shape [Hui et al., 2024] and WaLa [Sanghi et al., 2024] use a wavelet-tree representation to enhance geometry encoding. TRELLIS [Xiang et al., 2024] is a very recent, open-source 3D native generative model, whose latent representation is based on the hierarchical and sparse 3D voxels. It is composed of a sparse 3D VAE and a rectified flow transformer, enabling the generation of versatile 3D representations. However, the extended training cycles and unpredictable scaling behaviors still constrain the efficiency of these 3D generation approaches.

Autoregressive models for image generation. Autoregressive models have revolutionized visual generation by sequentially creating images using discrete tokens, produced by image tokenizers [Van Den Oord et al., 2017; Esser et al., 2021]. Models like DALL-E [Ramesh et al., 2021], RQ- Transformer [Lee et al., 2022], and Parti [Yu et al., 2022] rely on raster-scan sequences for "next token prediction" within a given scale. VAR [Tian et al., 2024b] introduces a novel "next-scale prediction" approach, which better preserves spatial locality and reduces computational costs. In this paper, we aim to explore the scalability potential of next-scale autoregressive modeling in 3D generation.

Figure 3: Illustration of the proposed CAR process in G3PT. The transformer predicts the next-scale token vector using features derived from the "upsampled" tokens of the previous scale. The "upsampling" process involves two layers of cross-attention to align the number of tokens across scales. First, features are "upsampled" with a learnable query ˜ es, and then "downsampled" using a "downsample" query es to match the token number of the next scale. A causal mask is applied to maintain the correct order and dependencies across different scales and input conditions, ensuring coherence in the model predictions.

3 G3PT

We introduce G3PT, a scalable and hierarchical 3D native generative model comprising two training stages, namely Cross-scale Vector Quantization (CVQ) and Cross-scale AutoRegressive Modeling (CAR). In the CVQ stage, we perform multi-scale residual quantization [Tian et al., 2024a], converting unordered, point-based 3D data into compact, discrete tokens across multiple scales. The core of this process is the Cross-scale Querying Transformer (CQT), which utilizes cross-attention mechanisms and learnable "downsample" and "upsample" queries to efficiently generate a level-of-detail latent 3D representation. In the subsequent CAR stage, this latent hierarchy is aligned by CQT and transformed into a sequential formulation for next-scale prediction.

In the following, we first review the key components of tokenization and autoregressive modeling in Section 3.1, followed by a detailed description regarding the Cross-scale Vector Quantization (CVQ) (Section 3.2) and the Cross-scale AutoRegressive Modeling (CAR) (Section 3.3).

3.1 Preliminaries

Tokenization. We use Lookup-Free Quantization (LFQ) [Yu et al., 2023] to tokenize the feature map Z ∈ RL×C with L tokens and C-dimensional embeddings into the quantized feature map ˆ Z. LFQ streamlines the quantization process by eliminating the need for explicit codebook lookups, thereby reducing the embedding dimension of the feature Z. Formally, the quantization is executed via a mapping function

ζ = q(z) = sign(z), which maps a feature vector z ∈ RC to an index vector ζ ∈ Rlog2 C , with each dimension of ζ being quantized independently. The token index for q(z) using LFQ is determined by:

Autoregressive modeling. For a sequence of discrete tokens x = (x1, x2, . . . , xN), the probability distribution over the sequence is defined as the product of the conditional probabilities of each token given its predecessors, expressed as P(x) = ∏Ni=1 P(xi | x1, x2, . . . , xi-1). This approach effectively models the dependencies between tokens, which is crucial in generating coherent sequences in tasks like language processing and image synthesis.

To account for the spatial dependencies and multi-scale characteristics of images, next-scale prediction [Tian et al., 2024a] is proposed to progressively refine the 2D latent representation across a sequence of varying scales. However, the original implementation constructs multi-scale token maps using average pooling and bilinear interpolation, both of which assume an inherent order on the tokens at each scale. This ordering becomes problematic when applied to unordered 3D data.

3.2 Cross-Scale Vector Quantization (CVQ)

Shape encoding. As illustrated in Figure 1 (a), we first follow the architecture described in 3DShape2VecSet [Zhang et al., 2023b] to encode 3D shapes. The input point cloud is represented as X ∈ RN×(3+3) , with each of the N points having 3 position and 3 normal point features. We employ a cross-attention layer to integrate the 3D information fromX into the learnable latent queries Lat ∈ RL×C , as follows:

where PosEmb represents Fourier positional encoding and Z ∈ RL×C is the output latent features.

CVQ. We then perform Cross-scale Vector Quantization (CVQ), a novel quantization process that extends the residual quantization steps [Tian et al., 2024a] leveraging the Crossscale Querying Transformer (CQT).

A high-level comparison between the quantization used in VAR and the proposed CVQ is illustrated in Figure 2. As shown in Figure 2 (a), the conventional downsampling and upsampling operations, such as average pooling and bilinear interpolation, require a sequential arrangement of tokens, which is problematic for unordered tokens. To address this issue, we propose Cross-scale Querying Transformer (CQT) to yield a level-of-detail latent 3D representation using a set of "downsample" and "upsample" learnable queries and crossattention, as shown in Figure 2 (b).

Specifically, we first apply a cross-attention layer and introduce a set of cross-scale learnable queries {e1, e2, . . . , eS} - with a hierarchy of decreased token numbers - to "downsample" the unordered 3D tokens Z. More formally, at each scale s, the current residual feature Zs ∈ RL×C (initialized by ZS = Z and L(S) = L), is "downsampled" into a compact latent vector Es ∈ RL(s)×C by a cross-attention layer with a "downsample" learnable query es ∈ RL(s)×C :

In this configuration, es serves as the query head, while Zs acts as the key and value heads. Following Eq. (1), LFQ is applied on Es, and we denote by ˆEs the resulting quantized token vector of the latent feature Es.

Then, we retrieve the "upsampled" feature ˜ Zs ∈ RL×C of the original scale by using another cross-attention layer with an "upsample" learnable query ˜ es ∈ RL×C :

The residual feature for the subsequent scale, Zs+1, is then calculated as: Zs+1 = Zs - ˜ Zs. This process iteratively continues until the final quantization step S.

Shape decoding. The final 3D shape decoder, as shown on the right side of Figure 1 (a), which consists of several selfattention layers and a cross-attention layer, decodes the sum of these "upsampled" features and a set of query points p into occupancy values:

3.3 Cross-Scale AutoRegressive Modeling (CAR)

After the stage of CVQ, we obtain a sequence of crossscale 3D token vectors ( ˆE1, ˆE2, . . . , ˆES) of varying lengths (L(1) , L(2) , . . . , L(S) ) to serve as inputs for AR modeling. To enable effective next-scale prediction, we introduce Crossscale AutoRegressive modeling (CAR), which reuses the CQT trained during the CVQ stage to align the token dimensions across different scales. Specifically, we first employ the "upsample" queries ˜ es-1 ∈ RL×C to elevate the token length L(s-1) to the original scale L, followed by using "downsample" queries es ∈ RL(s)×C that compress them to the scale L(s) to be predicted next.

To perform next-scale prediction [Tian et al., 2024a], we model the probability distribution P( ˆE) over the sequence of the cross-scale 3D token vectors ( ˆE1, ˆE2, . . . , ˆES), where the token vector at each scale ˆEs is conditioned on that at the preceding coarser scales:

In this manner, the token vector at each scale is added with more detailed information based on the previous one, allowing the model to progressively refine the data from a rough approximation to a detailed representation.

3.4 Conditional Autoregressive Modeling

Image conditioning. In each CAR block, pixel-level information from the conditional image and point-level information from the point cloud are seamlessly integrated, so as to align the image feature space with the 3D latent space for high-quality controllable 3D generation. The overall framework of the conditional CAR model is depicted in Figure 1 (b). Specifically, we employ the pre-trained DINO-v2 (ViTL/14) [Oquab et al., 2023] to extract image features, leveraging its strength in capturing the structural information crucial for 3D tasks. A linear layer projects the LI image tokens Idino ∈ RLI×CI , derived fromDINO-v2, to Zdino ∈ RLI×C , which matches the channel dimension of the cross-scale 3D tokens ( ˆE1, ˆE2, . . . , ˆES). These projected image tokens are then concatenated with the cross-scale 3D tokens and regulated through an attention mask in the causal transformer, as shown in Figure 3, to ensure that only subsequent 3D tokens are predicted.

Text conditioning. Text conditions can also be easily added to ensure semantic consistency using the pre-trained CLIP model, which extracts semantic tokens from the conditional text input. We use AdaLN [Wu et al., 2024a] for effective signal control. Note that this approach only serves as a preliminary attempt to validate the flexible conditioning capabilities of the proposed G3PT, while other advanced conditioning mechanisms are left for further exploration.

4 Experiments

4.1 Implementation Details

CVQ. Each input point cloud to CVQ contains 16384 points uniformly sampled from the 3D model in the Objaverse dataset [Deitke et al., 2023], accompanied by a learnable latent query with length L = 2304 and channel dimension C = 512. The vocabulary size of the codebook in LFQ is 8192. The shape encoder network includes one cross-attention layer and 12 self-attention layers. The shape decoder network contains a cross-attention layer and 16 selfattention layers with the same channel dimension of the encoder. When training CVQ, 8192 uniform points and 8192

near-surface points are sampled for supervision. The AdamW optimizer is employed with a learning rate of 1 × 10-4 , and the CVQ model is trained for 60,000 steps on 8 NVIDIA A100 GPUs with 80GB memory.

CAR. The transformer in CAR shares a similar architecture with the standard decoder-only transformer used in GPT-2. To stabilize training, queries and keys are normalized to unit vectors before calculating the attention weights. All models are trained with a learning rate of 1×10-5 and a batch size of 1600, using the AdamW optimizer with β1 = 0.9, β2 = 0.95, and a weight decay of 0.05 for every 1000 steps. The 1.5B model is trained for two weeks on 136 NVIDIA H20 GPUs with 96GB memory.

4.2 Training Details

CVQ. Directly training with a large number of discrete tokens is highly time-consuming. To mitigate this, during the initial training phase for CVQ, the quantization layer in between the encoder and decoder is replaced by a layer normalization to facilitate convergence. Once this model is adequately trained, the CVQ is further finetuned with the quantization layer.

CAR. We implement a progressive training strategy for CAR. Specifically, instead of processing the tokens across all scales (L(1) ,L(2) , . . . ,L(S) ) at once, the training begins with tokens before the S/2 scale (L(1) , L(2) , . . . , L(S/2) ) and progressively includes finer scales. This approach accelerates convergence and improves training stability.

4.3 3DGeneration Results

Evaluation protocols. Wemainly benchmark on the task of image-to-3D shape generation, where a single RGB image is used as the conditional signal and the output is the generated 3D mesh. We evaluate the mesh quality using Intersectionover-Union (IoU), Chamfer distance (Cham.), and F-score (with a threshold of 0.01), which reflect the overall proximity from the generated mesh to the ground-truth. The experiment is conducted on 120 randomly selected testing objects from the Objaverse dataset [Deitke et al., 2023]. We compare against LRMs including InstantMesh [Xu et al., 2024a], CRM [Wang et al., 2024], Triposr [Tochilkin et al., 2024], and LGM [Tang et al., 2024], as well as diffusion-based approaches like Michelangelo [Zhao et al., 2024], ShapE [Jun and Nichol, 2023], CraftsMan [Li et al., 2024], MakeA-Shape [Hui et al., 2024], WaLa [Sanghi et al., 2024], CLAY [Zhang et al., 2024], and TRELLIS [Xiang et al., 2024].

Image-to-3D generation. The results in Table 1 highlight the superiority of G3PT for image-to-3D shape generation, particularly the model with 1.5 billion parameters, which outperforms all other methods with a substantial margin in all metrics, demonstrating the unparalleled generation quality and fidelity. We also conduct qualitative evaluations by comparing the proposed G3PT with other state-of-the-art methods (including commercial software like Meshy 1 and Tripo2.0 2 ) using the Objaverse dataset [Deitke et al., 2023]. As shown in Figure 4, LRMs like CRM and Triposr suffer from noisy artifacts. Diffusion-based methods like Michelangelo and TRELLIS produce plausible geometry but struggle to align with the semantic structure of the conditional images. In contrast, G3PT achieves a superior balance between geometry quality and controllability, consistently producing highquality meshes that align well with the conditional images. Note that the proposed G3PT, only trained on the Objaverse dataset, performs on par with Meshy and Tripo2.0, which span a large amount of internal data for training. Please refer to the supplement for more quantitative and qualitative evaluations on other datasets.

Text-to-3D generation. In Figure 5, we present the qualitative generation results of G3PT, given only text prompts as conditions. The generated meshes align well with the conditional semantics while exhibit remarkable diversity and highquality geometry.

Scaling Behaviors. Figure 6 illustrates the scaling laws observed in G3PT by examining the relationship between the number of model parameters (in billions) and the test loss (measured as cross-entropy). Both plots demonstrate a clear

1 https://www.meshy.ai/discover

2 https://www.tripo3d.ai/

(a) Codebook sizes.

(b) Number of tokens.

Table 4: Comparison of various 3D representations paired with different encoding methods.

trend where the test loss decreases as the number of model parameters increases, with a power-law relationship. This further validates the potential of G3PT in handling complex 3D generation tasks.

4.4 Ablation Studies

In the following, we perform comprehensive ablation studies on the task of image-to-3D shape generation, strictly following the evaluation protocols outlined in Section 4.3.

Tokenization. We first demonstrate the effectiveness of the proposed CVQ by comparing to other tokenizers. Table 2 presents the comparison against two commonly used approaches: VAE and VQVAE. We implement VAE following [Zhang et al., 2024], which applies KL regularization between the encoder and decoder, differing from the quantization module utilized in diffusion models. We implement VQVAE by incorporating LFQ quantization, which maintains the same quantization structure as described by [Yu et al., 2023]. The quantitative metrics additionally include prediction accuracy (Acc.) of the occupancy value (0 or 1), which is determined by evaluating points that are randomly sampled in the vicinity of the ground-truth mesh. The "Usage" metric indicates the efficiency of codebook usage. As can be seen, CVQ outperforms both VAE and VQVAE across multiple metrics with near-complete codebook usage.

Codebook size and the number of tokens. A detailed quantitative comparison given different codebook sizes and the number of tokens (#token) are presented in Table 3 (a) and (b), respectively. Due to memory constraints, the evaluation is limited to a maximum of 8192 tokens. We refer readers to the supplement for relationships between #token and #scale.

3D representation. The comparison of different 3D representations and encoding methods is reported in Table 4. We consider three types of 3D representations: volumetric feature grids (Volume), Triplane [Wang et al., 2023], and 1D

latent vectors. The Volume representation is encoded using a 3D Convolutional Neural Network (3D CNN) following the architecture of SDFusion [Cheng et al., 2023], while the Triplane representation utilizes the architecture from Direct3D [Wu et al., 2024a]. All encoders are designed with similar parameter counts. For 1D Latent, a comparison is also conducted between CVQ and a baseline setup (Pooling + LFQ), i.e., a similar implementation to VAR [Tian et al., 2024a] by incorporating a 1D average pooling module, a bilinear upsampling module, and an additional 1D convolutional layer (Figure 2 (a)), which forces the tokens to learn an ordered sequence.

The results demonstrate that CVQ achieves superior performance with near-complete codebook usage, emphasizing the effectiveness of CVQ in preserving detailed structural information during quantization. When compared to the baseline setup with average pooling and bilinear upsampling, the results are in line with our intuition that 1D latent tokens do not possess an inherent sequential order, unlike image data.

5 Conclusion

This paper introduces G3PT, a scalable hierarchical 3D generative model featuring a Cross-scale Querying Transformer (CQT) to map unordered 3D data into discrete tokens across various levels of detail. By establishing a natural sequential relationship among these tokens, G3PT enables Cross-scale AutoRegressive modeling (CAR) in a manner that aligns well with the inherent unordered characteristics of 3D data. This novel CAR framework tailored for unordered data offers new insights into autoregressive algorithm design. Extensive experiments demonstrated that G3PT achieves superior generation quality compared to existing 3D generation methods, setting a new state-of-the-art in 3D content creation.