PointTFA: Training-Free Clustering Adaption for Large 3D Point Cloud Models

1 Introduction

"Is it always advisable to fine-tune models with high-dim point cloud inputs in the same manner as for low-dim ones (text)?" Probably Not, as this question is particularly relevant in light of "Curse of Dimensionality" (COD) [Bellman, 1966]. According to COD, as dimensions increase, an exponential growth of samples' amount is required to maintain average distances between them. Otherwise, sparse distributions due to limited samples in high-dim spaces can easily lead to overfitting when training large models.

Recall in the convolution era, 3D cloud models were custom-designed for a variety of tasks, such as point cloud classification [Qiu et al., 2021], object detection [Li et al.,

∗Corresponding Author

2021; Jiao et al., 2024], and segmentation [Hu et al., 2020; Li et al., 2024]. These models, due to their limited parameters, required individual fine-tuning on different downstream datasets and often paid less attention to the COD issue. However, the sweep of foundational models to point cloud areas has significantly enlarged the scale of model parameters, resulting in an unavoidable COD challenge of data scarcity.

Foundational Model (FM) originated from Natural Language Processing (NLP) [Song et al., 2023; Wu et al., 2023] and became well-known with GPTs [Brown et al., 2020]. They then moved into 2D vision-language areas (CLIP [Radford et al., 2021],[Zhou et al., 2024]) and are now also used in 3D understanding fields (ULIP [Xue et al., 2023a]). Its success comes from two main factors: (1) it uses hyper-scale training data, and (2) it relies on Transformer architecture with hyper-scale parameters. This helps them work well across different tasks, as shown by their excellent zero-shot results [Xue et al., 2023a].

But, when it comes to learning from a small amount of specific data, FMs are less economical than traditional models since large-scale FM would easily suffer from overfitting and incur its original generalizability [Wortsman et al., 2022] with few training samples. Moreover, simply increasing the number of samples for downstream tasks is also not an optimal solution. High-dim samples demand substantially more computational resources than lower-dim ones. Thus, we must consider whether collecting and processing more high-dim data points is more efficient than before, just for one specific downstream 3D cloud task. It would be more practical to explore alternative methods beyond the current practice.

As in Figure 1, transferring a 3D point cloud model (3DPCM) to downstream tasks typically operates under three conditions: zero-shot, few-shot, and fully fine-tuning. Zeroshot learning is training-free, whereas both few-shot learning and fine-tuning require updating partial or all parameters of the backbone/adaption module. This situation changes when a large-scale foundational model emerges with strong incontext learning ability by presenting a few relevant examples without training. It gains success on 1D text [Brown et al., 2020] and 2D image tasks [Alayrac et al., 2022]. This leads to a new, stricter condition of Training-Free, Few-Shot Learning for Large 3D-PCM than before. Training-free condition is particularly useful for adapting downstream 3D tasks, as it saves computations while approximating the performance of few-shot/fully tuned, achieving high cost-effectiveness.

In this paper, we introduce PointTFA, a Training-Free Clustering Adaptation of large 3D models, aimed at enhancing point cloud classification. Our PointTFA is built on top of the frozen ULIP [Xue et al., 2023a], with three new parameter-free modules: Representative Memory Cache (RMC), Cloud Query Refactor (CQR) and Training-Free 3D Adapter (3D-TFA). Specifically, the RMC selects the most representative samples into the support memory set from the training data for each new cloud category via unsupervised clustering algorithms. We validate that this data selection process is necessary and outperforms the randomly sampled support set (i.e., vanilla few-shot) by a large margin. The CQR serves to rebuild a test query with support set via a simplified attention module by eliminating all projection layers in [Vaswani et al., 2017]. By treating the support set directly as "key" and "value", the query feature is shifted in the identical distributions as the support set. Our 3D-TFA is inspired by 2D TIP-Adapter [Zhang et al., 2022b], designed initially to adapt the CLIP for image few-shot learning. However, 3DTFA is different because we customize the adaption for 3D point cloud inputs and integrate it with the 3D foundational model, extra data selection (RMC), and feature reconstruction modules (CQR).

We are the first to apply training-free conditions from zero-shot to few-shot for 3D point cloud understanding. By proposing a training-free, few-shot learning method, we omit the training process as zero-shot while approaching the performance of supervised few-shot learning. Experiments on ModelNet10, ModelNet40, and ScanObjectNN show the high cost-efficacy of PointTFA. We briefly summarize our contributions as follows.

• PointTFA: We introduce PointTFA, a new way to classify 3D point clouds without training. It has high training efficiency as a zero-shot method, saving on training computing, but still performs close to those finely tuned methods with a few-shot examples.
• Representative Memory Cache (RMC): Our research reveals that using clustering algorithms to create a support set for few-shot tasks is more effective than a randomly gathered set.
• Cloud Query Refactor (CQR): We demonstrate that a cloud query can be efficiently adapted to the distribution of the support set using a zero-parameter, simplified attention module.
• Training-Free 3D Adapter (3D-TFA): We've replicated the success of the Training-Free Adapter used in 2D vision models for 3D cloud models. This achieves high efficiency and effectiveness for 3D tasks.

2 RelatedWork

Our PointTFA is related to the following areas: training-free zero-shot, fine-tuned few-shot 3D cloud classification, and efficient cache models. We delve into each area below.

2.1 Training-Free, Zero-Shot 3DModel

Zero-Shot 3D cloud classification is naturally a training-free task, as no downstream training samples are available. Most current approaches use knowledge from similar fields, such as CLIP and ULIP, which are already trained on 2D/3D visionlanguage tasks. They then apply this pre-trained knowledge to new tasks they haven't seen before. Notable techniques using 2D-CLIP include PointCLIP-V1/2, CLIP2Point, and ViTLens. For example, PointCLIP [Zhang et al., 2022a] converts 3D clouds into several 2D depth images and then makes average predictions by processing these depth images through CLIP. PointCLIP-V2 [Zhu et al., 2023] further expands from its V1 predecessor by using descriptive sentences from GPT3 [Brown et al., 2020] for category words. Similar in spirit, CLIP2Point [Goyal et al., 2021] pre-trains adapters for CLIP to merge depth and rendered images using upstream datasets, then transfer learned weights to zero-shot downstream tasks. ViT-Lens [Lei et al., 2023] integrates multiple modalities, including 3D clouds, into pre-trained vision transformers. ULIP [Xue et al., 2023a] learns 3D encoders given 2D projected image and text features obtained by frozen CLIP. These training-free methods largely leverage pre-trained knowledge abstracted from large-scale upstream multimodal data.

2.2 Fine-Tuned, Few-Shot 3DModel

Few-shot learning typically includes a learnable adaptation module to bridge the distribution gap between upstream and downstream tasks. This module is plugged into a pre-trained base network and is then fine-tuned with a few training examples. For example, with few samples, the PointCLIPs [Zhang et al., 2022a; Zhu et al., 2023] incorporates a learnable adapter. This adapter adjusts individual 2D depth features with global information to better suit downstream tasks.

Moreover, CLIP2Point [Goyal et al., 2021] fine-tunes the gate unit in feature fusion given downstream samples.

2.3 Efficient Cache Model

Efficient cache models, also a training-free paradigm, store training samples and features in a key-value database for adapting to downstream tasks without training. Models like [Khandelwal et al., 2019], TIP-Adapter [Zhang et al., 2022b], and Point-NN [Zhang et al., 2023] use test features as queries for similarity-based retrieval from this database. Specifically, Point-NN pre-extracted hand-crafted query cloud feature, then matched it with training features stored in the memory bank. The TIP shares a similar strategy by replacing features extracted from the frozen CLIP model. Similar to TIP-Adapter, our method distinguishes itself by adding strategies for selecting support samples and aligning query features with the support set's distribution. We extend these trainingfree benefits to 3D cloud tasks.

3 Method

We first revisit ULIP for 3D recognition in §3.1. Next, we prepare the support memory for the whole training set in §3.2. Finally, we introduce PointTFA, which combines ULIP's pretrained knowledge with cache information in a training-free manner in §3.3.

3.1 ARevisit of ULIP

ULIP aligns the semantical meanings of three modalities: text, 2D image, and 3D point cloud. Only the 3D encoder is tuned during pre-training on large-scale point cloud datasets (i.e., ShapeNet [Chang et al., 2015]), while CLIP's text and image encoders remain frozen. This ensures that CLIP's knowledge is broadcasted intact to 3D understanding.

For a downstream zero-shot 3D cloud classification task with N new categories, we start by generating the categoryspecific classifier WU ∈ RN×D , where D is the dimension of the textual feature. We form a sentence using the template "point cloud of [category]" and input it into the textual encoder to generate textual feature si. The si is further normalized by the L2 norm. Repeating this for all N categories, we obtain the classifier WU, storing N categorical features. This procedure is detailed in Functions (1)-(2).

Given an unseen test point cloud T with Z number of 3D coordinates. We feed T into pre-trained ULIP 3D cloud encoder to get cloud feature ftest ∈ R1×D . We also postprocess ftest with the L2 norm. By multiplying cloud feature ftest with textual classifier WU, we could get the probability for N categories. Softmax function serves to normalize probability y ∈ RN .

Algorithm 1: Data-Efficient RMC

Input: point cloud tensor Ftrain ∈ R∑Ni=1 Ki×D Output: point cloud tensor C ∈ RMN×D representative memory from K-means centroid 1 Let clusters M 2 for i = 1 to N do 3 Select the i-th Fi ∈ RKi×D from Ftrain. 4 Randomly initialize M centroids: Ci = {c1, c2, . . . , cM}; 5 while Not Converged do 6 Assign each point cloud feature to the nearest centroid; 7 Update centroids based on the assigned point cloud features; 8 end 9 return Cluster centroids Ci ∈ RM×D 10 end 11 Concatenate Ci outputs C ∈ RMN×D . 12 return C

3.2 Support Memory Preparation

Even though we don't update the model's parameters with training data under training-free conditions, we can still store annotated data in a database for nearest-neighbor searches. To do this, we'll prepare a large support memory to store the features of all training samples using a frozen ULIP model.

Given a training set containing N new classes, where each i-th class has Ki samples (for i = 1, 2, · · · ,N), we extract 3D cloud features for each class. The labels are also converted into one-hot vectors and stored in the support memory. For example, the j-th data sample in i-th class is denoted as Pi,j, a bag of 3D coordinates. We extract its cloud feature pi,j using ULIP 3D encoder and collect its labels Li,j using the following functions. All feature pi,j are post-processed with L2 norm.

We collect features from all categories into Ftrain and corresponding labels in Ltrain, where Ftrain ∈ R∑Ni=1 Ki×D and Ltrain ∈ R∑Ni=1 Ki×N . By doing this, we convert the training split into a large support memory storing knowledge of a downstream task.

3.3 PointTFA

We present an overview of PointTFA, a training-free adaptation created to customize ULIP for the downstream 3D cloud classification in Figure 2. PointTFA consists of three components: the Representative Memory Cache (RMC), Cache Query Refactor (CQR), and 3D Training-Free Adapter (3DTFA). The RMC is used to gather knowledge in pre-memory efficiently. The CQR transfers cache knowledge to the test features. Lastly, the 3D-TFA is responsible for predicting the final category. Let's explore each of these three modules in more detail.

Representative Memory Cache (RMC). In §3.2, we have stored all training samples in support memory to keep knowledge of downstream tasks. However, since samples in the same category are similar, keeping them all can waste computing resources because of redundancy.

To solve the issue, we suggest compressing the support memory based on the "Data-Efficiency" principle. This principle focuses on retaining maximum information relevant to the downstream task while reducing the number of samples stored in the support memory. One effective way to do this is by selecting representative samples from each category through an unsupervised clustering algorithm.

Specifically, we individually select a few samples from each category to form a smaller Representative Memory Cache than before. This is done by applying the K-Means clustering algorithm category-by-category. Denoting data features in the i-th category as pi =

[pi,0; pi,1; pi,2; · · · ; pi,Ki ], we cluster them into M centers. This process reduces the number of data samples in a specific category from Ki to M, with M being set significantly smaller than Ki. We use matrix Ci to store the features of M centers, and collect all M one-hot vectors Li,0∼Li,M into Li as labels for this category. Below Functions (10-11) show this process.

After getting representative data for a category, we repeat this process for all categories, collecting features and labels separately into CRMC ∈ R(M·N)×D and LRMC ∈ R(M·N)×N . Algorithm 1 shows the complete process of obtaining the efficient RMC.

After being processed by Data-Efficient strategy, the support memory, which previously contained full-set, is upgraded to a condensed memory containing MN samples, only covering important knowledge about each category.

Specifically, the cloud features Ftrain ∈ R∑Ni=1 Ki×D are simplified to CRMC ∈ RMN×D , and the label features Ltrain ∈ R∑Ni=1 Ki×N are simplified to LRMC ∈ RMN×N .

We further utilize the <feature, label> from the condensed RMC as "key-value" caches for retrieving downstream point cloud information. We verified in §4 that the RMC approaches the accuracy of using the full training set as support memory while with fewer query-key computations.

Cloud Query Refactor (CQR). To mitigate the distribution gap between the query point cloud and support point clouds in the RMC. We project the test query into the supportset space. Then, we reconstruct the query using the support set. We adopt the attention mechanism to achieve this target but remove all learnable linear projections.

Given a query ftest ∈ R1×D and a support set of cache CRMC, our goal is to generate a new ˜ ftest using CRMC ∈ RMN×D . Specifically, the new of ˜ ftest projected into the cache space is obtained by a weighted sum of each element in the cache as Function (14). Hereby, ci ∈ R1×D denotes the i-th row of the CRMC, τ is a constant temperature value for adjust density in weighting and set to 100.

After reconstructing the query using samples from the RMC support set, the query features effectively shift to match the distribution of the downstream training set. We also observe that samples belong the same category are grouped denser after re-construction, with t-SNE visualization in §5. We experimentally verify the Cloud Query Refactor's (CQR) effects in §4.3 ablations. All these together demonstrate that the CQR is both simple and effective.

Training-Free 3DAdapter (3D-TFA). Our 3D-TFA, functioned by φ(·), is designed to generate categorical predictions for the query cloud ˜ ftest, using a support memory that contains "key-value" pairs. To achieve this, we initially match the query feature with the "keys" in the support set to find the most relevant samples. This requires a similarity function θ (·) to measure the "query-key" distance, as outlined in Function (16).

We use the similarity function from 2D-TIP, as in Function (17), to measure the distances of 3D cloud features. This function adapts well to 3D features, despite the switch the feature extraction encoder from 2D-CLIP to 3D-ULIP.

We define ˜w ∈ R1×MN as the affinity vector of the query to the keys, where β is a positive constant. Thus, when ˜ f test is close to a key in CRMC, the corresponding element in ˜w will be larger. By combining Functions (16) and (17), we can obtain the prediction y which consists of N categorical probabilities.

Finally, we integrate the predictions from the 3D-TFA (Function (18)) with the zero-shot predictions (Function (5)) using a weighted sum.

The constant α determines the amount of information from the support cache that influences the final query cloud predictions. Specifically, a large α is appropriate when there's a significant distribution shift between downstream and upstream data. Conversely, a small α suggests retaining more information from the upstream data. We set α in around 20 and β in around 10.

4 Experiments

We evaluate our plug-and-play PointTFA on five major Large 3D Point Cloud Models: PointNet2 (ssg) [Qi et al., 2017], PointMLP [Ma et al., 2022], PointBERT [Yu et al., 2022], PointNEXT [Qian et al., 2022], and PointBERT-ULIP-2 [Xue et al., 2023b]. We assess the performance in a Training-Free Few-Shot scenario across three downstream datasets. Details of datasets are below.

Datasets. We tested onModelNet10&40 [Wu et al., 2015], [Wu et al., 2015] and ScanObjectNN datasets [Uy et al., 2019]. Specifically, ModelNet10 covers ten categories, with 3991 training samples and 908 test samples. ModelNet40 extended categories to forty, with 9843 training and 2648 testing samples. ScanObjectNN contains 2902 object scans across fifty categories. Moreover, this dataset provides three variants, namely OBJ ONLY, OBJ BG, and OBJ T50RS, representing raw data, data with added background, and data with different noises, respectively.

4.1 ComparisonWith Train-Free, K-Shot Methods

Training-free few-shot learning is a relatively new area, emerging with the development of large foundational models. Currently, only a few pioneering works exist in 3D point cloud understanding. To provide a broader context, we extend our comparison to include both Training-Free Zero-Shot Learning and Fine-Tuned Few-Shot Learning.

We compare PointTFA with strong transfer learning SOTAs, including PointCLIP, CLIP2Point, RECON, OpenShape, ViT-Lens, ULIP, Point-NN, and TIP-3D in Table 1. Our PointTFA, built on top of ULIP-2 (PointBERT), shows competitive performance under strict conditions of TrainingFree, few-shot settings for downstream tasks. Our PointTFA, with 16 shots, even surpasses fine-tuned few-shot methods PointCLIP and CLIP2Point.

To test PointTFA's upbound, we adopt the full set as a support set. This setting further improves on 16-shot settings. However, we could still balance performance and computations during testing with few-shot settings.

4.2 Comparison to Vanilla ULIP

We plug PointTFA into five vanilla frozen ULIP backbones without training for downstream tasks. We apply PointTFA with full set as support memory and present comparison in Figure 3. We observe that the Training-Free PointTFA significantly introduces improvement for all backbones. The most noteworthy improvement is observed for PointNEXT, where PointTFA boosts the final performance of the pretrained PointNEXT model by 46.8%. This substantiates the effectiveness of our approach in bridging the domain gap between ULIP pre-training and unknown point clouds.

4.3 Ablations

We conducted an ablation study in three areas: the proportion of training samples, the validity of three modules, and the cache-building strategy. This section's 3D model adapted to PointTFA is PointBERT ULIP-2.

Proportion of Training Set. We randomly select different percentages of samples for each category in the training set and utilize these samples as support memory to construct cache models. We gradually increased the sample proportion, i.e., from 10% to 100% on the three datasets.

As depicted in Table 2, the accuracy gradually converges as the percentage of samples increases. Notably, when the training samples of ScanObjectNN(OBJ ONLY) grow from 10% to full-set, the performance fluctuates, hovering close to 10%. This observation underscores the impact of adopting different cache sizes or structures on the final performance. This analysis motivates our ongoing research to identify a balance between cache size and final performance-(RMC), leveraging a small amount of data to achieve excellent performance for enhanced data efficiency.

Validity of ThreeModules. We validated the effectiveness of RMC, CQR, and 3D-TFA by adding modules incrementally. In Table 3, with the sequential addition of the RMC and CQR modules, the complete entity PointTFA leads the TIPAdapter's [Zhang et al., 2022b] 3D version (only-3D-TFA) by 12.92% at 1-shot on ModelNet40. We can conclude that the three modules of PoinTFA each synergize with each other.

we separately test {yfuse,yTFA,y} of Function 19 in Table 4 to verify 3D-TFA adaptability.

Cache Construction Strategies. We test three different cache settings on ModelNet40: a cache constructed with different percentages of the training set, a random cache, and a catch with RMC. To show the effects of CQR under different cache settings, We denote the performance w/ and w/o CQR as PointTFA and PointTFA (-CQR).

In Figure 4 (a), PointTFA consistently outperforms PointTFA (-CQR) across all ratios of training data. This indicates that the Cloud Query Refactor (CQR) is effective when the support memory reaches a certain size.

Figure 4 (b) shows that PointTFA consistently outperforms PointTFA (-CQR) under a different number of K-means cluster settings. Notably, when number of clusters is set to 1, PointTFA improves by 4.58%. This result highlights the RMC could effectively select representative few-shot samples while balancing cache size.

In Figure 4 (c), we analyze results for PointTFA (-CQR, RMC), showing that reshaping test features with random fewshot cache keys initially leads to decreased performance. The shows that CQR needs to be used with RMC.

In short, our ablation study shows that RMC can effectively select representative samples into a small cache size. Additionally, the CQR enhances the generalizability of PointTFA, even with limited cache availability.

5 Visualization

To show the effect of the CQR and RMC, we used t-SNE to visually compare the support sets (Random vs RMC) and test queries (Raw vs CQR). Figure 5 (a-b) show that the RMC support set selects more representative categorical samples than a random cache. Figure 5 (c-d) show that The test features after CQR become denser within the same class and more separable across different classes.

6 Conclusions

We introduce PointTFA, a training-free adaption of large 3D point cloud models that constructs a cache model using fewshot knowledge to create an adapter. This approach effectively bridges the domain gap between ULIP pre-training and downstream point clouds. We propose a "Data-efficiency" (RMC) to capture representative information in support memory, forming a cache model that balances data dimension and baseline accuracy. Additionally, we present a reshaping technique (CQR) for projecting cache knowledge onto test features, enhancing the comprehensive understanding of cache information by test point clouds. Furthermore, PointTFA significantly improves the original capabilities of the ULIP pretrained 3D backbone, achieving state-of-the-art performance in training-free few-shot 3D Classification.

Acknowledgements

This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2-RP2020-016).