Latte: Transfering LLMs' Latent-level Knowledge for Few-shot Tabular Learning

1 Introduction

Given the high cost of sample annotation, both in terms of financial and temporal resources, learning from a limited number of labeled samples presents a more cost-effective approach for adapting machine learning models to practical deployment [Snell et al., 2017; Wang et al., 2020; Oreshkin et al., 2018; Wang et al., 2021a]. This scenario, commonly referred to as few-shot learning, has recently attracted significant research attention over various domains, including computer vision (CV) [Chen et al., 2019] and tabular learning [Nam et al., 2023; Shi et al., 2025]. However, without adequate supervisory signals, traditional supervised learning in this situation cannot effectively yield desirably parameterized models, as its performance heavily depends on the statistical convergence of a large number of labeled samples. This challenge becomes more intractable in the context of tabular data, owing to the inherent scarcity of labeled tabular data [Liu et al., 2024] in many real-world applications, such as fraud detection [Cao, 2022] and disease diagnosis [Shailaja et al., 2018].

With the rapid advancement of Large Language Models (LLMs), several pioneering studies have explored their potential in few-shot tabular learning and achieved promising results: The extensive knowledge encoded during pre-training makes LLMs a natural choice for analyzing tabular data, as it provides a rich prior understanding of the relationships between diverse tabular features and the task objective. For instance, knowledge regarding the increased risk of certain diseases in older individuals can be helpful in identifying truly predictive tabular features in disease prediction tasks [Han et al., 2024a]. However, most existing LLM-based methods [Slack and Singh, 2023; Hegselmann et al., 2023] adopt test-time knowledge extraction strategies, where each test sample is directly fed into a pre-trained or fine-tuned LLM as a query to obtain its final prediction. This approach necessitates at least one LLM inference per sample, leading to undesirable response latency and increased computational costs.

To solve above limitations, FeatLLM [Han et al., 2024a] is first proposed by designing a training-time extraction strategy. Specifically, in this framework, the LLM functions as a "feature engineer" to reduce the inherent redundancy in the original tabular features: Before each tabular learning task begins, FeatLLM instructs the "feature engineer" to analyze the metadata (including task and feature descriptions) and generate insightful rules for each answer class. These rules guide the subsequent feature engineering, i.e., transforming heterogeneous column features of a tabular dataset into unified binary features. This approach reduces the reliance on complex model architectures, thereby alleviating the urgent need for a large number of labeled samples in few-shot settings.

However, FeatLLM still faces several bottlenecks that limit its performance potential: ❶ Unreliable knowledge extraction: FeatLLM prompts LLMs to generate textual rules as the criteria for feature engineering. However, this kind of textlevel knowledge, i.e. textual responses auto-regressively generated by LLMs, often suffers from hallucinations and might include plausible yet factually wrong knowledge [Duan et al., 2024; Perkovi´c et al., 2024; Li et al., 2024], thereby leading to unreliable feature engineering. In addition, [Zhou et al., 2024; Chen et al., 2024] have demonstrated that the latentlevel knowledge, i.e. hidden states directly hooked from LLMs, is more informative and discriminative than the textlevel knowledge due to the randomness and information loss during the next-token sampling; ❷ Overlook of unlabeled tabular data: Constructing pseudo-supervision (e.g., by data clustering) based on unlabeled data has been proven to be an effective practice to facilitate few-shot tabular learning [Nam et al., 2023]. However, the tabular learning framework of FeatLLM cannot make use of these cluster pseudo-labels, as its feature engineering process is designed exclusively for ground-truth classes, implying its performance upper-bound is strictly constrained by the extremely limited labeled samples.

Contributions: To accommodate the above bottlenecks, we propose a training-time knowledge extraction framework to transfer LLMs' Latent-level Knowledge for few-shot tabular learning (Latte). Unlike FeatLLM, which treats the LLM as a "feature engineer", Latte positions the LLM as a "teacher" to instruct the training process of the downstream tabular learning model. Specifically, for a tabular learning task, Latte inputs its metadata (i.e., task and feature descriptions) into the LLM and retrieves the hidden states produced by the final transformer layer as task-relevant knowledge. To distill this latent-level knowledge into downstream training, we propose two key components: 1) a semanticaware tabular encoder that integrates feature semantics into the representation of feature values; and 2) a knowledge adapter that takes this latent-level knowledge as the guideline to weightedly fuse information from different feature values, thereby constructing more predictive row embeddings. Together, these two components enable a general knowledgeguided training process, mitigating the risk of overfitting to limited labeled samples-a persistent challenge in few-shot settings-and thereby promoting convergence toward a more generalized model checkpoint. (Bottleneck ❶). In addition, Latte includes an additional unsupervised pre-training stage, making full use of the available unlabeled samples to provide a robust parameter initialization for the subsequent knowledge-guided training (Bottleneck ❷).

We conducted extensive experiments to validate Latte 's effectiveness on few-shot tabular learning tasks. Latte consistently outperforms all competing methods by a significant margin across various datasets and prediction tasks. Comprehensive ablation studies further highlight the superiority of latent-level knowledge over text-level knowledge, as well as the effectiveness of the individual components in this framework.

2 RelatedWork

2.1 Classical Tabular Prediction

A wide range of machine learning methods have been developed for tabular data. For modeling linear relationships, logistic regression (LR) [LaValley, 2008] is commonly used. In contrast, tree-based models, such as decision trees (DT) [Loh, 2011] and ensemble methods like XGBoost [Chen and Guestrin, 2016], random forests [Breiman, 2001], CatBoost [Prokhorenkova et al., 2018], and LightGBM [Ke et al., 2017], are preferred for modeling non-linear relationships. With the rise of deep learning, there has been growing interest in applying neural networks to tabular data. These approaches can be broadly categorized into four types: (1) Standard Neural Networks: This group includes methods like SNN [Klambauer et al., 2017], AutoInt [Song et al., 2019], and DCN V2 [Wang et al., 2021b]. (2) Hybrid Methods: These combine decision trees and neural networks for endto-end training, with notable examples being NODE [Popov et al., 2019], GrowNet [Badirli et al., 2020], TabNN [Ke et al., 2018], and DeepGBM [Ke et al., 2019]. (3) TransformerBased Methods: These models, such as TabNet [Arik and Pfister, 2021], TabTransformer [Huang et al., 2020], and FT Transformer [Gorishniy et al., 2021], utilize attention mechanisms to learn from tabular data. (4) Representation Learning Methods: These methods leverage self-supervised and semi-supervised learning to extract meaningful representations, with prominent examples including VIME [Yoon et al., 2020], SCARF [Bahri et al., 2021], SAINT [Somepalli et al., 2021], and Recontab [Chen et al., 2023].

2.2 Few-shot Tabular Prediction

Few-shot tabular prediction holds significant promise in fields like agriculture, industry, and finance, where obtaining and annotating data can be challenging. As a result, this area has garnered increasing attention. However, traditional tabular prediction methods often require large amounts of annotated data and tend to perform poorly in few-shot scenarios. To address these challenges, two main approaches have emerged: (1) TNNs-based methods: These approaches generate tasks from unlabeled data for meta-learning and use a small amount of labeled data to create prototypes for tabular classification tasks [Nam et al., 2023]. (2) LLMs-based methods: These rely on the inference capabilities of large language models. For example, the In-context method [Wei et al., 2022] directly converts tabular data into text prompts for the LLMs. TABLET [Slack and Singh, 2023] further enhances this by adding task-specific instructions to the prompts, improving the model's ability to reason about tabular. FeatLLM [Han et al., 2024b] uses LLMs as a feature engineer to automatically perform feature extraction, which is then used to train specialized classification networks. Alternatively, LLMs can be fine-tuned using a small number of labeled samples, as demonstrated by methods like TabLLM [Hegselmann et al., 2023]. Additionally, GTL [Wen et al., 2024] fine-tunes LLMs using a large amount of tabular data to obtain a tabular foundation model.

3 Method

In this section, we innovatively propose Latte to transfer LLMs' latent-level knowledge for few-shot tabular learning. As shown in Fig. 1, Latte first utilizes a semantic-aware tabular encoder to obtain the representation of feature values integrated with their corresponding semantics (Section 3.2).

Next, Latte leverages task metadata to extract task-relevant knowledge from LLMs (Section 3.3) and transfers this latentlevel knowledge into downstream training via a specially designed knowledge adapter (Section 3.4). Finally, we outline the overall downstream training process (Section 3.5).

3.1 ProblemFormulation

In this section, we formally define the problem settings for few-shot tabular learning: Consider a tabular dataset D = {Dl,Du,M}, where Dl represents the set of labeled samples, Du the set of unlabeled samples, and M the corresponding metadata, respectively. Specifically, each sample in the tabular dataset is in this form (x, y), where x denotes all feature values of a single row, and y represents its corresponding label. The labeled set Dl = {(xil , yi l )}Nl i=1 contains Nl labeled instances, and the unlabeled set Du = {xiu}Nu i=1 consists of Nu unlabeled instances, with Nl ≪ Nu. Metadata M = {F, T} provides textual descriptions of the task objective and the definitions of features in the dataset, where F = {fj : dj}dj=1 represents the column names and their descriptions, and T denotes the task description. Formally, the goal is to train a downstream modelfθ : X → Y, parameterized by θ, to map the row feature space X to the label space Y, where Y could be {0, 1}C with C classes or R, depending on the specific task (classification or regression).

3.2 Semantic-aware Tabular Encoder

Tabular data presents unique challenges compared to text and images due to its inherent heterogeneity, permutation invariance, and the rich semantic information provided by column names. These characteristics of tabular complicate the encoding process. Traditional feature value encoders struggle with limited labeled samples, making it difficult to infer the semantic meaning of feature names implicitly due to the scarcity of supervisory signals. To address this limitation and better handle the heterogeneity of tabular data, we propose an encoding method that explicitly incorporates feature semantic information into the representation of feature values:

Here, fj denotes the j-th feature (e.g., "Gender"), xj is its corresponding value (e.g., "Male"), and × represents element-wise multiplication between numerical features and values, because BERT's tokenizer typically splits numerical values into subcomponents, which can hinder its effectiveness for encoding numerical features. This approach generates effective tabular semantic representations for both categorical and numerical features, resulting in a feature set H = {hj}nj=1 , where n is the total number of features. To obtain a global representation of the sample, we introduce an extra token [CLS] to the feature set H. This token serves as an aggregate representation of the entire sample, capturing the overall context and facilitating downstream tasks.

The interaction between tabular features is crucial for capturing meaningful patterns. For instance, the relationship between weight and blood pressure (BP) can provide valuable insights into cardiovascular health, which is vital for heart disease classification. To model these interactions while preserving the inherent permutation invariance of tabular data, we employ a Transformer architecture without positional embeddings to further refine the quality of feature value embeddings:

Here, ˆH = {ˆhCLS, ˆhj}nj=1 , where ˆhCLS and {ˆhj}nj=1 refers to the global representation and the set of tabular feature representations, respectively. These capture the implicit relationships between features, enabling the model to effectively understand and leverage feature interactions.

3.3 Task-relevant Prior Knowledge Extraction

The above semantic-aware tabular encoder effectively processes heterogeneous tabular data and incorporates its semantic information into the representations of feature values. However, when labeled data is limited, capturing task-specific semantics becomes more challenging. Thanks to large-scale pretraining, LLMs encode substantial general knowledge within their parameters. For a given a tabular learning task, to effectively obtain the task-relevant knowledge within LLMs, we formulate the following knowledge extraction process: Based on the metadata M of the given tabular dataset D, we elaborately craft the input prompt, describing the task objective and feature semantics (See Fig. 1 for an example of input prompt), in order to retrieve taskrelevant knowledge from the pre-trained LLM. Specifically, after the LLM processes this prompt, we hook the hidden states produced at the last transformer layer of all input tokens and apply average pooling to them. This operation results in a dense vector that encapsulates the LLM's prior understanding of the task, serving as our latent-level knowledge for subsequent knowledge-guided downstream training. The process can be formally described as follows:

where hi represents the last-layer hidden states of the i-th token after processing the input prompt. We obtain the prompt representation, i.e. the latent-level knowledge, by applying average pooling to the hidden states of all tokens:

where Average(·) represents the average pooling operator and hM denotes the representation of the input prompt. We regard this dense vector, which contains abundant prior understandings of the given tabular dataset, as the useful latent-level knowledge for subsequent model training.

3.4 Knowledge Adapter

The semantic spaces of LLMs and the BERT-based Semanticaware Tabular Encoder differ significantly in both dimension and representation, making it challenging to directly integrate the Latent-level knowledge from the LLM into our model. To address this, we design a knowledge adapter that aligns these two semantic spaces and facilitates the transfer of LLMgenerated knowledge into our model. We introduce a novel GTransformer to generate the global query vector q:

where both ˆhCLS and {ˆhj}nj=1 are representations encoded by the Semantic-Aware Tabular Encoder. To distill taskrelated semantic knowledge from the LLM into the global query vector q and obtain the task-relevant global query vector qLLM, we apply the following knowledge distillation formula:

where τ is the temperature, the matrix W0 is initialized randomly using Kaiming initialization, which aligns the dimensions of hM and q. Meanwhile, through the attention mechanism, we obtain the LLMs semantic knowledge guided taskrelevant semantic representation hLLM as:

where the ˆH generated by the Semantic-aware Tabular Encoder is directly used as K and V, ensuring that the semantic information of the sample remains unaffected and d is the feature dimension of qLLM. Finally, we combine the general tabular semantics ˆhCLS from the Semantic-aware Tabular Encoder with the task-relevant semantic representation hLLM to obtain a high-quality representation ˆhLLM, which effectively integrates both task-relevant and general tabular semantic in-

where g(·; ϕ) is a learnable transformation designed to align the semantic spaces of hLLM and ˆhCLS and η is a constant that allows for fine-tuning the influence of the LLM's prior knowledge on the prediction process, ensuring an optimal balance between the model's learned features and the external knowledge introduced.

3.5 Model Training

To fully leverage the available unlabeled samples and ensure robust parameter initialization for subsequent knowledgeguided training, we include an unsupervised pre-training stage, in addition to downstream task fine-tuning. During the pre-training stage, we generate N-way K-shot meta-training tasks using unlabeled datasets and perform meta-learning, guided by task-relevant knowledge extracted from LLMs. To obtain the meta-training task, we first generate pseudo-labels ˜y for the unlabeled data using the following cluster procedure:

where k represents the number of centroids, 1k is a vector of ones, C is the centroid matrix, and m is random noise used to corrupt features, aiding in the learning of robust embeddings for downstream tasks. Next, we randomly select k samples from each cluster to create an N-way K-shot meta-training task. By introducing different noise m, we generate a variety of meta-tasks. Finally, we compute the meta-learning loss as follows:

where LKL = KL(W0hM/τ, q/τ) denotes the KL divergence loss, which is used to guide the distillation of task-relevant knowledge from LLMs and Lpseudo is the loss associated with the pseudo-labels, given by Lpseudo =

-˜y log(MLP(ˆhLLM)). This stage of training enables the model to learn task-relevant semantic information under the guidance of LLM task-relevant knowledge, preparing the model for adapting to downstream tasks. Therefore, during the downstream task fine-tuning stage, Our model is capable of rapidly generalizing to downstream tasks with the guidance of limited labeled data. The loss is computed as:

where Ltrue depends on the type of task:

This final step refines the model's performance on the specific task using the limited number of labeled examples available.

4 Experiment

In this section, we first introduce the setup for few-shot tabular data prediction and provide the implementation details. We then evaluate the proposed model against with baseline, including traditional machine learning methods, few-shot tabular algorithms, and LLM-based frameworks. Additionally, we conduct extensive ablation studies to test the effectiveness of the proposed modules.

4.1 Experimental Settings

Dataset. As shown in Table 1, we evaluate the proposed method using nine real-world datasets, including six classification tasks and three regression tasks. Each dataset provides metadata such as a clear description for each attribute and task. Refer to Appendix A for details of the dataset

Baselines. We compare the proposed method against ten baseline models. The first is conventional tabular learning approaches: (1) Logistic Regression (LogReg) [LaValley, 2008], (2) XGBoost [Chen and Guestrin, 2016], and (3) Random Forest [Breiman, 2001]. The next two baselines leverage unlabeled datasets: (4) SCARF [Bahri et al., 2021] and (5) STUNT [Nam et al., 2023], though it is worth noting that in real-world scenarios while obtaining unlabeled data is typically straightforward, the process of labeling it can be expensive and time-consuming. Another recent baseline, (6) TabPFN [Hollmann et al., 2023], generates synthetic datasets with diverse distributions to pretrain the model. The final group of baselines utilizes LLMs: (7) In-context learning [Wei et al., 2022], which embeds few-shot training samples directly into the input prompt without fine-tuning; (8) TABLET [Slack and Singh, 2023], which enriches the prompt with additional hints through an external classifier using rule sets and prototypes to improve inference quality; (9) TabLLM [Hegselmann et al., 2023], which applies parameter-efficient tuning techniques such as IA3 to train the LLMs with few-shot samples; and (10) FeatLLM [Han et al., 2024b], which treats the LLMs as a feature engineer, automating feature extraction and using the engineered features for few-shot tabular classification.

Implementation details. We obtain task-relevant knowledge from the LLaMA2 series models [Touvron et al., 2023]. The hidden layer dimensions of the Feed Forward Network (FFN) and other components are set to 256 and 128, respectively. For the Semantic-aware Tabular Encoder, we use a 2-layer, 8-head transformer, while the knowledge adapter consists of a 2-layer, 2-head transformer. Dropout is set to 0.1. During the meta-learning phase, The Adam optimizer is used for training, with the learning rate set to 1e-4. The activation vector in the LLMs is obtained from the 30 layers. For the fine-tuning phase, the learning rate is reduced to 1e-5 due to the limited number of labeled samples. For evaluation, we use the Area Under the Curve (AUC) metric for classification tasks and Mean Squared Error (MSE) for regression tasks.

4.2 Results and Analysis

Latte offers an effective and comprehensive solution for few-shot tabular prediction. For Effectiveness: As shown in Table 2, Latte outperforms or comparable of the current state-of-the-art baseline across all datasets and shot setting. For example, in classification tasks, our method exceeds the performance of the state-of-the-art method FeatLLM in all shot settings, achieving an average performance improvement of 4.22%. This improvement is primarily due to Latte's ability to leverage both unlabeled data and taskrelated knowledge from LLM, which is something that existing methods cannot achieve.

For Comprehensiveness: Latte directly applicable to regression tasks without the need for modifications. Detailed experimental results can be found in Appendix B. In contrast, existing few-shot learning methods are often designed specifically for classification tasks and cannot be directly applied to regression. While large language models (LLMs) can handle regression tasks, they tend to produce inaccurate predictions, often due to spurious patterns or "hallucination."

4.3 Ablation Study

Each module in Latte plays a crucial role. We conduct ablation studies to assess the contribution of key components to the model's performance. Specifically, we evaluate four factors: (1) SaTE explores the decision of whether to use a semantic-aware table encoder. If a semantic-aware approach is not employed, a non-semantic-aware encoder from STUNT [Nam et al., 2023] is used as an alternative, (2) LLM indicates whether task-relevant knowledge is extracted from the LLMs, (3) Meta refers to the process of applying metalearning techniques to unlabeled data, and (4) Size represents the effect of varying LLM scales on performance.

Table 3 presents the impact of various ablation experiments on the AUC across the Heart datasets. In all cases, modifying any of the ablated components leads to a decline in performance. Specifically, we replaced SaTE with the encoder from STUNT [Nam et al., 2023] and the result reveal a maximum performance drop, particularly with few-shot labeled samples. For instance, with just 4 labeled samples, performance decreased by nearly 10%. This highlights the critical role of capturing the unique properties of tabular data and the importance of incorporating semantic information from column names, especially when working with limited labeled data. Besides, without extracting task-relevant knowledge from the LLMs using metadata, or meta-learning with unlabeled data, also decrease the model's performance. This highlights the importance of learning task-related semantic knowledge in the meta-learning process. Interestingly, increasing the size of the selected LLMs does not result in significant performance gains, possibly due to the 7B model has already contained all the necessary task-related semantic knowledge.

4.4 Impact of Labeled Sample Size and LLM Activation Layers on Latte

The deeper of LLM, the more sufficient knowledge it contains. Fig. 2 (a-b) illustrates the effect of activation in different layers L of the LLM and the number of labeled samples on model performance. Fig. 2 (a) presents AUC values for classification tasks on the Blood dataset, while Fig. 2 (b) shows MSE values for regression tasks on the Abalone dataset. As the number of labeled samples increases, model performance improves across both tasks. In classification tasks, performance improves as deeper layers of the LLM are activated. This suggests that higher-order knowledge is accumulated in the later layers. In contrast, for regression tasks, optimal performance is achieved at around 30 layers. In summary, adding more layers may not improve the model's performance and could instead introduce noise, leading to a decline in the quality of learned representations.

4.5 What Latent Knowledge Does Latte Transfer From the LLM

Latte can retrieve task-related semantic information from metadata. To examine the task-related semantic information embedded in the representations learned by Latte during the unlabeled data meta-learning stage, we measured the semantic similarity between the learned representations and the LLM activation at the word level using metadata. This was visualized through a heatmap, as shown in Fig. 3. Higher values, indicated by redder colors, reflect greater semantic similarity. As an example of a downstream task, this involves predicting whether a patient has heart disease. The results of the heatmap demonstrate that our model, after meta-learning on unlabeled data, captures task-relevant semantic information. For example, the representations learned by Latte exhibit high semantic similarity with "patient" and "heart disease," indicating its effective learning of task-relevant semantic knowledge. Furthermore, the model identifies and emphasizes key features such as "age," "cholesterol," "fast blood sugar," "maximum heart rate," and "peak exercise ST segment," which are crucial for determining heart disease.

4.6 Comparison of LLM Invocation Costs

Latte has an efficiency advantage in calling LLM. Table 4 compares the number of LLM calls required by Latteand other LLM-based baselines across the data preprocessing, training, and inference stages. Specifically, TABLET, Incontext and TabLLM require calling the LLM once for each test sample during inference, resulting in significant overhead. While FeatLLM limits LLM calls to the training phase to assist in learning, the number of calls still scales with the size of the training dataset. Our method calls the LLM only once during preprocessing to gather task-relevant knowledge, which is then stored locally. As a result, our approach significantly reduces the overall number of LLM calls throughout the entire process.

4.7 Visualization Experiments

Latte can generate high-quality representations with limited labeled data. The visualization of representations on the Bank dataset is shown in Fig. 2 (c-e), where the original representations refer to those generated without training. Latte learns effective representations in few-shot settings. With just four labeled samples, it produces a separable representation, and this separability improves as the number of labeled samples increases to 64. This demonstrates the model's strong ability to learn useful features with minimal supervision.

5 Conclusion

This paper introduces a novel framework, Latte, aimed at addressing the challenges of few-shot tabular learning by extracting latent-level knowledge from large language models (LLMs) during the training process. Latte positions the LLM as a "teacher" that provides task-relevant guidance to the downstream tabular model and enable a knowledgeguided training process that mitigates overfitting, a common issue in few-shot settings, and promotes convergence toward a generalized model. Additionally, Latte incorporates an unsupervised pre-training stage that leverages unlabeled data for robust parameter initialization, further enhancing the framework's performance. Extensive experiments demonstrate the effectiveness of Latte on a variety of few-shot tabular learning tasks.

Acknowledgments

This work was supported by a grant from the National Natural Science Foundation of China under grants (No.62372211, 62272191), and the Science and Technology Development Program of Jilin Province (No.20250102216JC).

Contribution Statement

Ruxue Shi and Hengrui Gu have equal contributions, with Xin Wang as the corresponding author.