Dynamic and Adaptive Feature Generation with LLM

1 Introduction

The success of machine learning (ML) algorithms generally depends on three aspects: data processing, feature engineering, and modeling [Jordan and Mitchell, 2015]. Among these, feature engineering is critical, directly influencing ML models' performance and effectiveness. Within feature engineering, feature generation is a vital process of transforming raw features into a structured format and optimizing the feature space by creating new features from the original ones [Nam et al., 2024]. This transformation usually involves mathematical or algorithmic operations on existing features. This optimization can significantly enrich the feature space, enabling ML models to draw and utilize data information more effectively and achieve superior performance.

∗Corresponding author.

Given the importance of feature engineering and feature generation, extensive research has focused on enhancing both the effectiveness and efficiency of these methods [Mumuni and Mumuni, 2024; Gu et al., 2024]. As deep learning has explosively developed, automated feature engineering has emerged as a mainstream approach due to its convenience and remarkable performance. These auto-engineering algorithms greatly reduce the need for manual calculation and determination, and thus reduce subjective errors and mistakes. In the field of automated feature generation, there are outstanding works such as [Khurana et al., 2018; Zhang et al., 2024b; Wang et al., 2022a]. These methods leverage advanced techniques like reinforcement learning to generate optimal feature sets that enhance the performance of downstream tasks while maintaining a degree of explainability. Despite these methods enriching the feature space and restructuring it into a more refined space structure, like other automated feature generation and broader automated feature engineering techniques, these works have not resolved three fundamental issues: 1) Lack of Explainability: Most feature engineering processes that utilize deep learning algorithms are result-oriented "black boxes". The specific operations on features and the process of restructuring feature space are usually hard for researchers to understand. 2) Limited Applicability: For different data types and downstream tasks, researchers need to manually select and switch among a wide range of feature engineering methods, which demands a high level of domain knowledge and expertise from the researchers. This limited applicability can be a significant barrier across varied ML scenarios. 3) Fixed Strategy: Existing methods usually require predefined strategies for exploration and generation. Models utilizing these methods are unable to adapt or refine their strategies based on insights acquired about the feature space during the model's learning process. Hence, these models lack the flexibility to dynamically adjust to the evolving understanding of the data.

Our Targets. We aim to develop an automated feature generation method that: (1) offers a transparent and understandable feature generation process; (2) adapts to various data types and downstream tasks by selecting best-suited operations automatically; (3) continuously learns from the feature space and refines its strategies based on new insights. For target (1), we explicitly document the operations applied during the feature generation and how they transform the input data into a structured feature space. By doing so, we intend to replace their lack of interpretability with a more open and interpretable framework. For target (2), we apply a large language model (LLM) that can assess the characteristics of the data and the requirements of the task and dynamically select the most appropriate strategies under proper guidance. For target (3), we develop multiple expert-level LLM agents with different strategies. These agents adjust their operations and strategies in response to evolving data patterns, task requirements, and actions of other agents.

OurMethod. We employ a novel approach to automate and enhance the feature generation process by adopting LLMs with expert-level guidance. Our method starts by inputting an original feature set and a predefined operation set into the LLM, which then instantiates several expert agents. Each agent generates new features with its exclusive strategy and integrates these new features with the original features to produce enriched feature sets. The newly created feature sets are applied to downstream tasks to evaluate their performance, and the performance outcomes are fed back to the agents. The agents "communicate" and share their inferences, and reflect on the impact of their own and other agents' operations on the downstream tasks. In this way, the agents refine and optimize their generation strategies to better capture how variations in the feature space impact downstream tasks and, consequently, reconstruct the feature space more adaptively and effectively. The iteration continues with new feature sets until the optimal feature set for the task is found or a predefined number of iterations is reached.

Contributions. Our main contributions include:

• We propose a novel automated feature generation methodology based on LLMs to restructure the feature space of a dataset. This end-to-end structure enables training with LLM agents on varied feature generation strategies without manual selection of feature operations.
• We introduce a dynamic and adaptive generation process based on feedback from downstream tasks. This method effectively utilizes the in-context learning and reasoning capabilities of LLMs in the feature generation process.
• It significantly enhances the applicability and effectiveness of the generated features across various machine learning scenarios without the need to create additional ML models.
• We conduct a series of experiments to validate the effectiveness and robustness of our method across different datasets and downstream tasks. Our results demonstrate that our method has clear advantages over existing methods and considerable potential to promote a broader range of feature engineering tasks.

2 Preliminary and RelatedWork

2.1 Feature Generation

Feature Engineering. Feature engineering is the process of selecting, modifying, or creating new features from raw data to improve the performance of machine learning models [Severyn and Moschitti, 2013]. This process can be described as a function ϕ : F →F ′ that transforms the original feature set F into a new feature set F ′ through certain operations. Feature generation is a decisive category in the field of feature engineering. Feature generation means generating new attributes from existing features in a dataset through various mathematical or logical transformation operations. Feature generation methods can primarily be categorized into two main classes: (1) latent representation learning based methods, such as deep factorization machines [Guo et al., 2017; Song et al., 2019] and deep representation learning [Zhong et al., 2016; Bengio et al., 2013]. These methods can create complex latent feature spaces; however, their generation processes often lack transparency and are difficult to trace or explain [Sch¨olkopf et al., 2021]; (2) feature transformationbased methods, which generate new features by arithmetic or aggregation operations [Nargesian et al., 2017; Wang et al., 2022c; Wang et al., 2022b]. These methods often require extensive manual operations and rely heavily on domain knowledge to select transformations and adjust parameters.

Automated Feature Generation. With the widespread adoption of large models and various deep learning methods in the feature engineering field, automated feature generation is experiencing significant development. Automated feature generation enhances the feature space by systematically creating and integrating new features to improve model performance [Xiang et al., 2021; Wang et al., 2022a; Pan et al., 2020; Wang et al., 2022a]. For example, [Katz et al., 2016] developed ExploreKit, which generates a large set of candidate features by combining information from the original features. [Khurana et al., 2016] explores the feature space using manually crafted heuristic traversal strategies, while [Shi et al., 2018] proposed a feature optimization approach using deep learning and feature selection to enhance traffic classification performance. Although these methods are more efficient than traditional manual feature engineering and enable faster processing of large datasets, they often ignore the semantic aspects of data. Furthermore, their "black box" operation process makes the results difficult to explain.

2.2 LLMs and Tree of Thoughts

LLMs' Capabilities. LLMs have revolutionized numerous fields with their extraordinary capabilities in in-context learning and step-by-step reasoning [Xie et al., 2024; Zhang et al., 2024a; Wei et al., 2022]. These capabilities enable LLMs to effectively understand and navigate complex tasks given appropriate guidance and precise prompts. Thus, they exhibit professional competence across various specialized domains [Wang et al., 2023; Zhang et al., 2024c; Zhang et al., 2025a; Wang et al., 2025]. Recent advancements in prompt engineering and reasoning structures have further enabled these models to tackle complex data analysis tasks and multi-step feature engineering processes [Peng et al., 2023; Liu et al., 2023].

Prompting. LLM prompting leverages pre-trained models to recall existing knowledge and generate output based on specific guideline prompts [White et al., 2023]. This process interacts closely with LLMs' capabilities for in-context learning and step-by-step reasoning [Chu et al., 2023]. Prompts containing context details guide models to think more coherently when handling long tasks. Moreover, step-bystep reasoning prompts can guide a model through logical thought processes by mimicking human-like reasoning to solve complex problems [Yu et al., 2023; Wei et al., 2022; Diao et al., 2023].

Tree of Thoughts. One line of work in prompting methods focuses on enhancing the structured reasoning capabilities of LLMs to guide them through logical analysis and problem-solving tasks. Input-Output (IO) Prompting [Wang et al., 2024; Hao et al., 2023] involves wrapping the input x with task-specific instructions or examples to guide the model to produce the desired output y. This method is straightforward, but its guiding capability is limited in complex tasks. Chain-of-Thought (CoT) Prompting [Wei et al., 2022; Besta et al., 2024; Zhang et al., 2025c] involves a sequence of intermediate, coherent language expressions z1, . . . , zn that logically bridge the input x to the output y. This method enhances the model's logical capabilities for complex problemsolving, but it is limited to linear, single-strategy reasoning, capturing only a subset of the possible solutions. Tree of Thoughts (ToT) Prompting [Yao et al., 2024] extends the CoT prompting by exploring multiple reasoning paths over thoughts. ToT operates as a search over a tree structure where each node represents a partial solution within the input and contextual thoughts. ToT has advantages over other methods by exploring multiple reasoning paths with different strategies [Zhang et al., 2025b]. Thus, it performs better in deep data analysis and multi-step feature engineering.

3 Methodology

In this section, we design a novel feature generation method called LLM Feature Generation (LFG), which is highly interpretable, dynamically adaptable, and employs an end-toend approach to significantly enhance downstream task performance. In this method, we apply an LLM as an autogenerator of features. We augment the LLM with the ToT technique to generate detailed inference along with each step of generation so that each decision is explainable. Furthermore, we collect the performance of downstream tasks to offer feedback to the LLM generator. Our entire workflow is divided into two main parts: (1) Automated Feature Generation with LLM Agents; (2) Feedback and Optimization.

3.1 Important Definitions

We first define the feature set and operation set, along with their corresponding mathematical symbols.

Feature Set. Let D = {F, y} be the dataset. Here, F = {f1, f2, . . . , fn} is the feature set, where each column represents a feature fi, and each row represents a data sample; y is the corresponding target label set for the samples. Our main purpose is to reconstruct the feature space of the dataset D = {F, y}.

Operation Set. The operation set O consists of mathematical operations performed on existing features to generate new ones. Operations are of two types: unary and binary. Unary operations include "square", "exp", "log", etc.; binary operations include "plus", "multiply", "divide", etc.

Optimization Objective. Our objective is to construct an optimal and interpretable feature space that can enhance the performance of downstream tasks. Given the feature set F and the operation set O, our optimization goal is to find a reconstructed feature set ˆ F:

where R is a downstreamML task (e.g., classification, regression, ranking, detection), θ is the performance metric of R, and ˆ F is a reconstructed feature set derived from F. Here, ˆ F is generated by applying operations from O to the original feature set F using a certain algorithmic structure.

3.2 Automated Feature Generation with LLM Agents

In this section, we introduce the concept of LLM agents and detail their role in feature generation. We utilize an LLM to generate several agents for feature-generation tasks. Here, each agent acts as an automated feature generator, which applies specific operations on a given feature set to generate new features or delete existing features based on iterative prompts. These agents can emulate expert-level logical reasoning and decision-making capabilities, thus optimizing the feature set for input data of downstream tasks and enhancing their performance.

Agents. We first define an agent Al that operates on a feature subset Fl = {f1, . . . , fn} by applying operations to the features fi and fj. The Fl is a subset of F. The agent draws operations from an operation set O, to generate new features:

Each new feature gk is produced by applying an operation o to the features fi and fj:

Here, Fl is the initial feature subset for Al; F ′ l is the output feature subset with original and all new features from Al; and gk is the new feature from applying operation o to features fi, fj ∈ Fl. Thus, we complete one generation node sl of Al, and layer Layer1 consists of l total generation nodes.

3.3 Feedback and Optimization

We now have a new feature set composed of all l subsets, denoted as F ′ l = {Fl, g1, . . . , gk}. After the agents generate all l subsets, we move forward to evaluate the performance of these subsets in downstream tasks and the effectiveness of the generated nodes at this layer. We then gather performance metrics from downstream tasks as feedback θ = {θt}Ti=1 , for each iteration t corresponding to each generation layer. This feedback reflects the effectiveness of the newly created feature subset and illustrates the agents' contribution to the task. The agents evaluate their performance by comparing the metric θt with θ(t-1). If θt significantly improves over θ(t-1), the feature generation strategy is considered "effective". The agents' self-evaluation positively correlates with the increase in these metrics; a larger improvement signifies a more successful feature generation strategy.

Furthermore, each agent shares its reasoning process during generating features, including the logic behind selecting specific operations. This transparency enables agents to understand and adopt diverse strategies to achieve collaborative improvement. Through self-evaluation and peer interactions, agents identify which strategies successfully enhanced performance and which may require adjustments or replacements. In the subsequent rounds, agents generate increasingly effective feature subsets adopting F ′ l (from the previous round) and O to form the next layer of generated nodes. The iterative process continues until the optimal feature set ˆ F is found or the maximum number of iterations T is reached:

where {F opt l | l ∈ L} is the set of optimal feature subsets from each iteration l, and θ(F) is the performance score of a feature set F.

Monte Carlo Tree Search. In order to further optimize the feature space and balance exploration with exploitation, we employ an improved Monte Carlo Tree Search (MCTS) [Yao et al., 2024; Browne et al., 2012] to explore new or underutilized feature space structures after the initial T iterations. We adopt this MCTS framework because of its random samplingbased search strategy, which enhances the probability of finding superior feature combinations. In our case, this flexible searching method constructs a tree from all agents' generation nodes and adjusts the search direction based on the performance feedback. Here, we define the root node as the original feature set given to the LLM, with each generation node representing a tree node. The search covers all generation nodes in the last iteration layer LayerT as leaf nodes.

We start with node evaluation by calculating the improvement in the downstream task's performance compared to the parent node. The evaluation reflects the effectiveness of newly generated features. Then, the MCTS iteratively builds a search tree by cycling through four phases: selection, expansion, evaluation, and searching. In the selection phase, children nodes are recursively selected from the root utilizing the Upper Confidence Bound (UCB) [Auer et al., 2002]:

where si and s′i are the visit counts for node i and its parent respectively, C is a hyperparameter balancing exploration and exploitation. The value wi represents the average performance improvement of all descendant nodes of node i, computed based on changes in the downstream task performance metric θ. Specifically, for each node i, the value of wi is calculated as the average performance improvement observed from its child nodes in the downstream tasks. This is given by:

Here, Ci is the set of child nodes of node i, and |Ci| is its cardinality (i.e., the number of child nodes of i). ∆θ(j) = (θ(j) t -θ(j) (t-1) ) represents the performance improvement contributed by child node j ∈ Ci, where θ(j) t and θ(j) (t-1) are the performance metrics associated with the state after exploring child j and the state before, respectively.

In the tree, we select the most promising node sl. We define the selection criterion for each node as:

where ψ(sl) and ϕ(sl) represent the current and future value of the node. We apply further strategies to generate and expand feature subsets upon selecting these nodes.

On the selected nodes sselect l , more complex or novel feature operations are implemented to generate additional feature subsets. These newly generated subsets, F ′ l , are then used for further evaluation in downstream tasks to validate their efficacy. The process is mathematically defined as:

where G is the transformation function generating F ′ l by applying operations from a selected subset O′ ⊆ O (the predefined complete operation set) to F select l . At the end of the MCTS process, the optimal feature subset is determined by choosing the leaf node with the highest wi value, representing the feature set that maximized performance improvement during simulations:

This F ∗ represents the optimal feature subset identified by the MCTS, which should then be validated further in real-world tasks to confirm its efficacy.

4 Experiments

In this section, we present three experiments to demonstrate the effectiveness of the LFG. First, we compare LFG against several baseline methods on multiple downstream classification tasks. Second, we perform a robustness check on LFG's performance improvement. Finally, we further study and analyze iterative performance improvements in experimental results and discuss their reasons.

4.1 Experimental Setup

Datasets. We evaluate the LFG method on four real-world datasets from UCI, including Ionosphere (Ino) [Sigillito et al., 1989], Amazon Commerce Reviews (Ama) [Liu, 2011], and Abalone (Aba) [Nash et al., 1995], as well as Diabetes Health Indicators Dataset (Dia) [Teboul, 2022] from Kaggle. The detailed information is shown in Table 1. For each dataset, we randomly selected 55% of the data for training.

Metrics. We evaluate themodel performance by the following metrics: Overall Accuracy (Acc) measures the proportion of true results (both true positives and true negatives) in the total dataset. Precision (Prec) reflects the ratio of true positive predictions to all positive predictions for each class. Recall (Rec), also known as sensitivity, reflects the ratio of true positive predictions to all actual positives for each class. FMeasure (F1) is the harmonic mean of precision and recall, providing a single score that balances both metrics.

Downstream Tasks. We apply the LFG model across a range of classification models, including Random Forests (RF), Decision Tree (DT), K-Nearest Neighbor (KNN) and Multilayer Perceptrons (MLP). We compare the performance outcomes in these models both with and without our method.

Baseline Models. We compare the LFG method with raw data (Raw), the Least Absolute Shrinkage and Selection Operator (Lasso), and Feature Engineering for Predictive Modeling using Reinforcement Learning [Khurana et al., 2018] (RL). Here, we set the same operation set for both LFG and RL method to consist of square root, square, cosine, sine, tangent, exp, cube, log, reciprocal, sigmoid, plus, subtract, multiply, and divide. For our LLM, we perform all the experiments on the OpenAI API, GPT-3.5 Turbo model [OpenAI, 2024].

4.2 Experimental Results

Overall Performance. Table 2 shows the overall performance results for LFG and the baseline models.

(1) Comparison with baseline models. From the table, we can see that the LFG method consistently surpasses baseline methods across a variety of metrics and datasets. Specifically, we show the results of LFG in 3 iterations of generation, denoted as LFG-3, and compare it with the full LFG in T ≤ 10. On dataset Ion, the LFG achieves the best accuracy of 95.6%, which is a 6.4% increase over raw data on task RF, and 4.5% over the RL method. On dataset Ama, the LFG-3 improves the accuracy of raw data of 59.6% to 61.5% while LFG performs an increase of 4.3% on RF. Beyond accuracy, LFG consistently demonstrates significant improvements over baseline methods across other metrics and datasets. For instance, on dataset Ion, the highest recall improvement of 9.8% on DT reflects LFG's ability to effectively handle positive samples. Similarly, the improvements in F1, such as increasing from 0.476 to 0.607 on dataset Ama on KNN and from 0.855 to 0.932 on dataset Ion on DT, showcase the model's adaptability across diverse tasks and feature distributions.

(2) Performance across different models and metrics. In precision, the LFG also shows superior performance. For example, on dataset Ion, the LFG reaches a precision of 90.6%, surpassing the highest precision of 86.7% among all baselines on KNN. This result indicates that the LFG is effective in reducing misclassifications. Regarding the metric recall, the LFG increases the Ion's recall by 9.8% compared to vanilla data and by 4.3% compared to best of baselines on DT task, which demonstrates that LFG brings higher improvement on identifying positive samples. On the Ama dataset, our method significantly improves F1. The 3-iteration version, LFG-3, increased the F1 from an initial 63.8% to 67.6%. The full LFG model further boosted this performance, reaching a final F1 of 68.0%. As the harmonic mean of precision and recall, this substantial improvement in F1 showcases the model's capability to effectively balance between reducing misclassifications and minimizing missed classifications.

Robustness Check. We validate the experiments through five-fold cross-validation. The LFG method demonstrates robust performance, achieving notable metric improvements. With LFG-3, it gains 2.72% in accuracy and approximately

5% in precision, recall, and F1. The full LFG model further boosts these gains to 4.81% in accuracy and approximately 7% in other metrics. This consistent enhancement is driven by the extended iterations, which continue to refine the feature space, and it illustrates the adaptability, effectiveness, and reliability of the LFG method.

Iterative Performance Improvements. We then compare the performance improvements of LFG-3 and LFG. On one hand, the incremental improvement from LFG-3 to LFG illustrates the continuous enhancement in all metrics, indicating that during additional iterations, the model refines and adjusts its generation strategy to explore better solutions for the feature space. On the other hand, the results of LFG3 with only three iterations already show notable enhancement in all datasets. This shows that the model effectively and rapidly finds the zones of optimal feature space within a limited number of steps. As shown in Figure 5, we can see that the average number of features in a feature subset increases continuously in the first 5 rounds of generation. The increase in features implies that the feature generation algorithm is actively finding new, potential feature combinations in the feature space, effectively expanding the feature space.

5 Conclusion

In this paper, we present a novel feature generation method LFG utilizing LLMs to enhance automated feature engineering's interpretability, adaptability, and strategic flexibility. Our target in designing the model is to address traditional feature generation challenges, including a lack of explainability, limited applicability, and rigid strategy formulation. Our approach utilizes expert-level LLM agents to overcome these. These problems can significantly limit the broader deployment of feature engineering on machine learning tasks in diverse scenarios. Thus, we present LFG with a dynamic, adaptive, automated feature generation approach to enhance interpretability and extend utility across various data types and tasks. Moreover, we present extensive experimental results demonstrating that our approach outperforms various existing methods. For future work, we plan to extend this automated feature engineering paradigm across various techniques and machine learning tasks and investigate transformations our approach applies to the feature space to gain deeper insights into its underlying mechanisms and impacts.

6 Limitations and Ethics Statements

While our LFG shows significant advancements and wide adaptability, there are several limitations that require further exploration, including: (1) high computational demands and limited scalability with very large or complex datasets. With LLM agents being the generator, the computational demands are comparatively higher than traditional methods; (2) the effectiveness of the generated features heavily relies on the quality of input data, which can affect the model's performance in scenarios with poorly curated or noisy datasets; (3) the current work focuses solely on the tabular feature generation process. More complex generation methods on different data are not considered. Extending the LFG to handle feature generation in non-tabular scenarios remains a challenge.

The LFG framework can enhance the effectiveness and explainability of the feature generation process. However, as the framework uses pre-trained GPT-3.5 Turbo as the generation model, it may inherit the ethical concerns associated with GPT-3.5 Turbo, such as responding to harmful queries or exhibiting biased behaviors.

Acknowledgements

Pengfei Wang is supported by the National Natural Science Foundation of China (Grant Nos. 62406306 and 92470204).