Towards Automated Kernel Generation in the Era of LLMs

1 Introduction

Rapid scaling of large language models (LLMs) has made efficient hardware utilization a central challenge in modern AI systems [Kaplan et al., 2020]. Consequently, specialized accelerators such as GPUs and NPUs have become the foundation of large-scale training and inference [Choquette et al., 2021; Liao et al., 2021]. Their performance is largely governed by kernels implementing fundamental operations, such as matrix multiplication and attention, which dominate execution time in LLM workloads. As a result, overall system throughput, efficiency, and cost are often determined more by kernel quality than by hardware peak performance.

Despite their foundational role, the development of efficient kernels remains a formidable engineering challenge. Achieving near-peak hardware utilization requires deep expertise in both algorithmic design and hardware-specific intricacies. Furthermore, kernel optimization is inherently nonscalable: implementations are often tightly coupled to particular hardware architectures and workload characteristics, which hinders their reuse and generalization across different GPU generations or hardware vendors [Wu, 2023]. Although compiler-based autotuning approaches partially alleviate these scalability challenges through automated schedule generation and optimization, they remain fundamentally constrained by manually designed search spaces, scheduling primitives, and optimization priors.

In response to these challenges, LLMs and agentic systems have emerged as a promising paradigm for kernel generation and optimization. Trained on large-scale code repositories and technical documentation, LLMs encode substantial expertise in hardware-aware programming, enabling them to bridge the gap between high-level algorithmic specifications and low-level implementations. Beyond one-shot code generation, agentic frameworks leverage iterative execution feedback to refine candidate kernels, facilitating scalable and more open-ended exploration of complex optimization spaces across workloads and hardware platforms. Consequently, LLMs and agents are increasingly becoming a foundation for automated kernel development.

Despite rapid progress, research on LLM-driven kernel generation remains fragmented and lacks a systematic synthesis. This survey addresses this gap by presenting a unified overview of the field, clarifying foundational concepts, and highlighting emerging methodologies and trends. A key contribution is our consolidated resource infrastructure, which features a structured organization of trainingready kernel datasets and a literature collection tailored for retrieval-augmented generation (RAG), designed to facilitate data-driven research in this specialized kernel-generation domain. Moving beyond a synthesis of existing methodologies, we also highlight critical open challenges and propose promising research directions, aiming to establish a foundational reference for the next generation of innovation in LLMdriven kernel generation.

2 Background

LLMs and LLM-based Autonomous Agents. The foundation of modern LLMs is the Transformer architecture [Vaswani et al., 2017], which functions as the probabilistic predictor trained via the next token prediction objective. Given a sequence of tokens x = (x1, . . . , xT ), the model maximizes the joint probability:

This objective enables the model to internalize world knowledge and reasoning patterns implicitly during pretraining.

While LLMs serve as the cognitive engine for reasoning and decision-making, autonomous agents extend this capability by integrating additional system components such as planning, memory, and tool-use mechanisms, and interact with the environment through trial and error [Wang et al., 2024]. In this framework, the LLM functions as the "brain", orchestrating actions through reasoning strategies. And agents utilize tools to perform actions beyond the model's internal knowledge and receive environmental feedback.

Kernel Programming and Code Generation. Kernel generation and optimization have traditionally followed two paradigms. The first relies on expert-written kernels and domain-specific abstractions, such as CUDA, CUTLASS, and TileLang, which enable fine-grained hardware optimization but require substantial architecture-specific expertise. The second centers on compiler-driven frameworks, such as Halide and TVM, which optimize kernels through scheduling and autotuning. While improving programmability, these approaches remain constrained by predefined search spaces, scheduling primitives, and handcrafted optimization rules. In contrast, LLM-based agents leverage knowledge distilled from large-scale code corpora and execution feedback to enable scalable and open-ended kernel optimization.

In parallel, LLMs have advanced code generation from code completion to complex software engineering tasks. However, kernel generation differs fundamentally from general-purpose code synthesis: beyond functional correctness, it must satisfy stringent performance requirements and adapt to hardware-specific execution characteristics. Consequently, it is more closely aligned with performance-oriented program synthesis and compiler optimization, necessitating specialized techniques beyond generic LLM-based code generation. This trend is also emerging in broader parallel programming systems, where LLMs have been applied to performance-oriented optimization beyond kernel generation [Wei et al., 2025a].

3 LLM for Kernels Generation

Building on advances in LLM-driven code generation, recent work has increasingly applied LLMs to the generation of high-performance GPU kernels. To highlight the methodological patterns that have emerged across this landscape, the following sections review two dominant families of posttraining techniques used to specialize LLMs for kernel generation: supervised fine-tuning and reinforcement learning.

3.1 Supervised Fine-Tuning

Supervised fine-tuning (SFT) has become a central methodology for enabling LLMs to synthesize high-quality kernels, relying on paired datasets that capture both high-level computational intent and low-level kernel implementation patterns. KernelLLM [Fisches et al., 2025] adopts this strategy by collecting samples and using the Triton compiler to produce aligned PyTorch-Triton examples, and applies instruction tuning with structured prompts that explicitly encode the mapping between computation and kernel structure. And ConCuR [Kong et al., 2025] further generates and curates high-quality kernel datasets with reasoning traces, motivated by the observation that the structure and clarity of model reasoning can strongly affect kernel correctness and performance. Fine-tuning on such data leads to KernelCoder, a model capable of generating CUDA kernels that achieve 17% on the fast1 metric of KernelBench [Ouyang et al., 2025] level 1. Moreover, to address the stringent demands of industrial software development, InCoder-32B [Yang et al., 2026] introduces a three-stage data curation pipeline consisting of pre-training, mid-training, and post-training, achieving a fast1 score of 22.2% on KernelBench Level 1.

3.2 Reinforcement Learning

Reinforcement learning further enhances kernel generation via iterative feedback. Kevin [Baronio et al., 2025] models kernel generation as a multi-turn optimization using crossturn reward attribution for long-horizon credit assignment. CUDA-L1 introduces contrastive RL with an LLM-as-ajudge for dense feedback, and is refined by CUDA-L2 [Su et al., 2025], which reports performance improvements over cuBLAS on its evaluated workloads. SparseRL [Wang et al., 2026] leverages RL to generate high-performance CUDA code for sparse matrix operations. CUDA Agent [Dai et al., 2026] introduces a large-scale agentic reinforcement learning system, including a skill-augmented CUDA development environment with automated verification and profiling to provide reliable reward signals, achieving state-of-the-art results on KernelBench, delivering a 99% faster rate over PyTorch Eager on KernelBench Level-1.

In addition to CUDA kernel generation, recent studies have investigated RL-based approaches for Triton kernel generation. AutoTriton [Li et al., 2025d] generates Triton kernel with the model trained by RL and addresses reward sparsity by combining structural assessments of generated kernels with execution-based runtime rewards, while TritonRL [Woo et al., 2025] extends this line of work through hierarchical reward decomposition and explicit verification of code outputs and intermediate reasoning traces. QiMeng-Kernel [Zhu et al., 2025] further structures optimization by applying RL hierarchically to macro-thinking strategies rather than lowlevel implementation. Dr. Kernel [Liu et al., 2026] introduces a robust distributed GPU environment for Triton kernel generations, combined with effective multi-turn RL methods to address the biased policy gradient issue and alleviates lazy optimization problem. Kernel-Smith [Du et al., 2026] presents a stable evolution-oriented post-training recipe, and the Triton kernels generated by this framework achieve a 70% fast1 score on KernelBench level 1. Finally, AscendKernelGen [Cao et al., 2026b] expands the alignment learning paradigm to Ascend NPUs and generates AscendC kernels, combining CoT-based SFT with DPO (Direct Preference Optimization).

4 LLMAgent for Kernels Generation

Relying on foundational LLMs alone typically reduces kernel development to a static one-pass inference process. In contrast, LLM-based agents introduce autonomy and feedback into the optimization loop by enabling planning, tool use, and evaluation of intermediate results. This closed-loop, selfimproving paradigm allows agent-based approaches to scale kernel optimization across diverse workloads and hardware platforms, while sustaining long-horizon, fatigue-free exploration. Concretely, we categorized recent agent-driven advancements into four structural dimensions: learning mechanisms, external memory management, hardware profiling integration, and multi-agent orchestration.

4.1 LearningMechanisms

This category studies how LLM agents interact with execution environments and learn effective kernel generation strategies through trial-and-error. Initial approaches view kernel generation as iterative refinement. Caesar in KernelBench utilizes simple feedback loops to refine kernels, while InferenceTime Scaling [Chen et al., 2025b] demonstrates that scaling test-time compute and reflection significantly boost kernel quality. To manage complexity, PEAK [Tariq et al., 2025] employs a modular iterative stepwise refinement strategy. AutoKernel [Jaber and Jaber, 2026] profiles the full model to identify computational bottlenecks and employs an iterative optimization process to refine the corresponding kernel implementations. MaxCode [Ou et al., 2026] further unifies existing iterative search methods under a max-reward reinforcement learning framework, combined with a natural language critique model converting raw execution feedback into diagnostic insights. K-Search [Cao et al., 2026a] leverages LLMs as world models, utilizing their prior domain knowledge to guide the search process. It decouples high-level algorithmic planning from low-level program instantiation, and demonstrates strong performance on diverse and complex kernels.

Furthermore, a large body of agent-based methods employs iterative refinement for GPU kernel generation targeting diverse model architectures, programming languages, and hardware backends. For example, DiffAgent [Zhu et al., 2026] adopts iterative refinement to accelerate diffusion models, TritonX [Hammond et al., 2025] uses iterative refinement within a state machine to cover kernels of complete PyTorch ATen backends, and KernelGen [BAAI, 2025] leverages testtime scaling and reflection techniques to enable kernel generation for multi-chip backends. Moreover, many existing CLI-based agents such as Claude Code or OpenCode also support iterative refinement, and AKO [Xie et al., 2026] provides harnesses for such coding agents to enable agentic kernel optimization and achieve state-of-the-art performance on challenging benchmarks.

To escape local optima, recent frameworks adopt population-based evolution. Lange et al. [Lange et al., 2025] optimize translation CUDA via mutation and crossover. FM Agent [Li et al., 2025a] includes an evolutionary stage with the principles of diversity preservation, adaptive evolution, and multi-population dynamics. Advanced population dynamics are also introduced in EvoEngineer [Guo et al., 2025], which decouples traversal techniques from population management. GPU Kernel Scientist [Andrews and Witteveen, 2025] employs a multi-stage evolutionary workflow to address the challenge of optimizing HIP kernels for the AMD accelerators. And cuPilot [Chen et al., 2025a] guides evolution through high-level semantic strategies.

4.2 External MemoryManagement

Complex kernel optimization often requires domain-specific knowledge, such as CUDA APIs and hardware instruction sets that may be hallucinated or forgotten by the LLM. Agents in this category augment generation with relevant skills, domain knowledge, or prior interaction experiences, which serve as an external memory for LLMs. KernelEvolve [Liao et al., 2025] advances the external knowledge management paradigm by integrating a sophisticated hardware-specific knowledge base specifically tailored for heterogeneous AI accelerators. Beyond retrieving unstructured textual context, recent work has explored utilizing structured representations as external memory to guide model inference. Work such as ReGraphT [Gong et al., 2025] proposes a novel framework that treats a reasoning graph as a domain-specific external memory for CUDA code optimization. In this approach, the logical transitions between optimization states of large language models are externalized into a static, navigable graph structure for the small language model to retrieve. KernelBlaster [Dong et al., 2026] enables agents to learn from experience by accumulating knowledge into a retrievable persistent CUDA knowledge base, EvoKernel [Zheng et al., 2026] further introduces a value-driven memory and retrieved prior experiences or trajectories with different priorities. KernelSkill [Sun et al., 2026] provides a dual-level memory architecture, where reusable expert skills are curated in the longterm memory.

4.3 Hardware Profiling Integration

The third dimension addresses the hardware-agnostic nature of standard LLMs by configuring the agent's persona profile with hardware specifications, and iteratively reasoning over performance profiling feedback.

QiMeng-TensorOp [Zhang et al., 2025a] triggers LLMs to analyze and distill low-level hardware documentation according to user input into the generation prompt, while QiMeng-GEMM [Zhou et al., 2025b] generates General Matrix Multiplication (GEMM) with the meta-prompt, which offers universal templates for various general optimization techniques and platform-specific optimization details. QiMengAttention [Zhou et al., 2025a] considers target GPU architecture and instruction set to convert the high-level thinking language into low-level CUDA code, and implements the highperformance FlashAttention on different GPUs. SwizzlePerf [Tschand et al., 2025] explicitly tackles the swizzling problem, which explicitly injects precise architectural specifications into the prompt context and restricts the search space specifically to swizzling patterns that focus solely on maximizing the L2 cache hit rate.

Complementing this, agents leverage dynamic feedback. CUDA-LLM [Chen et al., 2025c] incorporates detailed target GPU specifications (e.g., warp size, cache size) into the agent's prompt. Simultaneously, compilation logs and runtime performance metrics are also aggregated to guide the optimization process. TritonForge [Li et al., 2025b] utilizes profiling-guided feedback loops to iteratively analyze and identify performance bottlenecks. PRAGMA [Lei et al., 2025] uses a specialized profiling module to parse low-level quantitative metrics into an interpretable natural language suggestion. KernelBand [Ran et al., 2025] clusters runtime behavior of potential kernels to reduce the exploration space and utilizes profiling data as context to guide the selection of optimization strategies.

4.4 Multi-Agent Orchestration

Recognizing that kernel development inherently involves heterogeneous abilities ranging from algorithmic planning to low-level coding and debugging, recent work increasingly adopts multi-agent designs that explicitly decompose these responsibilities into coordinated roles.

STARK [Dong et al., 2025] structures generation into PlanCode-Debug phases to emulate the human team, AKG [Du et al., 2025] leverages similar modularity to achieve crossplatform synthesis, while Astra [Wei et al., 2025b] specializes this multi-agent approach for production-grade SGLang kernels. CudaForge [Zhang et al., 2025b] employs a Coder-Judge framework driven by hardware-level feedback, whereas KForge [Sereda et al., 2025] adapts this dual-agent model to new platforms using only single-shot example supervision. Addressing scale, KernelFalcon [Team and Contributors, 2024] employs a multi-agent system to tackle the challenge of GPU kernel generation of full machine learning architectures, where the system specifically addresses hierarchical task decomposition and delegation through coordinated manager and worker agents. Moreover, GEAK [Wang et al., 2025] targets AMD GPUs, integrating the generator, reflector, evaluator and optimizer within a Triton-based workflow.

5 Datasets

The effectiveness of LLM-driven kernel generation depends critically on the availability of domain-specific data. Unlike general software engineering tasks, kernel generation requires models to capture hardware intrinsics, parallel execution semantics, and memory hierarchy constraints. We categorize existing resources into two groups: (1) Training Corpora, comprising structured datasets and kernel code repositories used for model training; and (2) Knowledge Bases, which typically provide domain-specific knowledge for RAG.

Training corpora include both structured datasets and raw kernel repositories. Structured datasets typically pair highlevel intents with optimized implementations, whereas repositories provide collections of expert-written kernels drawn from open-source libraries, training and inference frameworks, and domain-specific languages. Complementing executable code, knowledge bases supply hardware and optimization knowledge through technical documentation, programming guides, tutorials, and community resources. Such knowledge can be incorporated into model pretraining or accessed dynamically through RAG. A comprehensive summary of these resources is provided in Table 1. The dates reported correspond to the initial release of each resource.

6 Benchmark

This section focuses on systematic benchmarking of kernel generation and provides a structured overview of representative evaluation benchmarks, including both evaluation metrics and benchmark datasets, to lay a solid foundation for subsequent method comparison and performance analysis.

6.1 Metrics

Kernel evaluation typically considers three aspects: correctness, performance, and composite quality metrics. Most benchmarks adopt execution-based testing, where generated kernels are executed and compared against reference CUDA or PyTorch implementations. Since kernel generation is inherently stochastic, evaluations are often repeated across multiple samples and random seeds.

Correctness measures whether a generated kernel produces outputs consistent with a reference implementation within predefined numerical tolerances. Evaluation criteria vary across kernel types and precision formats (e.g., FP16, BF16, and FP8), reflecting their distinct numerical characteristics. Detailed evaluation protocols are provided in FlashinferBench [Xing et al., 2026].

Performance is primarily measured by runtime speedup relative to a reference implementation, typically using wallclock execution time averaged over repeated trials. Some benchmarks further compare achieved performance against Speed-of-Light (SOL) estimates, which approximate the theoretical efficiency limit of the target hardware.

In addition, Efficiency refers to how effectively the generated operators utilize computation resources during execution, and Compatibility is considered when evaluating operator generation techniques across different hardware platforms or languages. Composite metrics jointly assess multiple dimensions of kernel quality. Representative examples include Similarity, which evaluates code similarity using lexical, syntactic, and dataflow features; and fastp, which combines correctness and performance by measuring the proportion of generated kernels that are both correct and achieve a speedup greater than p:

6.2 Benchmark Datasets

As summarized in Table 2, kernel benchmarks are progressively evolving toward more realistic and comprehensive evaluation. This evolution is reflected in three dimensions: metrics, where evaluation has expanded from correctness and runtime speedup (ParEval [Nichols et al., 2024]) to comprehensive measures of efficiency, robustness, and overall kernel quality. Examples include fastp in KernelBench, efficiency metrics in TritonBench [Li et al., 2025c]; hardware coverage, where benchmarks increasingly extend beyond NVIDIA GPUs to encompass AMD GPUs, Huawei NPUs, and Google TPUs (such as MultiKernelBench [Wen et al., 2025]); and workload diversity, where evaluation has shifted from simple operators toward production-grade kernels derived from realworld frameworks, systems, and AI workloads. For example, FlashInfer-Bench and SOL-ExecBench [Lin et al., 2026] focus on critical kernels derived from production systems and emerging AI models, and BackendBench [Saroufim et al., 2025] targets complex edge cases.

7 Challenges and Opportunities

Despite rapid progress, LLM-driven kernel generation remains a nascent research area. Advancing from proof-ofconcept systems to robust and scalable deployment requires addressing challenges spanning evaluation, data, agent capabilities, infrastructure, and human-AI collaboration. This section reviews these challenges and outlines promising directions for future research.

EvaluationReliability andGeneralization. Reliable evaluation remains a fundamental challenge for LLM-driven kernel generation. Existing systems are vulnerable to reward hacking, where kernels achieve favorable benchmark scores without delivering corresponding benefits in practical deployments. At the same time, current benchmarks often cover limited workloads, hardware platforms, and execution settings, raising concerns about the generalization of reported methods. Furthermore, the impact of kernel-level improvements on end-to-end AI systems remains insufficiently understood. Future evaluation frameworks should emphasize robustness against reward hacking, broad generalization across workloads or platforms, and system-level assessment, providing a more reliable measure of real-world effectiveness.

Data Scarcity and Synthetic Scaling. Data scarcity remains a major bottleneck for LLM-driven kernel generation. High-performance kernels are sparsely represented in existing corpora, and predominantly contain final implementations while largely omitting optimization trajectories and hardware-aware expertise. As a result, critical signals for learning kernel optimization remain limited. Promising directions include large-scale kernel dataset construction, synthetic data generation, and the collection of execution-driven optimization traces. Such resources could support both model training and agentic learning, and may be essential for scaling kernel generation capabilities.

Agentic Training and Harness Engineering. Kernel optimization poses a challenging long-horizon task that requires reasoning over iterative cycles of generation, execution, profiling, and refinement. Current foundation models are not explicitly trained for such long-horizon optimization trajectories, while existing agentic systems often rely on handcrafted workflows that struggle with exploration efficiency, context management, and long-term credit assignment. Future progress may require advances in both long-horizon agentic training and autonomous harness engineering. Moreover, recent advances in general-purpose coding agents (e.g., Table 2: Benchmark datasets for kernel generation and optimization. Metrics: C Correctness, S Speedup, E Efficiency, f fastp, S Similarity. Hardware Platforms: N NVIDIAGPUs, H HUAWEI NPUs, G Google TPUs, A AMDGPUs.

Claude Code, OpenCode) suggest that kernel optimization may not require developing task-specific agents from scratch. Instead, foundation agents can be specialized through harness engineering, leveraging domain-specific environments, tools, and feedback to enable scalable agentic performance engineering.

Scalable Infrastructure for Synthesis and Training. Scalable infrastructure remains a critical bottleneck for largescale data synthesis and agentic training. Key challenges include building distributed and securely isolated sandbox environments for execution, addressing latency mismatches between agent rollout and kernel compilation and verification, and designing fault-tolerant, multi-device services that ensure stable and efficient training at scale. Advances in such infrastructure are essential for transforming kernel synthesis and data sampling from low-throughput, ad-hoc experimentation into a systematic, data-driven evolutionary process, enabling continuous improvement of agentic kernel optimization systems.

Human-AI Collaboration for Kernel Generation. Human-AI collaboration represents a promising alternative to fully autonomous kernel optimization. A central challenge is to establish effective feedback loops between human expertise and agentic exploration. While human-in-theloop guidance can guide optimization through high-level objectives and constraints, human-from-the-loop learning enables knowledge discovered by agents to be transferred back to developers. Given the verifiable nature of kernel optimization, such bidirectional collaboration may create an effective cycle of human and agent co-evolving, expanding the frontier of automated performance engineering.

8 Conclusion

This survey highlights the transformative potential of LLMs and agentic systems for automating kernel generation and optimization, synthesizing recent advances in methodologies alongside the development of kernel-centric datasets and benchmarks. Looking ahead, future progress will depend on more reliable evaluation protocols, as well as the data infrastructure and harness engineering required for agentic kernel generation. These advances have the potential not only to alleviate the burden of manual kernel engineering, but also to unlock substantial productivity gains for rapidly scaling AI infrastructure.