A Survey on the Feedback Mechanism of LLM-based AI Agents

1 Introduction

An artificial intelligence (AI) agent refers to an autonomous system that performs tasks, makes decisions, or interacts with its environment based on a set of rules, training, or learned knowledge [Wang et al., 2024b]. Traditional feedback systems, such as symbolic [Xi et al., 2025] or reinforcement learning-based agents [Xi et al., 2025; Cheng et al., 2024], are often limited by rigid predefined rules or heuristic policy functions, which markedly diverge from the human learning process, where individuals demonstrate the ability to adapt flexibly across diverse contexts and learn from unstructured feedback. Consequently, the agents developed in these studies often fall short of replicating human-level generalpurpose decision-making abilities [Durante et al., 2024].

∗ corresponding author

Recently, large language models (LLMs) [Ouyang et al., 2022; OpenAI, 2023] have shown remarkable success across diverse tasks, exhibiting human-like reasoning and decisionmaking capabilities [Xi et al., 2025]. Building upon this success, there has been a surge of research interest [Cheng et al., 2024] in employing LLMs as central controllers to construct general-purpose AI agents capable of processing input and executing actions in open-domain scenarios. However, a critical challenge remains: LLM-based AI agents struggle to learn quickly and efficiently from trial-and-error processes in a manner comparable to humans. Consequently, the feedback mechanism has emerged as an increasingly vital component in LLM-based AI agents. This mechanism analyzes and evaluates behavioral data, environmental responses, and historical trajectories to guide agents in refining their strategies and actions. Researchers have developed numerous promising feedback mechanisms, spanning from reinforcement learning [Akyürek et al., 2023], self-refine [Madaan et al., 2024] to multi-model debate [Du et al., 2024a]. However, these models were developed independently, with limited efforts made to summarize and compare them holistically. While several surveys [Pan et al., 2024; Madaan et al., 2024] offer a general perspective of feedback, they focus on LLMs rather than the AI agents which operate within a broader ecosystem of interactions and decision-making processes. The feedback mechanism for LLM-based AI agents is more complex than for standalone LLMs, as it involves not only the refinement of language generation but also the agent's ability to interpret, learn from, and adapt to feedback continuously.

To fill this gap, we shed light on recent advances in feedback mechanisms for LLM-based AI agents. We start by introducing a generalized framework of LLM-based AI agents (§2). We then present a taxonomy of recent studies on feedback mechanisms for LLM-based AI agents (§3), categorized as internal feedback, external feedback, multiagent feedback, and human feedback. After that, we summarize the evaluation method and benchmarks specifically tailored for assessing feedback mechanisms (§4). Finally, we compare the representative methods of each category on three benchmarks (§5) and analyze the limitations of existing work and highlight important directions for future research (§6). We advocate collaborative efforts within the community to pave the way toward the ultimate goal of Artificial General Intelligence (AGI). The contributions of this work can be summarized as: (1) we present a structured taxonomy that categorizes existing works into four categories; (2) we provide the first comprehensive survey of recent advancements in feedback mechanisms for LLM-based AI agents, encompassing methodologies, evaluation protocols, and benchmarks; (3) we discuss the remaining limitations of current research and point out potential directions for future work.

2 Preliminary: AUnified Framework for LLM-based AI agents

Recent advances in LLM-based AI agents have tended to be modularized to improve the agent's efficiency, adaptability, and scalability. Inspired by previous studies [Cheng et al., 2024; Durante et al., 2024], we propose a unified framework to summarize these modules. As shown in Figure 1, an LLMbased AI agents is composed of five modules: perception, planning, feedback, memory, and action. The core functions and roles of each module are described below:

Perception: The perception module enables AI agents to analyze and understand environmental inputs, identifying key patterns to support task execution.

Planning: The planning module helps AI agents break down complex tasks into simpler sub-tasks, generating plans and guiding actions to ensure logical and reliable behavior.

Memory: The memory module stores past experiences to support decision-making, with long-term memory preserving stable knowledge and short-term memory handling real-time task information, thus enabling AI agents to accumulate expertise and transfer knowledge between different scenarios.

Action: The action module translates decisions into operational outputs, interacting with the environment through tools, APIs, and embodied actions, bringing the agent's capabilities to real-world applications.

Feedback: The feedback module serves as a critical component in equipping AI agents with self-reflection and self-optimization capabilities. Leveraging the power of LLMs, this module facilitates the agent's ability to critically evaluate its prior decisions and actions, enabling dynamic adjustments and optimizations that enhance both intelligence and adaptability. This mechanism is particularly crucial for achieving continuous evolution in complex environments, ensuring more reliable and efficient task execution.

Moreover, the feedback module is tightly coupled with the memory and planning modules, collectively optimizing the agent's decision-making and execution. Taking Voyager [Wang et al., 2023] iron-smelting task as an example: Check furnace inventory → Discover no furnace → Retrieve skills → Refer to chest crafting skill to craft a furnace → Place the furnace → Check raw iron inventory → Smelt 5 raw iron. When detecting the absence of a furnace in the inventory, the feedback module critiques this failure state, triggering a skill retrieval mechanism, thereby guiding the agent to learn furnace crafting.

3 Taxonomy of FeedbackMechanisms

As shown in Figure 2, we present a structured taxonomy that organizes existing research into four core categories: (1) Internal Feedback, where AI agents generate feedback internally to refine their strategies and actions; (2) External Feedback, where AI agents leverage external models or tools to enhance feedback quality; (3) Multi-agent Feedback, where multiple agents interact and provide feedback to one another; and (4) Human Feedback, where humans act as the primary source of feedback to guide agent behavior.

We illustrate the basic flow of each category in Figure 3 and summarize the representative works, feedback format, learning strategy, iteration (or not), and application tasks associated with each category in Table 1.

3.1 Internal Feedback

As shown in Figure 3(a), Internal feedback originates from the agent itself and serves to motivate proactive exploration, learning, and self-improvement. This type of feedback does not rely on explicit external goals; instead, it encourages the agent to discover latent patterns, generalize strategies, and develop adaptive behaviors [Liu et al., 2025]. Internal feedback often manifests as signals generated and utilized by the model itself [Pan et al., 2024]. For example, in internal feedback mechanisms based on Chain of Thought (CoT) reasoning, the model is prompted to generate answers with explanations. By identifying and reinforcing reasoning chains that lead to correct answers, the model's behavior can be fine-tuned, improving performance even without explicit external supervision. This process can be iterative, continuously enhancing the model's reasoning and generalization capabilities. Depending on whether cross-task knowledge is used, internal feedback can be categorized into intra-task and inter-task feedback.

Intra-task feedback refers to the feedback derived from an agent's historical steps during its trial-and-error interactions with the environment, serving as the most relevant and informative signals to guide the agent's future actions. For example, ReAct [Yao et al., 2023] uses LLMs to generate reasoning trajectories and task-specific actions in an interleaved manner, improving the ability to solve complex tasks. SelfRefine [Madaan et al., 2024] improves the initial output of LLMs by iterative feedback and refinement during the same task, leading to higher quality and more accurate results. Inter-task feedback involves the transfer of knowledge and experience across tasks, enabling the agent to apply lessons learned from previous tasks to adapt to new or similar tasks. Reflexion [Shinn et al., 2024] derives experiences from past tasks and applies them in subsequent tasks to improve performance in new scenarios. ExpeL [Zhao et al., 2024] retrieves and analyzes similar past trajectories, identifying success patterns by contrasting positive and negative examples.

Discussion: Intra-task feedback enables rapid optimization for the current task, but its task-specific nature limits broader knowledge learning and accumulation. Conversely, inter-task feedback facilitates cross-task learning and longterm experience integration, while it demands substantial memory to store historical task experience. The fundamental challenge lies in effectively combining these complementary approaches to harness their respective strengths.

3.2 External Feedback

External feedback, as illustrated in Figure 3(b), comprises environment-defined signals that evaluate and guide an agent's behavior toward goal achievement. For example, in reinforcement learning, scores, task completion indicators, and success rates are all forms of external feedback. In AI

agents, external feedback is usually provided by independent feedback models or external tools (such as compilers, search engines, or world models). Based on the external modules involved, external feedback can be categorized into web knowledge, game API, code interpreter, and world model.

Web knowledge is widely used to enhance AI agents' feedback capabilities. WebGPT [Nakano et al., 2021] enables web-based question answering by searching and browsing online content, then refining answers through user feedback. Game API serves as a key resource for improving agent functionality and adaptability. Voyager [Wang et al., 2023] introduces an iterative hinting mechanism that combines execution errors and self-validation to enhance program quality. Some approaches employ code interpreters to boost feedback effectiveness. StepCoder [Dou et al., 2024] learns from compiler feedback, improving its ability to understand complex requirements and generate better code. World model facilitates feedback loops by allowing AI agents to incorporate and adapt to real-world sensory and environmental data. PaLME [Driess et al., 2023] integrates real-world sensor feedback and visual feedback into a language model to establish a link between words and perceptions.

Discussion: External feedback offers LLM-based AI agents additional insights that they might not obtain through self-improvement alone. However, the effectiveness of such feedback is highly dependent on the quality and relevance of external information. Therefore, optimizing the acquisition and processing mechanism of external feedback is crucial to boosting the performance of LLM-based AI agents.

3.3 Multi-Agent Feedback

Multi-agent feedback, shown in Figure 3(c), mimics multiagent interactions where diverse perspectives converge to improve solutions. This process promotes knowledge sharing and stronger decision-making in either collaborative or adversarial ways.

Collaborative approaches encourage divergent thinking, where each agent presents and discusses its own points of view in multiple rounds, with the optimal solution as the final answer. MetaGPT [Hong et al., 2024] employs an executable feedback loop for iterative code refinement, with agents collaboratively enhancing code quality using execution history and debug logs. InteRecAgent [Huang et al., 2023] adopts a dual-role feedback mechanism, pairing a recommender agent with a reviewer agent to evaluate results and identify execution errors. Adversarial approaches aim to improve the quality of the output by employing multiple agents to debate the same viewpoints in multiple rounds until a consensus is reached. Du et al. [2024a] proposed a multiagent debate framework, featuring multiple agents expressing their arguments in a "tit-for-tat" state, and a judge manages the debate process toward final resolutions.

Discussion: In a multi-agent system, agents take on distinct roles and responsibilities, coordinating their actions to achieve shared goals, which reduces the burden on individual agents and enhances task performance. However, challenges such as coordination, reward alignment, and learning stability persist. For coordination, when three or more agents freely express their views, discussions may become uncontrolled; Solutions like introducing a dedicated coordination agent (e.g., ChatLLM [Hao et al., 2023]) can integrate responses to refine the final answer. Regarding reward alignment, misaligned goals and reward functions among agents may lead to conflicts or suboptimal outcomes. CollaQ [Zhang et al., 2020] addresses this by decomposing reward allocation to provide an innovative solution for decentralized Q-functions. For learning stability, multi-agent negotiations may converge on incorrect consensuses, PRD [Li et al., 2023] peer-ranking algorithm analyzes pairwise preferences to generate a final ranking, ensuring more accurate consensus answers.

3.4 Human Feedback

As shown in Figure 3(d), human feedback enables agents to iteratively refine their behavior according to human feedbacks while performing the task. Human feedback manifests in three primary forms: instructional feedback, corrective feedback, and preference-based feedback.

Instructional feedback provides direct task guidance through explicit human instructions. For instance, WebGPT [Nakano et al., 2021] uses imitation learning to train models by setting tasks in a way that humans can perform them and then optimizes the quality of answers through user feedback. Corrective feedback is provided when the agent's behavior deviates or contains errors, and humans intervene to correct these mistakes or guide improvements. Abramson et al. [2022] developed a 3D simulation framework capturing human corrections when agents stray from goals. ReHAC [Feng et al., 2024b] includes a policy model designed to determine the most opportune stages for human intervention within the task-solving process. Preference-based feedback shapes agent behavior through human preferences or choices. PrefCLM [Wang et al., 2025] introduces a human-in-the-loop pipeline that facilitates collective refinements based on user comparative feedback to train robots.

Discussion: Human feedback is closely tied to human values, guiding an agent's behavior and learning through direct human input. However, relying solely on human feedback is often impractical due to limited resources or realtime constraints, as it often requires continuous monitoring, evaluation, and adjustment by experts to ensure the agent aligns with desired outcomes. Several solutions have been proposed, such as LLM-as-a-judge [Zheng et al., 2023], which leverages the scalability and interpretability of LLMs to reduce the need for human intervention, providing scores and explanations, but it suffers from positional bias, verbosity bias, and limitations in evaluating mathematical problems. Agent-as-a-judge [Zhuge et al., 2024] integrates agent capabilities to provide intermediate feedback during task resolution, achieving better consistency with human evaluators.

Ethical concerns about human feedback are growing. Specifically, human feedback mechanisms have the potential to introduce biases into the training data, particularly along lines of gender, race, and culture, which in turn can amplify the inherent biases within LLMs. To mitigate these risks, future research should prioritize the development of more diverse learning methodologies, such as the integration of culturally representative datasets and the enhancement of cultural awareness in model design.

4 Evaluation and Benchmark

4.1 Evaluation

Although LLM-based AI agents have demonstrated remarkable capabilities in downstream tasks, the question of how to effectively evaluate feedback mechanisms remains unresolved. Current studies can be broadly categorized into Outcome-based evaluation and Process-based evaluation.

Outcome-based evaluation: Presently, most evaluations are Outcome-based evaluation, relying on the assessment of feedback mechanisms through end-to-end agent task performance. These methods typically measure the effectiveness of feedback mechanisms based on task success rates. Olausson et al. [2023] analyze LLMs' self-repair capabilities on the APPS coding benchmark, where the feedback mechanism is evaluated by the pass rate of programming tasks. TravelPlanner [Xie et al., 2024] assess feedback mechanisms by assigning agents constrained travel planning tasks and measuring tool usage error rates and planning failure rates. While providing valuable insights, these indirect evaluations often oversimplify the assessment process by focusing on aggregate outcomes rather than granular performance details.

Process-based evaluation: Traditional agent evaluation methods tend to focus solely on final outcomes, overlooking critical details during execution or relying heavily on manual assessment. Recently, DevAI [Zhuge et al., 2024] enables agents to evaluate each other, assessing both the final outcomes and intermediate execution steps for richer feedback. AMOR [Guan et al., 2024] solves problems through autonomous executions and transitions over disentangled modules, allowing to provide feedback to the individual modules, and thus naturally forms process supervision.

Discussion: Gaps remain in developing robust quantitative indicators for self-correction across diverse domains. Factors like learning stage, task complexity, and environmental diversity can significantly impact feedback effectiveness, highlighting the need for a more comprehensive evaluation system. Therefore, it is necessary to develop a more comprehensive feedback evaluation framework that integrates both Outcome-based and Process-based metrics.

In the future, standard evaluation methods in the field of code generation may use datasets like DevAI for intermediate process optimization and MBPP [Austin et al., 2021] for overall performance testing.

4.2 Benchmarks

As presented in Table 2, we summarize and categorize mainstream benchmarking approaches for evaluating LLM-based AI agents, with emphasis on their feedback mechanisms.

Reasoning: Reasoning tasks aim to test an agent's ability to handle complex reasoning by answering questions. Common benchmarks include HotPotQA [Yang et al., 2018], ScienceQA [Lu et al., 2022], FEVER [Thorne et al., 2018], and StrategyQA [Geva et al., 2021]. ScienceQA involves multiple-choice questions, but recent studies suggest agents may exhibit option biases (e.g., favoring option C) rather than truly understanding the problem. Moreover, this Q&A format overlooks the reasoning process, where an incorrect process might still yield a correct result.

Virtual World: Virtual world tasks evaluate an agent's decision-making and execution abilities in dynamic, complex environments. Common benchmarks include ALFWorld [Shridhar et al., 2020], IGLU [Mehta et al., 2024], and Minecraft [Wang et al., 2023]. ALFWorld's grid-world simplicity may fail to capture real-world ambiguity. Minecraft, a widely adopted open-world simulation, offers diverse tasks and vast potential for evaluating simulated environments. However, this platform encounters difficulties concerning the transferability of skills from simulation to real-world application, i.e., the "sim-to-real" transfer problem.

Embodied Action: Embodied tasks test an agent's integrated abilities in perception, decision-making, and action execution, typically simulating real-world physical actions like moving a cup, pouring water, or cleaning an environment. Common benchmarks include Franka-Kitchen [Gupta et al., 2020], Meta-World [Yu et al., 2020], and RT-X [Padalkar et al., 2023], with task success rate as the evaluation metric. Franka-Kitchen focuses on kitchen environments, limiting generalization to other physical scenarios. RT-X evaluates diverse physical settings.

Web Navigation: Web navigation tasks simulate human processes of solving problems through querying and filtering information. Common benchmarks include WebShop [Yao et al., 2022], WebArena [Zhou et al., 2023], Mind2Web [Deng et al., 2024], and WebVoyager [He et al., 2024]. Mind2Web interacts only with static website states, while WebArena creates realistic, dynamic, and reproducible web environments through simulated websites. WebVoyager goes further, using Selenium to directly interact with real web pages.

Code Generation: Code generation tasks test an agent's logical reasoning, planning, and interaction with compilers in programming-related problems. Typical benchmarks include MBPP [Austin et al., 2021], HumanEval [Chen et al., 2021a], and SWE-Bench [Jimenez et al., 2024]. However, these rely heavily on success rates, which fail to provide specific feedback on each stage or capture the dynamic performance of agent systems. DevAI [Zhuge et al., 2024] not only focuses on final outcomes but also tracks and evaluates each stage of task execution for more comprehensive feedback.

Social Simulation: Social simulation tasks involve agents playing roles and interacting with other virtual characters in simulated social environments. Common benchmarks include SocialBench [Chen et al., 2024] and SocKET [Choi et al., 2023]. However, these benchmarks often feature overly idealized social scenarios, lacking the complexity and unpredictability of real-world social interactions. Park et al. [2023]'s AI Town is a notable attempt to address this.

Tool Usage: Tool use tasks test an agent's ability to leverage external tools to achieve complex task goals. Common benchmarks include ToolBench [Qin et al., 2024], TravelPlanner [Xie et al., 2024], and ToolEyes [Ye et al., 2025]. These tasks may overly focus on specific tools, limiting flexibility in tool selection and combination. HuggingGPT [Shen et al., 2023] uses a broader range of tools. Beyond evaluating tool use, future assessments might explore an agent's ability to create tools tailored to specific tasks.

Multi-agent Collaboration: Multi-agent collaboration tasks examine the ability of multiple agents to work together in shared environments. Common benchmarks include RocoBench [Mandi et al., 2024], PARTNR [Chang et al., 2024], and VillagerBench [Dong et al., 2024]. VillagerBench relies on Minecraft environments, while RocoBench assumes perfect perception (e.g., object detection, pose estimation, collision checking), which may fail in real-world scenarios like industrial production or medical collaboration.

Mathine Translation: Machine translation tasks assess an agent's ability to accurately and contextually translate text across languages, preserving meaning, tone, and cultural nuances. Common benchmarks include WMT [Specia et al., 2021] and FLORES-200 [Guzmán et al., 2019]. FLORES200 covers multiple languages but focuses on document translation, falling short in general translation. WMT is widely used for general translation evaluation.

Financial: Financial tasks typically require understanding numerical data, interpreting market signals, and generating actionable insights. Common benchmarks include FiQA_SA [Cheng et al., 2023] and FinQA [Chen et al., 2021b], with evaluation based on prediction accuracy or financial calculation correctness. However, these tasks limit the ability to test agents in dynamic market scenarios like stock trading. Increasingly, financial agents are tested in realworld stock trading scenarios.

Multidimensional evolution: Multidimensional evolution evaluates an agent's general capabilities across multiple domains, including reasoning, decision-making, and execution. For example, AgentBench [Liu et al., 2024] includes diverse scenarios covering operating systems, databases, knowledge graphs, card games, lateral thinking puzzles, household tasks, online shopping, and web browsing, assessing an agent's comprehensive abilities across eight distinct task scenarios.

5 Experimental Comparison

This section selects several representative feedback mechanisms for comparative experiments on HotpotQA [Yang et al., 2018], ALFWorld [Shridhar et al., 2020], and WebShop [Yao et al., 2022] (some experimental results are borrowed from ExpeL [Zhao et al., 2024] and AutoGuide [Fu et al., 2024]), as shown in Table 3.

In terms of reasoning capability, ReAct [Yao et al., 2023] demonstrates limited performance improvement on complex reasoning tasks such as HotpotQA and StrategyQA, whereas Reflexion [Shinn et al., 2024] exhibits more significant enhancements on HotPotQA. For sequential decision-making tasks (e.g., ALFWorld and WebShop), ReAct shows outstanding performance, while Reflexion further improves ALFWorld's performance through its multi-step environmental navigation capability, with the most notable improvements observed between the first and second trials. ExpeL [Zhao et al., 2024], AdaPlanner [Sun et al., 2023], and AutoGuide [Fu et al., 2024] also demonstrate applicability in such tasks.

From the stability perspective, non-learning-based methods like CoT, Act, and ReAct are suitable for immediate single-run tasks but face accuracy bottlenecks. In contrast, Reflexion, ExpeL, and AutoGuide exhibit greater stability in long-term dynamic tasks through iterative learning mechanisms. AdaPlanner mitigates hallucination issues by employing code-style prompts, demonstrating stability performance.

Compared to traditional reinforcement learning, feedback mechanisms like Reflexion offer significant advantages over traditional reinforcement learning (RL) across four dimensions: (1) Trial-and-Error learning; it improves performance by reflecting on errors and incorporating experience into subsequent decisions, suitable for tasks requiring trial-anderror learning, such as decision-making, reasoning, and programming; (2) Efficiency; it does not require fine-tuning language models, offering higher data and computational efficiency; (3) Detailed feedback; it uses linguistic feedback, which is more specific and detailed than the scalar rewards of traditional RL, helping agents understand errors and make targeted improvements; (4) Interpretability and memory; It provides explicit, interpretable staged memory, storing self-reflections to facilitate analysis of the learning process, outperforming traditional black-box RL.

6 Challenges and Future Directions

6.1 Real-Time Feedback andMulti-Agent System Implementation

Challenge: As the number of agents grows, computational demands rise sharply, increasing the need for efficient architecture and optimization. More complex communication networks slow information flow, make coordination harder, and lower collaboration efficiency, hindering the achievement of shared goals. Additionally, a lack of standardized protocols also complicates interactions, especially when agents come from different vendors and use different architectures.

Future Direction: Developing efficient and scalable frameworks, such as agent systems based on the MARL framework [Ma et al., 2024], to reduce communication costs and computational complexity while enhancing adaptability to heterogeneous agents and systems. Additionally, promoting protocol standardization, such as MCP, A2A, ANP, and Agora, for building large-scale agent systems.

6.2 Integration of Multi-modal Feedback

Challenge: The core challenge lies in the complexity of aligning different modalities. Effective multimodal selfcorrection requires agents to process and combine diverse data-such as text, images, audio, and sensor inputs-where unique representations for each modality make integration difficult. Current research faces difficulties in establishing unified fusion or representation learning strategies to convert these diverse signals into cohesive features.

Future Direction: Developing unified frameworks for multimodal feedback. For instance, Tactical Rewind [Ke et al., 2019] leverages self-correction strategies to optimize visual and language navigation. MM-React [Yang et al., 2023] achieves multimodal reasoning and action by integrating ChatGPT with a pool of visual experts. A promising research avenue involves cross-modal representation learning to find shared feature space across modalities.

6.3 Meta-Learning Adaptive Feedback

Challenge: The primary challenge is creating a system that effectively balances learning from multiple feedback sources while maintaining stability and consistency in agent behavior. For instance, AutoGen [Wu et al., 2023] highlights that agents must continuously adapt to evolving user needs and contexts, which can result in feedback conflicts or overfitting.

Future Direction: Future research should focus on developing more efficient and robust adaptive mechanisms to enhance agent performance in diverse applications. One potential direction is mutual learning, such as the two-stage mutual learning framework [Wang et al., 2024a]. Another direction involves learning unified latent representations to integrate multi-source feedback and minimize conflicts.

6.4 Explainability of FeedbackMechanisms

Challenge: Current feedback mechanisms often lack transparency in how they make and adjust decisions, which undermines user trust-especially in critical areas like healthcare and transportation.

Future Direction: Developing explainable feedback models with visualization tools and transparency metrics to clarify how feedback affects agent behavior. For example, natural language explanation tools can generate easy-to-understand feedback [Feng et al., 2024a], while AMOR [Guan et al., 2024] improves explainability and safety by providing stepby-step feedback similar to a chain of thought.

6.5 Feedback in Embedded AI Agent

Challenge: In embedded AI, deploying models trained in simulated environments to the real world faces significant "sim-to-real" issues. Embodied agents trained with reinforcement learning might struggle to fully replicate realworld disturbances, lighting, gravity, and other physical properties, leading to poor model performance in reality. Approaches like domain randomization [Saito et al., 2022], domain adaptation [Rao et al., 2020], and simulation improvement [Martinez-Gonzalez et al., 2020] are commonly solutions to address these gaps.

Future Direction: Future developments include designing adaptive learning algorithms that dynamically adjust in realworld settings, rapidly adapting to unseen physical properties through online learning or meta-learning; exploring hybrid training paradigms that combine simulated and real-world data to optimize generalization; developing high-fidelity simulators that integrate vision, touch, and mechanics to achieve more robust transfer to real environments.

6.6 Inherent Limitations of LLMFeedback

Challenge: LLMs and VLMs often produce hallucinations due to biases and spurious features in training data, such as incorrectly associating objects with visual cues or generating factually inaccurate text. Moreover, models may generate biased or incorrect outputs due to limitations in pre-trained knowledge or insufficient understanding of the dynamics of the deployment environment. Such issues are especially noticeable in AI agents with minimal fine-tuning, leading to unreliable feedback and reduced trust in the agent system.

Future Direction: Addressing hallucinations and biases might require the integration of retrieval-augmented generation (RAG) to cross-validate outputs with external sources, or integrating external databases to improve factual accuracy. In multi-agent systems, iterative interaction and debate among agents can help correct individual hallucinations and reasoning errors [Xi et al., 2025]. Additionally, incorporating world models (environmental models) can also support systematic fact-checking of generated content.

7 Conclusion

In this survey, we provide a systematic review of feedback mechanisms in LLM-based agents. First, we introduce the overall framework of LLM-based agents to help readers understand their fundamental components and operational models. Next, we summarize the current research status of feedback mechanisms, evaluation protocols, and benchmarks in LLM-based agents, offering clear classifications and intuitive insights. Finally, we conduct an in-depth analysis of the main limitations of existing feedback mechanisms and highlight several research directions worth further exploration. Through this comprehensive review, we aspire to provide a comprehensive reference for readers interested in this rapidly evolving field and to inspire future research.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (62406091, 62276077, 62306085, U23B2055, 62350710797), Guangdong Basic and Applied Basic Research Foundation (2024A1515011205), Shenzhen Science and Technology Program (KQTD2024072910215406, ZDSYS20230626091203008), and Shenzhen College Stability Support Plan (GXWD20231130104007001, GXWD20220817123150002, GXWD20231130151329002, GXWD20220811170358002).