ARMOR: Adaptive Curriculum Meta-Learning for Noise-Robust RAG Reasoning

1 Introduction

Retrieval-Augmented Generation (RAG) has emerged as a critical paradigm for mitigating hallucinations in Large Language Models (LLMs) by incorporating external knowledge [Lewis et al., 2020; Gao et al., 2023]. However, the retrieval process is inherently imperfect and inevitably introduces irrelevant or misleading Retrieval Noise, which poses a severe challenge to the robustness of RAG systems [Cuconasu et al., 2024; Wu et al., 2025]. Fundamentally, noise robustness is not a standalone trait but a composite capability. It requires generative systems to simultaneously perform across three distinct dimensions: identifying knowledge boundaries, filtering noise from retrieved context, and executing correct reasoning [Thakur et al., 2024].

Unfortunately, current methods often focus on specific facets of noise robustness [Chen et al., 2024]. Some studies employ supervised fine-tuning to train generators to detect noise and execute robust reasoning within noisy contexts [Zhou et al., 2025; Wei et al., 2025]. Conversely, others leverage generative systems' intrinsic knowledge boundary perception to reduce unnecessary retrieval [Su et al., 2024; Wang et al., 2023]. However, these methods generally restrict optimization to isolated capabilities, lacking the synergy required for comprehensive robustness. Reinforcement Learning (RL) offers a paradigm to address this by enabling the co-evolution of these capabilities via end-to-end policy optimization [Shao et al., 2024; Feng et al., 2025].

Nevertheless, applying standard RL to enhance retrieval noise robustness poses two critical challenges. First, the reward signals derived from the multi-dimensional optimization objectives of noise robustness may result in mutual interference or cancellation, thereby preventing the generative system from converging to an optimal policy for any specific capability [Li et al., 2025]. Second, these capabilities exhibit varying degrees of complexity and follow an underlying sequential logic. Just as a student must learn addition and subtraction before multiplication and division, the optimization of noise robustness necessitates a sequential learning trajectory. However, standard RL algorithms typically disregard this inherent progression, attempting to optimize all noise robustness capabilities simultaneously [Huang et al., 2025]. This simultaneous learning strategy often induces an overwhelming optimization burden, restricting the generative system from effectively learning these capabilities.

To address these challenges, we propose ARMOR, an Adaptive Curriculum Meta-Learning for Noise-Robust RAG Reasoning. ARMOR leverages curriculum-based Reinforcement Learning to enhance the comprehensive noise robustness of generative systems. Accordingly, we decouple the composite objective of noise robustness into three sub-tasks: (1) knowledge boundary perception, (2) retrieval verification, and (3) robust reasoning. Specifically, knowledge boundary perception determines whether the model's intrinsic knowledge suffices or external retrieval is required. Retrieval verification distinguishes relevant evidence from irrelevant noise within the retrieved context. Robust reasoning integrates valid information to generate correct answers even in noisy contexts. Systematically acquiring these capabilities is essential for establishing a resilient defense against retrieval noise.

We draw inspiration from the human teaching process, where teachers dynamically adjust the curriculum based on the student's learning feedback [Portelas et al., 2019]. ARMOR models the training process as an adaptive curriculum meta-learning framework, and introduces an adaptive task scheduler as a teacher. In each training step, the teacher dynamically selects a specific sub-task for the student (the RAG generator), leveraging Absolute Learning Progress (ALP) to quantify the student's real-time proficiency and prioritize the capability with the highest potential for improvement.

By prioritizing the most critical capability at each stage, ARMOR allows the optimizer to focus on a single dimension of robustness, effectively avoiding the mutual interference and cancellation of reward signals. To further stabilize this decoupled training, we introduce inter-group advantage estimation. Distinct from standard group relative policy optimization (GRPO) [Shao et al., 2024], inter-group advantage estimation extends advantage estimation to a batchlevel global scope. By combining inter-group advantage estimation with the adaptive curriculum meta-learning framework, ARMOR effectively eliminates task heterogeneity between different inputs and mitigates mode collapse stemming from limited output spaces. Experiments on three complex reasoning datasets (2WikiMultiHopQA [Ho et al., 2020], MusiQue [Trivedi et al., 2022], and HotPotQA [Yang et al., 2018]) indicate that ARMOR achieves synergistic improvements in retrieval noise robustness, significantly outperforming baselines in output quality.

We summarize our contributions as follows:

• We propose ARMOR, utilizing an ALP-based adaptive task scheduler to dynamically adjust the learning curriculum based on student's feedback.
• We introduce inter-group advantage estimation to stabilize training and mitigate mode collapse stemming from limited output spaces.
• Experiments on three noise-injected complex reasoning datasets demonstrate that ARMOR achieves synergistic improvements in retrieval noise robustness.

2 RelatedWork

Noise Robustness in RAG. Enhancing the robustness of RAG systems against retrieval noise has become a pivotal research direction. A primary defense against noise involves explicit mechanisms to evaluate and filter retrieved documents before or during generation. CRAG [Yan et al., 2024] employs a lightweight retrieval evaluator to assess document quality, triggering corrective actions when the confidence is low. Similarly, TRUSTRAG [Zhou et al., 2025] utilizes clustering and self-assessment to identify and filter out adversarial or misleading information. Rather than filtering inputs, CHAIN-OF-NOTE [Yu et al., 2024] trains models to generate sequential reading notes, explicitly assessing document pertinence before formulating an answer. INSTRUCTRAG [Wei et al., 2025] further advances this by instructing models to synthesize rationales that explain how the answer is derived from noisy contexts. Additionally, RAAT [Fang et al., 2024] employs adaptive adversarial training, exposing the model to various noise types to internalize robust reasoning capabilities. SKR [Wang et al., 2023] utilizes the model's selfknowledge to classify questions as "known" or "unknown" before invoking a retriever. Similarly, DRAGIN [Su et al., 2024] introduces Real-time Information Needs Detection to trigger retrieval only when the model exhibits uncertainty during generation. However, existing methods typically address noise robustness as an isolated capability by focusing solely on filtering, reasoning, or retrieval triggering.

RL for Robust RAG. Recent advancements have increasingly leveraged RL to enhance the robustness of RAG systems against retrieval noise. Some studies focus on optimizing the interaction policy between reasoning and retrieval. For instance, SEARCH-R1 [Jin et al., 2025] and R1SEARCHER++ [Song et al., 2025] utilize outcome-based RL to train models to autonomously search queries within reasoning steps, enabling dynamic knowledge acquisition. To enhance noise filtering, R3-RAG [Li et al., 2025] employs a relevance-based process reward to verify retrieved documents at each step. Similarly, LETS [Zhang et al., 2025a] integrates knowledge redundancy and matching rewards to penalize irrelevant retrieval and prevent error propagation in multi-hop reasoning. EVINOTE-RAG [Dai et al., 2025] restructures the workflow into a "retrieve-note-answer" paradigm, using an evidence quality reward to filter noise and distill key evidence before generation. Furthermore, RAG-RL [Huang et al., 2025] investigates static curriculum strategies to improve model robustness against distractors. In contrast, ARMOR decomposes robustness into three distinct sub-tasks, utilizing an ALP-based adaptive task scheduler to dynamically adjust the learning curriculum based on the generator's feedback.

3 Preliminaries

Task Definition: We focus on the task of RAG under noisy retrieval conditions. Given a QA pair (q, a) ∈ Ttrain, the RAG system utilizes a retriever to fetch a set of external documents D = {d1, d2, · · · , dk} based on the question q from a large corpus. The retrieved set D inevitably contains noise due to the inherent imperfections of the retriever. We denote the subset of golden documents as D+ ⊆ D and the noisy documents as D- = D \ D+. The robust RAG generator πθ aims to generate the correct answer a by effectively filtering D- and reasoning over D+ in conjunction with its intrinsic knowledge. We formulate this as maximizing the probability of the correct answer a given the query q and retrieved set D:

Warm-Up: To equip the RAG generator with the fundamental capabilities of problem decomposition and step-bystep reasoning, we employ SFT on reconstructed multi-hop RAG datasets as a warm-up stage. This phase lays the foundation for the subsequent RL phase. For each pair of <main-question> and <main-answer>, we construct a reasoning chain to serve as the training target. A single reasoning chain consists of structured nodes: <think>, <sub-question>, <search>, <doc>, <doc-type>, and <sub-answer>. Specifically, we train the generator to decompose the <main-question> into sub-questions, determine retrieval policies via <search>True<search> or <search>False<search>, verify retrieved documents using <doc-type> tags, and generate corresponding sub-answers based on the reasoning process. This structured supervision ensures that the generator explicitly internalizes the "retrieve-verify-reason" workflow before exploring complex noisy environments. Detailed dataset construction procedures are provided in Appendix A.

4 Methodology

To achieve comprehensive noise robustness, we propose ARMOR, an Adaptive Curriculum Meta-Learning framework. As illustrated in Figure 1, ARMOR consists of two core components: (1) Adaptive Task Scheduler that dynamically curates the learning curriculum based on the learning feedback, and (2) Inter-Group Advantage Estimation mechanism that stabilizes optimization across heterogeneous sub-tasks.

4.1 Adaptive CurriculumMeta-Learning

We decouple the composite objective of noise robustness into three sub-tasks: Knowledge Boundary Perception (τret), Retrieval Verification (τfil), and Robust Reasoning (τans). To avoid the mutual interference of reward signals from distinct tasks, we employ an adaptive task scheduler. In each training round n, the scheduler selects a task τs from the task set T = {τret, τfil, τans} for optimization, prioritizing tasks with high potential for improvement based on learning feedback. We initialize the task weights w1(τ) = 1/|T | for all τ ∈ T . In the n-th round, the selection probability pn(τ ) is computed to balance exploration and exploitation:

where T is the predefined training rounds, and γ serves as an exploration coefficient to prevent premature convergence to local optima. The scheduler samples a task τs ∼ pn(·) at the round n. Upon completing the training for the selected task τs, we quantify the model's increase in competence using Absolute Learning Progress (ALP) [Portelas et al., 2019]. A higher ALP indicates that the task offers significant potential for performance improvement. The raw ALP is defined as:

where rn(τs) denotes the task reward received in the current training round n (detailed in section 4.3), rold(τs) represents the maximum historical reward achieved for task τs. To correct for the bias introduced by probabilistic sampling, we compute the unbiased reward estimation ˆ rn(τ ):

detailed proofs are provided in Appendix B.2. Finally, the scheduler updates the task weights for the next round, incorporating a forgetting factor α = 1/T to smooth updates and retain historical experience:

4.2 Inter-Group Advantage Estimation

Standard Group Relative Policy Optimization (GRPO) normalizes advantages within a group of rollouts corresponding to a single input. However, intra-group normalization results in unstable learning signals for individual tasks during rollouts. In our decoupled training setting, the constrained output space induces high similarity among rollouts generated from the same input, which reduces intra-group diversity and exacerbates the risk of mode collapse. To overcome this limitation, ARMOR introduces Inter-Group Advantage Estimation, which performs advantage normalization across the entire training batch rather than within isolated inputs.

Given a batch of questions Q = {q1, . . . , qm}, we generate l rollouts for each question, denoted as:

where oi,j denotes the j-th rollout of question qi. For the selected task τs, each rollout oi,j receives a sequence of tasklevel rewards, denoted as:

where r(k) i,j represents the reward of k-th task span in the rollout oi,j. In addition, each rollout sequence oi,j is associated with a format reward rfmt i,j , which evaluates whether the rollout sequence conforms to the required format. We define the global sets of task and format rewards in current batch as:

As shown in Figure 1, we normalize rewards using the global statistics of the entire batch to compute the task-level advantage Atask(k) i,j and format-level advantage Afmt i,j :

where mean(·) and std(·) compute the mean and standard deviation over the entire batch. By combining inter-group advantage estimation with the adaptive curriculum metalearning framework, ARMOR effectively eliminates task heterogeneity between different inputs and provides a more stable training signal.

The final token-level inter-group advantage A(t) i,j combines dual-level signals:

where t is the token index in rollout oi,j, tokens(i, j, k) denotes the token index of the k-th task span in rollout oi,j . A(t) i,j represents the token advantage assigned to token t in rollout oi,j . The weighting coefficient ω controls the relative strength of the format signal. Then the clipped policy optimization objective of ARMOR is as follows:

After the training for the task τs in the n-th batch, the task reward rn(τs) = mean(Rtask) is calculated on the validation dataset, and returned to the adaptive task scheduler to update the task weights.

4.3 RewardModeling

To precisely guide the generator toward robust behaviors, we design tailored reward functions for each sub-task, addressing their specific optimization challenges.

Knowledge Boundary Perception. The primary objective of this task is to enable the generator to accurately perceive the boundaries of its intrinsic knowledge. For each generated sub-question <sub-question>, the RAG generator predicts a <search> tag to determine whether to invoke the retriever. The action space is defined as:

To align the model's retrieval behavior with the actual necessity of external information, we design a reward function that evaluates whether the model should retrieve from external knowledge based on the correctness of its direct answer. We denote the correctness of the answer generated without retrieval as vans ∈ {correct, error}, and the model's chosen retrieve action as aret. The reward rret is defined as follows:

Retrieval Verification. This task requires the generator to classify each retrieved document into three categories: Cdoc = {golden, inconsequential, irrelevant} [Zhang et al., 2025b]. Standard coarse-grained binary rewards (correct/incorrect classification) often lead to training instability. To mitigate this, we propose a fine-grained Retrieval Verification Reward based on the probability margin. For a document d, let c∗ be the ground-truth label and p(c) be the predicted probability of class c. We first calculate the confidence margin m(d) between the target class and the most competitive incorrect class:m(d) = p(c∗ ) - max

The final retrieval verification task reward is:

where scaling factor β is used to control the sensitivity of the reward scaling.

Robust Reasoning. The goal of robust reasoning is to generate correct sub-answers <sub-answer> and a final answer <final-answer> given noisy context. We compute the answer reward rans by first obtaining a raw correctness score sans from an LLM evaluator, then linearly mapping it to the range [-1, 1]:

where [smin, smax] is the range of correctness score. Details of the LLM-based evaluation are provided in Appendix E.

5 Experiment

In this section, we present empirical evaluations of the effectiveness of ARMOR in defending against retrieval noise across three complex reasoning datasets. Specifically, we aim to validate the following: (1) ARMOR's capability in enhancing the noise robustness of RAG generators; (2) the contribution of core components via ablation studies; (3) the dynamic trends of learning feedback pn(τ) during training.

5.1 Experiment Setup

Datasets. As detailed inAppendixA, we reconstruct 2WikiMultiHopQA and MuSiQue datasets. For each dataset, training set consists of two stages: a warm-up stage and a RL stage. The validation set derives from the reconstructed datasets, while the test set consists of two reconstructed subsets and HotpotQA, which serves as an out-of-distribution dataset for evaluating model generalization. The detailed dataset statistics are presented in Table 1.

EvaluationMetrics. We evaluated the model using the following metrics: (1) Exact Match (EM) measures the strict accuracy by determining if the predicted answer matches the ground truth verbatim. (2) F1-Score (F1) captures the tokenlevel overlap between prediction and reference, offering flexibility for answers with minor phrasing variations. However, these traditional metrics rely on surface-level matching and often fail to capture semantic validity in generative tasks. To address this, we introduce (3) Correctness (Cor.), an LLMbased metric utilizing GPT-5 to assess the factual accuracy and completeness of answers against references. For clarity, correctness scores are normalized to the range [0, 1]. Detailed evaluation prompts and results aligned with manual annotations are presented in the Appendix E.

Baselines. We compare ARMOR against several established baselines: (1) Adaptive Retrieval: We compare against SKR [Wang et al., 2023] and DRAGIN [Su et al., 2024], which selectively trigger retrieval based on model's intrinsic knowledge boundary to mitigate noise introduction. (2) Robust Reasoning: This category includes CHAIN-OFNOTE [Yu et al., 2024] and IRCOT [Trivedi et al., 2023]. We employ standard COT without retrieval as a lower bound to benchmark pure reasoning robustness. (3) RL-Enhanced RAG: This includes methods like RAG-RL [Huang et al., 2025] and SEARCH-R1 [Jin et al., 2025] that optimize retrieval and reasoning policies via reinforcement learning. In addition, we include NO-RAG (direct inference) and VANILLA-RAG (standard RAG workflow), which serve as performance lower bounds, as they lack specific mechanisms to defend against retrieval noise. We employ two mainstream LLM series with varying parameter scales as the RAG generators: Llama-3.2-1B, Llama-3.2-3B [Team, 2024], Qwen-30.6B, Qwen-3-1.7B and Qwen-3-4B [Yang et al., 2025]. For all baselines, we strictly follow the hyperparameter settings and configurations reported in their original studies. Detailed baseline configurations are provided in Appendix C.

Training Details. We train the RAG generator for one epoch using Adam optimizer [Kingma and Ba, 2015] with a learning rate of 5e-5. One epoch includes 857 training rounds, and each round consists of 10 questions. The clipping coefficient εlow and εhigh are set to 0.2 and 0.28, separately. The default rollout number l is set to 2. The weighting coefficient ω is set to 1, and the scaling factor β in the retrieval verification task is set to 6. During the rollout process, we employ e5-base-v2 [Wang et al., 2022] as the retriever, retrieving a maximum of three documents per retrieval triggering. For generation, we adopt a sampling temperature of 0.85, along with top-p = 0.95 and top-k = 50. All reported results are averaged over three independent runs to ensure statistical reliability.

5.2 Main Results

As demonstrated in our preliminary study (Appendix C.2), standard retrieval systems inevitably retrieve irrelevant or inconsequential documents, introducing natural noise into the generation process. Table 2 presents the comparative performance of ARMOR and baselines under this realistic setting. ARMOR achieves SOTA performance across all datasets and metrics, demonstrating remarkable consistency on both Qwen and Llama backbones. Specifically, ARMOR surpasses the strongest baseline (IRCOT) by a substantial margin, achieving an average Correctness improvement of 38.8% on Qwen3-1.7B and 19.9% on Llama-3.2-3B. Standard VANILLARAG shows limited improvement over direct inference (NORAG) and performs poorly on complex datasets. This confirms that without explicit robustness mechanisms, generators are easily misled by the inherent noise in natural retrieval. IRCOT remains competitive by interleaving reasoning and retrieval steps. CON struggles to maintain stability, suggesting that their sequential noise-filtering logic is prone to error propagation in open-domain settings. ARMOR's decoupled training objective effectively mitigates this issue. Existing RL-based approaches (RAG-RL, SEARCH-R1) exhibit suboptimal performance, highlighting the difficulty of optimizing RAG generators with sparse rewards under natural noise.

We conducted a controlled experiment by manually injecting noise into the retrieval. We fixed the number of retrieved documents to 5 and varied the noise ratio from 0% to 100%, as shown in Figure 2. Baseline methods exhibit a precipitous performance decline as the noise ratio increases. IRCOT and SEARCH-R1 suffer significant losses, indicating they struggle to filter out distracting information effectively when valid evidence is scarce. In contrast, ARMOR demonstrates exceptional stability, maintaining a high correctness score of 69% even at an 80% noise ratio. ARMOR often outperforms the DIRECT ANSWER baseline or remains comparable at 100% noise. This indicates that ARMOR has effectively learned a rejection mechanism via Inter-Group Advantage Estimation, allowing it to suppress the influence of irrelevant contexts rather than being hallucinated.

5.3 Ablation Study

We compare ARMOR with the following variants: (1) w/o CRL: replaces the ALP-based scheduler with random sampling (-random), rigid sequential ordering (-seq), or outcomesupervised training (-outcome sup: removing process rewards); (2) w/o Inter-Group Adv: reverts global advantage normalization to standard intra-group normalization; (3) Base Setting: removes both adaptive curriculum and inter-group estimation, reverting to standard outcome-supervised RL. As shown in Table 3, ARMOR consistently outperforms all variant settings across all metrics. Specifically, w/o CRL -seq suffers a 47.51% drop in average correctness, indicating that rigid task ordering leads to catastrophic forgetting among decoupled sub-tasks. Similarly, w/o CRL -random underperforms relative to ARMOR, confirming that the ALP-based scheduler is essential for dynamically balancing exploration and exploitation based on learning feedback. Reverting to standard intra-group normalization (w/o inter-group adv.) degrades performance by 14.58%. This decline highlights that standard intra-group normalization yields unstable signals in our decoupled setting. In contrast, ARMOR's batch-level advantage estimation provides robust supervision, effectively preventing mode collapse and stabilizing optimization.

5.4 Analysis of Dynamic Learning Feedback

We visualize the dynamic trends of the task selection probability pn(τ) throughout the training process. Figure 3 illustrates these trends for both Llama-3.2-3B and Qwen-31.7B backbones. Observing the training dynamics of Qwen3-1.7B, the scheduler assigns high probability to the Knowledge Boundary Perception task in the early stages. This indicates that the generator primarily focuses on distinguishing "what it knows" vs. "what it needs to retrieve." The scheduler shifts its focus toward Retrieval Verification around step 300. This transition suggests that once the generator learns "when to retrieve," it begins to focus on "how to filter noise" from the retrieval. Meanwhile, the attention on Robust Reasoning remains relatively stable, serving as a consistent regularization throughout the process. The trends for Llama-3.2-3B exhibit a different pattern, highlighting the scheduler's adaptability to different generator architectures. Unlike Qwen, Llama shows a continuous and increasing demand for Knowledge Boundary Perception. This suggests that for the Llama backbone, accurately determining the necessity of retrieval is a more challenging task. These findings confirm that ARMOR does not impose a rigid "one-size-fits-all" curriculum. Instead, it acts as a dynamic meta-learner, tailoring the training strategy to the learning trajectory of the underlying generator.

5.5 Case Study

We analyze a multi-hop query: "When was Ole Rynning's father born?" (answer: "May 30, 1778"), detailed case is provided in Appendix D.2. ARMOR successfully decomposes the logic: identifying the father ("Jens Rynning") first, then retrieving his birth year. Crucially, ARMOR effectively filters out distractor documents describing the son's birth "1809", leading to the correct answer. In contrast, SEARCH-R1 exhibits chaotic reasoning, generating hallucinated contexts and failing to align the answer type. IRCOT suffers from distraction failure, although it retrieves the correct evidence, it lacks explicit noise rejection and is misled by the high semantic overlap of the son's birth date. This case validates that ARMOR's adaptive curriculum meta-learning effectively equips the generator with the ability to distinguish golden evidence from noise in complex reasoning scenarios.

6 Conclusion

In this work, we presented ARMOR, a novel framework designed to fortify RAG systems against the pervasive challenge of retrieval noise. ARMOR decouples the training objective into three distinct sub-tasks: Knowledge Boundary Perception, Retrieval Verification, and Robust Reasoning. To coordinate these tasks, ARMOR introduces an Adaptive Curriculum Meta-Learning strategy, where a teacher scheduler dynamically prioritizes learning objectives based on the generator's Absolute Learning Progress. Moreover, ARMOR addresses the optimization instability inherent in decoupled reinforcement learning by introducing Inter-Group Advantage Estimation, which mitigates mode collapse and ensures stable gradient optimization via global batch normalization. Extensive experiments across three complex multi-hop reasoning datasets demonstrate that ARMOR exhibits exceptional resilience, maintaining high correctness even under extreme noise ratios. This confirms that Adaptive Curriculum Scheduling and Inter-Group Advantage Estimation are essential for comprehensive RAG noise-robust reasoning.

Acknowledgments

This work is partially supported by National Nature Science Foundation of China under No. 62476058. We thank the Big Data Computing Center of Southeast University for providing the facility support on the numerical calculations in this paper.

Contribution Statement

Yan Wang and Yuxin Zhang conceived the study, designed the experiments, and wrote the manuscript. Yan Wang collected the data and performed the experiments. Shenyu Zhang, Yongrui Chen, and Sheng Bi reviewed and edited the final manuscript. Yan Wang and Yuxin Zhang contributed equally to this work. Guilin Qi is the corresponding author.