Bridging Generative and Discriminative Learning: Few-Shot Relation Extraction via Two-Stage Knowledge-Guided Pre-training

1 Introduction

Relation Extraction (RE) focuses on identifying semantic relationships between entities in unstructured text, serving as a cornerstone for knowledge graph construction and event analysis. Although pre-trained language models (PLMs) [Wei et al., 2020; Li et al., 2022; Wang et al., 2023a; Zhu et al., 2024] have significantly advanced RE performance, their success depends on the availability of large-scale annotated datasets, a requirement that is often unattainable in real-world applications such as rare disease relationship mining and financial event detection. These scenarios are particularly challenging due to the severe scarcity of annotated examples. As a result, Few-Shot Relation Extraction (FSRE) has emerged as a critical research area, necessitating the development of models capable of generalizing from only a handful of labeled examples per relation type.

∗Corresponding Author

Existing FSRE methodologies face three critical challenges. Data Scarcity: Traditional approaches rely on extensive annotations to distinguish between relations. While distant supervision [Vrandeˇci´c and Kr¨otzsch, 2014] generates labeled data aligned with knowledge bases, it introduces noisy labels and domain mismatches [Soares et al., 2019;

Peng et al., 2020; Qin et al., 2021], as shown in Figure 1(a). Limited Generalization: Models trained on indistribution data struggle to adapt to cross-domain or finegrained relations. For example, discriminative models frequently misclassify semantically similar relations, such as confusing "Place of birth" with "Place of death", with error rates exceeding 30% [Alt et al., 2020]. GenerativeDiscriminative Disconnect: Large language models (LLMs) like GPT-4 [Achiam et al., 2023] show potential to generate synthetic data, but treat RE as a black-box text generation task [Li et al., 2023; Xu et al., 2023b]. This results in a misalignment between the outputs of LLMs and the structured learning objectives required by FSRE models.

To address these challenges, we propose Two-stage Knowledge-guided Relational Extraction (TKRE), a novel framework that integrates generative and discriminative paradigms through structured knowledge infusion. We first leverage large language models (LLMs) to generate relational logic explanations (e.g., "Subject X exhibits relation R with Object Y because ..."), explicitly capturing fine-grained semantic dependencies between entities. These explanations create interpretable reasoning pathways that guide downstream model training. Then, we employ constrained decoding strategies to generate schema-compliant training instances by enforcing entity type compatibility and relationspecific lexico-syntactic patterns. This dual approach ensures that the synthetic data preserve domain-specific relational semantics while mitigating distributional changes, as illustrated in Figure 1(b).

Finally, we introduce a two-stage knowledge-guided pretraining strategy that combines Masked Span Language Modeling (MSLM) and Span-level Contrastive Learning (SCL). MSLM focuses on selectively masking relation-indicative spans and optimizing their reconstruction, enhancing the model's ability to capture contextual dependencies and infer relational semantics. Complementing this, SCL operates within a contrastive learning framework, where positive spans (semantically aligned with target relations) are contrasted with negative spans (relationally discordant but contextually plausible). This approach sharpens the model's discriminative boundaries for entity interaction modeling, fostering a more precise understanding of relational structures.

As illustrated in Figure 2, TKRE establishes a selfreinforcing knowledge integration cycle: (a) relational explanations guide the generation of synthetic data, ensuring semantic consistency; (b) the synthetic data is leveraged during pre-training to incorporate generative knowledge; and (c) pre-training objectives explicitly align the representations of synthetic and golden data. This closed-loop architecture facilitates the iterative refinement of relational understanding through knowledge distillation and representation alignment, effectively bridging the gap between generative and discriminative paradigms.

In summary, our contributions are as follows:

• We leverage LLMs to automatically generate explanation-driven knowledge and structured synthetic data, addressing data scarcity while enhancing the traditional RE model's comprehension of underlying relational logic.
• We introduce a novel two-stage knowledge-guided pretraining framework that seamlessly integrates generative knowledge with discriminative objectives, enabling models to effectively learn from sparse annotations and noisy synthetic data simultaneously.
• Extensive experiments on four benchmark datasets demonstrate that TKRE significantly improves the performance of traditional FSRE models, achieving F1 gains of 7.8% and 5.0% over strong baselines (TYP Marker, GenPT) and outperforming both pure LLMbased and hybrid methods in few-shot settings.

2 RelatedWork

Few-Shot Relation Extraction with Traditional Models. Traditional FSRE methods predominantly rely on discriminative learning to optimize P(y|x), but they face significant challenges under low-resource conditions. Approaches such as [Peng et al., 2020], [Zhou and Chen, 2022] and [Zhang and Kang, 2024] enhance intra-class representation consistency but struggle to generalize to fine-grained relations, due to their reliance on rigid in-distribution assumptions. Promptbased methods such as [Han et al., 2022] and PTRE [Zhang et al., 2024b]) reformulate classification as an infilling task, yet require precise semantic alignment and fail to scale with extreme data scarcity. Distant supervision [Mintz et al., 2009] alleviates annotation costs via knowledge base alignment, and many recent works investigate how to improve learning with such distant data [Soares et al., 2019; Peng et al., 2020; Qin et al., 2021]. Despite its benefits, distant supervision introduces noisy labels and domain mismatches, which significantly undermine model robustness.

These limitations arise from a common bottleneck: discriminative models excel at distinguishing predefined classes but lack the ability to reason over unseen relational logic or contextual dependencies. This deficiency is particularly critical in FSRE, where success depends on capturing subtle semantic distinctions and adapting to cross-domain variations. To address these challenges, our proposed TKRE framework integrates generative knowledge to bridge this gap, enabling more robust relational reasoning and improved adaptability.

Large Language Models for FSRE. The emergence of LLMs like GPT-4 [Achiam et al., 2023] and LLaMA-2 [Touvron et al., 2023] has generated significant interest in leveraging their in-context learning capabilities for FSRE. Early efforts [Xu et al., 2023a; Li et al., 2023; Xu et al., 2023b] query LLMs using task descriptions and demonstrations, achieving only marginal improvements over traditional baselines. These approaches, however, treat LLMs as black-box generators and fail to align their generative knowledge (e.g., contextual reasoning and dependency modeling) with the structured discriminative objectives of RE models. As a result, the generated outputs often lack task-specific structural awareness, leading to suboptimal performance on finegrained relations [Li et al., 2023]. Furthermore, directly fine-tuning LLMs is computationally prohibitive [Yang et al., 2024], and their zero-shot performance is constrained by the complexity of prompt design [Whitehouse et al., 2023; Guo et al., 2025]. These limitations underscore the urgent need for frameworks that effectively bridge generative and discriminative learning paradigms to unlock the full potential of LLMs for FSRE.

Hybrid Methods: Combining LLMs with Traditional Models. Recent hybrid approaches attempt to address data scarcity by leveraging LLMs to augment training data for smaller RE models. Editing-based methods [Wei and Zou, 2019; Jiao et al., 2020] replace tokens with synonyms or back-translated sentences but often fail to maintain relational consistency. Generative methods [Anaby-Tavor et al., 2020; Schick and Sch¨utze, 2021] use LLM prompting to synthesize labeled instances, yet their outputs frequently suffer from noise and lack diversity in structured RE tasks. Semisupervised techniques [Vu et al., 2021] combine synthetic and golden data using confidence filtering or auxiliary tasks but treat synthetic data as static inputs rather than dynamic knowledge carriers. For instance, [Xu et al., 2023a] introduces a two-stage self-training framework but does not explicitly model entity-span interactions. Similarly, [Wang et al., 2023b] generates label-conditioned data but neglects alignment between synthetic examples and relational reasoning. These methods fail to fully exploit the potential synergy between generative and discriminative paradigms, limiting their ability to transfer LLM-derived knowledge into taskspecific representations effectively.

3 Methodology

In this section, we present the detailed modules of our TKRE: explanation-driven knowledge generation, twostage knowledge-guided pre-training, and task-oriented finetuning. Figure 2 depicts the overall framework.

3.1 Task Formulation

Let R = {r1, r2, . . . , r|R|} represent the set of |R| relation types in the dataset (e.g., TACRED contains |R| = 42). Given a sentence x with two marked entities esub and eobj, the goal is to predict the relation r ∈ R between these entities. In the few-shot setting, only K labeled examples Dtrain = {(x, esub, eobj, r)i}K×|R| i=1 per relation type are provided for training, where K is a small constant (e.g., K = 8).

3.2 Data Preparation

To address the challenges of data scarcity and limited generalization, we employ LLMs to generate knowledge explanations and synthetic examples for each relation type, followed by multi-task pre-training and fine-tuning.

Explanation-Driven Knowledge Generation

To further improve domain adaptation and enhance the task relevance of the pre-training strategy, we utilize LLMs to construct task-oriented, generated knowledge. LLMs, trained on diverse corpora that likely encompass domains relevant to NER tasks, provide a rich source of contextual and domainspecific information. However, directly fine-tuning LLMs is computationally expensive and resource-intensive, making it impractical for many downstream tasks.

To address this, we design an intuitive instruction to guide the LLM in explaining why a given text span should be recognized as an entity, thereby generating a task-oriented corpus. For a sentence x ∈ Dtrain, the golden relation type r, and an entity pair (esub, eobj) ∈ x, the instruction X is constructed as described in Figure 3.

Figure 3: Instruction of explanation-driven knowledge generation.

Given this instruction X, we denote LLM as fLLM. The generated sequence regarding the explanation of relation < y > is denoted as Y = fLLM(X) = {yi}Li=1 , where L is defined as the length of the generated sequence, i.e., the number of tokens in Y . For the classic auto-regressive generative model, the generated sequence Y is predicted by the following conditional probability:

where y<t represents the tokens before yt.

Consequently, we can obtain several sentences of a relation extraction flow by reasoning in the raw textual context < x >, such as the grey part in Figure 1. Then, with respect to all relation types in Dtrain, we employ the frozen LLMM to obtain K explanations cluster of each relation type. Formally,

where Xri denotes the instruction X with the corresponding slots of relation ri. Following [Qin et al., 2021], we build the explanation-driven knowledge corpus C from the labeled raw texts in Dtrain.

Schema-Constrained Data Generation

In this section, we aim to perform data augmentation using LLMs to enrich the training data for RE. Inspired by [Xu et al., 2023b], we design prompts that inform the LLM about the essential components of an RE training sample: context text, subject entity, object entity, subject entity type, object entity type, and the relation. The LLM is then guided to generate additional pseudo RE samples based on this structure. To ensure the augmented data aligns with the desired format, we apply predefined rules, such as regular expressions, to process and transform the LLM-generated outputs. The specific instruction provided to the LLM is illustrated in Figure 4.

Figure 4: Instruction of schema-constrained synthetic data generation.

3.3 Two-Stage Knowledge-guided Pre-training

In this section, we present a two-stage knowledge-guided pretraining framework that integrates Masked Span Language Modeling (MSLM) and Span-level Contrastive Learning tasks. This framework is designed to equip traditional FSRE

models with the ability to effectively capture contextual relationships between entities while simultaneously learning robust span-level representations. By combining these complementary tasks, the pre-training strategy enhances the model's capability to reason over entity relationships and adapt to fine-grained relational distinctions.

Masked Span Language Modeling

MSLM is an extension of traditional Masked Language Modeling (MLM) [Devlin et al., 2019], where instead of masking individual tokens, we mask entire spans (sequences of tokens). Inspired by the better learning ability of span masking [Liu et al., 2021] and recent advances in span-based masking methods [Zhang et al., 2024a], we adopt this strategy to improve the model's ability to capture long-range dependencies and complex relations between entities.

Let x = ⟨x1, x2, . . . , xn⟩ represent a given sentence, where xi ∈ C is a token from the explanation-driven knowledge corpus C, and n is the sentence length. In MSLM, we mask a continuous span S = ⟨xi, xi+1, . . . , xj⟩ by replacing it with a special sentinel token, such as [MASK]. The task is to predict the masked span S based on the surrounding context.

Given a masking strategy with different probabilities for different types of spans:

• 80% probability to mask relation spans,
• 50% probability to mask subject/object entity spans,
• 20% probability to mask other words.

We define a custom probability distribution for masking tokens based on their linguistic role in the sentence. The mask distribution M = ⟨m1,m2, . . . ,mn⟩ is determined by the specific type of each token xi. For each token xi, the masking probability pi is defined according to the following rules:

(3)

This strategy allows for a more targeted masking process, where different parts of the sentence are masked with different probabilities based on their syntactic roles. We model this process using a Bernoulli distribution for each token, where the probability of masking a token xi is determined by its type:

where mi ∈ {0, 1} indicates whether the token xi is masked (mi = 1) or not (mi = 0).

The model is trained to predict the masked span by minimizing the cross-entropy loss:

where γ represents the maximum length of input sequence, and yi denotes the predicted output for each masked token xi. The loss function trains the model to recover the masked spans, encouraging it to learn rich contextual information for span prediction.

Span-level Contrastive Learning

In addition to MSLM, we apply Span-level Contrastive Learning to further refine the traditional model's ability to differentiate between different spans in the context of specific relations. Contrastive learning encourages the model to learn useful embeddings that can distinguish positive spans (relevant to the relation) from negative spans (irrelevant to the relation).

For each sentence x, the model learns to predict both the positive span sp (e.g., "was born in") and the negative span sn (e.g., "New York"). Let ha, hp, hn represent the embeddings of the anchor (relation type), positive, and negative spans, respectively. We optimize the following contrastive loss:

where sim(ha, hx) is the cosine similarity between the anchor ha and the span hx, and τ is the temperature parameter controlling the sharpness of the similarity function.

This loss function encourages the model to bring positive spans closer to their respective anchor contexts and push negative spans further away, improving the model's span-level understanding.

Multi-task Training Objective

The total loss for pre-training is a weighted combination of MSLM and SCL losses:

where λ1 and λ2 control the relative importance of the two tasks.

3.4 Task-Oriented Fine-tuning

After pre-training, the model is fine-tuned on the specific fewshot relation extraction task. Fine-tuning is performed using both the original small-shot examples and the synthetic samples generated in the previous stages. The goal of fine-tuning is to adapt the pre-trained model to the downstream task of relation classification, which involves predicting the correct relation between two entities in a given sentence.

For each input sentence x with entities esub and eobj , the model is fine-tuned to predict the correct relation r ∈ R from the set of relations. The input to the model consists of the sentence xi, along with the two entities esubi and eobji . The output is the predicted relation label ri. The final relation prediction is made by passing the output of the encoder through a fully connected classification layer:

where h(xi, esubi , eobji ) is the representation of the sentence and entities from the pre-trained model, and W and b are the learned weights and bias of the classifier. The classification loss is computed as:

where N ≈ 2 ×K × |R| is the total number of samples in the training set, because the number of samples for some relation types may be less than K. And P (ri|xi, esubi , eobji ; θ) is the predicted probability for the correct relation ri.

4 Experiments

4.1 Experimental Design

Datasets. SemEval 2010 Task 8 [Hendrickx et al., 2010] is a widely used testbed for relation extraction. The schema is targeted at semantic relations between pairs of nominals, which requires a certain level of abstractive capabilities. TACRED [Zhang et al., 2017] is a large-scale dataset annotated using Amazon Mechanical Turk crowdsourcing. It was initially created for the TAC knowledge base population and mainly covers common relations between people, organizations, and locations based on the TAC KBP scheme. TACREV [Alt et al., 2020] is a label-corrected version of the TACRED dataset, which motivates from the unresolved challenging cases in the original TACRED dataset. ReTACRED [Stoica et al., 2021] further conducted a more comprehensive analysis and re-annotated the entire dataset. Besides, it made alternations to the schema to make it more clear and intuitive, which greatly improved the dataset quality. Following the strategy adopted by [Gao et al., 2021; Xu et al., 2023b], for each relation type, we randomly sample K instances per relation (K-shot) for the training phase. The whole validation set and test set are preserved to ensure the effectiveness of the evaluation. The statistics of datasets are presented in Table 1.

Baselines. We compare our proposed TKRE with Traditional RE, LLM-based and Hybrid Methods. The Traditional RE methods include CP [Peng et al., 2020], TYP Marker [Zhou and Chen, 2022], Knowprompt [Chen et al., 2022], GenPT [Han et al., 2022], and PTRE [Zhang et al., 2024b]. The LLM-based methods include LLaMA-2 [Touvron et al., 2023], GPT-3.5 [Ouyang et al., 2022], and GPT-4 [Achiam et al., 2023]. The Hybrid methods include Unleash [Xu et al., 2023b], and S2 ynRE [Xu et al., 2023a].

Implementation Details. In the LLM data generation part, we leverage the 13B version for LLama-2 ( llama-2-13b-chathf ) to conduct experiments, and we double the K-shot training set through LLMs. For example, for an 8-shot training set, we construct 8 pieces of pseudo data per relation, thereby creating the final augmented training set.

In the Traditional RE Model Pre-Training part, we adopt TYP Marker and GenPT as the base architecture. Regarding the evaluation metrics, we adopt the micro F1 scores of RE as the primary metric to evaluate models, considering that F1 scores can assess the overall performance of precision and recall. More details are shown in Appendix.

4.2 Main Results

The main results are illustrated in Table 2. Our proposed TKRE model outperforms all baselines across all metrics. Particularly compared to the base architecture TYP Marker and GenPT, our method manifests a significant advantage. This demonstrates the effectiveness of our designs and the benefits of integrating traditional RE models and LLMs. Furthermore, there are also some interesting phenomena:

1) The vast majority of methods exhibit superior performance on the ReTACRED dataset compared to the TACRED and TACREV datasets. This is reasonable as Re-TACRED is an improved version among these three datasets, which addresses some shortcomings of the original TACRED dataset and refactors its training/development/test sets. The more precise labels contribute to the learning process of these models, thereby yielding superior performance.

1. Among these LLM-based methods, GPT-4 demonstrates competitive performance and significantly outperforms GPT-3.5 and LLama-2 on all datasets. This proves its strong ability, as claimed in [Achiam et al., 2023]. However, it exhibits the generation performance in FSRE. This may be because the training objective of GPT-4 is on generative tasks, which predict the next word based on context, rather than optimizing specifically for RE tasks even though it utilized various very largescale corpora for training.

3) Unleash introduces a schema-constrained data augmentation method using LLMs to enhance the TYP Marker baseline by 6.4%. Similarly, S2 ynRE presents a two-stage self-training framework leveraging LLM-generated synthetic data, improving the CP baseline by 2.6%. Both approaches demonstrate a measurable improvement over their respective baselines, validating the feasibility of this line of research.

In comparison, our TKRE model significantly outperforms TYP Marker by 7.8% and GenPT by 5.0%, further confirming the effectiveness of our approach from a different perspective.

4.3 Ablation Study

To validate the effectiveness of main components in TKRE, we introduce the following variants for the ablation study:

TKRE w/o EDKG: This variant removes explanationdriven knowledge generation at the data preparation stage.

TKRE w/o SCDG: This variant removes schemaconstrained data generation at the data preparation stage.

TKRE w/o MSLM: This variant abides by the mask setting of BERT[Devlin et al., 2019] instead of masked span language modeling at the two-stage knowledge guidance preTraining stage.

TKRE w/o SCL: This variant skips the span-level contrastive learning at the two-stage knowledge guidance preTraining stage.

TKRE w/o ALL: This variant addresses the FSRE task by fine-tuning the baseline model (e.g., TYP Marker) solely on the golden training dataset, without incorporating any of the enhancements introduced by our method.

From Table 3, we observe the following:

1. Removing EDKG significantly reduces performance, especially in smaller settings, highlighting its importance in generating relation-relevant explanations for FSRE.
2. The absence of SCDG leads to a performance decline, especially on TACRED with fewer training examples. This demonstrates the importance of schema constraints help LLMs toward more accurate and relevant data generation.
3. Replacing MSLM with traditional BERT-style masking results in a modest performance reduction, emphasizing the significance of span-based masking in capturing relational dependencies.
4. Omitting SCL leads to a slight drop in performance, suggesting that it plays a key role in differentiating relevant spans, which is crucial for fine-grained RE.
5. The variant with all enhancements removed exhibits the lowest performance, particularly for smaller K values. This reinforces the necessity of these enhancements in boosting the performance of FSRE.

4.4 Analysis and Discussions

Explanation-driven Knowledge Generated from Different LLMs. We evaluate the impact of different LLMs applied to generate explanation-driven knowledge. We extend GPT-3.5 and Qwen2.5-7B [Team, 2024] as other knowledge generators under the TACRED dataset, and continue model pre-training as well as fine-tuning under the same setting of Llama-2. As shown in Figure 5, the models pre-trained on different LLMs have similar performance. This indicates that our framework is not sensitive to different LLMs for FSRE.

Scale of Synthetic data. As shown in Figure 6, performance initially improves with increasing volume of schemaconstrained data, but eventually declines, while still consistently surpassing the performance of the scenario without generated data. This observation suggests that incorporating schema-constrained data generated by LLMs can enhance FSRE performance up to a certain threshold. However, beyond this point, the inclusion of low-quality or noisy data begins to negatively affect the training process. This behavior is consistent with previous studies indicating that excessive noise in training data can undermine model performance, even considering the noise-resilience capabilities of LLMs [Song et al., 2022].

We also report supervised results using additional golden training data to measure the utility of synthetic data. We can achieve two conclusions: 1) the advantage of golden training data are more significant when it is scaled up. However, this also takes more expensive costs. 2) TKRE demonstrates the ability to achieve performance on par with models trained using significantly larger volumes of manually annotated golden data, despite being trained with only a small amount of synthetic data. Furthermore, the generation of the requisite synthetic data requires only a few minutes of GPU computation.

5 Conclusion

In this work, we propose TKRE, a Two-Stage Knowledge Guided framework for FSRE, which effectively bridges the gap between generative and discriminative learning paradigms. By leveraging LLMs for explanation-driven knowledge generation and schema-constrained synthetic data, TKRE addresses key challenges in FSRE, including data scarcity and limited generalization ability. Our two-stage knowledge-guided pre-training approach integrates Masked Span Language Modeling and Span-Level Contrastive Learning, enhances traditional FSRE models' ability to capture relational reasoning and fine-grained semantic distinctions. Extensive experiments on multiple benchmark datasets demonstrate that TKRE significantly improves the performance of traditional FSRE models. Ablation studies further validate the effectiveness of each proposed component. Future work could explore extending TKRE to broader relation extraction tasks, such as zero-shot scenarios, or further optimizing the generation process to mitigate noise in synthetic data.