Debate with Myself: Zero-Shot Event Causality Identification with Adversarial Evidence Integration via Large Language Models

1 Introduction

Event Causality Identification (ECI) is a critical task in Natural Language Processing (NLP) that systematically identifies and validates causal relationships among events in unstructured text, facilitating automated extraction of structured knowledge [Cheng et al., 2025]. Revealing causalities between events in text is of significant importance for advancing automated knowledge extraction and decision-making processes in complex domains, such as military intelligence [Cockburn, 2010], scientific research [Gopalakrishnan et al., 2025], and social development [Radinsky et al., 2012]. In the ECI task, events are represented by their triggers, referred to as "event mentions." Event causality in text is classified into two types according to the distribution of event mentions: intra-sentence causality (or sentence-level causality), in which all related event mentions appear within a single sentence, and inter-sentence causality (or document-level causality), in which event mentions may span multiple sentences.

∗Corresponding author

Early ECI relied on lexical, temporal, and co-occurrence features [Chan et al., 1999; Mirza, 2014; Do et al., 2011]. With deep learning and Pre-trained Language Models (PLMs) [Devlin et al., 2019], performance improved through richer semantic representations [Zhao et al., 2024; Li et al., 2024; Hu et al., 2023; Kadowaki et al., 2019; Liu et al., 2021]. However, these supervised methods require large-scale annotated datasets, which are often inaccessible for domain-specific texts. Large Language Models (LLMs) exhibit strong few-shot and zero-shot capabilities [Kiciman et al., 2024], yet they face significant challenges in ECI tasks, primarily: (1) Causal hallucination, where LLMs systematically overestimate causal links due to architectural and pretraining biases [Gao et al., 2023]; and (2) Underestimation of long-distance causalities, as cross-sentence event pairs are often overlooked due to lexical distance and sequence modeling constraints. Figure 1 intuitively illustrates the two challenges mentioned above.

We argue that zero-shot ECI should return to the essence of causality, making judgments based on definitions. Although predictions from an individual LLM may be unreliable, their reasoning exhibits consistent internal patterns [Wang et al., 2023], suggesting that systematic consensus from multiple perspectives can lead to more reliable causal inference [Cai et al., 2025]. Inspired by this, we propose LLM Self-Debate (LLM-SD). Unlike recent multi-agent methods that deploy different LLMs for the same task, our approach employs multiple identical LLM agents. Each agent is assigned a unique, non-overlapping role to evaluate a specific dimension of causality, with the final decision emerging from the adversarial integration of their heterogeneous evidence, not from model diversity. Specifically, the framework formalizes ECI as a structured debate process, consisting of affirmative and negative teams that collect multi-dimensional evidence, and an adjudication committee that evaluates the evidence to deliver a final verdict. This role-decomposed design is key to reducing causal hallucination while improving the recall of long-distance causal relations. It effectively mitigates spurious causal links and false negatives caused by single-evidence errors, as illustrated in Figure 2.

Our contributions are threefold:

• We systematically summarize prerequisite conditions for causal relations and their supporting/rebuttal evidence based on causality definitions, providing a comprehensive argumentative foundation for multi-evidence causal reasoning.
• We innovatively reformulate ECI as an LLM debate, where identical LLMs act as debaters and adjudicators to present, evaluate, and adjudicate causal evidence.
• Extensive experiments on three benchmarks show that LLM-SD achieves state-of-the-art zero-shot performance, with notable robustness in long-text and long-distance ECI tasks.

2 RelatedWorks

2.1 Event Causality Identification

Early research on ECI primarily focused on sentence-level ECI (SECI) [Liu et al., 2023; Liu et al., 2021; Zuo et al., 2020]. However, document-level ECI (DECI) has recently gained significant attention and is an emerging research area. Current DECI strategies include: (1) Promptbased fine-tuning models [Xiang et al., 2023; Yuan et al., 2023], which adapt pre-trained models through innovative prompt engineering. (2) Event graph reasoning methods [Liu et al., 2024; Chen et al., 2022], which model causality as graph inference problems using structured event representations. (3) Contextual encoding techniques [Man et al., 2024; Huang et al., 2024], which leverage deep learning architectures to capture semantic and contextual features. In the past two years, LLM-based zero-shot ECI models have been explored but are limited by causal hallucination. Models such as KnowQA [Wang et al., 2024] and Dr.ECI [Cai et al., 2025]

proposed targeted solutions: the former utilizes event argument structures to enhance reasoning, while the latter addresses diverse causal structures by incorporating prior causal knowledge.

However, existing LLM-based ECI models oversimplify causality analysis by neglecting essential causal definitions and supporting/contradictory evidence, leading to persistent hallucination. Our LLM-SD mitigates this by simulating evidence-based human reasoning through LLMs' selfconsistent causal understanding.

2.2 Multiple LLMAgents Debate

Multiple LLM agents debating has emerged as a promising approach to enhance reasoning and factual accuracy by letting models challenge and refine each other's outputs. [Estornell and Liu, 2025] provide a formal framework for debate dynamics and propose interventions to overcome majority convergence. [Ku et al., 2025] demonstrate that structured two-agent debates can effectively recover implicit premises, outperforming single-agent models. MRBalance[Zou et al., 2025] employs multiple LLM agents in a joint questionanswering framework for ECI. However, it mainly aggregates agents that repeatedly reason about causality at a coarse granularity. More broadly, most of current multi-agent LLM paradigms rely on heterogeneous agents (different models or differently tuned agents) to solve the same task instance, expecting diversity to improve accuracy.

By contrast, LLM-SD uses identical LLM instances and achieves diversity through role decomposition: each agent is constrained to produce a specific type of evidence. Therefore, LLM-SD is not "multiple models doing the same reasoning," but rather "one model performing multiple complementary causal tests," enabling adversarial evidence integration that is directly grounded in the causality definition.

3 Preliminaries

3.1 ProblemFormulation

Given an input document D comprising multiple sentences {S1, S2, . . . , SN}, containing a set of annotated event mentions {ei, ej, . . . , eM}, ECI seeks to identify all causal event pairs {p1, p2, . . . }, where each pair pl = (ei, ej) denotes a causal relationship C(pl) ∈ {0, 1}, where i, j ∈ 1, 2, . . . ,M.

3.2 Event Causality Definition

Grounding in both logical/commonsense causality definitions [Cheng et al., 2025] and established annotation protocols from prior work [Caselli and Vossen, 2017; Wang et al., 2022; Lai et al., 2022], we define event causality as follows:

Definition 1. For a given context D, causality ei → ej between two events ei, ej ∈ D exists if and only if they satisfy at least one of the following three conditions:

• Enablement/Prevention: ei occurs before ej and has effect on (enables or prevents) the occurrence of ej;
• Counterfactual Dependence: ej would not occur if ei did not occur (counterfactual necessity);
• Deterministic Implication: The occurrence of ei necessarily entails the occurrence of ej (logical sufficiency).

4 Methodology

LLM-SD consists of three modules: Affirmative Team (AT), Negative Team (NT), and Adjudication Committee (AC), each comprising identical LLM agents to collect different evidence or issue a verdict across two distinct reasoning phases (see Figure 3 ).

4.1 Reasoning Phrases

The first phase is the Evidence Presentation Phase, AT and NT deploy LLM agents to collect supporting or rebuttal evidence for the existence of causality. In this phase, the LLM agents function as "debaters," articulating their positions by leveraging allocated reasoning evidence to support their claims. This step parallels the Constructive Speech stage in formal debates, where teams systematically present their core arguments.

The second phase is the Adjudication Phase, which involves evaluating the strength (reliability) of both supporting and rebuttal evidence in the first phase through two adjudication mechanisms: 1) An automatic adjudication method that applies pre-determined rules to cases where the evidence strength demonstrates a clear disparity; and 2) A panel adjudication method where the collected evidence is submitted to AC for contradiction analysis and final verdict. This phase resembles the Open Argumentation stage in formal debates, where the debaters present and rebut claims, the adjudication mechanisms systematically weigh opposing evidence. The automatic adjudication handles clear-cut cases efficiently, while the panel adjudication resembles moderated deliberation.

4.2 Debate Teams

Aligned with standard debate formats, LLM-SD comprises two opposing teams: AT (arguing for causality) and the NT (rejecting it). For an ordered query event pair (ei, ej), these two teams target for collecting and presenting evidence that supports/rebuts the causality ei → ej (see Figure 4).

Supporting Evidence of Affirmative Team

Building upon formal definitions of causality, our analysis employs six categories of supporting evidence collected through LLM agents: (1) direct causal question answering result Es dir , (2) temporal ordering validation result Es tem, (3) necessity testing result via counterfactuals Esnec , (4) sufficiency analysis result Es suf (if ei occurs, ej must occur), and (5) explicit lexical evidence Es exp identifying causal markers that linguistically connect the events in context. (6) mediating factor evidence result Esmed : whether there exists a mediator em that satisfies ei → em and em → ej. The supporting evidence can be formalized as:

Explicit lexical evidence Es exp is used only for intra-sentence event pairs due to the difficulty of accurately identifying causal markers across sentences. Mediating factor evidence Esmed is applied only to inter-sentence event pairs, as mediators are less common within single sentences and prone to confusion in intra-sentence contexts.

Rebuttal Evidence of Negative Team

The opposing debater agents gather the following three categories of rebuttal evidence: (1) reverse temporal evidence Er tem, which demonstrates that the cause event does not precede the effect event; (2) non-dependency evidence Er dep , indicating that the change of ei has no effect on ej; (3) contradictory relation evidence Er con , where the relationship rab between the two events contradicts causality, such as coreferential events or inclusive (sub-event) relations. The rebuttal evidence can be formalized as:

We only selects reverse temporal evidence for rebuttal and does not select the evidence that contradicts necessity and sufficiency, because according to Definition 1, sufficiency and necessity are not prerequisites for the existence of causality. The purpose of non-dependency evidence is to prevent LLMs from engaging in excessive reasoning.

Evidence Strength Grading

As the affirmative and negative teams may present perspectively divergent evidence, we evaluate the strength of both supporting and rebuttal evidence via a predefined rule-based approach. Strength grading rules are designed according to the importance and decisiveness of different evidence types (denoted fs for support and fr for rebuttal). Temporal precedence (Es tem) is treated as a near-necessary constraint and combined with complementary tests-direct causal judgment (Es dir ) and counterfactual necessity (Esnec )-to avoid over-reliance on any single cue. Strong support is assigned only under multi-test agreement accompanied by a decisive connector. On the rebuttal side, higher strength grades require multiple independent refutation cues, enhancing robustness against noisy single-evidence rebuttals.

Table 1 presents the grading rules 1 . These rules are consistent with the fundamental causality logic in Definition 1, and the multi-evidence grading reduces misidentification risks caused by individual misinterpretations and overestimation biases.

1 Where 1 denotes 'TRUE' and 0 denotes 'FALSE' for the evidence.

Evidence Adjudication

Based on the supporting and rebuttal evidence strengths, Gs = fs(E s ) and Gr = fr(E r ), we employ two adjudication methods: automatic adjudication and LLM-based (AC) adjudication. The automatic method follows the rules in Table 2. If it cannot reach a conclusion, the evidence is submitted to AC, which then reasons comprehensively to deliver a final verdict. The causality verdict C(ei, ej) thus integrates rule-based (Φ) and LLM-based (Ψ) adjudication via Equation 4.

Here Φ(·) denotes leveraging prompt template to guider LLM for causal determination according to support evidence and rebuttal evidence.

5 Experiments

To evaluate the effectiveness and advancement of LLM-SD and verify the rationality of the framework design, we conducted extensive comparative experiments on three benchmark datasets. In our experiments, we primarily investigate the following five research questions (RQs):

• RQ1: Can LLM-SD effectively mitigate the excessive false positives caused by causal hallucination?
• RQ2: Are all "debaters" (i.e., LLM agents for evidence collection) in the affirmative and opposing teams necessary?

• RQ3: Are the two adjudication mechanisms necessary for the model?
• RQ4: Are the evidence strength grading rules effective and necessary for the model?
• RQ5: How robust is LLM-SD with respect to event distance and text length?

5.1 Experimental Setup

Datasets

We selected three datasets for comparative validation.

• Causal-TimeBank (CTB) [Mirza et al., 2014] focuses on explicit causality. It is commonly used for evaluating intrasentence ECI performance. We conduct 10 independent trials on CTB, each time randomly selecting 10% of the dataset as the test set, and report the average results.
• EventStoryLine Corpus (ESL) [Caselli and Vossen, 2017] is designed for cross-document reasoning and narrative generation. We perform 5 independent trials, each excluding 2

topics and taking 20% of the remaining data as the test set, then report the average results.

• MAVEN-ERE [Wang et al., 2022] is a large-scale event relation extraction dataset covers coreference, temporal, causal, and subevent relations. Following the original split, we directly use its development set as the test set for evaluation.

Baselines

Given that we focus on model performance in zero-shot scenarios, we employed different reasoning methods based on LLMs as unsupervised baselines. Specifically, we selected five prompt-based reasoning paradigms for comparison:

• Simple Prompt (SP) [Brown et al., 2020]: We use a simple prompt to guide the LLM to directly judge and answer whether a causal relationship exists based on the context and event pair.
• In-Context Learning (ICL) [Min et al., 2022]: In addition to the simple prompt, we provide positive and negative examples for the LLM to perform in-context learning. Following the findings in [Gao et al., 2023], we selected 2 positive and 4 negative examples as demonstrations.
• Zero-Shot CoT (ZSCoT) [Wei et al., 2022]: Based on the simple prompt, we add the instruction "think step-by-step" to require the LLM to reason step-by-step.
• Self-Ask [Press et al., 2023]: Following the causal judgment logic, we crafted a prompt template to guide the LLM to reason sequentially from perspectives of temporality, sufficiency, and necessity.
• Definition Prompt: We formulated the definition of event causality proposed in this paper into a prompt, requiring the LLM to judge based on this definition and select the corresponding causal relation type.

In addition, we compared LLM-SD with two advanced unsupervised LLM-based models, KnowQA [Wang et al., 2024] and Dr.ECI [Cai et al., 2025]. To quantify the gap between unsupervised and supervised models, we also selected multiple supervised models for comparison, including BERT [Devlin et al., 2019], RoBERTa [Zhuang et al., 2021], Longformer [Beltagy et al., 2020], KADE [Wu et al., 2023], ERGO [Chen et al., 2022] and iLIF [Liu et al., 2024].

Evaluation Metrics

Consistent with the previous works, we use Precision (P), Recall (R), and F1-score (F1) to evaluate model performance.

Implementation Details

We selected two advanced LLMs as the backbone models: GPT-4o-mini 2 and Deepseek-V3 (DeepSeek-Chat) 3 for main experiment. In Section 5.6, we compared the performance of LLM-SD on different LLMs.

In experiments, we utilized Python's concurrent. futures.ThreadPoolExecutor 4 to implement multi-threaded parallel calls to all LLM agents, thereby saving inference time. We set temperature = 0 to reduce the randomness of generated outputs while maintaining logical consistency.

5.2 Main Results (RQ1)

Tables 3-4 show that LLM-SD achieves state-of-the-art performance among unsupervised models across all benchmarks. It significantly improves precision by reducing false positives from causal hallucination and enhances recall on inter-sentence ECI through multi-evidence reasoning. Compared to supervised baselines, LLM-SD remains competitive and even outperforms several supervised models on ESL, highlighting its strong potential. Simpler strategies like SelfAsk or providing the Definition yield only marginal gains, confirming that explicit instructions alone cannot fix LLMs' consistency errors in causal reasoning. The consistent results across different LLMs (GPT-4o-mini and DeepSeek-Chat) urther validate the robustness of our approach.

2 https://openai.com/api/

3 https://api.deepseek.com/v1

4 https://docs.python.org/3/library/concurrent.futures.html

5.3 Ablation Studies (RQ2-RQ3)

This section presents ablation studies from three perspectives to verify the effectiveness of each evidence type and module. All experiments were conducted on a test subset of the ESL dataset, with LLM parameters kept identical to the main experiments. The ablation experiments results are shown in Figure 5.

Evidence Ablation To systematically evaluate the independent contribution and indispensability of each type of evidence, an ablation study on evidence was conducted. Specifically, according to the rules in Table 1, when a specific evidence-collection agent is ablated, the corresponding term is excluded from the evidence aggregation formula (with the threshold kept unchanged), and the assessment outcome of causal relations is re-determined. The experimental results in Figure 5 demonstrate that removing any supportive or rebuttal evidence leads to a decline in overall model performance, indicating that all the aforementioned evidence is essential for the framework.

LLM Adjudication Mechanism Ablation We further ablated the LLM-based adjudication mechanism (AC) by forcing uncertain cases to be judged as either always non-existent (-adj(-)) or always existent (-adj(+)). Both ablation settings lead to performance degradation, confirming that the LLM adjudicator is an indispensable component of the framework. This indicates that while our rule-based automatic adjudication is logically grounded, it benefits from the flexible judgment of an LLM to handle nuanced and complex scenarios that pre-defined rules may not fully cover. It also compensates for potential errors arise from the subjectivity of rules.

Automatic Adjudication Mechanism Ablation We removed the predefined adjudication rules and relied solely on the LLM adjudicator (AC) to judge all cases. As shown in Figure 5, this leads to a significant performance drop, especially in precision. This result demonstrates that relying exclusively on LLM judgment is unreliable; without structured rules to guide the synthesis of multiple evidences, the LLM tends to make simplistic decisions based on evidence counts, which increases false positives and confirms the necessity of our hybrid adjudication design.

5.4 Validating the Evidence Grading Rules (RQ4)

To further examine whether the evidence strength grading rules in Table 1 are reasonable and not overly tailored, we keep the debater agents and evidence extraction prompts unchanged, and only replace the mapping from evidences to (Gs,Gr) and the subsequent adjudication. We consider the following alternatives:

(1) Count-based grading, which assigns strength levels based solely on the count of satisfied evidences:

(2) Simple-voting grading, which determines causality directly by comparing the counts of supporting and rebuttal evidences with a threshold k:

In this method, k serves as a hyperparameter representing the minimum number of supporting evidences required when no rebuttal evidence is present.

Table 5 shows that our evidence grading rules significantly outperform the other two alternatives, confirming their effectiveness. The count-based method performs worst, as merely tallying evidence amplifies the LLM's systematic errors and yields low precision. Simple-voting grading strongly depends on threshold k, with precision rising and recall falling as k increases, making it difficult to achieve an effective balance.

5.5 Impact of Lexical Distance and Text Length (RQ5)

Prior research suggests that LLM-based ECI performance degrades with increasing document length and event distance. To assess LLM-SD's robustness, we analyzed its performance across different lexical distance and document length intervals on CTB and ESL datasets (Figure 6).

The results confirm that all unsupervised baselines suffer significant performance drops as distance and length increase, with the Definition baseline showing a near-linear decline. In contrast, LLM-SD exhibits strong resilience, maintaining stable performance (F1 > 0.3) with only minor degradation. This demonstrates LLM-SD's superior robustness in handling long-range dependencies and confirms the effectiveness of its multi-evidence reasoning approach for complex ECI tasks.

5.6 Evaluating LLM-SDwith Different LLM Architectures

To assess the sensitivity and practical deployability of the LLM-SD framework across different LLM architectures, we conduct additional experiments on the ESL dataset using the same zero-shot prompts, identical evidence role assignments, and consistent decoding configurations (temperature = 0). We evaluate several widely adopted LLMs with distinct architectures, including LLaMA3-8B, GPT-3.5, Qwen3max, and GLM4-32B, and compare their performance with DeepSeek-Chat (our backbone).

Figure 7 reveals two key findings. First, all LLM-SD variants outperform unsupervised baselines using the same corresponding LLM (overall F1 < 0.2), confirming that its roledecomposition framework structurally reduces causal hallucination and enhances performance independently of the backbone model. Second, performance does not scale monotonically with parameter count: DeepSeek-Chat performs best, while the larger Qwen3-max scores lower; the smaller GPT3.5-turbo outperforms the larger GLM4-32B. This indicates that ECI capability depends on factors beyond scale, such as pre-training data, architecture, and specialized tuning. The backbone model's inductive biases and training alignment are also crucial for effective deployment within LLM-SD.

6 Conclusion

We propose LLM-SD, a framework that casts ECI as a structured debate among specialized LLM agents. LLM-SD employs Affirmative and Negative Teams to collect supporting and rebuttal evidence, evaluated by an Adjudication Committee using rule-based and LLM-driven methods. Experiments on three benchmarks show state-of-the-art zero-shot performance, with improved precision and robust recall. However, the reliance on multiple LLM invocations raises cost, and performance on inter-sentence causality remains below that of supervised models. Future research could explore fine-tuning strategies or alignment mechanisms that constrain LLM outputs to further mitigate hallucinations, enhance evidence consistency, and improve multi-hop reasoning capabilities.

Acknowledgments

Zefan Zeng and Yuehang Si contributed equally to this work.