Falsdo: Benchmarking Artifact-Controlled Multimodal Fake News Verification via Failure-Aligned Auditing

1 Introduction

In recent years, fake news has increasingly evolved into complex forms, combining textual narratives with highly realistic images or AI-generated content. Such multimodal content poses a severe threat, as visual cues significantly enhance perceived credibility [Newman and Schwarz, 2024; Pulm et al., 2024]. Moreover, emotional rendering in these fabrications further exacerbates users' gullibility and accelerates the dissemination of deceptive content [Ali Adeeb and Mirhoseini, 2023; Vosoughi et al., 2018]. With the rapid growth of synthetic media in social ecosystems [Bui et al., 2024; Wilson et al., 2023; Mirsky and Lee, 2021], traditional singlemodal detection or shallow fusion methods struggle to effectively cope with the structural complexity of multimodal fake news in real-world scenarios.

∗Corresponding author.

To address this, researchers have proposed various multimodal detection methods, typically attempting end-to-end discrimination through feature fusion. However, as illustrated in Figure 1, existing paradigms face limitations when heterogeneous tasks are compressed into the same discrimination channel. In high-stakes scenarios requiring the synthesis of multi-source evidence, these models often fail to capture fine-grained manipulation clues stably, lacking both robustness and interpretability.

These issues are compounded by insufficient characterization at the evaluation level. While existing datasets have laid a foundation [Boididou et al., 2018; Jin et al., 2017; Wang, 2017], most focus on short text or limited social media contexts. Many benchmarks merely require models to judge image-text consistency rather than factual veracity [Shu et al., 2020]. Even recent works introducing external knowledge [Luo et al., 2021; Shao et al., 2023; Papadopoulos et al., 2024; Schlichtkrull et al., 2023] are dominated by overall authenticity judgments [Thorne et al., 2018; Augenstein et al., 2019]. Consequently, a critical evaluation confound remains: it is difficult to distinguish whether a model truly performs evidence grounding or merely exploits statistical shortcuts (fake patterns) in the data [Geirhos et al., 2020; Arjovsky et al., 2020].

To bridge this gap, we construct FALSDO (Fine-grained Analysis of Logical and Semantic Deception in Online content), a diagnostic benchmark designed to systematically measure robust, grounded semantic verification. As shown in Figure 2, FALSDO differs from binary-label datasets by providing instruction-based fine-grained annotations. Importantly, each instruction is paired with explicit grounding evidence (image boxes / text spans), aligning our benchmark with established region-level phrase grounding settings [Yu et al., 2016; Mao et al., 2016]. Crucially, we introduce an Artifact Control protocol: a counterfactual stress test that equalizes low-level generation traces while preserving event semantics. Under this setting, we observe that monolithic multimodal validators tend to over-rely on shortcut cues learned from generation artifacts, and even unconstrained verification pipelines can drift toward shortcut-dominated consensus under noisy evidence.

These observations motivate a failure-aligned design: semantic verification should be decomposed into representation-incompatible channels. We specify DREA (Decoupled Reasoning Experts with Adaptive Arbitration), a reference evidence-auditing system (Figure 1c). DREA treats detection as an investigation rather than static classification. It processes textual logic checking, visual forensic analysis, and cross-modal consistency through independent reasoning channels, all guided by retrieval-augmented evidence. Finally, an adaptive fusion mechanism arbitrates these multi-perspective conclusions to output an explained verdict. DREA serves as a reproducible baseline for instantiating the benchmark, not as a standalone novelty claim.

The main contributions of this paper are summarized as follows:

• We implement a web-mining pipeline with instructionlevel grounding, advancing evaluation from label accuracy to verifiable evidence.
• We construct FALSDO as a diagnostic benchmark with a
• logic-centric taxonomy, long-form narratives, and counterfactual artifact-control evaluation that makes semantic robustness directly measurable.
• We provide DREA as a reference evidence-auditing baseline that specifies evidence acquisition and constrained channel-wise auditing with reliability-aware fusion.
• FALSDO exposes shortcut collapse under artifact equalization; strong auditing baselines improve grounding and reduce degradation in the artifact-controlled setting.

2 RelatedWork

Robust and Grounded Multimodal Verification. High accuracy on unconstrained benchmarks often masks brittle behavior, where models rely on spurious correlations rather than true understanding [Geirhos et al., 2020; Sagawa et al., 2020; Arjovsky et al., 2020]. For multimodal misinformation, auditability requires grounded verification, localizing specific image regions and text spans [Shao et al., 2023; Yu et al., 2016; Krishna et al., 2017], instead of producing single holistic labels. However, most fake news setups emphasize end-to-end veracity, obscuring the distinction between genuine semantic checking and shortcut exploitation [Shu et al., 2020]. Our FALSDO benchmark bridges this gap by pairing instruction-level grounding with a counterfactual artifact-control protocol to directly test semantic robustness.

Multimodal Fake News Detection. Early detection methods relied on feature fusion for overall discrimination, such as EANN [Wang et al., 2018], MVAE [Khattar et al., 2019], MGN [Qian et al., 2021], SpotFake [Singhal et al., 2019], and Fakeddit [Nakamura et al., 2020]. Later works modeled finegrained associations through cross-modal attention [Wu et al., 2021; Chen et al., 2022]. Recent research attempts to introduce localization; for instance, HAMMER [Shao et al., 2023] grounds tokens and CMA-VRP [Li et al., 2025] reveals inconsistencies. However, these methods are still dominated by overall judgments, making it difficult to systematically characterize complex, multi-location semantic manipulations.

Retrieval-Augmented and Agentic Verification. RAG [Lewis et al., 2020; Karpukhin et al., 2020] introduces external knowledge, but verification is often bottlenecked by evidence noise. While RARG [Yue et al., 2024], DEFAME [Braun et al., 2024], and vision-native indexing [Yu et al., 2024] extend this to multimodal scenarios, distinguishing robust reasoning from artifact shortcuts remains challenging. Concurrently, multi-agent frameworks like MIRAGE [Shopnil et al., 2025], RAMA [Yang et al., 2025], EXCLAIM [Wu et al., 2025], and MAD-Sherlock [Lakara et al., 2025] focus on generating interpretable verdicts. We complement them by emphasizing diagnostic evaluation. Our proposed DREA serves as a principled reference system, reducing shortcut-driven convergence via artifact-only information propagation and reliability routing.

3 FALSDO Dataset

We propose FALSDO, a multimodal fake news evaluation benchmark oriented towards semantic and logical inconsistencies and robust, grounded verification under counterfactual artifact equalization. FALSDO contains 12,740 highquality image-text pairs, equipped with long-form narratives and dense instruction-conditioned annotations.

Unlike datasets primarily relying on visual artifact clues or coarse-grained image-text mismatches, FALSDO jointly provides in a unified benchmark: (i) a fine-grained manipulation taxonomy oriented towards world knowledge and narrative reasoning; (ii) instruction-level grounding evidence (visual box/text span) corresponding to each manipulation instruction; (iii) an artifact-control comparative setting to suppress low-level generation trace shortcuts, thereby measuring semantic robustness and evidence grounding in a way that is directly auditable.

3.1 Systematic Data Acquisition Pipeline

To ensure high semantic diversity, we implement a staged web-mining pipeline targeting three heterogeneous sources: Authoritative News, including Reuters, The Washington Post, The Guardian, and AP News (60%); Collaborative Knowledge Bases such as Wikipedia (20%); and Professional Factchecking Repositories like Snopes and FactCheck.org (20%).

We employ on-demand semantic load balancing to prevent overfitting to hot topics. Specifically, we enforce a coarse domain distribution during crawling: Protocol (40%), SpatioTemporal (30%), and Context (30%). To operationalize this distribution, we constructed taxonomy-anchored seed lexicons for each domain. Specifically, we curated a set of highfrequency discriminative queries like bilateral treaty for Protocol, historical landmark for Spatio-Temporal, to perform targeted query-driven retrieval. This ensures that the retrieved raw corpus is explicitly aligned with our predefined semantic typology, rather than relying on random web crawling. To support complex reasoning, only samples with captions exceeding 100 words are retained. Crucially, we maintain a pure real distribution via dual near-duplicate de-duplication: (i) Image-level using Perceptual Hashing (dimg = 6), and (ii) Text-level using semantic similarity filtering (τtext = 0.92). This guarantees that the base samples are distinct and highquality.

3.2 Semantics-DrivenManipulation Generation

FALSDO adopts a semantics-driven plan-and-execute paradigm, emphasizing high semantic impact and low perceptual distortion. The generation process follows a strict protocol to simulate sophisticated fabrication tactics:

1. Intent-conditioned Planning: A SOTA Large Language Model acts as a planner to output a multi-step manipulation plan. We enforce a complexity constraint: each sample must contain 3 to 6 modifications, including at least one image edit and one text edit. The planner prioritizes logical contradictions ,e.g., protocol/fact/causality violations, over random noise.
2. Instruction-based Multimodal Execution: The visual side employs text-guided inpainting/completion [Ho et

al., 2020]; the text side executes counterfactual rewriting to flip key assertions while maintaining narrative fluency. This creates concealed but critical cross-modal conflicts.

1. Instruction-level Grounding & Sanitization: Each manipulation step generates explicit evidence: visual grounding boxes or text spans. We apply geometric sanitization, e.g., validity checks ymax > ymin, to ensure all grounding annotations are physically valid and auditable.

To strictly isolate semantic inconsistencies from the artifacts introduced during the generation process, we generate a Control Group by applying the same inpainting backend to Real images using an identity-preserving prompt, followed by identical post-processing, e.g., JPEG compression Q = 85. This protocol ensures that low-level generation traces are equalized across Real (Control) and Fake samples. This forces the model to ignore pixel-level artifacts and focus

(a) CLIP Score Comparison (b) Text Length vs. Edit Distance

Figure 5: Dataset statistical characteristics: (a) The CLIP similarity distribution of original and manipulated images overlaps highly, indicating manipulations preserve global alignment; (b) The scatter relationship between text length and edit distance reflects the concealed nature where "minor changes trigger major semantic reversals".

on high-level semantic logic inconsistencies. Thus, FALSDO explicitly measures reasoning ability under Artifact Control, rather than simple knowledge memorization or statistical pattern matching. The protocol is a standardized diagnostic stress test for one controlled generation pipeline, not a guarantee over all editing backends.

3.3 Fine-Grained Taxonomy

To simulate the complexity of real-world multimodal misinformation, where fabrications often retain high semantic consistency while violating specific world knowledge, FALSDO introduces a comprehensive taxonomy of 15 fine-grained categories. As shown in Figure 3 and Figure 4, these categories encompass both visual and textual modalities, shifting the detection focus from low-level artifacts to reasoning over crossmodal discrepancies and narrative logic.

Visual Manipulation Dimensions. Visual modifications fundamentally restructure the narrative context by introducing semantic conflicts into the scene. Temporal & Chronological Manipulation (21%) stands out as the most prevalent category, targeting the scene's temporal integrity by falsifying textual date markers or introducing anachronisms that contradict the event's historical grounding. A comparable proportion is attributed to Sociopolitical Identity Tampering (20%), which recontextualizes the event's ideological stance through the precise modification of cultural symbols and protest signage. Additionally, Contextual Object Incongruity (16%) presents a unique challenge to common sense reasoning by embedding objects that disrupt the scene's established atmospheric or physical causality.

Textual Manipulation Dimensions. Textual fabrications aim to distort the precise factual constraints and named entities within the narrative. We observe that Institutional & Protocol Fabrication (23%) constitutes the primary textual distortion, involving the hallucination of incompatible administrative roles or entities to violate established diplomatic protocols. Ranking second, Geospatial & Geopolitical Displacement (18%) generates location-based conflicts by attributing events to geographically or geopolitically incongruent regions. Finally, a significant volume of cases involves Chronological & Temporal Inconsistency (16%), which disrupts the narrative timeline by asserting dates or durations that are factually irreconcilable with the visual evidence.

3.4 Dataset Analysis and Characterization

Dataset Statistics. FALSDO ultimately contains 12,740 class-balanced samples (Train: 10,192 / Val: 1,274 / Test: 1,274). Figure 5(a) shows that the CLIP similarity distributions [Radford et al., 2021] of original and manipulated samples overlap highly, indicating that manipulations preserve global alignment. Figure 5(b) shows that under long texts (Avg. 465 words), small edit distances can trigger significant factual reversals, reflecting the core difficulty of the benchmark.

Comparison with Existing Datasets. Table 1 compares FALSDO with mainstream benchmarks. LIAR, FakeNewsNet, etc., are mostly coarse-grained binary classifications; NewsCLIPpings focuses on out-of-context pairing. In contrast, FALSDO is a benchmark that jointly supports instruction-level grounding, artifact-control evaluation, longform narratives, and a logic-centric taxonomy, systematically measuring the robustness of semantic verification.

Quality Control and Auditability. We use 500 aligned Real/Control/Fake triplets for QA. Manual validation on 100 triplets yields 99% semantic preservation, 100% contradiction validity, and 99% grounding reasonableness. Sanitization discards 6.2% of samples; the geometric-sanitization false-positive rate is 2.8%, the control semantic failure rate is 4.5%, and accuracy varies by at most 0.7 points across JPEG

Q = 70/85/95.

4 Reference Auditing Baseline: DREA Specification

To instantiate FALSDO's diagnostic protocols with a reproducible baseline, we specify DREA (Decoupled Reasoning Experts with Adaptive Arbitration). We design DREA as a streamlined evidence-auditing specification. Unlike monolithic architectures, it focuses on structural mitigation of the failure modes observed in monolithic validators. As shown in Figure 6, DREA replaces end-to-end discrimination with a decoupled evidence-auditing pipeline defined by strict information flow constraints.

Diagnostic Target (Failure Modes). We explicitly design DREA to address five failures identified in FALSDO: (F1) Representation Interference in shared parameters; (F2) Artifact Shortcut Learning exposed by artifact-control; (F3) Retrieval Noise; (F4) Evidence Conflict across heterogeneous channels; and (F5) Evidence Scarcity in long-tail samples.

4.1 ProblemFormulation

We model multimodal fake news detection as an evidencedriven binary classification task. Given a news sample N = {T, I}, where T is text and I is image, the system aims to predict the veracity label y ∈ {0, 1} (0: Real, 1: Fake) and generate an explanation E grounded in verifiable evidence. Formally, the inference process is defined as a mapping:

where Mmeta denotes the Meta-Auditor, {Ok} represents the outputs from K decoupled expert channels, and {Feam,Aeam} represent structured retrieval artifacts.

4.2 Phase 1: Auditable Evidence Acquisition

Standard RAG often injects noise directly into reasoning. To mitigate (F3) Retrieval Noise, DREA enforces a "Broad Entry, Strict Exit" protocol.

Vision-Native Retrieval and Filtering. We encode the query qN (using both image and text) and retrieve top-K candidates from the knowledge base K [Karpukhin et al., 2020]. To prevent irrelevant context injection, we apply a strict exit threshold τret:

If Cfiltered = ∅, the system executes a Zero-Day Fallback, blocking retrieval inputs to downstream experts to prevent hallucination.

The Evidence Acquisition Mechanism and Kill-Shot Scoring. Instead of end-to-end generation, an Evidence Acquisition Mechanism (EAM) distills Cfiltered into Textual Facts (Feam) and a calibrated Kill-Shot Signal (sks). The Kill-Shot signal quantifies the presence of decisive debunking patterns (e.g., known fake templates). We formulate sks as a continuous variable derived from retrieval similarity ri and the EAM's pattern-match confidence ki ∈ [0, 1]:

This formulation turns decisive evidence into a differentiable signal that can be weighed against other channels, avoiding hard-coded rules.

4.3 Phase 2: Decoupled Auditing and Arbitration

To prevent (F1) Representation Interference and (F2) Artifact Shortcut Learning, DREA enforces deep decoupling: reasoning is split into representation-incompatible channels that communicate only via compact artifacts.

Independent Expert Channels. We instantiate three experts: Narrative Fact Verifier (Enarr), Scene Logic Verifier (Escene), and Cross-Modal Verifier (Ecm). Crucially, to ensure independence, experts receive verifiable facts Feam but are blinded to the forensic advice Aeam. Each expert k outputs a normalized suspicion score ˜ Sk ∈ [0, 1] and a selfconfidence score ck.

Formalizing Reliability Arbitration. To handle (F4) Evidence Conflict, the final decision is mediated by a MetaAuditor via a learned gating mechanism. Unlike static fusion, we propose a Sample-Adaptive Reliability Router gϕ that dynamically estimates the trustworthiness of each channel. We construct a routing feature vector x by concatenating expert confidences, embeddings, and the kill-shot score. The channel weights w are computed via a temperature-scaled Softmax:

where τ controls the sharpness of the selection. The final verdict logic z is a reliability-weighted sum of expert scores and the kill-shot signal:

Finally, the probability is given by pmeta = σ(z). This mechanism ensures that if an image expert relies on artifacts (low reliability under control settings) or retrieval is noisy (low sks), the router can dynamically suppress their influence, achieving robustness.

Audit-Triggered Refinement. For (F5) Evidence Scarcity, we define a conditional feedback loop. If the Meta-Auditor's fusion confidence cmeta = 2|pmeta - 0.5| falls below a safety threshold τaudit:

This triggers targeted natural language feedback to the most uncertain expert, simulating a "second look" mechanism to improve coverage on long-tail samples.

5 Experiment

To validate FALSDO as a diagnostic benchmark, we evaluate three dimensions: veracity (Acc/AUC), auditability (instruction-level grounding), and robustness (counterfactual artifact-control). Figure 7 visualizes the results.

Setup Protocol: All trainable baselines are supervised using the same SSP JSON targets and identical training budgets. We first apply SFT/LoRA [Hu et al., 2021] to the backbone under SSP to obtain expert checkpoints, then train the reliability router (gϕ) with experts fixed. This ensures gains stem from structural decoupling rather than extra parameters.

5.1 Main Results: Diagnostics on Falsdo

Mitigating Shortcut Learning. As shown in Table 2, DREA achieves the highest Acc of 75.88% and AUC of 75.18% on Falsdo. Crucially, under the Artifact-Control (C) stress test, monolithic baselines FT-Qwen3-VL-8B (referred to as Qwen3-VL for brevity) and CoT-SC suffer a collapse of approximately -8.7%, revealing their heavy reliance on superficial generation traces. In contrast, DREA degrades by only 2.68%. This diagnostic gap confirms that FALSDO's artifact-control protocol successfully exposes shortcut learning, and DREA's decoupled auditing effectively shifts reliance towards semantic consistency.

Budget-matched Efficiency. Compared to CoT-SC@5 which also uses 5 inference calls, DREA leads significantly with 75.88 vs. 70.50 while using fewer tokens and lower latency (4.5k tokens / 1.8s vs. 5.0k / 2.0s). This indicates that performance gains derive from evidence-guided structural auditing rather than simply scaling inference compute via multiple sampling.

The "Artifact Probe" Baseline. We treat the standalone Scene Logic Verifier (Escene) as an "artifact probe". It performs poorly on the original set scoring only 62.15 and deteriorates further under Control. This confirms that FALSDO cannot be solved by visual forensics alone, validating the dataset's emphasis on semantic rather than pixel-level manipulation.

5.2 Generalization to Public Benchmarks

DREA achieves optimal Acc on all four benchmarks (Table 4). Notably on DGM4 and NewsCLIPpings, DREA improves by +2.9 and +4.3 points relative to DEFAME. This suggests that reliability-aware evidence auditing improves robustness against open-world evidence noise, validating the system's generalization capability beyond the specific distribution of FALSDO.

5.3 Fine-Grained Grounding

We evaluate the model's ability to output verifiable evidence chains. We report box localization quality using mAP@50 and mAP@75 following common object detection evaluation protocols [Yu et al., 2016]. We report Joint-F1, which counts a prediction as correct only if both the verdict and the evidence (box/span) are accurate. Results in Table 3 show DREA's Joint-F1 reaches 49.60, a +14.4 leap over the Base model and a +4.5 lead over GPT-4o. This significant gap quantifies the "hallucination vs. grounding" difference: while monolithic models may guess the right label, DREA provides auditable proof, aligning with the benchmark's goal of trustworthy verification.

5.4 Manipulation Type Prediction

We evaluate the distinguishability of specific tampering patterns like Temporal Displacement, Semantic Conflict. DREA leads significantly in all categories (Table 5), especially in "Text Fact/Logic" +20.6 and "Image Temporal Displacement" +20.4. This confirms that decoupling evidence channels allows for sharper diagnosis of complex causal and temporal contradictions compared to entangled monolithic representations. In other words, DREA is explicitly "checking the story," not just "reading the pixels."

5.5 Ablation Study: Validating Decoupling

Table 6 shows complementarity rather than a single driver. Evidence acquisition determines what the system can reason over: removing it causes the larger regression (70.42 vs. 73.42 for monolithic auditing). Decoupling determines how conflicting evidence is used: the full system still gains 2.46 points over monolithic EAM and has a much smaller artifactcontrol drop. The weak Scene Logic Verifier confirms that FALSDO cannot be solved by visual artifacts alone.

6 Conclusion

This work presents FALSDO, an artifact-controlled diagnostic benchmark for grounded multimodal misinformation verification, and DREA, a reference evidence-auditing baseline for instantiating its protocols. Experiments show that artifact equalization exposes shortcut reliance, while reliabilityaware auditing improves robustness and grounding. Future directions include broader generator sensitivity analysis and lower-latency deployment.

Ethical Statement

FALSDO is intended for research on misinformation detection and grounding. We do not release any automated tool for generating misinformation at scale, and we recommend downstream users follow institutional review and dataset governance practices. We further discourage any use that facilitates deception, manipulation, or targeted influence operations.