ICFD-31k: A Large-Scale Dataset and Benchmark for Real-Time Conversational Fraud Detection

1 Introduction

AI plays a crucial role in advancing social good by safeguarding marginalized communities from exploitation. Conversational fraud uses phone calls as a primary medium, resulting in financial loss and emotional harm to millions of people. India, home to over a billion telecom users, has advanced its digital payment ecosystem. However, the payment ecosystem is threatened by evolving social engineering techniques utilized by conversational fraud. Moreover, these attacks harness urgency and trust to disproportionately affect vulnerable groups such as the elderly, populations with limited digital literacy, and individuals in linguistically diverse or low-resource environments. Also, it decreases public trust and economic progress for digital financial inclusion. Traditional AI-driven detection systems can support fraud prevention, enabling inclusive digital ecosystems. Nevertheless, such systems are hampered by the non-availability of a largescale public dataset for conversational fraud. Existing fraud detection research has made considerable progress using graph-based and transactional approaches [Bai et al., 2025; Wu et al., 2024; Zou and Cheng, 2024]. However, these approaches do not address bilingual conversational fraud detection in Indian contexts. Moreover, they primarily emphasize detection performance rather than granular, conversationlevel explanations. Additionally, the non-availability of annotated datasets with granular and explainable labels denies actionable insights for real-time streaming call analysis (during ongoing calls) and transparent "slow-thinking" rationales. Therefore, such scarcity of resources limits not only rigorous benchmarking and comparative evaluation of detection approaches but also the deployment of practical solutions in resource-constrained settings where protection is most urgently needed, perpetuating a cycle where those most vulnerable to exploitation have the least access to protective technologies. Hence, there is a strong need for comprehensive, multilingual conversational fraud benchmark datasets with rich annotations that enable both real-time detection and explainable decision-making in diverse, resource-constrained environments.

∗Corresponding Author.

The main contributions of the proposed work are threefold:

• The work introduces ICFD-31k, the first large-scale Indian conversational fraud benchmark designed for bilingual (English/Hinglish) contexts, directly addressing SDG 10 (Reduced Inequalities) by enabling protective technologies for vulnerable, linguistically diverse populations disproportionately targeted by telephone scams.
• The proposed work establishes baselines with finetuned RoBERTa models (M1 for static, M2 for streaming), enabling chunk-level real-time fraud identification supported by rationale annotations and providing a lightweight baseline for future low-resource deployment studies.
• The study develops a complete methodology using LLMs and efficient architectures designed with lowresource and mobile deployment constraints in mind,

balancing accuracy, interpretability, and computational cost.

2 RelatedWork

This section reviews key literature on fraud detection, focusing on datasets, synthetic data generation, conversational AI applications, real-time/streaming techniques, and explainable AI approaches in security contexts.

2.1 Fraud Detection Datasets

The scarcity of publicly available conversational datasets has hindered the development of robust fraud detection systems. Furthermore, existing work predominantly relies on proprietary data from financial institutions or telecommunications providers, limiting reproducibility and cross-study comparison. The most closely related prior work is TeleAntiFraud28k [Ma et al., 2025], a recent dataset of 28,000 Chinese telecom fraud conversations with audio and text modalities. TeleAntiFraud-28k provides slow-thinking rationales for explainability. However, it is limited to the monolingual Chinese context, lacks chunk-level temporal annotations essential for real-time streaming detection, and fails to capture code-switching (Hinglish) or region-specific social engineering tactics prevalent in the Global South, such as in India. Moreover, real Indian telecom fraud calls are difficult to redistribute because recordings may contain sensitive personal data and raise consent and data-protection concerns, therefore necessitating an approach that can address bilingual conversational fraud detection in Indian contexts with granular, conversation-level explanations. Besides, other fraud-related datasets focus on different modalities. For instance, financial transaction datasets [Dal Pozzolo et al., 2014] provide tabular format data but lack the conversational context required. Similarly, email phishing datasets [Fette et al., 2007] and SMS spam corpora [Almeida et al., 2011] operate on written text rather than conversational interactions, therefore necessitating a resource that can jointly address real-time conversational fraud detection and explainability requirements.

2.2 Conversational AI for Security

Recent advancements in large language models have demonstrated strong performance on various NLP tasks, including zero-shot and few-shot learning scenarios [Brown et al., 2020; Chung et al., 2024]. However, their application to security domains like fraud detection remains underexplored. The zero-shot baseline with Gemma 1.1 2B [Gemma Team et al., 2024] is used as a compact, on-device reference point rather than a proxy for all frontier LLMs. Its 35.83% F1 on ICFD-31k shows that this lightweight prompt-only baseline is insufficient for reliable fraud screening without domain adaptation, while evaluating larger cloud LLMs remains complementary future work. Additionally, the literature includes related work on dialogue safety that focuses primarily on toxicity detection [Zhang et al., 2018], hate speech identification [Davidson et al., 2017], and misinformation in social media [Shu et al., 2017]. Those tasks, while related, differ fundamentally from fraud detection, which requires recognizing subtle social-engineering tactics and policy violations, and reasoning about the temporal progression of deceptive behavior within a conversation.

2.3 Synthetic Data Generation

The use of LLMs for synthetic data generation has gained traction across various NLP applications [Puri et al., 2020; Schick and Schütze, 2021]. Recent work [Ye et al., 2022] shows that carefully controlled synthetic data can augment or even replace human-annotated datasets for certain tasks. However, concerns about distribution shift, hallucination, and lack of authentic complexity remain significant challenges. Therefore, ICFD-31k adopts a generation approach different from prior synthetic data efforts in three key aspects. First, a structured generation pipeline is employed with explicit persona and scenario combinations to ensure even coverage and diversity. Second, ICFD-31k quality is evaluated through rigorous human evaluation with a mean pairwise Cohen's Kappa of 0.534, indicating moderate inter-annotator agreement, rather than only relying on automatic metrics. Third, utility is demonstrated through comprehensive benchmarking, showing that models trained on ICFD-31k achieve strong performance on realistic fraud detection tasks.

2.4 Streaming and Real-Time Detection

Most fraud detection systems operate in batch mode, analyzing complete transactions or conversations post-hoc. However, real-world deployments require real-time detection to enable immediate intervention against fraud. In the literature, prior work on streaming classification has focused primarily on concept drift adaptation [Gama et al., 2014] and online learning [Hoi et al., 2021], rather than the temporal progression of fraud signals within individual conversations. Consequently, ICFD-31k employs a chunk-level annotation schema that enables research into early detection systems that must balance speed (detecting fraud quickly) against accuracy (avoiding false alarms with incomplete information).

2.5 Explainable AI for Security

The need for explainable AI in security applications has been widely recognized [Barredo Arrieta et al., 2020], particularly for high-stakes decisions involving financial loss or legal consequences. Recently, several authors explored various approaches, including attention visualization [Wiegreffe and Pinter, 2019], rationale extraction [Lei et al., 2016], and chain-of-thought reasoning [Wei et al., 2022]. Therefore, ICFD-31k integrates explicit reasoning rationales. Each conversation in the proposed ICFD-31k contains reasoning chains detailing specific policy violations and conversational evidence using slow-thinking rationales [Ma et al., 2025] presented in TeleAntiFraud-28k.

3 The ICFD-31k Dataset

The proposed work uses a novel generational pipeline to produce ICFD-31k that addresses data limitations and ensures contextual applicability for the Indian fraud scenarios.

Listing 1. Complete annotation schema for a single conversation in ICFD-31k.

{ "transcript": [ {"speaker": "Agent", "text": "Namaste...", " timestamp_end": 5.0}, {"speaker": "Customer", "text": "Achha...", " timestamp_end": 10.0} ], "multimodal_analysis": { "dominant_emotion": "Trust", "pace": "Fast", "confidence_score": 0.8 }, "key_entities": { "organization": ["Canara Bank"], "product": ["Card"], "pii_requested": ["Card number", "Expiry", "CVV"] }, "chunk_level_analysis": [ {"timestamp": 10.0, "verdict_at_chunk": "NO", "rationale_at_chunk": "Introduction"}, {"timestamp": 25.0, "verdict_at_chunk": "YES", "rationale_at_chunk": "Card number requested (Rule A) "} ], "final_slow_thinking_rationale": "Agent requested full card...", "final_verdict": "YES", "violated_policies": ["Rule A: no full card requests"], "scam_outcome": "Successful", "session_id": 213, "agent_persona": "Empathetic helper persona", "customer_persona": "Busy professional persona", "scenario": { "scenario_id": 3, "description": "Debit card CVV scam" , "case_type": "Clear Fraud", "policy": "Rule A: no card data" } }

3.4 Dataset Statistics

The dataset ICFD-31k expands 31,000 source conversations into 1,111,071 temporal chunks. Specifically, the 3-second interval was selected to approximate short conversational turns and support research into early intervention. Further, in order to control a fraud-heavy training expansion, all nonClear-Normal cases are fully chunked while Clear Normal calls are sparsely sampled (1-2 chunks). It may be noted that the asymmetric construction is explicit, should not be read as deployment prevalence, and consequently motivates future interval/sampling ablations. Moreover, the ICFD-31k dataset

3.1 Design Principles

The study covers 10 fraud categories ("umbrellas") such as bank & payment fraud, E-commerce scams, job scams, government impersonation, and many more, with each umbrella containing 30 scenarios with varying difficulty.

3.2 Data Generation Pipeline

The generation of ICFD-31k followed a systematic, two-stage pipeline as in Figure 1. First, it employs foundational components: 10 agent and customer personas, over 300 scenarios across 10 fraud umbrellas (each with description, case type, and policy rules), and extensive entity lists including Indian banks, e-commerce platforms, and region-specific names. Thereafter, the pipeline uses the cross-product of personas and scenarios, yielding approximately 31,000 structured JSON conversations using an LLM-based generation system. Finally, in the second stage, each conversation expands into temporal chunks at 3-second intervals, producing 1,111,071 training samples for real-time streaming models.

3.3 Annotation Schema

Each 1 conversation is stored as structured JSON (Listing 1) containing timestamped transcripts with speaker turns. The schema includes two specific innovations for training nextgeneration detection systems as discussed below:

Chunk-level streaming labels. The schema provides verdict and rationale annotations at approximately 3-second intervals throughout each conversation, enabling training and evaluation of real-time detection systems that must make progressive decisions with incomplete information.

Slow-thinking rationales. Each conversation includes a detailed reasoning chain explaining the final verdict [Ma et al., 2025], citing specific policy violations and conversational evidence. Furthermore, it facilitates the development of explainable AI systems that can provide transparent justifications. Besides, the schema also captures entity extraction (pii_requested, organizations, products), violated policies, scam outcomes, and the generative context (scenario, personas) for comprehensive analysis.

1 Large language models were used only to assist with grammar refinement and manuscript editing. All scientific content, experimental results, and conclusions are the sole work of the authors.

is structured across 10 primary fraud umbrellas to ensure a diverse and balanced benchmark. Specifically, each umbrella contributes approximately 3,000 source conversations, ensuring no single fraud typology dominates the ICFD-31k dataset as shown in Table 2. Furthermore, the main training set consists of these 10 umbrellas, while an additional 1,000 conversations across 5 completely unseen domains (e.g., Cryptocurrency, Romance scams) were generated to create a dedicated cross-domain test set for evaluating model generalization. In addition, within each umbrella, the distribution of scenario complexity is maintained according to the adopted design principles as illustrated in Figure 2, providing a rich mix of clear, ambiguous, and subtle cases. Besides, Table 1 summarises the overall statistics for ICFD-31k.

4 Quantitative Dataset Analysis

The study evaluates four corpus properties to test whether ICFD-31k is fluent, diverse, and not merely templated: (1) linguistic fluency, (2) structural novelty, (3) semantic complexity, and (4) vocabulary distribution. Subsequently, these diagnostics directly target synthetic-artifact risks and thereby complement, rather than replace, real-call validation.

4.1 Linguistic Fluency (Perplexity)

A core requirement for a synthetic dataset is that it is fluent and natural-sounding. Hence, ICFD-31k measures fluency using Perplexity (PPL) [Jelinek et al., 1977], the standard metric for evaluating language model naturalness, where a lower score indicates better performance. Specifically, it calculates the PPL of conversations using a pre-trained GPT2 model. The ICFD-31k dataset achieves a low mean PPL of 20.44 for normal conversations and 22.37 for fraud conversations as described in Table 3. The obtained scores provide model-based evidence that the Llama 3.3 70B [Meta, 2024] generator produced linguistically fluent text, while human evaluation provides complementary evidence regarding perceived realism and naturalness.

4.2 Structural Novelty (N-gramOverlap)

A major risk in synthetic generation is "templating," where the model produces conversations that are structurally identical but with different entities. Therefore, the proposed work measures the N-gram Overlap between 10,000 random pairs of conversations for validation. The result shows a vanishingly small overlap, providing evidence that ICFD-31k is not simply "copy-pasting" templates as summarized in Table 4. Additionally, the obtained average 3-gram overlap is only 1.96%, suggesting that the used generation pipeline, with its 300 unique scenarios and 20 persona combinations, produced structurally varied conversations.

4.3 Semantic Complexity (t-SNE& Similarity)

The study designs a robust benchmark for fraud detection that deliberately emphasizes difficulty. Consequently, fraudulent messages display subtle linguistic patterns that overlap substantially with legitimate conversations. Such semantic complexity was evaluated using two distinct methods. First, the proposed work computes the Semantic Similarity (cosine similarity of all-MiniLM-L6-v2 embeddings)

• Clear fraud (n=661) Clear normal (n=665)

Subtle fraud (n=300)

Ambig. normal (n=319)

Table 5: Semantic similarity scores showing high class overlap, indicating dataset complexity.

within classes (e.g., fraud-to-fraud) and between classes (e.g., fraud-to-normal). Also, the obtained result, as shown in Table 5, indicates that the dataset contains a low separation score of 0.017. It suggests that fraudulent and normal conversations are linguistically close, making the task difficult for models that rely only on shallow lexical cues. Additionally, the study visualizes such complexity in Figure 3 using a tSNE [van der Maaten and Hinton, 2008] plot. The "Fraud vs Normal" panel (top) shows substantial class overlap. The "By Case Type" panel (bottom) is even more revealing: the "Subtle Fraud" (orange) and "Ambig. normal" (light blue) cases are semantically mixed in the "battleground" area between the clear cases. The results provide quantitative evidence that ICFD-31k constitutes a difficult benchmark, demanding models to perform deeper contextual reasoning beyond keyword matching.

4.4 Vocabulary Distribution (Distinct-n &Zipf's Law)

The study analyzes the ICFD-31k's vocabulary. Specifically, the Linguistic Diversity (Distinct-n) scores described in Table 6 (e.g., Distinct-1: 0.0653, Distinct-2: 0.3371, Distinct-3: 0.6113) indicate a trade-off between lexical diversity and repeated domain-specific terminology. In this regard, the controlled temperature (T=0.7) may contribute to the frequent occurrence of terms such as 'OTP', 'bank', and 'CVV'. Furthermore, the vocabulary distribution is further examined using Zipf's Law [Zipf, 1949], as presented in Figure 4. Accordingly, the MSE of 1.81 and the curve's deviation from the ideal line suggest a vocabulary distribution shaped by recurring fraud-domain terminology. Collectively, these findings should be interpreted alongside the structural-diversity and human-evaluation results.

5 Experimental Evaluation

The performance of ICFD-31k is evaluated through benchmarking with a strong baseline. In particular, the experimental design considers two questions: 1) How effectively does the model classify in-domain conversations? 2) How well does the model generalize to novel scam types? The results reveal that the fine-tuned classifier obtains near-perfect in-domain performance, while performance decreases on unseen scam types.

5.1 Experimental Setup

The developed models are evaluated on three tasks to test both real-world deployment and offline analysis capabilities.

Task 1: Static Classification. Given a complete conversation transcript, predict whether the interaction is fraudulent. This represents the best-case scenario where all information is available for batch processing.

Task 2: Streaming Classification. Given conversation chunks at 3-second intervals, predict fraud status at each timestamp, simulating real-time detection where decisions must be made with incomplete information.

Task 3: Cross-Domain Generalization. Evaluation on completely unseen fraud types to test the model's ability to adapt to novel scam typologies that emerge after training.

ICFD-31k is partitioned as follows: Training set contains 21,000 conversations (70%) from the main 10 fraud umbrellas; Validation set contains 4,500 conversations (15%); Indomain test set contains 4,500 conversations (15%) from the same 10 umbrellas; Cross-domain test set contains 1,000 conversations from 5 completely unseen fraud types (cryptocurrency scams, romance scams, investment fraud, charity scams, and tax impersonation). The proposed work ensures that for streaming experiments, all chunks inherit the split of their source conversation. Consequently, it yields 1,111,071 temporal chunks while preventing chunks from the same session from appearing in both training and test sets. Furthermore, the obtained F1-score, along with Precision and Recall are reported in Table 7. Additionally, Table 8 reports the measurement report for early detection rate (percentage of fraud cases detected within the first 30 seconds) and mean detection latency (average time to first fraud prediction) for streaming classification.

5.2 Baseline Models

The proposed work establishes three benchmarks: a compact zero-shot LLM reference (Gemma 1.1 2B), Fine-Tuned RoBERTa (static), and streaming RoBERTa. Specifically, Gemma is prompted with the full transcript and policy rules to measure prompt-only performance under an on-device model scale. Subsequently, RoBERTa-base [Liu et al., 2019] is finetuned on full transcripts for static classification and on cumulative 3-second chunks for streaming classification. In this regard, the RoBERTa models use a 512-token limit, a binary classification head, 5 epochs with early stopping, learning rate 2e-5, and effective batch size 16, thereby prioritizing a lightweight baseline for future low-resource deployment studies over architectural novelty.

5.3 Main Results

Table 7 presents the core performance comparison across all baseline models and evaluation scenarios. Specifically, Gemma 2B achieves 35.83% F1 with high precision but low recall (22.16%), showing that this compact zero-shot baseline misses most fraudulent cases under the prompt-only setting and is unsuitable where false negatives carry high societal cost. In contrast, the fine-tuned RoBERTa achieves 99.40% F1 with near-perfect precision and recall in-domain; five-fold cross-validation yields 99.51% ± 0.07% F1. Accordingly, we interpret the in-domain result as an upper-bound synthetic benchmark score, not evidence that real-world fraud detection is solved. Furthermore, the streaming RoBERTa achieves 95.59% F1, a 3.81% decrease, quantifying the cost of incomplete information. Notably, the model detects 99.82% of fraud cases with an average latency of 32.2 seconds. In addition, Table 8 shows performance varies across timeline: early chunks (0-30s) achieve 95.78% F1, while very late chunks (120s+) reach 97.12% F1 as context accumulates.

5.4 Analysis of the Generalization Gap

The 99.40% F1-score on in-domain data appears near-perfect, but it should be read together with the cross-domain stress test. Specifically, when the same RoBERTa model is evaluated on the cross-domain test set containing 1,000 conversations from 5 completely unseen fraud types (cryptocurrency scams, romance scams, investment fraud, charity scams, and tax impersonation), the performance degrades by 6.43% from 99.40% to 92.97% F1 as discussed in Table 9. Accordingly, the drop suggests a genuine shift in fraud language and tactics rather than a solved artifact-matching task, thereby motivating continual benchmark extension as scams evolve. In this regard, static classifiers trained on historical data may degrade as fraud tactics evolve, consequently making historical accuracy an incomplete predictor of future utility.

5.5 Expert Validation and Stakeholder Analysis

The study employs a multidisciplinary expert panel comprising cybersecurity analysts, computational linguists, and fraud-detection practitioners. Their expertise respectively covered fraud logic and policy violations, Indian English/Hinglish code-switching, and social-engineering realism, collectively shaping scenario coverage, boundary cases, and the human validation protocol. Specifically, each expert evaluated 25 conversations (15 overlapping) across quality dimensions (Fluency, Coherence, Realism, Emotion) and fraud classification. Consequently, expert validation yielded a mean pairwise Cohen's Kappa [Cohen, 1960] of 0.534, indicating moderate inter-annotator agreement, alongside 80% consensus, realism 3.96/5, and confidence 4.29/5. Notably, disagreements were concentrated in Subtle Fraud and Ambiguous Normal cases, indicating boundary-case difficulty rather than annotation collapse. Accordingly, ICFD-31k should therefore be interpreted as a controlled benchmark for conversational fraud research rather than a replacement for real-call deployment validation. In this regard, important remaining gaps include ASR errors, noisy audio, accent and dialect variation, realistic deployment class priors, and the asymmetric chunk-sampling protocol. In order to address these gaps, future evaluation will couple the benchmark with speech-to-text front ends, simulate ASR perturbations, and run interval/sampling ablations to quantify the trade-off between earlier warnings and false alarms. Furthermore, for societal use, the intended deployment role is a human-in-the-loop warning layer, not autonomous call blocking or punitive decision-making. In particular, the stakeholder path is to work with cybercrime support teams, telecom safety groups, and digital-literacy organizations to collect privacy-preserving feedback on emerging scam patterns, warning language, and intervention timing. Moreover, since no comparable Indian bilingual fraud benchmark currently exists, the released cross-domain split provides an initial reproducible stress test until independent real-world evaluation sets become available. Collectively, the feedback loop can keep the benchmark current while preserving the restricted defensive-use license and auditability requirements.

6 Conclusion

The paper tackles a major bottleneck in conversational fraud detection research - the lack of large-scale, public, contextually relevant benchmarks - by introducing ICFD-31k, a novel dataset comprising over 31,000 richly annotated bilingual (English/Hinglish) conversations across 10 prevalent Indian fraud typologies. ICFD-31k features a novel annotation schema with chunk-level labels for real-time streaming analysis and detailed slow-thinking rationales to support nextgeneration explainable and streaming-capable AI systems. The paper also presents a comprehensive benchmark with hierarchical baselines to validate the ICFD-31k's quality. The obtained results show that fine-tuned classifiers achieve nearperfect in-domain performance (99.40% F1) in static settings. However, performance degrades substantially on novel, unseen threats, exposing a critical generalization gap underscoring the limitations of relying solely on historical accuracy and highlighting the need for more adaptable, robust models in evolving fraud landscapes.

Ethical Statement

ICFD-31k is released for defensive research only under a restricted license to reduce dual-use risk. All names, organizations, account details, phone numbers, and other PII are synthetic and do not correspond to real victims or real call recordings. Expert annotators participated with informed consent and were briefed on the defensive purpose of the study. The dataset is intended to support fraud prevention, safety research, and auditable countermeasures; offensive use, scamscript generation, or malicious social-engineering training is prohibited.

Acknowledgments

The authors acknowledge the support of the Ministry of Education, Government of India, and the Department of Information Technology, Dr. B. R. Ambedkar National Institute of Technology Jalandhar, Punjab, India for academic and institutional facilities.