Proceedings

An LLM-based Chain-of-Response Counter-Scam System

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. AI and Social Good. Pages 7266-7274.

https://doi.org/10.24963/ijcai.2026/808

Abstract

The rapid evolution of online scams, driven by transnational networks and mass-produced social engineering scenarios, has exposed the speed limitations of conventional detection, necessitating tighter inter-agency coordination. While LLMs show promise in scam identification, their role in accelerating integrated response frameworks remains underexplored. We propose Counter-Scam, a unified LLM-based multi-agent framework that orchestrates end-to-end response from initial detection to crime investigation. The framework first proposes safe data guidelines, emphasizing non-public scam data and secure dataset construction via scam-specific NER. Developed with insights from 37 stakeholders to reduce delays and improve analytical efficiency, the system integrates CSRA (multi-agent mitigation), CSRT (nine role-aligned NLP tasks), and CSRD (a corpus of 185,300 scam cases and 38,587 knowledge entries). Experiments show that fine-tuned sLLMs surpass commercial models with over 10% in all CSRT tasks and a 0.24 F1 improvement in scam-specific NER. This proves the framework's capability for enabling rapid, collaborative mitigation of online scam.

  1. [Afane et al., 2024] Khalifa Afane, Wenqi Wei, Ying Mao, Junaid Farooq, and Juntao Chen. Next-generation phishing: How llm agents empower cyber attackers. In 2024 IEEE International Conference on Big Data (BigData), pages 2558-2567. IEEE, 2024.
  2. [Agency, 2023] Korean National Police Agency. Criminal charge classification table. https://www.police.go.kr/com ponent/file/ND_fileDownload.do?q_fileSn=156181&q _fileId=0e28f484-9ad5-4ebf-b3d2-0ce03c4eeb3c, 2023. Accessed: 2025-09-25. Link
  3. [Ai et al., 2024] Lin Ai, Tharindu Sandaruwan Kumarage, Amrita Bhattacharjee, Zizhou Liu, Zheng Hui, Michael S Davinroy, James Cook, Laura Cassani, Kirill Trapeznikov, Matthias Kirchner, et al. Defending against social engineer- ing attacks in the age of llms. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12880-12902, 2024.
  4. [Amnesty International, 2025] Amnesty International. Cam- bodia: 'i was someone else's property': slavery, human trafficking and torture in cambodia's scamming compounds. https://www.amnesty.org/en/documents/asa23/9447/2025 /en/, June 26 2025. Accessed: 2026-01-16. Link
  5. [Boussougou et al., 2024] Milandu Keith Moussavou Bous- sougou, Prince Hamandawana, and Dong-Joo Park. Korean voice phishing detection dataset with multilingual back- translation and smote augmentations, 2024. IEEE Dataport.
  6. [Cao et al., 2025] Tri Cao, Chengyu Huang, Yuexin Li, Wang Huilin, Amy He, Nay Oo, and Bryan Hooi. Phishagent: a robust multimodal agent for phishing webpage detection. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 39, pages 27869-27877, 2025.
  7. [Chiang and Lee, 2023] Cheng-Han Chiang and Hung-Yi Lee. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607-15631, 2023.
  8. Clinic, ] Data Clinic. Vera 911 consolidated dataset. https: //github.com/tsdataclinic/Vera. Accessed: 2025-09-25. Link
  9. Data, ] San Francisco Open Data. San francisco law enforce- ment dispatched calls for service. https://data.sfgov.org/P ublic-Safety/Law-Enforcement-Dispatched-Calls-for-Ser vice-Real-/gnap-fj3t. Accessed: 2025-09-25. Link
  10. [FBI, 2025] FBI. Fbi internet crime report, 2025. Accessed on 2025-10-04.
  11. [Goebel et al., 2024] Randy Goebel, Yoshinobu Kano, Mi- Young Kim, Juliano Rabelo, Ken Satoh, and Masaharu Yoshioka. Overview of benchmark datasets and methods for the legal information extraction/entailment competition (coliee) 2024. In JSAI International Symposium on Artifi- cial Intelligence, pages 109-124. Springer, 2024.
  12. [Holgersson and Gottschalk, 2008] Stefan Holgersson and Petter Gottschalk. Police officers' professional knowledge. Police Practice and Research: An International Journal, 9(5):365-377, 2008.
  13. [Hwang et al., 2022] Wonseok Hwang, Dongjun Lee, Ky- oungyeon Cho, Hanuhl Lee, and Minjoon Seo. A multi-task benchmark for korean legal language understanding and judgement prediction. Advances in Neural Information Processing Systems, 35:32537-32551, 2022.
  14. [Kim and Lim, 2022] Hee-Dou Kim and Heuiseok Lim. A named entity recognition model in criminal investigation domain using pretrained language model. Journal of the Korea Convergence Society, 13(2):13-20, 2022.
  15. [Kim et al., 2024] Heedou Kim, Dain Kim, Jiwoo Lee, Chan- woong Yoon, Donghee Choi, Mogan Gim, and Jaewoo Kang. Lapis: language model-augmented police investiga- tion system. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 4637-4644, 2024.
  16. [Kim et al., 2026] Heedou Kim, Changsik Kim, Sanghwa Shin, and Jaewoo Kang. SCRIPTMIND: Crime script inference and cognitive evaluation for LLM-based social engineering scam detection system. In Yevgen Matusevych, Gülşen Eryigit, and Nikolaos Aletras, editors, Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: In- dustry Track), pages 11-38, Rabat, Morocco, March 2026. Association for Computational Linguistics.
  17. [Koide et al., 2024] Takashi Koide, Naoki Fukushi, Hiroki Nakano, and Daiki Chiba. Chatspamdetector: Leveraging large language models for effective phishing email detec- tion. arXiv preprint arXiv:2402.18093, 2024.
  18. [Kumarage et al., 2025] Tharindu Kumarage, Cameron John- son, Jadie Adams, Lin Ai, Matthias Kirchner, Anthony Hoogs, Joshua Garland, Julia Hirschberg, Arslan Basharat, and Huan Liu. Personalized attacks of social engineering in multi-turn conversations-llm agents for simulation and detection. arXiv preprint arXiv:2503.15552, 2025.
  19. [Law&Company, 2022] Law&Company. Klaid. https://hugg ingface.co/datasets/lawcompany/KLAID, 2022. Accessed: 2025-09-25. Link
  20. [Lee and Han, 2024] Yunseung Lee and Daehee Han. Ko- rsmishing explainer: A korean-centric llm-based frame- work for smishing detection and explanation generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 642-656, 2024.
  21. [Lee et al., 2026] S. Lee, H. Kim, and H. Kim. Evaluating llms for police decision-making: A framework based on police action scenarios. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 40, pages 38817- 38825, 2026.
  22. [Loggen and Leukfeldt, 2022] Joeri Loggen and Rutger Leukfeldt. Unraveling the crime scripts of phishing net- works: an analysis of 45 court cases in the netherlands. Trends in Organized Crime, 25(2):205-225, 2022.
  23. [Luong and Ngo, 2024] Hai Thanh Luong and Hieu Minh Ngo. Understanding the nature of the transnational scam- related fraud: Challenges and solutions from vietnam's perspective. Laws, 13(6):70, 2024.
  24. [Ma et al., 2025] Zhiming Ma, Peidong Wang, Minhua Huang, Jinpeng Wang, Kai Wu, Xiangzhao Lv, Yachun Pang, Yin Yang, Wenjie Tang, and Yuchen Kang. Teleantifraud-28k: An audio-text slow-thinking dataset for telecom fraud detection. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 5853-5862, 2025.
  25. [Mitchell, 2025] Ottilie Mitchell. Nearly 60 south koreans repatriated from cambodia over alleged scams, October 2025.
  26. [of Korean Language, 2024] National Institute of Ko- rean Language. Nikl korean dialogue corpus(transcription) 2023(v.1.0). https://kli.korean.go.kr/corpus, 2024. Accessed: 2025-09-25. Link
  27. [Permana and Jamaludin, 2023] Flea Akbar Permana and Ah- mad Jamaludin. Personal data vulnerability in the digital era: Study of modus operandi and mechanisms to pre- vent phishing crimes. Jurnal Al-Hakim: Jurnal Ilmiah Mahasiswa, Studi Syariah, Hukum dan Filantropi, pages 201-216, 2023.
  28. [Roberts, 2012] Paul Roberts. Law and criminal investiga- tion. In Handbook of criminal investigation, pages 92-145.
  29. Willan, 2012.
  30. [Shen et al., 2025] Zitong Shen, Sineng Yan, Youqian Zhang, Xiapu Luo, Grace Ngai, and Eugene Yujun Fu. " it warned me just at the right moment": Exploring llm-based real-time detection of phone scams. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1-7, 2025.
  31. [Stainton et al., 2025] Iain Stainton, Robert Ewin, and Tony Blockley. Criminal investigation. Routledge, 2025.
  32. [Tanbhir et al., 2024] Gazi Tanbhir, Md Farhan Shahriyar, Khandker Shahed, Abdullah Md Raihan Chy, and Md Al Adnan. Hybrid machine learning model for de- tecting bangla smishing text using bert and character-level cnn. In 2024 13th International Conference on Electrical and Computer Engineering (ICECE), pages 57-62. IEEE, 2024.
  33. [Timko and Rahman, 2024] Daniel Timko and Muham- mad Lutfor Rahman. Smishing dataset i: Phishing sms dataset from smishtank. com. In Proceedings of the Four- teenth ACM Conference on Data and Application Security and Privacy, pages 289-294, 2024.
  34. [Tundis and Mühlhäuser, 2019] Andrea Tundis and Max Mühlhäuser. The role of information and communication technology (ict) in modern criminal organizations. In Or- ganized Crime and Terrorist Networks, pages 60-77.
  35. Rout- ledge, 2019.
  36. [Westmore et al., 2022] Kathryn Westmore, Simon Miller, Jonathan Frost, and Diana Foltean. Enabling cross-sector data-sharing to better prevent and detect scams, September 2022.
  37. [Xia et al., 2024] Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caim- ing Xiong. Fofo: A benchmark to evaluate llms' format- following capability. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 680-699, 2024.
  38. [Xue et al., 2025] Yinuo Xue, Eric Spero, Yun Sing Koh, and Giovanni Russello. Multiphishguard: An llm-based multi- agent system for phishing email detection. arXiv preprint arXiv:2505.23803, 2025.
  39. [Yosiandra and Sakariah, 2024] Syakira Yuniar Yosiandra and Dewi Saraswati Sakariah. Unveiling the romance scam scheme: Psychological manipulation and its impact on vic- tims. Humanika, 31(2):185-199, 2024.
  40. [Zheng et al., 2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023.
  41. [Zhong et al., 2020] Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. How does nlp benefit legal system: A summary of research on legal judgment prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5218-5230, 2020.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards