Proceedings

ContextAware: A Multi-Agent Framework for Detecting Harmful Image-Based Comments on Social Media

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. AI and Social Good. Pages 9927-9935.

https://doi.org/10.24963/ijcai.2025/1103

Abstract

Detecting hidden stigmatization in social media poses significant challenges due to semantic misalignments between textual and visual modalities, as well as the subtlety of implicit stigmatization. Traditional approaches often fail to capture these complexities in real-world, multimodal content. To address this gap, we introduce ContextAware, an agent-based framework that leverages specialized modules to collaboratively process and analyze images, textual context, and social interactions. Our approach begins by clustering image embeddings to identify recurring content, activating high-likes agents for deeper analysis of images receiving substantial user engagement, while comprehensive agents handle lower-engagement images. By integrating case-based learning, textual sentiment, and vision-language models (VLMs), ContextAware refines its detection of harmful content. We evaluate ContextAware on a self-collected Douyin dataset focused on interracial relationships, comprising 871 short videos and 885,502 comments—of which a notable portion are image-based. Experimental results show that ContextAware not only outperforms state-of-the-art methods in accuracy and F1 score but also effectively detects implicit stigmatization within the highly contextual environment of social media. Our findings underscore the importance of agent-based architectures and multimodal alignment in capturing nuanced, culturally specific forms of harmful content.

  1. [Achiam et al., 2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Alt- man, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. [Bai et al., 2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforce- ment learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
  3. [Cao et al., 2023] Rui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, and Jing Jiang. Prompting for multimodal hate- ful meme classification. arXiv preprint arXiv:2302.04156, 2023.
  4. [Cao et al., 2024] Rui Cao, Roy Ka-Wei Lee, and Jing Jiang. Modularized networks for few-shot hateful meme detec- tion. In Proceedings of the ACM on Web Conference 2024, pages 4575-4584, 2024.
  5. [Cheng et al., 2024] Yuheng Cheng, Ceyao Zhang, Zheng- wen Zhang, Xiangrui Meng, Sirui Hong, Wenhao Li, Zi- hao Wang, Zekai Wang, Feng Yin, Junhua Zhao, et al. Exploring large language model based intelligent agents: Definitions, methods, and prospects. arXiv preprint arXiv:2401.03428, 2024.
  6. [Cheong et al., 2024] Inyoung Cheong, Aylin Caliskan, and Tadayoshi Kohno. Safeguarding human values: rethinking us law for generative ai's societal impacts. AI and Ethics, pages 1-27, 2024.
  7. [Dong et al., 2022] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Shuyang Gu, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Clip itself is a strong fine-tuner: Achieving 85.7% and 88.0% top-1 ac- curacy with vit-b and vit-l on imagenet. arXiv preprint arXiv:2212.06138, 2022.
  8. [Farías et al., 2016] Delia Irazú Hernańdez Farías, Viviana Patti, and Paolo Rosso. Irony detection in twitter: The role of affective content. ACM Transactions on Internet Technology (TOIT), 16(3):1-24, 2016.
  9. [Gallegos et al., 2024] Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Der- noncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, pages 1-79, 2024.
  10. [Hong et al., 2023] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent col- laborative framework. arXiv preprint arXiv:2308.00352, 2023.
  11. [Hou et al., 2024] Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. Large language models are zero-shot rankers for recommender systems. In European Confer- ence on Information Retrieval, pages 364-381. Springer, 2024.
  12. [Hu et al., 2022] Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. Prompt- cap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.09699, 2022.
  13. [Ji et al., 2024] Junhui Ji, Xuanrui Lin, and Usman Naseem. Capalign: Improving cross modal alignment via informa- tive captioning for harmful meme detection. In Proceed- ings of the ACM on Web Conference 2024, pages 4585- 4594, 2024.
  14. [Kiela et al., 2019] Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXiv:1909.02950, 2019.
  15. [Li et al., 2019] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019.
  16. [Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
  17. [Li et al., 2024] Lingyao Li, Lizhou Fan, Shubham Atreja, and Libby Hemphill. "hot" chatgpt: The promise of chat- gpt in detecting and discriminating hateful, offensive, and toxic comments on social media. ACM Transactions on the Web, 18(2):1-36, 2024.
  18. [Liao et al., 2025] Junxiang Liao, Zheng Wei, Zeyu Yang, Xian Xu, Pan Hui, Changyang He, and Muzhi Zhou. "even when success seems impossible, i keep streaming": How do chinese elderly streamers interact with platform algo- rithmic (in) visibility. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1-15, 2025.
  19. [Meng et al., 2024] Zixiang Meng, Qiang Gao, Di Guo, Yunlong Li, Bobo Li, Hao Fei, Shengqiong Wu, Fei Li, Chong Teng, and Donghong Ji. Mmlscu: A dataset for multi-modal multi-domain live streaming comment under- standing. In Proceedings of the ACM on Web Conference 2024, pages 4395-4406, 2024.
  20. [Mossie and Wang, 2020] Zewdie Mossie and Jenq-Haur Wang. Vulnerable community identification using hate speech detection on social media. Information Processing & Management, 57(3):102087, 2020.
  21. [Pourpanah et al., 2022] Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, and QM Jonathan Wu. A review of gen- eralized zero-shot learning methods. IEEE transactions on pattern analysis and machine intelligence, 45(4):4051- 4070, 2022.
  22. [Pramanick et al., 2021] Shraman Pramanick, Shivam Sharma, Dimitar Dimitrov, Md Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. Momenta: A multi- modal framework for detecting harmful memes and their targets. arXiv preprint arXiv:2109.05184, 2021.
  23. [Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
  24. [Rao et al., 2014] Yanghui Rao, Qing Li, Xudong Mao, and Liu Wenyin. Sentiment topic models for social emotion mining. Information Sciences, 266:90-100, 2014.
  25. [Schwartz et al., 2022] Reva Schwartz, Reva Schwartz, Apostol Vassilev, Kristen Greene, Lori Perine, Andrew Burt, and Patrick Hall. Towards a standard for identifying and managing bias in artificial intelligence, volume 3. US Department of Commerce, National Institute of Standards and Technology, 2022.
  26. [Singgalen, 2024] Yerik Afrianto Singgalen. Analyzing an interest in gpt 4o through sentiment analysis using crisp- dm. Journal of Information Systems and Informatics, 6(2):882-898, 2024.
  27. [Talebirad and Nadiri, 2023] Yashar Talebirad and Amirhos- sein Nadiri. Multi-agent collaboration: Harnessing the power of intelligent llm agents. arXiv preprint arXiv:2306.03314, 2023.
  28. [Wang and Lee, 2024] Han Wang and Roy Ka-Wei Lee. Memecraft: Contextual and stance-driven multimodal meme generation. In Proceedings of the ACM on Web Conference 2024, pages 4642-4652, 2024.
  29. [Wang et al., 2024] Jingwei Wang, Ziyue Zhu, Chunxiao Liu, Rong Li, and Xin Wu. Llm-enhanced multimodal detection of fake news. PloS one, 19(10):e0312240, 2024.
  30. [Wei et al., 2024a] Zheng Wei, Yixuan Xie, Danyun Xiao, Simin Zhang, Pan Hui, and Muzhi Zhou. Social media discourses on interracial intimacy: Tracking racism and sexism through chinese geo-located social media data. In Proceedings of the ACM on Web Conference 2024, pages 2337-2346, 2024.
  31. [Wei et al., 2024b] Zheng Wei, Xian Xu, and Pan Hui. Dig- ital democracy at crossroads: A meta-analysis of web and ai influence on global elections. In Companion Proceed- ings of the ACM Web Conference 2024, pages 1126-1129, 2024.
  32. [Wu et al., 2023] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: En- abling next-gen llm applications via multi-agent conversa- tion framework. arXiv preprint arXiv:2308.08155, 2023.
  33. [Yang et al., 2022] Jinyu Yang, Zhe Li, Feng Zheng, Ales Leonardis, and Jingkuan Song. Prompting for multi-modal tracking. In Proceedings of the 30th ACM international conference on multimedia, pages 3492-3500, 2022.
  34. [Yang et al., 2024] Xiaocui Yang, Wenfang Wu, Shi Feng, Ming Wang, Daling Wang, Yang Li, Qi Sun, Yifei Zhang, Xiaoming Fu, and Soujanya Poria. Mm-instructeval: Zero-shot evaluation of (multimodal) large language mod- els on multimodal reasoning tasks. arXiv preprint arXiv:2405.07229, 2024.
  35. [Yin et al., 2016] Xu-Cheng Yin, Ze-Yu Zuo, Shu Tian, and Cheng-Lin Liu. Text detection, tracking and recognition in video: a comprehensive survey. IEEE Transactions on Image Processing, 25(6):2752-2773, 2016.
  36. [Zhang et al., 2019] Shiwei Zhang, Xiuzhen Zhang, Jeffrey Chan, and Paolo Rosso. Irony detection via sentiment- based transfer learning. Information Processing & Man- agement, 56(5):1633-1644, 2019.
  37. [Zhou et al., 2024] Muzhi Zhou, Zheng Wei, and Junxiang Liao. How can the universal disclosure of provincial-level ip geolocation change the landscape of social media analy- sis. ACM SIGWEB Newsletter, 2024(Autumn):1-9, 2024.
  38. [Zhu et al., 2016] Yingying Zhu, Cong Yao, and Xiang Bai. Scene text detection and recognition: Recent advances and future trends. Frontiers of Computer Science, 10:19-36, 2016.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards