Proceedings

Detect, Attend and Extract: Keyword Guided Target Speaker Extraction

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 5784-5792.

https://doi.org/10.24963/ijcai.2026/644

Abstract

Target speaker extraction (TSE) aims to extract the speech of a target speaker from mixtures containing multiple competing speakers. Conventional TSE systems predominantly rely on speaker cues, such as pre-enrolled speech, to identify and isolate the target speaker. However, in many practical scenarios, clean enrollment utterances are unavailable, limiting the applicability of existing approaches. In this work, we propose DAE-TSE, a keyword-guided TSE framework that specifies the target speaker through distinct keywords they utter. By leveraging keywords (i.e., partial transcriptions) as cues, our approach provides a flexible and practical alternative to enrollment-based TSE. DAE-TSE follows the Detect-Attend-Extract (DAE) paradigm: it first detects the presence of the given keywords, then attends to the corresponding speaker based on the keyword content, and finally extracts the target speech. Experimental results demonstrate that DAE-TSE outperforms standard TSE systems that rely on clean enrollment speech. To the best of our knowledge, this is the first study to utilize partial transcription as a cue for specifying the target speaker in TSE, offering a flexible and practical solution for real-world scenarios. Our code (https://github.com/GnafiY/DAE-TSE) and demo page (https://gnafiy.github.io/DAE-TSE_demo) are now publicly available.

  1. [Borsdorf et al., 2021] Marvin Borsdorf, Haizhou Li, and Tanja Schultz. Target Language Extraction at Multilingual Cocktail Parties. In 2021 IEEE Automatic Speech Recog- nition and Understanding Workshop (ASRU), pages 717- 724, 2021.
  2. [Borsdorf et al., 2022] Marvin Borsdorf, Kevin Scheck, Haizhou Li, and Tanja Schultz. Experts Versus All- Rounders: Target Language Extraction for Multiple Target Languages. In Proc. IEEE ICASSP, pages 846-850, 2022.
  3. [Chen et al., 2022] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505-1518, 2022.
  4. [Chung et al., 2018] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. VoxCeleb2: Deep Speaker Recogni- tion. In Proc. ISCA Interspeech, pages 1086-1090, 2018.
  5. [Cosentino et al., 2020] Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vin- cent. LibriMix: An Open-Source Dataset for Generaliz- able Speech Separation. arXiv preprint arXiv:2005.11262, 2020.
  6. [Desplanques et al., 2020] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. ECAPA-TDNN: Em- phasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. In Proc. ISCA Interspeech, pages 3830-3834, 2020.
  7. [Dong et al., 2023] Hao-Wen Dong, Naoya Takahashi, Yuki Mitsufuji, Julian J. McAuley, and Taylor Berg- Kirkpatrick. CLIPSep: Learning Text-Queried Sound Separation with Noisy Unlabeled Videos. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
  8. [Hao et al., 2026] Xiang Hao, Jibin Wu, Jianwei Yu, Chenglin Xu, and Kay Chen Tan. Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extrac- tion. IEEE Transactions on Cognitive and Developmental Systems, 18(2):361-372, 2026.
  9. [He et al., 2024] Shulin He, Jinjiang Liu, Hao Li, Yang Yang, Fei Chen, and Xueliang Zhang. 3S-TSE: Efficient Three-Stage Target Speaker Extraction for Real-Time and Low-Resource Applications. In Proc. IEEE ICASSP, pages 421-425, 2024.
  10. [Heo et al., 2024] Woon-Haeng Heo, Joongyu Maeng, Yoseb Kang, and Namhyun Cho. Centroid Estimation with Transformer-Based Speaker Embedder for Robust Target Speaker Extraction. In Proc. ISCA Interspeech, pages 4333-4337, 2024.
  11. [Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Con- ference Track Proceedings, 2015.
  12. [Kuchaiev et al., 2019] Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Beliaev, Vitaly Lavrukhin, Jack Cook, et al. NeMo: A Toolkit for Building AI Applications Using Neural Modules. arXiv preprint arXiv:1909.09577, 2019.
  13. [Lee et al., 2018] Younggun Lee, Suwon Shon, and Taesu Kim. Learning Pronunciation From a Foreign Lan- guage in Speech Synthesis Networks. arXiv preprint arXiv:1811.09364, 2018.
  14. [Li et al., 2024] Junjie Li, Ruijie Tao, Zexu Pan, Meng Ge, Shuai Wang, and Haizhou Li. Audio-Visual Active Speaker Extraction for Sparsely Overlapped Multi-Talker Speech. In Proc. IEEE ICASSP, pages 10666-10670, 2024.
  15. [Lin et al., 2023] Jiuxin Lin, Xinyu Cai, Heinrich Dinkel, Jun Chen, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Zhiyong Wu, Yujun Wang, and Helen Meng. AV- Sepformer: Cross-Attention Sepformer for Audio-Visual Target Speaker Extraction. In Proc. IEEE ICASSP, pages 1-5, 2023.
  16. [Liu et al., 2022] Xubo Liu, Haohe Liu, Qiuqiang Kong, Xinhao Mei, Jinzheng Zhao, Qiushi Huang, Mark D. Plumbley, and Wenwu Wang. Separate What You De- scribe: Language-Queried Audio Source Separation. In Proc. ISCA Interspeech, pages 1801-1805, 2022.
  17. [Liu et al., 2024] Yun Liu, Xuechen Liu, Xiaoxiao Miao, and Junichi Yamagishi. Target Speaker Extraction with Curriculum Learning. In Proc. ISCA Interspeech, pages 4348-4352, 2024.
  18. [Luo and Mesgarani, 2019] Yi Luo and Nima Mesgarani. Conv-TasNet: Surpassing Ideal Time-Frequency Magni- tude Masking for Speech Separation. IEEE/ACM Trans. ASLP., pages 1256-1266, 2019.
  19. [Luo and Yu, 2023] Yi Luo and Jianwei Yu. Music Source Separation With Band-Split RNN. IEEE/ACM Trans. ASLP., 31:1893-1901, 2023.
  20. [Ma et al., 2024] Hao Ma, Zhiyuan Peng, Xu Li, Mingjie Shao, Xixin Wu, and Ju Liu. CLAPSep: Leveraging Contrastive Pre-Trained Model for Multi-Modal Query- Conditioned Target Sound Extraction. IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, 2024.
  21. [Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing Data Using t-SNE. Journal of machine learning research, 9(Nov):2579-2605, 2008.
  22. [Panayotov and others, 2015] Vassil Panayotov et al. Lib- riSpeech: An ASR Corpus Based on Public Domain Audio Books. In Proc. IEEE ICASSP, pages 5206-5210, 2015.
  23. [Pandey et al., 2024] Ashutosh Pandey, Sanha Lee, Juan Az- carreta, Daniel Wong, and Buye Xu. All Neural Low- latency Directional Speech Extraction. In Proc. ISCA In- terspeech, pages 4328-4332, 2024.
  24. [Peng et al., 2024] Junyi Peng, Marc Delcroix, Tsub- asa Ochiai, Oldřich Plchot, Shoko Araki, and Jan Černocký. Target Speech Extraction with Pre-Trained Self-Supervised Learning Models. In Proc. IEEE ICASSP, pages 10421-10425, 2024.
  25. [Shi et al., 2025] Bowen Shi, Andros Tjandra, John Hoff- man, Helin Wang, Yi-Chiao Wu, Luya Gao, Julius Richter, Matt Le, Apoorv Vyas, Sanyuan Chen, et al. SAM Audio: Segment Anything in Audio. arXiv preprint arXiv:2512.18099, 2025.
  26. [Sun et al., 2024] Tianchi Sun, Tong Lei, Xu Zhang, Yuxi- ang Hu, Changbao Zhu, and Jing Lu. A Lightweight Hy- brid Multi-Channel Speech Extraction System with Direc- tional Voice Activity Detection. In Proc. IEEE ICASSP, pages 1486-1490, 2024.
  27. [Tao et al., 2024] Ruijie Tao, Xinyuan Qian, Yidi Jiang, Jun- jie Li, Jiadong Wang, and Haizhou Li. Audio-Visual Tar- get Speaker Extraction with Reverse Selective Auditory Attention. arXiv preprint arXiv:2404.18501, 2024.
  28. [Wang et al., 2023] Hongji Wang, Chengdong Liang, Shuai Wang, Zhengyang Chen, Binbin Zhang, Xu Xiang, Yanlei Deng, and Yanmin Qian. Wespeaker: A Research and Pro- duction Oriented Speaker Embedding Learning Toolkit. In Proc. IEEE ICASSP, pages 1-5, 2023.
  29. [Wang et al., 2024a] Shuai Wang, Ke Zhang, Shaoxiong Lin, Junjie Li, Xuefei Wang, Meng Ge, Jianwei Yu, Yanmin Qian, and Haizhou Li. WeSep: A Scalable and Flexible Toolkit Towards Generalizable Target Speaker Extraction. In Proc. ISCA Interspeech, 2024.
  30. [Wang et al., 2024b] Yichi Wang, Jie Zhang, Shihao Chen, Weitai Zhang, Zhongyi Ye, Xinyuan Zhou, and Lirong Dai. A Study of Multichannel Spatiotemporal Features and Knowledge Distillation on Robust Target Speaker Ex- traction. In Proc. IEEE ICASSP, pages 431-435, 2024.
  31. [Wu et al., 2024] Tianci Wu, Shulin He, Jiahui Pan, Haifeng Huang, Zhijian Mo, and Xueliang Zhang. Unified Audio Visual Cues for Target Speaker Extraction. In Proc. ISCA Interspeech, pages 4343-4347, 2024.
  32. [Yang et al., 2023] Lei Yang, Wei Liu, Lufen Tan, Jaemo Yang, and Han-Gil Moon. Target Speaker Extraction with Ultra-Short Reference Speech by VE-VE Framework. In Proc. IEEE ICASSP, pages 1-5, 2023.
  33. [Yıldırım et al., 2025] Mehmet Sinan Yıldırım, Ruijie Tao, Wupeng Wang, Junyi Ao, and Haizhou Li. Leveraging Language Information for Target Language Extraction. In 2025 Asia Pacific Signal and Information Processing As- sociation Annual Summit and Conference (APSIPA ASC), pages 837-842, 2025.
  34. [Yu et al., 2023] Jianwei Yu, Hangting Chen, Yi Luo, Rongzhi Gu, Weihua Li, and Chao Weng. TSpeech-AI System Description to the 5th Deep Noise Suppression (DNS) Challenge. In Proc. IEEE ICASSP, pages 1-2, 2023.
  35. [Zhang et al., 2024] Yiru Zhang, Linyu Yao, and Qun Yang. OR-TSE: An Overlap-Robust Speaker Encoder for Target Speech Extraction. In Proc. ISCA Interspeech, pages 587- 591, 2024.
  36. [Zhang et al., 2025] Ke Zhang, Junjie Li, Shuai Wang, Yangjie Wei, Yi Wang, Yannan Wang, and Haizhou Li. Multi-level Speaker Representation for Target Speaker Ex- traction. In Proc. IEEE ICASSP, pages 1-5, 2025.
  37. [Zhao et al., 2024] Shengkui Zhao, Yukun Ma, Chongjia Ni, Chong Zhang, Hao Wang, Trung Hieu Nguyen, Kun Zhou, Jia Qi Yip, Dianwen Ng, and Bin Ma. MossFormer2: Combining Transformer and RNN-Free Recurrent Net- work for Enhanced Time-Domain Monaural Speech Sepa- ration. In Proc. IEEE ICASSP, pages 10356-10360, 2024.
  38. [Zmolikova et al., 2023] Katerina Zmolikova, Marc Del- croix, Tsubasa Ochiai, Keisuke Kinoshita, Jan Černocký, and Dong Yu. Neural Target Speech Extraction: An Overview. IEEE Signal Processing Magazine, pages 8- 29, 2023.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards