PCToolkit: A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models

1. [Bai et al., 2024] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
2. [Dong et al., 2024] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning, 2024. [Ghalandari et al., 2022]
3. Demian Ghalandari, Chris Hokamp, and Georgiana Ifrim. Efficient unsuper- vised sentence compression by fine-tuning transformers with reinforcement learning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1267-1280, Dublin, Ireland, May 2022. Association for Computational Linguistics.
4. [Grosse et al., 2023] Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hub- inger, Kamil Lukoit, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influ- ence functions, 2023.
5. [Jiang et al., 2023] Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language mod- els. In Conference on Empirical Methods in Natural Lan- guage Processing, 2023.
6. [Jiang et al., 2024] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, ed- itors, Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1658-1677, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
7. [Laban et al., 2021] Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A. Hearst. Keep it simple: Unsuper- vised simplification of multi-paragraph text. In Chengqing
8. Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 6365-6378, Online, August 2021. Association for Computational Linguistics.
9. [Lewis et al., 2020] Patrick Lewis, Ethan Perez, Aleksan- dra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th International Con- ference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020.
10. [Curran Associates Inc. Li et al., 2023] Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance inference efficiency of large language models. In Conference on Em- pirical Methods in Natural Language Processing, 2023.
11. [Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Annual Meeting of the Asso- ciation for Computational Linguistics, 2004.
12. [Pan et al., 2024] Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Vic- tor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. LLMLingua-2: Data distil- lation for efficient and faithful task-agnostic prompt com- pression. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar, editors, Findings of the Association for Computa- tional Linguistics ACL 2024, pages 963-981, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics.
13. [Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for auto- matic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Penn- sylvania, USA, July 2002. Association for Computational Linguistics.
14. [Wang et al., 2024] Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Ar- maghan Eshaghi. Beyond the limits: A survey of tech- niques to extend the context length in large language mod- els. In Kate Larson, editor, Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pages 8299-8307. International Joint Confer- ences on Artificial Intelligence Organization, 8 2024. Sur- vey Track. IJCAI
15. [Wei et al., 2024] Jason Wei, Xuezhi Wang, Dale Schuur- mans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompt- ing elicits reasoning in large language models. In Proceed- ings of the 36th International Conference on Neural In- formation Processing Systems, NIPS '22, Red Hook, NY, USA, 2024.
16. [Curran Associates Inc. Yang et al., 2024] Haoran Yang, Yumeng Zhang, Jiaqi Xu, Hongyuan Lu, Pheng-Ann Heng, and Wai Lam. Un- veiling the generalization power of fine-tuned large lan- guage models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 884-899, Mexico City, Mexico, June 2024. Association for Computational Lin- guistics.
17. [Zhang* et al., 2020] Tianyi Zhang*, Varsha Kishore*, Fe- lix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Con- ference on Learning Representations, 2020.

References were extracted automatically from the PDF and may contain errors.

Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.

1 Introduction

Large Language Models (LLMs) have demonstrated remarkable generalization capabilities [Grosse et al., 2023; Yang et al., 2024], allowing them to adapt to a wide range of tasks through prompt engineering techniques such as CoT [Wei et al., 2024], ICL [Dong et al., 2024], and RAG [Lewis et al., 2020] without necessitating fine-tuning. However, this advantage comes with an obvious drawback: increasing the length of prompts to encompass the necessary information, which subsequently escalates computational overhead [Wang et al., 2024]. Also, for online models such as ChatGPT and Claude, lengthy prompts inflate the economic cost associated with API calls.

To address this issue, prompt compression is the most straightforward strategy. As illustrated in Figure 1, it aims to reduce the length of prompts while retaining the essential information. However, the deployment of prompt compression methods varies between different approaches. There is not yet a general toolkit that can invoke multiple types of compressors. Thus, with the aim of providing plug-andplay services, easily customizable interfaces, and supporting common datasets and metrics, we propose Prompt Compression Toolkit (PCToolkit), a unified plug-and-play toolkit for Prompt Compression of LLMs, making prompt compression methods accessible and portable to a wider audience. Our plug-and-play design enables users to deploy and use the toolkit without any further model training. Meanwhile, users are also able to plug in their custom-trained models in PCToolkit.

∗Corresponding author.

Key features of PCToolkit include:

• (i) Reproducible methods. PCToolkit offers a unified interface for six different compressors: KiS [Laban et al., 2021], SCRL [Ghalandari et al., 2022], Selective Context [Li et al., 2023], LLMLingua [Jiang et al., 2023], LongLLMLingua [Jiang et al., 2024], and LLMLingua-2 [Pan et al., 2024].
• (ii) Modular design. Featuring a modular structure that simplifies the transition between different methods, datasets, and metrics, PCToolkit is organized into four distinct modules: Compressors, Datasets, Metrics and Runner.

(iii) User-friendly interface. Facilitating portability and ease of adaptation to different environments, the interfaces within PCToolkit are designed to be easily customizable.

2 PCToolkit

2.1 Modular Design

As shown in Figure 2, PCToolkit is designed with a modular architecture, consisting of Compressors, Datasets, Metrics and Runner.

Compressors. pctoolkit.compressors module encompasses six compression methods tailored for prompt optimization. All compressors can be invoked through a unified interface shown in Section 2.2. Figure 3 divides them into three categories: (1) RL-based: KiS, SCRL, (2) LLM scoringbased: Selective Context, and (3) LLM annotation-based: LLMLingua, LongLLMLingua, LLMLingua-2. Among them, KiS does not typically trim words but uses an autoregressive approach to regenerate a shorter context.

Datasets. pctoolkit.datasetsmodule includes a diverse collection of datasets, each curated to cover a wide array of natural language tasks. As shown in Table 1, the datasets are systematically organized by task requirements. For instance, reconstruction tasks leverage domain-specific corpora like BBC and Arxiv, while complex reasoning tasks utilize mathematical benchmarks like GSM8K. From tasks like reconstruction, summarization, question answering, to more specialized domains such as code completion and lies recognition, PCToolkit offers a comprehensive testing ground for assessing prompt compression techniques.

Table 1: Task-dataset mapping in PCToolkit. The table illustrates a structured breakdown of supported NLP tasks and their corresponding evaluation datasets across reconstruction, reasoning, and generation paradigms.

Task	Datasets
Reconstruction	BBC, ShareGPT, Arxiv, GSM8K
Mathematical Problems	GSM8K, BBH
Boolean Expressions	BBH
Multiple Choice	BBH
Lie Detection	BBH
Summarization	BBC, Arxiv, Gigaword, DUC2004, BNC, Broadcast, Google, LongBench
Question Answering	BBH, LongBench
Few-Shot Learning	LongBench
Synthetic Tasks	LongBench
Code Completion	LongBench

Metrics. pctoolkit.metrics module quantifies the performance of the compression methods across different tasks. Key metrics include accuracy, BLEU [Papineni et al., 2002], ROUGE [Lin, 2004], BERTScore [Zhang* et al., 2020], Token-F1 [Bai et al., 2024], and edit-distance. All necessary metrics can be easily organized into a list, which instructs the Runner on what to measure. As detailed in Table 2, task requirements dictate metric selection: accuracy dominates mathematical and reasoning evaluations (GSM8K, BBH), while text generation tasks (BBC, Arxiv) employ composite metrics including ROUGE and BERTScore.

Runners. pctoolkit.runners module serves as the engine that drives the evaluation process. Users can seamlessly execute experiments, compare results, and analyze the performance of different compression techniques using the Runner component.

Table 2: Dataset-metric mapping in PCToolkit. The table presents correspondences between evaluation datasets and their specialized metrics, emphasizing accuracy for reasoning tasks (e.g., BBH) and text generation metrics (e.g., ROUGE) for summarization.

Dataset	Metrics
BBH	Accuracy
Gigaword	ROUGE, Token-F1
BNC	ROUGE, Token-F1
DUC2004	ROUGE, Token-F1
Broadcast	ROUGE, Token-F1
Google	ROUGE, Token-F1
GSM8K	Accuracy, BLEU, ROUGE, BERTScore
BBC News	BLEU, ROUGE, BERTScore
Arxiv articles	BLEU, ROUGE, BERTScore
ShareGPT	BLEU, ROUGE, BERTScore
LongBench	Accuracy, BLEU, ROUGE, BERTScore, Edit-distance

2.2 Unified Interface

In PCToolkit, a unified interface for invoking prompt compression methods is provided. In the following example, we show how to simply invoke the compressing methods within few lines.

from pctoolkit.compressors import PromptCompressor compressor = PromptCompressor( type='SCCompressor', device='cuda') prompt = 'This is a prompt.' ratio = 0.5 result = compressor. compressgo(prompt, ratio)

For simple compression task, one compressor is selected. Following the example given above, the original prompt is input to the compressor, and the compressor outputs the compressed prompt. For datasets evaluation, one datasets and multiple metrics are selected, along with the compressor chosen, these three parts are deployed in Runner. The Runner will provide the evaluation results according to the metrics list. The following example shows how to use PCToolkit to evaluate a dataset.

from pctoolkit.runners import run from pctoolkit.datasets import load_dataset from pctoolkit.metrics import load_metrics compressor = PromptCompressor( type='SCCompressor', device='cuda') dataset_name = 'arxiv' dataset = load_dataset(dataset_name) run(compressor=compressor, dataset=dataset, metrics=load_metrics, ratio=0.5)

Currently, the supporting dataset calls are implemented inside run. Users can also following the format in run to adapt their own datasets or metrics.

3 Conclusion

We introduced PCToolkit, an open-source project designed for prompt compression and evaluation. This toolkit provides a user-friendly and comprehensive resource, featuring six common compression methods and over ten diverse datasets that encompass a wide range of natural language tasks.

Acknowledgments

This research is supported by the National Natural Science Foundation of China (No. 62406267), the Guangzhou-HKUST(GZ) Joint Funding Program (Grant No.2025A03J3956), the Guangzhou Municipal Science and Technology Project (No. 2025A04J4070), and the Guangzhou Municipal Education Project (No. 2024312122).

Machine-readable versions of this paper, for tools and AI agents:

• Markdown — metadata, abstract, full text and references
• JSON — metadata and structured references
• BibTeX