LPDetective: Dusting the LLM Chats for Prompt Template Abusers

1 Introduction

In recent years, chatbot technology driven by large language models (LLMs) has made tremendous progress, such as ChatGPT [Ouyang et al., 2022; Wikipedia, 2022], GPT-4 [OpenAI, 2023], etc., exhibiting near-human or even superhuman capabilities in natural conversation, question answering, and content creation [Frieder et al., 2023; McGee, 2023]. These chatbots have been widely applied in customer service [Følstad and Skjuve, 2019], education [Wollny et al., 2021], healthcare [Ayanouz et al., 2020], and other fields, profoundly changing our work and lifestyles [Hussain et al., 2019; Gupta et al., 2020; Paliwal et al., 2020].

However, the open interaction interfaces of LLM chatbots also introduce new security risks. In 2023, OWASP listed Model Denial of Service (MDoS) attacks as the 4th biggest threat faced by LLM systems [OWASP, 2023]. In MDoS attacks, malicious attackers can use automated tools to generate conversation requests in bulk, rapidly consuming the computing and storage resources of LLM systems, affecting the experience of normal users, and even leading to complete service unavailability. For example, in November 2023, OpenAI's ChatGPT service suffered a targeted distributed denial of service (DDoS) attack, causing the system to be down for an extended period [CNBC, 2023]. In August of the same year, security researchers successfully constructed malicious conversation requests by reverse engineering the API interface of the new Bing Chat, confirming its risk of MDoS abuse [Integration-Automation, 2023].

∗Corresponding author

To counter LLM robot abuse attacks, academia and industry have proposed a series of detection and defense methods, mainly focusing on strategies such as adversarial training [Xu et al., 2021], access control [Gondaliya et al., 2020], and anomaly detection [Qian et al., 2023]. However, these methods struggle to cope with increasingly complex abuse scenarios, especially the problem of Chatbot interfaces being massively abused by robots. Robots quickly call APIs to generate harmful content, seriously wasting computing power, a phenomenon that has become particularly prominent after Chatbots opened their APIs [Economist, 2023; Hurler, 2023]. To address this issue, this paper innovatively proposes utilizing unsupervised learning to automatically mine robot Prompt templates from historical chat logs and construct regular expressions for real-time detection. This method can adaptively handle template syntax changes, resist template obfuscation, and is computationally efficient, overcoming the limitations of existing work.

This paper proposes LPDetective, an unsupervised LLM robot detection method based on chat log mining. This method leverages the fact that robots often reuse similar prompt templates in bulk, leading to a certain level of similarity and repetition in the syntactic and semantic structures of conversations. We design a multi-stage chat log mining framework, including message grouping, text similarity calculation, hierarchical clustering, and regular expression extraction, to automatically discover potential robot prompt templates from massive human-machine conversation records and generate corresponding detection rules. Unlike existing methods, LPDetective does not require predefined template libraries or manually labeled data and can adaptively mine new, unknown patterns.

We evaluate the performance of LPDetective on three real-world datasets: Bing Copilot, Wildchat [Zhao et al., 2024], and ChatLog [Tu et al., 2023]. The experimental results demonstrate that this method can effectively mine high-frequency robot prompt templates in multiple scenarios. For example, on 118,000 chat records in the Bing Copilot dataset, the regular expression rules generated by LPDetective achieve an average accuracy of 94.6%, recall of 93.4%, and F1 score of 94.0%, outperforming the best baseline method XLNet by 7.3%, 7.7%, and 7.5%, respectively, in the same hardware environment. Moreover, the detection latency is only 0.294 seconds, significantly lower than XLNet's 52.438 seconds.

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 introduces the overall process and key algorithms of LPDetective. Section 4 presents the experimental results. Section 5 discusses limitations and future work. Section 6 concludes the paper.

2 RelatedWork

In recent years, chatbot technology driven by large language models (LLMs) has made tremendous progress, but its rapid development has also brought numerous security risks [Van Dis et al., 2023]. Malicious attackers may exploit Chatbots to generate harmful content, such as fake information [Shu et al., 2020; Karanjai, 2022], malware [Mulgrew, 2023; VARINDIA, 2022], phishing emails [Baki et al., 2017; Giaretta and Dragoni, 2020], etc. [Point, 2023]. Mulgrew et al. [Mulgrew, 2023] demonstrated how to use ChatGPT to write an undetectable data-stealing virus. Baki et al. [Baki et al., 2017] found that utilizing natural language generation technology can automatically generate large-scale, personalized spoof emails, greatly enhancing the power of phishing attacks. Although Zellers et al. [Zellers et al., 2019] proposed a defense method against neural fake news, and Stiff et al. [Stiff and Johansson, 2022] attempted to detect fake content generated by Chatbots, there are currently no mature and effective defense measures.

Furthermore, the security and privacy of Chatbots themselves are also threatened [Chung et al., 2017]. Chung et al. [Chung et al., 2017] discussed the security risks of voice assistants like Alexa. Some Chatbots have design flaws or software vulnerabilities that can be exploited by attackers [Lei et al., 2017; Security, 2018]. Lei et al. [Lei et al., 2017] systematically analyzed the vulnerabilities of home digital voice assistants using Alexa as an example. Moreover, users' private conversation data with Chatbots may be leaked [Agomuoh, 2022; Coles, 2023]. Recently, The Economist [Economist, 2023] reported that Samsung employees misused ChatGPT, leading to the disclosure of commercial secrets. The advice provided by Chatbots also lacks security reviews and may mislead users into dangerous behaviors [Prakken, 2020; Mijwil and Aljanabi, 2023]. Prakken [Prakken, 2020] pointed out that persuasive Chatbots may induce users to make inappropriate decisions when using crowd opinion graphs.

Existing work on Chatbot abuse mainly focuses on defense strategies such as adversarial training [Xu et al., 2021], access control [Gondaliya et al., 2020], and anomaly detection [Qian et al., 2023]. Xu et al. [Xu et al., 2021] proposed a method based on adversarial training to generate safer Chatbot responses. Qian et al. [Qian et al., 2023] designed an anomaly detection method to identify out-of-bounds answers from Chatbots. However, these methods struggle to cope with increasingly complex abuse scenarios. In contrast, we focus on the problem of Chatbot interfaces being massively abused by robots. Robots quickly call APIs to generate harmful content, seriously wasting computing power [Economist, 2023; Hurler, 2023]. We innovatively propose utilizing unsupervised learning to automatically mine robot Prompt templates from historical chat logs and construct regular expressions for real-time detection. This method can adaptively handle template syntax changes, resist template obfuscation, and is computationally efficient, overcoming the limitations of existing work.

3 Methodology

3.1 ProblemDefinition

In this paper, we assume that the LLM chatbot platform to be protected provides public API interfaces similar to ChatGPT and Bing Copilot, which can be accessed without user login. The attacker has mastered the method of calling the platform's API and the parameter format, and their goal is to maximize the consumption of the LLM system's computing resources by sending a large number of malicious requests in batches, resulting in a decline in service quality or denial of service. We focus on attackers using reusable prompt templates to implement MDoS attacks, i.e., using tools to automatically generate a large number of homogeneous conversation requests. This approach can simplify the attack process and improve attack efficiency. Since the platform does not require user login, attackers can easily change IP addresses, User Agents, and other request parameters frequently to circumvent traditional rate limiting mechanisms. At the same time, the behavior of batch reuse of prompt templates also provides a unique entry point for our detection method. This paper fully utilizes this feature to automatically identify potential robot prompt templates by mining real humanmachine conversation logs, thereby achieving efficient detection of MDoS attacks.

This paper proposes an innovative method based on LPDetective for automatically detecting abnormal dialogue behavior of chatbots. Our method is based on the assumption that the prompt text generated by robots repeatedly uses the same or highly similar phrases and sentence structures across multiple sessions, which is significantly different from the pattern of human natural language. As shown in Figure 1, we designed a multi-stage workflow, including message grouping, text similarity measurement, hierarchical clustering analysis, and regular expression extraction.

3.2 Message Grouping

We first group messages according to the user's fingerprint features. Formally, given a set of messages M = m1,m2, . . . ,mn and a set of users U = u1, u2, . . . , uk, we define the fingerprint feature extraction function ϕ : U → 0, 1d , which maps each user to a d-dimensional binary vector. We divide the messages into different groups G = {G1,G2, . . . ,Gl} according to the user's fingerprint feature vector, where Gi contains all messages sent by user ui.

To encode the fingerprint feature vector into a compact representation, we use the SimHash algorithm [Sadowski and Levin, 2007] for hashing. The SimHash algorithmmaps highdimensional feature vectors to low-dimensional binary codes, and the distance between codes is positively correlated with the distance between the original feature vectors.

Given a fingerprint feature vector x ∈ 0, 1d , the i-th bit of its SimHash code c ∈ 0, 1b is calculated as follows:

where b is the number of bits in the SimHash code, and wij ∼ N(0, 1) is an element of the random projection matrix.

3.3 Text SimilarityMeasurement

After obtaining the message groups G = {G1,G2, . . . ,Gl}, we calculate the normalized Levenshtein distance [Levenshtein, 1966] for each pair of messages within each group to measure their text similarity. The normalized Levenshtein distance maps two strings to their edit distance under unit length:

where lev(·, ·) is the Levenshtein distance, representing the minimum number of edit operations required to convert one string to another.

Based on the normalized Levenshtein distance, we can define the similarity matrix S(i) ∈ [0, 1]ni×ni for messages within group i, where S(i) jk = 1 - d(m(i) j ,m(i) k ) represents the text similarity between messages m(i) j and m(i) k within group i.

3.4 Hierarchical Clustering

Based on the similarity matrix {S(1) , S(2) , . . . , S (l) } for messages within each group, we use hierarchical clustering algorithms to further group messages at a finer granularity.

Hierarchical clustering algorithms generate a clustering tree T (i) = (V (i) ,E(i) ,W (i) ) by iteratively merging the most similar clusters.

During the construction of the clustering tree, we use the Complete Linkage method [Defays, 1977] to measure the distance between clusters. Given two clusters C (i) j and C (i) k within group i, their distance is defined as:

which takes the maximum distance among all pairs of messages between the two clusters.

The hierarchical clustering algorithm proceeds as follows:

1. Initialization. Treat each message within group i as an independent cluster, forming the initial cluster set.
2. Iteration. Find the two closest clusters and merge them, while updating the clustering tree.
3. Termination. The algorithm terminates when there is only one cluster left in the cluster set.

To obtain the final clustering result, we need to cut the clustering tree. A common strategy is to set a distance threshold θ(i) , and stop merging when the weight of an edge exceeds θ(i) . The choice of threshold θ(i) affects the granularity of clustering.

To intuitively evaluate the quality of clustering results, we introduce Silhouette Coefficient [Rousseeuw, 1987] as an unsupervised evaluation metric. Given the clustering result C (i) = {C (i) 1 ,C (i) 2 , . . . ,C (i) ki } for group i, the Silhouette Coefficient of data point m(i) j is defined as:

where a(m(i) j ) represents the average distance between m(i) j and other points within the same cluster, and b(m(i) j ) represents the minimum average distance between m(i) j and points in other clusters.

We can calculate the average value of Silhouette Coefficients for all data points within group i:

The average Silhouette Coefficient ¯ s(i) reflects the overall quality of the clustering result for group i and can be used to compare the performance of different clustering algorithms or parameter settings.

Figure 2 shows the result of sampling and visualization of Bing Copilot's chat data on a workday. It can be seen from the figure that different clusters form several compact clusters in the low-dimensional space, with high semantic similarity within clusters and large semantic differences between clusters. We calculated the average Silhouette Coefficient of these 20 clusters to be 0.713, further verifying the high quality of the clustering results.

3.5 Regular Expression Extraction

From the clustering results, we need to extract regular expressions that can match most messages. We designed a difference comparison algorithm LCS-Diff based on the Longest Common Substring (LCS), which automatically generates regular expressions that match the cluster by comparing the differences between messages within the cluster. Algorithm 1 shows the complete process of regular expression extraction, where n is the size of cluster C, l is the average length of strings, k is the number of regular expression clusters.

3.6 Matching Optimization

To achieve real-time robot detection, we store the extracted regular expressions in a template database. We designed a matching algorithm based on inverted indexing and regular expression optimization to efficiently match new messages with a large number of regular expressions.

Inverted Index Design. We first build an inverted index for the keywords (ordinary strings) of each regular expression. Keywords can be generated by word segmentation algorithms. Given a new message M, we quickly find the candidate regular expression set RM through the inverted index. This step can avoid the expensive overhead of matching with

Algorithm 1 Regular rxpression extraction

Require: Cluster C = {s1, s2, . . . , sn} Ensure: Regular expression r 1: R← ∅ 2: for i ← 1 to n-1 do 3: diff ←LCS-Diff(si, si+1) 4: ri ← ε 5: for (op, c, p, l, t) in diff do 6: if t = 0 then 7: ri ← ri ◦ c 8: else 9: ri ← ri ◦ .*' 10: end if 11: end for 12: ri ← Optimize(ri, λ) 13: R←R∪ {ri} 14: end for 15: {R1,R2, . . . ,Rk} ←Cluster(R, τ) 16: r ← ε 17: for j ← 1 to k do 18: rj ←MSA(Rj).replace( ', .*') 19: r ← r ◦ rj ◦-' 20: end for 21: r ← r[: -1] 22: return r

Algorithm 2 Optimized regex matching

Require: Message M, inverted index I, regular expression set R Ensure: Matching result Rmatch 1: KM ←ExtractKeywords(M) 2: RM ← ∅ 3: for k ∈ KM do 4: RM ←RM ∪ I(k) 5: end for 6: RM ← SortByHitRate(RM) 7: RM ← SortByLength(RM) 8: Rmatch ← ∅ 9: for Ri ∈ RM do 10: if Match(M, Ri) then 11: Rmatch ← Rmatch ∪ {Ri} 12: end if 13: end for 14: return Rmatch

all regular expressions, significantly improving matching efficiency.

Regular Expression Optimization. To further accelerate the matching process, we executed a series of optimization strategies on the candidate regular expression set RM , such as sorting according to historical hit rate, prioritizing matching shorter expressions, structurally preprocessing and indexing regular expressions, etc.

After optimization, we obtained a refined and efficient regular expression matching workflow. Algorithm 2 shows the optimized matching process.

Finally, we comprehensively calculate the risk score Risk(M) of the message according to the matching results:

where wi is the risk weight of regular expression Ri. When Risk(M) ≥ 1, we can directly execute an interception operation, i.e., refuse to provide service, to quickly filter out highrisk messages and improve the system's security.

4 Experiments

4.1 Dataset

We obtained real chat log data of Bing Copilot on a certain day in March 2024 from the Bing platform for analysis in this study. We randomly sampled 118,000 messages and labeled each message as either bot or human using Bing's existing hundreds of mature detection rules. These detection rules include various types such as fake User-Agent detection, IP reputation, mouse trajectory, JavaScript fingerprinting, and JA3 fingerprinting. Ultimately, we identified 33,294 botgenerated messages out of the 118,000 messages, accounting for 28.2% of the total. During the preprocessing stage, we grouped all messages using user fingerprint features. Table 1 shows the top 10 most frequent regex templates extracted from the dataset. It can be seen that these templates cover various types, including translation, text rewriting, e-commerce promotion, etc. At the same time, most templates have an average intra-class distance (Distance) less than 0.1, indicating that the internal consistency of the corresponding message clusters is high, and the extracted regular expressions can effectively summarize these messages.

We observed some typical prompt templates, as shown in Figure 3. These two cases demonstrate an Amazon sellerbased promotional bot and a history event-based Q&A bot, respectively. It can be seen that although there are certain differences in the details within each cluster of messages, they all follow obvious structured patterns. LPDetective can automatically extract highly generalized regular expressions from these similar texts, thereby effectively identifying the corresponding types of bot messages.

We evaluated the performance of LPDetective on three datasets: Bing Copilot, WildChat [Zhao et al., 2024], and ChatLog [Tu et al., 2023]. Among them, Bing Copilot contains real conversation data from Bing AI, WildChat selects 15,783 ChatGPT interaction data, and ChatLog extracts 28,412 conversations. We manually annotated the prompt abuse in WildChat and ChatLog and found that bot-generated messages accounted for 15.8% and 14.0%, respectively. The annotation work was completed by 27 experienced engineers, all with more than 2 years of relevant work experience. Each message was independently annotated by 3 engineers to ensure accuracy and consistency. Fleiss' Kappa [Fleiss, 1971] evaluation showed high consistency among annotators, with a Kappa value of 0.93. The annotators were mainly 25-35 years old, with a bachelor's degree or above in computer-related majors and more than 2 years of relevant work experience. In this paper, these final labels are assumed to be accurate benchmark ground truth. We randomly divided each website's dataset into training set (70%), validation set (10%), and test set (20%). We ran each experiment 10 times and reported the average and standard deviation of the metrics. The standard deviation of all experiments was guaranteed to be within 0.5%. Unless otherwise specified, all experimental re-

Example 1: Amazon seller promotion bot

Message 1: When you are an amazon seller.You plan to run a cpc campaign for product:Bingo G

Message 2: When you are an amazon seller.You plan to run a cpc campaign for product:Pet U S

Message 3: When you are an amazon seller.You plan to run a cpc campaign for product:Lucky C

Extracted regex: When you are an amazon seller.You plan to run a cpc campaign for product:.*

Example 2: Verb query bot

Message 1: 10 phrasal verbs with N different from the above searched

Message 2: 17 phrasal verbs with Q different from the above searched

Message 3: 10 phrasal verbs with K different from the above searched

Extracted regex: .* phrasal verbs with .* different from the above searched*

Figure 3: Examples of typical prompt templates extracted by LPDetective.

sults reported in this section were obtained using this method. All experiments were conducted on an Ubuntu 20.04 server equipped with an Intel Xeon 8369B CPU, 96 GB memory, and an NVIDIA V100 GPU. The experimental code was implemented based on PyTorch 2.2.0.

4.2 Baseline Comparison

We implemented several baseline models, including: LSTM [Hochreiter, 1997], N-gram [Cavnar et al., 1994], FastText [Joulin et al., 2016], TextCNN [Kim, 2014], XLNet [Yang et al., 2019], DRNN [Wang, 2018], and HiAGM [Zhou et al., 2020]. All models used the Adam optimizer, and we searched for the learning rate initial value between 0.0001 and 0.1, the batch size between 16 and 128, and the number of training iterations between 10 and 1000. We selected the hyperparameter combination with the highest F1 value on the validation set as the final setting. The learning rate initial value for all models was 0.001, the batch size was 64, and the number of training iterations was 100.

Table 2 shows the performance comparison of LPDetective with baseline methods on the three datasets. Traditional machine learning methods such as N-gram have an F1 value not exceeding 0.70. Deep learning-based methods such as LSTM, FastText, TextCNN, etc. have improved F1 values, but they are still not sufficient to effectively characterize the prompt text features, and their inference latency is high. XLNet outperforms other methods in F1 value, but its inference latency is as high as tens of seconds. The F1 values of DRNN and HiAGM increase to around 0.80, but inference speed is still a bottleneck. LPDetective achieves F1 values as high as 0.940, 0.931, and 0.947 on the three datasets, respectively, with Latency controlled within 0.3 seconds. We used the Wilcoxon signed-rank test to compare the differences in F1 values between LPDetective and each baseline method at a significance level of p¡0.01, and the results showed that the superiority of LPDetective was statistically significant (p¡0.01), rather than random noise. The excellent performance of LPDetective is attributed to: 1. Automatically mining key patterns from massive real data to construct a highly structured and fine-grained prompt template database; 2. Systematic exploration in matching optimization and introduction of a risk scoring mechanism to balance precision and recall.

Table 1: Top 10 most frequent regex templates, sorted by the number of matched messages

4.3 Prompt Language Analysis

We sorted the regular expressions extracted from the dataset in descending order of length. Table 3 shows the distribution of regular expressions in various languages. It can be seen that English is the most widely used language for prompt templates, accounting for 40.84%. For languages with larger character lengths such as Chinese and Japanese, the P1 length of their regular expressions is significantly higher than other languages, reaching 142, 145, and 148, respectively. In contrast, for languages such as Indonesian and Portuguese, the P1 length of their regular expressions is relatively shorter, at 117 and 122, respectively. This indicates that bots in these regions tend to use shorter prompt templates.

4.4 Prompt Topic Analysis

We used the GPT-4 model to classify the content topics of the extracted regular expressions. We divided the regular expressions into 9 topic categories, as shown in Table 4. It can be seen that Programming is a key area of bot applications, accounting for 8.40%. This indicates that some developers are trying to leverage Bing Copilot's code generation capabilities to assist with programming or debugging tasks. The proportion of the Math Problems topic is also relatively high, reaching 5.73%, and the P1 value of the regular expression length is as high as 162. This may be due to the complexity of mathematical problems themselves, resulting in a relatively large length of the corresponding regular expressions. Additionally, we found that 54.19% of the regular expressions were difficult to classify into existing topics and were marked as the Others category. We also discovered that some regular expressions involved pornographic and gambling content, highlighting the necessity of strengthening content moderation.

4.5 Ablation Study

We conducted ablation experiments on three datasets to examine the following variants:

• w/o Grouping. Remove the message grouping step and directly cluster and extract regular expressions.
• w/o Clustering. Remove the hierarchical clustering step and directly compare messages within each group to extract regular expressions.
• w/o Optimization. Do not optimize during regular expression extraction and retain all non-wildcard parts.
• w/o Index. Do not use inverted index during matching and match with all regular expressions.

Table 5 shows the results of the ablation experiments. Removing message grouping (w/o Grouping) reduces the F1 value by 0.085, indicating that pre-clustering using user fingerprints can improve the cohesion of prompt templates. Removing hierarchical clustering (w/o Clustering) leads to a decrease of 0.124 in F1, indicating that two-stage clustering can effectively organize similar messages and facilitate the extraction of more accurate regular expressions. Removing regular expression extraction optimization (w/o Optimization) generates lengthy expressions, reduces interpretability, and increases detection latency by 3.628 seconds. Removing the inverted index (w/o Index) increases the detection latency from 0.294 seconds to 8.947 seconds.

We also analyzed the impact of key parameters: the minimum non-wildcard substring length threshold L and the clustering stop distance threshold θ on LPDetective. Table 6 shows that increasing L makes regular expressions more concise, but when L exceeds 5, overly simple expressions instead cause the F1 value to decrease, dropping to 0.820 at L = 10. This indicates that excessive simplification of regular expressions loses important pattern information and leads

Prec.: Precision, Rec.: Recall, F1: F1-score, Latency: Processing time in seconds per 1,000 messages

Table 4: Prompt template topics

to a decrease in generalization ability. θ controls the granularity of clustering, and as θ increases from 0.01 to 0.3, the F1 value generally decreases, reaching 0.762 at θ = 0.3. The larger θ is, the coarser the clustering, the greater the difference within clusters, and the worse the generalization ability of the extracted regular expressions. When θ is less than 0.05, the clustering is too fragmented, slightly improving the F1 value but significantly reducing detection efficiency. Reasonable selection of L and θ is crucial for LPDetective, and experiments show that L = 5 and θ = 0.05 is the optimal parameter combination.

5 Discussion

Although LPDetective demonstrated superior performance in our experiments, it still has some limitations. First, LPDetective mainly relies on prompt templates mined from historical conversation data, and it may be difficult to identify new, unseen abnormal conversation patterns in a timely manner. Second, LPDetective has been primarily optimized for English conversation data, and its adaptability to other languages needs further verification. Third, LPDetective needs to match a large number of regular expressions during inference. Although we have reduced the matching overhead through optimization measures such as inverted indexes, the inference latency may still fail to meet real-time requirements in scenarios with an extremely large number of templates.

6 Conclusion

To address the threat of bot abuse faced by large language model-driven dialogue systems, this paper proposes LPDetective, a novel unsupervised detection framework. Our largescale experiments on three real-world datasets, Bing Copilot, Wildchat, and ChatLog, show that LPDetective significantly outperforms existing methods in terms of accuracy, recall, and real-time performance. For example, on the Bing Copilot dataset, LPDetective achieves an F1 score of 94.0%, an improvement of 7.5% over the best baseline XLNet, with a detection latency of only 0.294 seconds, which is 1/178 of XLNet. In the future, we plan to explore combining fewshot learning, multilingual optimization, hierarchical indexing, and other techniques to enhance its practicality.

Acknowledgments

This work was supported by the National Key R&D Program of China under Grant No. 2022YFB2703301.

Ethical Impact

During the development of LPDetective, we placed great emphasis on protecting user privacy and data security. We ensure that all data used for analysis has been desensitized and anonymized.