Automated Detection of Pre-training Text in Black-box LLMs

1 Introduction

The extensive corpus used during the pre-training phase is a major factor behind the extraordinary performance of Large Language Models (LLMs) [Brown et al., 2020]. In fact, some corpora potentially include sensitive information, such as copyrighted materials. This could give rise to legal disputes, such as The New York Times lawsuit against OpenAI [Grynbaum and Mac, 2023] for unauthorized use of its articles in training LLMs. Pre-training data detection task in LLMs aims to determine whether a given material is included in the target LLM's (i.e., the LLM to be detected)

∗Code: github.com/STAIR-BUPT/VeilProbe

†Corresponding Author

pre-training corpus [Shi et al., 2024]. It has become a research hotspot due to its important value in addressing issues such as detecting data contamination [Oren et al., 2023; Ye et al., 2024] and protecting the rights of copyright holders [Karamolegkou et al., 2023; Chang et al., 2023].

Pre-training detection methods typically rely on the target LLM's distinct output behavior when the given pretraining texts are used as input. According to the detector's access levels, they can be categorized as the whitebox [Wang et al., 2024], grey-box [Shi et al., 2024; Zhang et al., 2024b], and black-box settings [Duarte et al., 2024; Chang et al., 2023]. The methods in white-box and greybox settings, as illustrated in Fig. 1 (a), generally follow a paradigm that calculates a distinguishable score based on the statistical characteristics of token probabilities or hidden parameters [Carlini et al., 2021; Shi et al., 2024; Zhang et al., 2024b]. For instance, Min-K% Prob [Shi et al., 2024] uses the sum of the lowest token log probabilities as the score. DC-PDD [Zhang et al., 2024b] introduces a calibrated score based on the cross-entropy between token probabilities and reference corpus frequencies. Nevertheless, advanced LLMs like ChatGPT and Claude are often in the black-box settings, where only the input and output texts are accessible. Whitebox and grey-box methods are infeasible in this scenario.

Existing black-box pre-training data detection methods typically enable the target LLM to perform complicated tasks such as question-answering tasks [Karamolegkou et al., 2023], multiple-choice questions [Duarte et al., 2024], and name-cloze tasks [Chang et al., 2023]. The LLM's responses distinguish between pre-training texts and non-training texts. Fig. 1 (b) shows an example that the name-cloze task requires deciding where to place blanks and creating customized options. Yet, it requires handcrafted instructions for each specific input, which is expensive and time-consuming and cannot adapt to real-world large-scale pre-training text detection.

The purpose of this study is to detect texts in the blackbox setting without human involvement. One natural idea is to infer the membership distinguishable features and use these features to train a membership classifier to identify pre-training texts, as shown in Fig. 1 (c). However, it is non-trivial due to the following two challenges. (1) The commonly used intuitive indicators [Dong et al., 2024; Abbasi-Yadkori et al., 2024], which are directly computed based on the input and output texts, such as semantic shifts or consistency of output texts, have been proven to be inadequate to distinguish members (see our experiments in Appendix C.1); (2) The number of ground-truth pre-training samples is quite limited as the LLM providers usually do not disclose [Karamolegkou et al., 2023; Shi et al., 2024; Chang et al., 2023]. This would result in overfitting issues when training the membership classifier.

To tackle the above challenges, we propose a novel pretraining text detection framework VeilProbe, consisting of two main parts: the membership feature inference module, and the prototype-based classifier. To capture the distinguishable features between training and non-training texts, the inference module first uses a sequence-to-sequence mapping model to simulate the mapping pattern of how the LLM completes a given text to its suffix. Additionally, inspired by [Liu et al., 2024], it integrates the perturbation calibration features by perturbing the key tokens in the input text to enrich the distinguishable features. Using these features, a prototype-based classifier [Snell et al., 2017] integrated with Information Bottleneck (IB) [Tishby and Zaslavsky, 2015] is proposed for the detection of pre-training texts. This classifier is capable of filtering out irrelevant features. Moreover, it allows for effective classification using only a limited number of ground-truth training instances. Our contributions are as follows:

• We propose a novel framework named VeilProbe. To the best of our knowledge, we are the first to automatically detect pre-training text in the black-box LLMs without human involvement.
• We propose a sequence-to-sequence model aiming to capture the intricate Text-to-Suffix pattern and infer the features that can distinguish membership. Additionally, we design a prototype-based classifier, which is capable of identifying the membership training data by leveraging merely a limited number of known samples.
• We compare the performance of VeilProbe to a wide

range of baselines. It shows state-of-art results on three widely used datasets and outperforms the existing methods by a good margin.

2 RelatedWork

2.1 Membership Inference Attack in LLMs

Membership Inference Attack (MIA) [Carlini et al., 2022; Shokri et al., 2017] aims to identify whether a specific data point is in the model's training set. The concept of MIA, which was initially applied in the Computer Vision domain [Shokri et al., 2017], has recently been extended to the Natural Language Processing tasks [Carlini et al., 2021; Ye et al., 2022; Duan et al., 2024]. MIA on LLMs can serve as a technique for several relevant tasks, such as Data Extraction Attacks [Carlini et al., 2021; Yu et al., 2023] and Personally Identifiable Information Attacks [Lukas et al., 2023; Shi et al., 2024; Kim et al., 2023], data contamination [Oren et al., 2023; Golchin and Surdeanu, 2024] and finetuning data detection [Mireshghallah et al., ; Song and Shmatikov, 2019]. Our works focus on an instance of MIA, pre-training data detection, a task that has become a hot issue in recent research.

2.2 Pre-training Data Detection

According to the access level for detectors, there are three categories: the white-box, grey-box, and black-box settings. As the settings become increasingly strict, the information available to the detector diminishes, making the detection more challenging. Most studies focus on the grey-box setting, where the token probability distribution is accessible. Most of them [Carlini et al., 2021; Mattern et al., 2023; Shi et al., 2024; Zhang et al., 2024a; Zhang et al., 2024b] typically rely on heuristic assumptions to calculate a membership score for identifying pre-training texts. Recently, some studies [Maini et al., 2024; Puerto et al., 2024]selectively combine the membership scores as the aggregated feature. We focus on the more challenging black-box setting. Some previous works [Duarte et al., 2024; Chang et al., 2023]in such settings usually design complicated tasks to identify pretraining texts. Unlike previous work, our work focuses on automating the detection process in the black-box setting without human intervention.

3 Problem Statement

Formally, given a text s ∈ S, where the S denotes the set of texts to be detected, and a target LLM Θ with pre-training data Dθ, we aim to determine whether s is a member in the pre-training dataset Dθ of Θ. Building upon previous work [Shi et al., 2024; Zhang et al., 2024b], the task is defined as follows:

where A denotes the detection algorithm, and K represents the information available to the detector. In the black-box setting, the detector can only access the input and output texts generated by Θ. We assume a few ground-truth pretraining and non-training text samples are available, which is in line with the practical scenarios. In particular, for the pre-training samples, a few classic popular published books, papers, and Wikipedia pages are well known to be included in the pre-training corpus [Chang et al., 2023; Karamolegkou et al., 2023; Zhu et al., 2015]. Moreover, there are also some unpublished manuscripts from some writers that can be the non-training samples, which is consistent with the setting in previous studies such as [Maini et al., 2024; Puerto et al., 2024].

4 Method

Our proposed VeilProbe framework is illustrated in Figure 2, which is composed of three modules: (1) For preprocessing, the text-to-suffix pairs sampling module is proposed to sample the completion suffix sout of each input text s ∈ S by requesting the target LLM response. Moreover, we perturb the key tokens of s to generate r, which is then fed into LLMs to obtain the corresponding suffix rout . (2) Based on multiple ⟨s, sout ⟩ and ⟨r, rout ⟩ pairs, the membership feature inference module is devised to infer the latent membership feature Z. (3) The prototype-based classifier is trained with the inferred membership features to obtain the prototypes for pre-training and non-training data. In the detection pipeline, given a text s to be detected, if its membership feature is closer to the pretraining prototype, it can be inferred that s is a member of the target LLM's pre-training data.

4.1 Text-to-Suffix Pairs Sampling

As shown in Figure 2 (a), for preprocessing, the sampling stage aims to generate a set of text-to-suffix pairs from the target black-box LLM, including text-to-suffix pairs ⟨s, sout ⟩ and perturbed text-to-suffix pairs ⟨r, rout ⟩.

Text-to-Suffix Pair Generation. The autoregressive completion task is employed to align with the training patterns of the corpus during the pre-training phase. To be specific, without additional instructions, each text s serves as a prefix, and the target black-box LLMΘ is requested to complete the sentence based on s. The target LLM autoregressively generates the subsequent tokens until the maximum generation length is reached. A pair of the text s and its corresponding response sout is denoted as ⟨s, sout ⟩. More than one suffix is generated for each text to reduce the randomness in the responses. Similarly, the corresponding suffix rout of the perturbed text r is generated in the same autoregressive way. The following will elaborate on how to obtain the perturbed r.

Key Token Perturbation. We carry out perturbations on the key tokens within the original input s to generate r. The key tokens are those that exert substantial influence on the generation of the suffix text. Inspired by [Li et al., 2024], we employ a feature attribution technique to identify key tokens. However, directly applying feature attribution analysis to the black-box LLM is non-trivial. Most of these methods depend on hidden information (prediction or token probability distributions), which is unsuitable for the black-box scenario.

To address the above issue, a perturbation-based feature attribution method is introduced, as illustrated in Figure 3. We use multiple small-size open-source LLMs (preferably from the same family) as proxy models to assist in selecting the key tokens. It is observed that when using LLMs from the same family as proxy LLMs, the feature attribution technique tends to select a similar set of informative tokens in a text. For instance, GPT-2 can serve as a proxy LLM for ChatGPT. (See Appendix C.2 for detailed analysis). Then the importance scores obtained for each token across multiple proxy LLMs are averaged to obtain the final importance score µ. Based on these scores, top-γ% tokens are selected and perturbed with synonyms to produce the perturbed text r.

4.2 Membership Feature Inference

This module is designed to infer membership features based on the ⟨s, sout ⟩ and ⟨r, rout ⟩ pairs obtained from the above module.

Text-to-Suffix Mapping Modeling. Since LLMs generate outputs one by one in an autoregressive manner, the inputoutput mapping f generally follows a sequence-to-sequence pattern [Sutskever et al., 2014; Vaswani et al., 2023]. To capture such a pattern, a sequence-to-sequence mapping model based on the transformer architecture [Vaswani et al., 2023] is proposed to model the mapping pattern from the input text to its suffix. Specifically, this mapping model is trained on ⟨s, sout ⟩ pairs to learn the input-output dependency, that is, f : s → sout . The training objective is to maximize the conditional probability of the output sequence sout given the input sequence s as the prefix, which is expressed as:

where T denotes the max length of the output sequence sout , and each wout t represents the token generated at time step t. The model learns to predict wout t based on the previously generated tokens wout 1 , . . . ,wout t-1 and the input sequence s. Upon completion of the training process, once a text input is provided to the sequence-to-sequence model, the hidden states of the model are extracted as the text-to-suffix mapping feature Zs . Similarly, the perturbed mapping features Zr can be obtained based on ⟨r, rout ⟩ pairs.

The obtained mapping feature can function as a part of the membership distinguishable features to distinguish between pre-training and non-training texts. The fundamental concept is based on previous research [Wang et al., 2024] which has shown that the hidden states in the internal layers of a whitebox LLM can tell the difference between the pre-training texts and non-training texts.

Perturbation Calibration Feature Extractor. In addition to the hidden states obtained from the aforementioned mapping model, we further explore the perturbation calibration feature as a complement. The intuition is based on the observation that the robustness of the target model exhibits marked differences when perturbed pre-training samples and nontraining samples are employed as inputs [Liu et al., 2024; Mattern et al., 2023].

Specifically, as shown in Figure 2 (b), we first obtain the raw perturbation feature Zc , which is the difference between the original Zs and perturbed mapping features Zr . Since the perturbation operations would probably induce noisy information [Liu et al., 2024; Hooker et al., 2019] for identifying members, it is crucial to select the most critical features to infer membership status. The significance test [WELCH, 1947] is applied to filter critical features. We formulate the null hypothesis (H0) and the alternative hypothesis (H1) for the values in each dimension of the feature with respect to their impact on the target variable (i.e., membership status Y ). The hypotheses are defined as follows:

where G1, and G0 represent the ground-truth pre-training and non-training samples, respectively. zc i represents the i-th dimension of the feature vector Zc . Then the p-value of each dimension of the feature vector is obtained. A lower p-value indicates strong evidence against the null hypothesis, suggesting that the pre-training and non-training samples are distinct in the i-th dimension, supporting the alternative hypothesis. We then retain the critical dimensions and set the others to zero. The processed features then serve as the final perturbation calibration feature Z˜c .

Consequently, the membership feature of s is the concatenation of the mapping feature Zs and the perturbation calibration feature Z˜c , which is defined as:

4.3 Prototype-based Classifier

Based on the membership features, we propose a prototypebased classifier to distinguish between pre-training and nontraining texts. We choose this classifier as a backbone because it works well even with limited labeled data.

Prototype Constructing. The prototype-based classifier constructs a metric space through learning [Snell et al., 2017]. In this space, samples belonging to the same class are proximal to each other, whereas samples from different classes are distantly separated.

Each class is represented by a prototype, which is the center of its samples in this space. As shown in Figure 2 (c), based on the membership features, we aim to construct the pre-training prototype and the non-training prototype. Specifically, the embedding function F is trained following the approach of Prototypical Networks [Snell et al., 2017] to mitigate overfitting issues caused by the limited labeled data. The training episodes are constructed by randomly sampling a subset of samples from the ground-truth set G to form the support set U, while a separate subset of the remaining samples serves as query points.

Formally, we define the support set as U = {(Z1, Y1), . . . , (Zm, Ym)}, where each Zi ∈ RD represents the D-dimensional feature vector of an sample, and Yi ∈ {0, 1} is the corresponding label. UY represents the set of samples labeled with pre-training (U1) or non-training (U0). The objective is to compute an K-dimensional representation, denoted as cY ∈ RK , where c0 represents the non-training prototype and c1 represents the training prototype. These prototypes can be obtained using an embedding function Fϕ : RD → RK with learnable parameters ϕ. Each prototype is defined as the mean vector of the embedded representations of the support samples belonging to their set:

We define the probability distribution over class labels for a query point Z using a softmax function, which is based on the distances between the query's embedding and the class prototypes. The probability of classifying Z as belonging to class Y is given by:

where d(Fϕ(Z), cY ) represents the distance between the query embedding Fϕ(Z) and the prototype cY of class Y , and the distance function d is squared Euclidean distance. The learning process minimizes the negative log-likelihood of the true class label via Stochastic Gradient Descent.

InformationBottleneckDenoising. To further filter out redundant features in Z, we integrate the Information Bottleneck technique (IB) into the prototype computation. This technique aims to extract a compact representation that retains the information relevant to Y while eliminating redundant information from the input feature Z. Specifically, we construct a compressed representation ˜ Z by optimizing the following objective:

where I(·) represents the mutual information, Z represents the input feature, Y is the ground-truth label, and β balances the trade-off between minimizing redundancy and retaining relevance to Y .

We then use the compressed representation ˜ Z as the feature to detect the pre-training text s. That is, the detection score δ is defined as:

where d represents the distance metric, c1 represents the pretraining prototype, and c0 represents the non-training prototype. If ˜ Z is closer to the c1 than c0, it can be inferred that the text s is a member in the pre-training data of the target LLM.

5 Experiments

5.1 Experimental settings

Datasets. The statistics of three datasets are shown in Table 1. WikiMIA [Shi et al., 2024] consists of Wikipedia event snippets. BookTection [Duarte et al., 2024] is a widely adopted dataset that contains 165 copyrighted books, which is expanded based on BookMIA [Shi et al., 2024]. arXivTection [Duarte et al., 2024] includes classic papers from arXiv. Target LLMs. Following [Shi et al., 2024], for WikiMIA, we select Pythia-6.9B [Biderman et al., 2023], LLaMA13B [Touvron et al., 2023a], and GPT-NeoX-20B [Black et al., 2022] as our target LLMs. Following [Duarte et al., 2024], for BookTection, Mistral-7B [Jiang et al., 2023], LLaMA2-7B [Touvron et al., 2023b] and ChatGPT (gpt-3.5instruct) are selected as the target LLMs. For arXivTection, Mistral-7B [Jiang et al., 2023], LLaMA2-13B [Touvron et al., 2023b], and Claude 2.1 are selected as our target LLMs. Evaluation Metrics. Following previous works [Shi et al., 2024; Zhang et al., 2024b; Duarte et al., 2024; Carlini et al., 2022] we use the Area Under the ROC curve (AUC) and True Positive Rate at low False Positive Rate (TPR@5%FPR) as our evaluation metrics. A higher AUC reflects better perfor-

mance with higher TPR and lower FPR across thresholds.

Baselines. We compare our framework with the following strong baselines. For the grey-box methods, eight methods are selected: PPL, Lowercase, Zlib [Carlini et al., 2021], Neighbor [Mattern et al., 2023], Min-K% Prob [Shi et al., 2024], Min-K%++ Prob [Zhang et al., 2024a], DCPDD [Zhang et al., 2024b], and FeatureAgg [Maini et al., 2024]. Two methods are selected for the black-box methods: DE-COP [Duarte et al., 2024] and Name-Cloze [Chang et al., 2023]. The details are described in Appendix A.

Implementation Details. In our work, all experiments are implemented on a workstation with five NVIDIA Tesla V100 32G GPUs, and Ubuntu22.04.4. For each text to be detected, three suffixes were generated using the target LLM, with the maximum suffix length set to 512 tokens. The parameter γ is set to 10 for obtaining the perturbed text r. The p-value significance threshold is chosen from {0.001, 0.01, 0.05, 0.1} to select the critical perturbation calibration features. We randomly sample approximately 50 ground-truth samples per dataset to train the prototype-based classifier, with the remaining samples serving as the texts to be detected. It should be noted that our evaluations are carried out at both sentence and document levels, and the document-level results are derived from the sentence-level ones. Specifically, we compute the average of the sentence-level detection scores within each document to obtain the document-level results.

5.2 Main Results

Table 2 and Table 3 show the comparison methods' AUC and TPR@5%FPR scores, respectively. We can observe that (1) VeilProbe significantly outperforms all the other methods, achieving an impressive improvement of over 25.1% in AUC and nearly doubling the TPR@5%FPR scores; (2) Compared to grey-box detection methods, VeilProbe can be implemented on ChatGPT and Claude because it does not rely on hidden information; (3) VeilProbe significantly improves text detection performance compared to existing black-box methods. One possible reason is that these methods often rely on complicated instructions, making some LLMs struggle to comprehend the task.

Additionally, we evaluate document-level detection on 165 bestselling books and 50 arXiv papers from the BookTection and arXivTection datasets by aggregating sequences' detection scores. As shown in Table 4, VeilProbe achieves an average AUC of 99.2% and an average TPR@5%FPR of 96.3% when detecting documents. The detailed results are provided in the Appendix C.4. The document-level results consistently outperform the sentence-level detection, which is in line with previous works [Puerto et al., 2024].

5.3 Ablation Studies

To further assess the importance of each module in VeilProbe, we perform ablation studies on WikiMIA-64 and BookTection-64. For the membership feature inference stage, we analyze the contributions of (i) perturbation calibration feature extraction, (ii) critical feature selection with significance tests, and (iii) key token perturbation. For the prototype-based classifier, we evaluate (i) the impact of the IB module and (ii) the impact of different distance metrics

(a) AUC w.r.t Model Size.

(b) AUC w.r.t Text Length.

Figure 5: VeilProbe's performance w.r.t #Ground-truth samples.

for prototype computation.

Table 5 shows that all ablation variants demonstrate lower performance than the full framework. The perturbation calibration features are essential, particularly in the TPR@5%FPR score. The critical feature selection can improve performance by reducing noise induced by perturbation operations. Key token perturbations are more effective than random perturbations. The IB module can eliminate redundant information regarding membership status to enhance performance. The most suitable distance function for prototype computing in our task is the Squared Euclidean distance.

5.4 Discussion and Analysis

In this subsection, we analyze five main factors that may affect the detection performance. Among them, the former two are general factors, while the latter three are frameworkspecific factors.

Size of target LLMs. To investigate the impact of different target LLM sizes, we analyze Pythia models of sizes 1B, 1.4B, 2.8B, and 6.9B on WikiMIA. Figure 4a shows that AUC scores improve with increasing model sizes, aligning with previous studies [Shi et al., 2024; Duarte et al., 2024]. Additionally, our proposal consistently outperforms baselines.

Length of texts. We evaluate performance with respect to different text lengths (Figure 4b). For BookTection, the detection performance increases with longer texts. The possible reason is that longer texts can contain more distinguishable features. For WikiMIA, the AUC score decreases at the length of 128, potentially due to the limited number of texts of this length (only 250 samples), much smaller in number than the samples of other lengths, as shown in Table 1.

Number of ground-truth samples. We further explore the impact of the number of ground-truth samples. Figure 5

shows that AUC scores improve as the number of samples increases. Nevertheless, even with only 10 ground-truth samples, our method is still capable of achieving an AUC detection rate of approximately 80-90% across three datasets.

Volume of text-to-suffixmapping features. Table 6 shows the impact of different volumes of mapping features on the detection performance. We use four extraction methods: (i) Full, which extracts the hidden states from all layers of the mapping model; (ii) Full-LT, extracting the last token's hidden states from all layers; (iii) LL, extracting the hidden states from the last layer of both encoder and decoder; and (iv) LLT, extracting the last token's hidden states from the last layer of both encoder and decoder. Despite a general slight decline in performance metrics as the quantity of mapping features is decreased, our approach stays effective at a high detection performance.

Significance threshold for critical perturbation calibration feature selection. Figure 6 shows that the optimal pvalue varies across different text domains. A p-value of 0.1 allows tolerance, 0.05 is standard, and 0.001 is stricter. When the p-values are large, there is a risk of underrepresenting crucial features. Conversely, when they are small, it may lead to the introduction of misleading features that impede detection.

6 Conclusion

In this paper, we propose VeilProbe, the first automatic pretraining text detection framework for black-box LLMs. We propose a novel sequence-to-sequence mapping model that can effectively capture the Text-to-Suffix mapping pattern and infer latent membership features. Given the scarcity of ground-truth labeled pre-training and non-training samples, we devise a prototype-based classifier to identify pre-training texts and mitigate the overfitting issue. Comprehensive experiments on three widely used datasets demonstrate that our framework significantly outperforms existing baselines. Notably, VeilProbe operates automatically, without any human intervention throughout the entire detection process.

Acknowledgments

This work was supported by the Natural Science Foundation of China (No. 62372057).