Information Entropy for LLM-generated Text Detection

1 Introduction

Large language models (LLMs) recently exhibited the powerful ability to generate textual data [Liu et al., 2023b; Li et al., 2024a; Li et al., 2025b], achieving great attention. However, LLMs may also be abused by malicious actors to generate large volumes of spam text, e.g., generating misleading public opinion content and fraudulent academic material [Wang et al., 2025b; Li et al., 2024b]. Considering this, there is an imperative for the development of efficient detection technologies capable of accurately discerning whether the text is generated by humans or LLMs.

Many approaches [Quidwai et al., 2023; Bhattacharjee et al., 2024; Mitchell et al., 2023; Bao et al., 2023] have been proposed to detect generated text, which can be generally divided into two classes depending on the type of internal information utilized by LLMs, i.e., weights and intermediate layer output. The methods utilizing weight information explore the magnitude of weight fluctuations to discern if the text has been generated by LLMs, i.e., substantial weight alterations pointing towards LLM-generated text, while slight modifications in weights are human-written [Bakhtin et al., 2019]. Other methods use intermediate layer output, such as log probability for each word, to perturb the text and observe the changes in log probability [Taguchi et al., 2024; Wang et al., 2023b; Mitchell et al., 2023; Wang et al., 2023a]. A decrease in log probability indicates text generated by LLMs, while an increase or a small change in log probability suggests text written by humans. While these techniques are highly effective for detecting text when internal model information (e.g., gradients, hidden states) from LLMs is accessible, their utility is limited in black-box settings where only the generated text is available.

∗Haozhao Wang and Jingcai Guo are corresponding authors.

In this work, we seek to develop a technique that relies not on internal mechanisms of LLMs but entirely on the informational content of the text itself to determine whether text is authored by humans or generated by LLMs. Particularly, we identify that human-written text exhibits higher information entropy (IE) than LLM-generated text. Inspired by this observation, we introduce an innovative methodology named IED, which leverages the information entropy for generative text detection. IED first calculates the information entropy value of each word and then transforms each given input text into an IE vector by using the calculated IE. Finally, IED inputs the transformed IE vector into a binary classifier. Furthermore, we propose a novel method named IEGD, which leverages the information gain to construct a text vector, thereby enhancing the differentiation between human-written and LLMgenerated texts. Experimental results on various datasets and models demonstrate that our method can significantly enhance detection accuracy. Our contributions are:

• We identify that human-written texts usually exhibit higher information entropy than LLM-generated text. As we know, this research is the first to explore the detection of LLM-generated text using information entropy gain from an information-theoretic standpoint.
• We propose the novel information entropy-based meth-

ods for distinguishing between human and LLMgenerated texts. Our information entropy detection method is a technique that functions independently of the internal specifics (e.g., architecture, weights) of LLMs, making it broadly applicable.

• We construct a new large-scale dataset comprised of generated texts using various LLMs and prompts from real datasets. Extensive experiments on 9 advanced and commercial LLMs (GPT-4, Qwen-2, GPT-3.5-turbo, LLAMA-13B, etc.) show that our method outperforms state-of-the-art detection methods by up to 10.81%.

2 Related work

In this section, we discuss representative approaches to determine whether humans or LLMs produce the text. These approaches mostly consist of leveraging internal mechanisms of LLMs, statistical analyses of feature information, and watermarking techniques.

Leveraging internal mechanisms of LLMs. Many approaches capitalize on the intricate workings of LLMs, including intermediate layer outputs and weight parameters, to discriminate between human-written and LLM-generated texts [Taguchi et al., 2024; Wang et al., 2023b; Wang et al., 2023a; Bakhtin et al., 2019]. These methods introduce subtle text modifications to track changes in log probabilities. A conspicuous decline in log probability typically points to text generated by large language models. Conversely, an elevation in log probability or insignificant variations tends to imply human authorship [Mitchell et al., 2023; Bao et al., 2023]. Other methods utilize weight information to explore the magnitude of weight fluctuations [Bakhtin et al., 2019; Wang et al., 2026]. While these methodologies demonstrate considerable efficacy in identifying text origins when afforded access to the internal workings of LLMs, their utility is constrained in scenarios where solely the text itself is accessible. Our approach differs fundamentally from theirs in both methodology and underlying assumptions. It is independent of the complex internal mechanics of LLMs and harnesses the metric of IE gain to distinguish human-written and LLM-generated text. Also, our approach's model-agnostic property confers notable universality, seamlessly detecting across a wide array of LLMs without necessitating bespoke optimization for each model [Wang et al., 2025a].

Statistical analyses of feature information. Aiming at detecting text generated by small models, multiple methods employ statistical analysis to examine various features of the text, such as word choice, sentence structure, and stylistic elements, comparing them against known human or generation patterns to detect discrepancies that might indicate nonhuman authorship [Wang et al., 2023b; Shi et al., 2024; Gao et al., 2018; Li et al., 2025a]. Nevertheless, statistical approaches are inherently limited in detecting the text generated by LLMs. Particularly, they rely solely on patterns within vocabulary and structure, which can lead to overfitting in high-dimensional [Wu et al., 2023]. As LLMs continue to advance, their capabilities have reached near parity with human performance, thereby progressively undermining the effectiveness of conventional statistical methods.

Watermarking techniques. Watermarking constitutes an intricate, embedded detection technique that entails the sophisticated embedding of cryptic identification information within the very substrate of the generated text [Gehrmann et al., 2019; Fr¨ohling and Zubiaga, 2021; Antoun et al., 2023]. It allows for identifying the source and validity of the content during subsequent detection [Lancaster, 2023; Sadasivan et al., 2023]. Although the watermarking methodologies assure robustness [Kirchenbauer et al., 2023], they are employed at the creation stage as opposed to being utilized for subsequent detection efforts.

3 Methodology

In this section, we introduce a framework named IED, which leverages Information Entropy to discern whether the text is human-written or LLM-generated. Specifically, IED first calculates the entropy value of each word and transforms the input text into an explicit information entropy vector. Then, it adopts a classifier to conduct the detection based on this vector. As shown in Figure 5, this architecture guides the model's semantic capacity with a discriminative, theory-driven signal, enhancing the differentiation between human-written and LLM-generated texts.

3.1 Motivation

By comparing the texts written by humans and generated by LLMs, we find that the information entropy of humanwritten text is higher than LLM-generated text. Specifically, we compiled a corpus of broader sample essays written for the TOEFL exam from online sources. We employed three advanced language models - GPT-2 [Radford et al., 2019] and LLaMA-7B [Touvron et al., 2023] for the essay generation task, intending to conduct a thorough comparison against essays authored by humans. Let q(xji ) be the frequency of word xji occurring in the document xi, and W be the number of different words. We then can compute the information entropy H(xi) of each input document xi as:

As shown in Figure 1, the IE of human-written texts is higher than LLM-generated texts overall. For instance, the IE of most human-written documents is higher than 3050 (indicated by the horizontal dashed line), while that of most LLM-generated documents is smaller than 3050. The main reason is higher IE generally indicates more variability, unpredictability, and randomness in the text, which corresponds to the characteristics of human writings, which are replete with a broad spectrum of lexical choices and syntactical configurations. Conversely, LLMs tend to generate texts that often conform to structured patterns, which are inherently limited by the scope of the training corpus and algorithmic logic, lacking intrinsic human creativity.

3.2 IED: Information Entropy based Detection

In this section, we introduce IED to discern whether text x originates from LLMs or humans based on the IE. IED mainly consists of two steps, i.e., calculating the information entropy to construct an entropy-word mapping table and then training the text classifier based on this table.

LLM-generated Text Dataset Construction. Due to the lack of readily available LLM-generated text databases, we collect sufficient and representative samples of generated text for each model to be evaluated. To achieve this, we first construct a diversified prompt set C = {c1, c2, · · · , cn} from datasets such as X-sum [Narayan et al., 2018] and RoFT [Dugan et al., 2023] by extracting the questions existed in their samples. Then, we randomly sample a prompt ci from the prompt set C and input it to each LLM j ∈ [S] to generate the text x(i-1)∗j+i = LLMj(ci), where S denotes the number of LLMs. Next, the generated text x(i-1)∗j+i is added to the generation dataset T by T = T ∪ {x(i-1)∗j+i}. This process performs MT iterations, i.e., constructing MT samples for the dataset T. This approach guarantees uniformity and equity in the sampling process, cementing a robust groundwork for LLMs' subsequent evaluation and analysis.

Calculating Information Entropy. The dataset D is composed of LLM-generated dataset T and human-written dataset U. we decompose all the texts in the dataset D into a 1-gram set G={g1,. . . ,gW}. Then, by counting the number r(gi, T) of occurrences of the word gi in the entire dataset T , we can calculate the frequency q(gi,D) of each word gi in dataset D:

we calculate the H(gi, D) with equation (3).

Finally, the IE of the word gi is added to the dictionary D by D = D ∪ {(gi, H(gi, D)}.

Text Classification. For ease of classification, we transform the input text into a vector composed of information entropy gain values by retrieving the mapping table. Then, this vector is adopted as the input for the classification network. This approach enhances the model's ability to discern subtle patterns and anomalies within the text, potentially improving the accuracy and reliability of the detection process.

Specifically, we refer to the mapping table M to get the corresponding information entropy vector ei for the input text xi remaining to be detected, i.e., ei = [M(x1i ),M(x2i ), . . . ,M(xWi )]. Then, we transform the information entropy vector ei to the backbone θb of the neural network, obtaining the hidden feature vector zi = fb(ei, θb). Let θh be the classifier parameter. The detection label is obtained by ˆ yi = fh(zi, θh). Let l(·) be the loss function. Then, our objective is to train the classifier θh by minimizing the average of all sample loss L( ˆ yi, yi; θh):

Note that we keep the parameter θb unchanged during training and only train the classification layer with θh = θh - η∇L( ˆ yi, yi, θh) to improve the efficiency of fine-tuning the model parameter, where η is the learning rate.

3.3 IEGD: Information Entropy Gain based Detection

Drawing upon our insightful observations, we endeavored to leverage information entropy gain as a pivotal element in our experimental design. Particularly, built upon the IE, information entropy gain quantifies the decrease in uncertainty pertinent to data categorization, further enriching the analytical depth of IE. While IED has demonstrated robust efficacy in characterizing textual patterns, the concept of IEG extends this theoretical foundation by providing a quantitative measure of distributional differences between human and machine-generated text features. Specifically, information entropy gain enhances discrimination capability when employed with information entropy metrics, offering a systematic approach to identifying distinctive linguistic patterns between the two text categories. Therefore, we further extends the IED to IEAD that uses the information entropy gain to vectorize the input texts.

Calculating Information Entropy Gain. Let D be the same dataset composed of LLM-generated dataset T and humanwritten dataset U. We first calculate the information entropy H(D) of the whole dataset D:

We divide the dataset D into two subsets: Dgi contains the text with gi, and D¬gi doesn't contain g. Then, we calculate the conditional entropy H(D|gi):

Next, we calculate the information gain ∆H(D|gi) by

Based on the information entropy gain of all words, we transform the texts into the vectors of information entropy gain. Finally, we train the classifier θh using the same equation (4). The algorithm and illustration of IEGD can be found in Appendix A.

4 Experiments

In this section, we first introduce the experimental setup for the datasets, implementation details of the IEGD, and the baselines. We evaluated the accuracy of our approach on four datasets and compared it with state-of-the-art algorithms. Finally, we assessed the impact of hyperparameters on our proposed method.

4.1 Experiment setting

Dataset&LLMs.Our experiment employs a meticulously curated collection of four datasets to rigorously appraise the capabilities of generated text detection across a spectrum of domains. For instance, news articles from the XSum dataset [Narayan et al., 2018] are used for news detection. Wikipedia paragraphs from SQuAD contexts [Rajpurkar et al., 2016] represent machine-written academic essays and prompted stories from the Reddit WritingPrompts dataset [Fan et al., 2018] gauge the models' ability to generate creative writing similar to human submissions. To further test the model's robustness against shifts in data distribution. We include various topics from the RoFT dataset [Dugan et al., 2023]. We use various LLMs to generate text, i.e., GPT2 [Radford et al., 2019], OPT-2.7B [Zhang et al., 2022], Neo-2.7B [Gao et al., 2020], LLaMA-7B [Touvron et al., 2023], LLaMA-13B, GPT-NeoX [Black et al., 2022], GPT3.5-turbo, GPT-4, Qwen-2 [Wang et al., 2024].

Metrics. To evaluate the detector's ability to distinguish between texts generated by LLMs or humans, we employ Accuracy (A), AUROC,F1 scores (F1) and Recall (R) as performance metrics.

Baselines. We compared our method with various approaches for detecting LLM-generated text. FastDetectGPT [Bao et al., 2023] is a method for detecting text generated by large language models (LLMs), which works by evaluating the conditional probability curvature associated with sampled alternative tokens. The Log-Rank is employed to assess whether text generated by a model is based on the value of the log perplexity. GLTR [Gehrmann et al., 2019] integrates a suite of statistical techniques, employing visual analytics to assist users in discerning text generated by models. DetectGPT [Mitchell et al., 2023] exploits the curvature characteristics of language model probability functions to discern synthesized text. LLMDet [Wu et al., 2023] assesses the provenance of generated text by calculating the surrogate perplexity for each LLM. SeqXGPT [Wang et al., 2023a] translates sentences into waveforms akin to those in speech processing, leveraging convolutional networks and self-attention mechanisms for sentence-level detection of generated text. GECScore [Wu et al., 2024] determines whether text originates from LLMs by feeding it into a grammar error correction model, quantifying the similarity between the original and corrected texts. COCO [Liu et al., 2023a] uses contrastive learning to detect text.

Implementation Details. To ensure the reproducibility and efficiency of our experiments, all computations were meticulously conducted on a high-performance computing platform running the Ubuntu operating system. The system is powered by an Intel Xeon Platinum 8176 CPU@2.10GHz, delivering robust computational capabilities for data processing. To handle the memory-intensive tasks associated with large language models, the platform is equipped with a substantial

503 GB of memory. To accelerate the inference and calculation of information entropy, we utilized four NVIDIA Corporation GA102GL RTX A6000 GPUs.

4.2 Performance Evaluation

Accuracy. As detailed in Table 1, our proposed method consistently demonstrates superior precision across diverse benchmarks, outperforming state-of-the-art baselines. Specifically, on the X-sum dataset for GPT-2 generated text, IEGD achieves a remarkable accuracy of 92.60%, surpassing competitive methods like DetectGPT (80.25%) and FastDetectGPT (72.87%). Beyond individual datasets, extensive experiments indicate that our method achieves an improvement over existing methods by up to 10.81%. This performance disparity validates our foundational premise: humanauthored texts exhibit higher information entropy due to the complexity of human thought and rich background knowledge. In contrast, LLMs optimize for probability maximization, resulting in structured, predictable patterns with lower entropy. By quantifying this uncertainty gap, IEGD effectively transforms subtle statistical variations into discriminative signals for accurate detection.

F1-score. As illustrated in Table 2, the experimental results demonstrate the superior performance of IED and IEGD in distinguishing LLM-generated text, maintaining a robust balance between precision and recall. On the WritingPrompts dataset, IEGD achieves an F1 score of 89.97% for GPT-2 generated content, significantly outperforming baselines such as Log-Rank (47.12%) and DetectGPT (74.06%). Similarly, for OPT-2.7, IEGD attains an F1 score of 90.10%, showing a substantial lead over Log-Rank's 48.86%. This dominance extends to the RoFT dataset, where IEGD consistently surpasses competitors (e.g., 88.40% on GPT-2). By analyzing information entropy at the lexical level, our methods effectively capture the intrinsic differences in statistical distributions and underlying structures of texts. Beyond merely identifying superficial discrepancies, these entropy-based measures amplify the underlying contrast by quantitatively reflecting the varying degrees of uncertainty, randomness, and informational richness. The subtle yet crucial distinctions in textual properties between human and LLM-generated content are translated into more pronounced and empirically measurable signals, significantly enhancing detection capabilities even in complex creative writing scenarios.

Recall. As illustrated in Table 3, the experimental results on the X-sum dataset reveal the superiority of IED and IEGD over other state-of-the-art detection methods when evaluating texts generated by various LLMs. Specifically, IEGD achieves a recall rate of 93.9% for content generated by GPT2, significantly outperforming baselines such as Log-Rank and COCO . This trend extends to the SQuAD dataset, where both IED and IEGD consistently achieve high recall rates across a variety of models. These findings underscore the consistent advantage of IED and IEGD in identifying and distinguishing between human-authored and LLM-generated texts. The superior performance is attributed to their ability to capture both statistical and structural differences.

Ablation Study. Table 4 presents an ablation study focusing on the performance of the IED and IEGD methods when combined with different transformer-based classifier backbones. Across all datasets, the IEGD variant consistently outperforms its IED counterpart. IEGD-BERT achieves 85.99% on X-sum compared to IED-BERT's 79.56%. Similarly, IEGDRoBERTa scores 92.41% on SQuAD, while IED-RoBERTa scores 86.91%. This suggests that using Information Entropy Gain provides more discriminative features for distinguishing between human-written and LLM-generated text than using Information Entropy alone. The IEGD feature engineering approach is more effective than the IED. Using more powerful classifier backbones like RoBERTa and BERT with IEGD features yields the best performance. Furthermore, even lighter models like DistilBERT can achieve competitive results when combined with the potent IEGD features. This demonstrates that the robustness of our feature engineering can compensate for reduced model complexity, making them viable options when computational resources are a concern.

5 Conclusion

We proposed a model-agnostic framework for detecting LLM-generated text named IEGD. Our findings confirm that human-written content exhibits inherently higher information entropy than the structured, predictable patterns produced by LLMs. IEGD leverages this distinction by quantifying uncertainty reduction, transforming subtle statistical variations into discriminative signals. Future work will incorporate conditional entropy to capture fine-grained distributional dynamics. By offering token-level resolution, this approach aims to overcome the limitations of aggregate metrics like perplexity, exposing the specific linguistic pacing essential for detecting high-fluency generated content.

Limitations

The empirical validation of our methods predominantly relies on datasets such as XSum, SQUAD, TOEFL, and Reddit WritingPrompts, which are primarily English-based. Future research needs to explore the information entropy characteristics of different languages, potentially revealing unique patterns in entropy distribution for certain languages (such as Chinese) due to the characteristics of their writing systems. Furthermore, the potential for adversarial attacks specifically designed to circumvent entropy-based detection (e.g., by manipulating text to match expected human entropy profiles) poses an ongoing challenge [Liu et al., 2026a]. While our methods provide a strong baseline, their long-term robustness against these evolving threats will require continuous evaluation and potential adaptation, possibly by incorporating features resilient to such manipulations or developing dynamic thresholding mechanisms [Huo et al., 2025].

Acknowledgments

This work is supported by the National Key Research and Development Program of China under grant 2024YFC3307900; the National Natural Science Foundation of China under grants 62302184, 62376103, 62436003 and 62206102; Major Science and Technology Project of Hubei Province under grant 2024BAA008; Hubei Science and Technology Talent Service Project under grant 2024DJC078; Ant Group through CCF-Ant Research Fund. The computation is completed in the HPC Platform of Huazhong University of Science and Technology. Hong Kong RGC General Research Fund (Grant Nos. 15221123, 15216424, and 15211525), and the Hong Kong PolyU Internal Research Fund (Grant Nos. P0058468 and P0056171).