VaryBalance: Detecting LLM-Generated Text Through Variation

1 Introduction

The strong text-generation capability of large language models (LLMs) and ease of access motivate users to rely on LLMs for content creation. However, machine-generated text has increasingly permeated the human information space consequently. According to [Liu et al., 2024], since the release of GPT-3 and ChatGPT, the number of academic papers on arXiv identified as LLM-generated has grown exponentially. Without explicit disclosure of text provenance, LLMgenerated content are easily misused across news, academia, and social media [Chen and Shu, 2023; 2024; Gressel et al., 2024], leading to the spread of misinformation, academic misconduct, and phishing. Liu et al. [Liu et al., 2024] also found that even experts perform only marginally better than random guessing on identifying LLM-generated texts, suggesting that humans are not capable enough to discern LLMgenerated texts anymore. As a result, developing reliable detectors for LLM-generated text is crucial.

To address this, multiple automatic LLM-generated text detectors have been proposed, which can be roughly divided into white-box and black-box two groups. White-box methods need to know the target LLMs and access the internals of LLMs, such as logits. Representative white-box methods include DetectGPT [Mitchell et al., 2023] and FastDetectGPT [Bao et al., 2023]. Black-box approaches make predictions based solely on observable features of the text, or information obtained from surrogate models (not the target model). N-gram statistics [Yang et al., 2023], Levenshtein score [Mao et al., 2024], grammar errors [Wu et al., 2025], and Neural-based methods [Hu et al., 2023; Wu et al., 2024; Chen et al., 2025] are commonly used black-box methods.

∗Corresponding author

However, both methodologies are constrained by their inherent limitation. For white-box detectors, it is generally impractical to precisely identify the underlying text generator in real-world scenarios, and access to the generator's logits is typically unavailable or infeasible. In contrast, as LLMs continue to advance, producing text that highly resembles human writing is no longer challenging, rendering these blackbox methods (such as simple text-level methods) fail to detect LLM-generated texts.

To address these limitations, we introduce a simple but effective LLM-generated text detection method, VaryBalance. The core idea of VaryBalance is that, compared to human texts, there is a lighter difference between LLM-generated texts and their rewritten versions LLM. Figure 1 shows texts from humans and LLMs with their corresponding rewritten versions by GPT-4o-mini. We can see that the LLMgenerated text and its rewritten version follow similar patterns, and thus are more similar after quantified by Mean Squared Deviation(MSD). Based on this observation, VaryBalance employs a rewriter to generate multiple rewrites of an input text, and a scorer to compute the log perplexity of both the original and rewritten texts. After that, it uses the log perplexity and the MSD of the rewritten texts' log perplexities, computed with respect to the original text's log perplexity as the reference, as an indicator for LLM-generated content detection. Moreover, we present an extended scoring variant tailored to shorts and more stylistically diverse social media text.

To evaluate the effectiveness of VaryBalance, we conduct comprehensive experiments on eight datasets and five models. Results demonstrate that in formal writing contexts, VaryBalance outperforms state-of-the-art methods, such as Binocular and Fast-DetectGPT, with an overall improvement of 34.5% in terms of AUROC.

In summary, our contributions are:

• We empirically reveal that, compared to LLM-generated texts, there is a greater difference between human texts and their rewritten version via LLMs.
• Based on our observation, we propose a novel black-box LLM-generated text detection method, VaryBalance.
• We conduct comprehensive experiments to showcase the SOTA performance of VaryBalance and its robustness under different models and languages.

2 RelatedWork

Detecting LLM-generated texts has become an increasingly challenging task. LLMs achieve superior performance on a wide range of language benchmarks and can produce convincing, contextually appropriate text [Mitchell et al., 2023]. Moreover. Liu et al. [Liu et al., 2024] show that even domain experts perform only slightly better than random guessing when attempting to identify LLM-generated text. Hence, developing reliable detectors for LLM-generated content is both important and urgent.

We introduce approaches of white-box settings or blackbox settings. There is also research [Kirchenbauer et al., 2023] on embedding watermarks into LLM or the generated texts for further detection. Here, we do not discuss watermark-based methods as they demand cooperation from commercial or open-source LLM providers based on their inherent requirement for accessing model generation processes or deployment infrastructure, which is often infeasible.

2.1 White-boxMethods

What characterizes white-box approaches is the direct access to the logits of the generative model, which evaluates the LLM's confidence over every possible next token. [Hans et al., 2024] proposed Binocular, which measures the ratio between perplexity and cross-perplexity as a detection criterion. [Wu et al., 2023] introduced LLMDet, a multi-model text attribution tool that constructs an n-gram-to-next-token probability mapping for each language model and computes proxy perplexities to classify the source of the text.

Beyond direct logit-based scoring, other white-box works leverage perturbation [Mitchell et al., 2023; Bao et al., 2023;

2025]. These methods perturb text using a language model and measure changes in log-probability or other scores under the target model.

Additional works that amend the existing works include TOCSIN [Ma and Wang, 2024], which combines token cohesiveness with conventional logit-based detection through a dual-channel architecture, and AdaDetectGPT [Zhou et al., 2025], which learns an adaptive witness function to improve log probabilities. Text Fluoroscopy [Yu et al., 2024] explores another methodology that exploits internal hidden representations of LMs for classification.

White-box detectors are typically zero-shot and easy to deploy. However, they rely on a less impractical scenario in which detectors have direct access to model logits. Most commercial LLM APIs do not expose logits, and many users lack the computational resources to deploy open-source models locally.

2.2 Black-boxMethods

Black-box detectors do not require access to the model or any proxy logits. Instead, they rely on observable textual signals. The way these methods process these features includes text truncation and continuation [Yang et al., 2023], grammar error correction [Wu et al., 2025] and regeneration [Mao et al., 2024].

Other works employ neural classifiers or fine-tuned language models. [Fagni et al., 2021] demonstrated that finetuned RoBERTa achieves state-of-the-art classification performance. [Hu et al., 2023] proposed RADAR, an adversarial learning framework that jointly trains a paraphraser and a detector to achieve robust AI-text detection.[Liu et al., 2024] constructed the GPABench dataset to study the detection of ChatGPT-generated academic content and trained an LSTMbased classifier. [Nguyen-Son et al., 2024] extended Raidar by incorporating multiple rewrites and ranking, followed by fine-tuned RoBERTa classification. [Chen et al., 2025] formulates AI-generated text detection as an inverse prompt reasoning problem, and detects LLM-generated text by inverting the generation prompt and validating its consistency with the input text.

Despite diverse efforts, both white-box and black-box methods face intrinsic limitations. As LLMs continue to evolve, it is close to the world where LLM-generated texts are indistinguishable from human-written text, risking many fundamental theories of existing black-box approaches and highlighting the need for new detection paradigms.

3 VaryBalance

We propose VaryBalance, a black-box LLM-generated text detector. As shown in Figure 2, VaryBalance consists of three key components: generation of rewritten texts, scoring texts, and prediction.

Generation of rewritten texts. We leverage the advanced linguistic capabilities of a modern LLM to generate rewritten variants of the input text. We name the LLM Rewriter. To minimize the effect of the prompt, we only instruct the rewriter with a simple prompt: Revise this text.

Scoring Texts. After the preparation of texts and their rewritten versions, a scorer is needed to transfer these texts to computable values for further comparison. VaryBalance employs Perplexity (PPL) as the quantification score. PPL measures how confused a model is in predicting the next token: lower perplexity indicates higher confidence and better predictive performance, making it a useful indicator for model-generated content, and has been widely used in quantifying texts [Solaiman et al., 2019; Mitchell et al., 2023; Bao et al., 2023; Yang et al., 2023; Hans et al., 2024]

Given a token sequence W = (w1,w2, . . . ,wn), the perplexity of a model M is defined as:

where PM(wi | w<i) denotes the conditional probability of token wi given its preceding context w<i, and n is the length of the token sequence. However, direct use of PPL is not ideal for a scoring-based detector for the instability introduced by the exponential operation. Instead, we adopt the logarithmic form log(PPL) as a more stable and interpretable component in our framework. VaryBalance employs a small LLM to compute the log(PPL) of the original texts and rewritten variants, according to Eq. (1).

Prediction. The next step is to quantify the difference between texts and their rewritten variants. VaryBalance integrates Mean Squared Deviation (MSD) for this quantification. Here, log(PPL) is used as the indicator, and the log perplexity of the original text log(PPL0) is selected as the central reference.

where log(PPLi) represents the log perplexities of rewritten texts, and n is the total number of rewritten texts.

We found that some machine-generated texts exhibit lower log perplexity scores than the rewritten texts, as they align more closely with the preferences of the relatively smaller scoring model. This results in inflated MSD values for some of LLM-generated texts. To address this issue, we introduce Eq. (3) to adjust the influence of MSD on the total score. With a negative sign, a larger MSD corresponds to a stronger indication of the LLM-generated text.

Finally, VaryBalance calculates the final score as:

We describe the influence of MSD on the original text's log perplexity using an exponential function, which allows human-written text to fully benefit from the amplification effect of MSD, thereby creating a clear separation from machine-generated text with similar log perplexity values. The score is then compared with a chosen threshold to classify LLM-generated text.

We also offer an expansion of the original VaryBalance score. We observe that for social media-style text, the gap in MSD between human-written text and the variance of its rewritten texts is substantially larger than that of machinegenerated text. This discrepancy can be leveraged to refine the original Score for social media texts for boosting the detection performance.

where Var is the variance, and ρ as the expansion coefficient that is applied to texts with a more informal style.

4 Preliminary Study

The proposed new method is motivated by the observation that, compared to LLM-generated texts, there is a greater difference between human texts and their rewritten version via LLMs. To systematically validate this assumption, we conduct a preliminary study.

Specifically, we first randomly sample 400 text pairs from the HC3 dataset [Guo et al., 2023]. Each pair in the dataset consists of a question with both a human-written answer and a machine-generated answer. Then, we use these questions as prompts to produce new machine-generated responses, and rewrite machine-generated responses and original human texts using LLMs. As a result, we have two groups of data, human-generated texts with their rewritten versions, and machine-generated texts with their rewritten versions. After that, we employ MSD to quantify the difference between texts and their rewritten versions.

We consider two types of LLMs for the rewritten, DeepSeek-chat and GPT-4o-mini, and two types of LLMs as scorers, Falcon and Qwen. Fig. 3 depicts the results. We can see that it is clear that the MSD of human text and its rewritten texts is greater than that of the counterpart of LLM-generated texts and their rewritten texts. The average MSD value of human texts is around 0.34, while that of machine texts is around 0.009. Moreover, we observe that 96% of total texts in the dataset follow this observation, which demonstrates that our assumption stands.

To assess the effectiveness of VaryBalance, we conduct comparative experiments across multiple languages, generation models, generation strategies, and application scenarios. It is worth noting that the part of the dataset used to design the VaryBalance scoring function is not involved in the performance evaluation of VaryBalance.

4.1 Datasets

Benchmarks. We evaluate our method on several datasets that are widely used in prior literature. Among them, a commonly adopted benchmark introduced by [Verma et al., 2024] consists of three datasets: Creative Writing, News, and Student Essay. For each dataset, the authors generate machinewritten counterparts of the human texts using gpt-3.5-turbo as the training set, while additional texts generated by Claude are provided as a held-out test set.

Datasets for Robustness Experiment We select three datasets to represent different genres of texts with different lengths. Drawing samples from dataset CNN [Hermann et al., 2015], reddit posts dataset RedditClean 1 and yelp review dataset YelpReviewFull [Zhang et al., 2015], we generate corresponding LLM-generated texts using Qwen3-Next80B-A3B-Instruct [Yang et al., 2025; Team, 2025] and grok4-fast 2 . All human texts were collected before the release of ChatGPT, which ensures that they are not contaminated by LLM-generated content.

Multi-language. We evaluate the multilingual capability of our detector using the M4 dataset[Wang et al., 2024], which contains texts in Chinese, Arabic, Urdu, and Bulgarian collected from multiple sources. We regenerate the machinegenerated texts with qwen3-235b-a22b-instruct-2507[Yang et al., 2025; Team, 2025] by following the prompts provided in the dataset, thereby better simulating outputs from more recent and capable large language models. To reduce bias towards non-native speaker, we additionally employ the EssayForum 3 dataset, which contains both original student-written essays and their grammar-corrected counterparts.

Rewrite. To evaluate the detector's ability to identify rewritten text, we construct a rewritten-text dataset based on the HC3 dataset[Guo et al., 2023] using two large language models, deepseek-chat and qwen3-235b-a22b-instruct-2507.

4.2 Baseline

Among the white-box detectors, we compare our detector with Fast-DetectGPT[Bao et al., 2023], Binoculars[Hans et al., 2024] and AdaDetectGPT [Zhou et al., 2025]. For blackbox detectors, we choose RADAR[Hu et al., 2023] and DNAGPT[Yang et al., 2023] as baselines. RADAR presents a neural-based detector trained through adversarial learning for robust AI-text detection, while DNA-GPT is mainly based on word-level features caused by text continuation through an LLM. All detectors are implemented in the setting of their original papers to avoid confounders.

4.3 Metric

Following previous works [Mitchell et al., 2023; Ma and Wang, 2024; Zhou et al., 2025], we evaluate the detection effectiveness using the area under the receiver operating characteristic curve (AUROC).AUROC indicates the probability of a detector to rank a randomly-selected positive sample higher than a negative counterpart, capturing the overall performance of the subject.

5 Experiment

In large-scale experiments, we employed Qwen3-14B as the scoring model with bfloat16 precision. All text rewriting and generation were performed under consistent settings: temperature=0.7, max tokens=1000, with each input text rewritten 10 times.

1 https://huggingface.co/datasets/SophieTr/reddit clean

2 https://x.ai/news/grok-4-fast

3 https://huggingface.co/datasets/nid989/EssayFroum-Dataset

5.1 Results

Benchmark Performance

As shown in Fig. 4, VaryBalance consistently outperforms DNA-GPT, RADAR, and AdaDetectGPT across most experimental settings, while achieving performance comparable to Fast-DetectGPT and Binoculars. When the generation model shifts to Claude, most detectors exhibit a slight performance degradation while VaryBalance maintains a high level of effectiveness and continues to surpass the baseline methods.

It is worth noting that the machine-generated portions of the Creative Writing, Essay, and News datasets are produced using gpt-3.5-turbo, which is relatively less capable and slower compared to more recent models. Evaluating performance solely on these datasets is insufficient.

Performance on Various Text Sources and Genres

In real-world applications, LLM-generated text exhibits substantial variation in length, style, domain, and generating models. Therefore, we evaluate our detector on the Reddit, Yelp, and CNN datasets, which represent different genres of texts with different lengths from different source models, enabling us to assess the detector's robustness to LLMgenerated content beyond the GPT and Claude model families. The result shown in Fig. 6 demonstrates that VaryBalance surpasses baselines by a substantial performance gap on Yelp and Reddit datasets while maintaining comparable effectiveness with baselines on CNN datasets. Combining the benchmark results, we observe that the baselines perform well on relatively formal and longer texts (e.g., mainstream news) but exhibit pronounced performance degradation on social media-style text. Our detector remains robust to the change in text genres with an AUROC of 0.949 on Reddit, 0.963 on Yelp, and 0.997 on CNN, surpassing other detectors by margins of 0.30.

In terms of the impact of generating models, we perform comparative evaluations on Reddit texts generated by three models, Grok, Qwen, and gpt-4o-mini(represented as GPT), with the results summarized in Table 1. VaryBalance outperforms baseline methods, with the expansion further enhancing its advantages, demonstrating robustness across generating models and text genres.

Multi-language Scenarios

Our detector nearly achieves the best performance across all evaluated languages. As shown in Table 2, on ChatGPTgenerated texts, VaryBalance attains state-of-the-art performance, with an AUROC of up to 0.986. On text generated by Qwen, VaryBalance consistently achieves AUROC values above 0.892, with the highest reaching 0.975, significantly surpassing the performance of other methods.

An important challenge in machine-generated text detection is that some detectors exhibit a bias toward classifying texts written by non-native English speakers as machinegenerated [Liang et al., 2023]. On the EssayForum dataset, VaryBalance achieves an accuracy of 0.98. Meanwhile, Figure. 5 shows that the score distributions of the original essay and grammar-corrected essay are highly similar, indicating that our detector does not exhibit bias against text written by non-native English speakers.

Performance on Rewritten Texts

Prior studies have shown that machine-rewritten text is particularly difficult to detect[Wahle et al., 2022; Pudasaini et al., 2024]. As shown in Table 3, VaryBalance significantly outperforms baselines: across three different models, it consistently surpasses other detectors, achieving AUROC scores above 0.82 in all cases. Moreover, VaryBalance with expansion further increases AUROC by 0.0256 on average across three settings.

Overall, the experimental results show that VaryBalance is robust to variations in text type, length, generation model, and language. It effectively identifies LLM-rewritten text and, through the incorporation of a single parameter, further enhances performance on shorter, social media-style texts, surpassing current state-of-the-art detection models.

6 Discussion

We investigate the contribution of each component in VaryBalance to its overall performance, including the scoring function, the number of rewrites, the text length, and the choice of the scoring model.

6.1 Size and Series of Scorers

We evaluate the impact of the scoring model on VaryBalance by selecting Qwen3 models with parameter sizes ranging from 0.6B to 14B, as well as several 7B models from other families, and applying the base VaryBalance framework across six datasets. As shown in the Fig. 7, on benchmarks, the choice of the scoring model has only a limited effect on performance, while a more pronounced performance gap shows on other datasets that were constructed using more recent models. On these datasets, we observe a clear trend of performance improvement with increasing model size. Among the 7B-scale models, DeepSeek-7B consistently outperforms Falcon-7B, while the slightly larger Qwen38B consistently surpasses both DeepSeek-7B and Falcon-7B. Notably, on the Reddit and benchmark datasets, even the smallest Qwen3-0.6B model outperforms DeepSeek-7B and Falcon-7B.

These results indicate that the choice of the scoring model has a significant impact on the performance of VaryBalance, encompassing both the selection of the model family and the model size. Due to computational constraints, we do not evaluate larger models; however, the experimental findings suggest that, when computational resources permit, larger Qwen models can further enhance the effectiveness of VaryBalance. The impact of model selection for the scorer is more pronounced when evaluated on texts generated by recent large language models.

6.2 Regeneration Times

The experiment shows that the number of regeneration times has only a marginal effect on overall performance, see Fig.8. We hypothesize that multiple rewritten outputs produced by the same LLM tend to have similar log perplexity values, resulting in low variance among the rewritten texts and consequently a stable MSD. Therefore, in practical applications, the number of rewrites can be selected flexibly according to API constraints without significantly affecting the performance of VaryBalance.

6.3 The influence of MSD

To verify the effectiveness of our modification to log PPL, we conduct comparative experiments on the Reddit, Essay, and Rewrite datasets, which respectively represent social media text, academic text, and machine-rewritten text. As shown in Table. 4, both variants of the VaryBalance score achieve better performance than log PPL score, and VaryBalance with expansion yields noticeable improvements on the Reddit and Rewrite dataset, while exhibiting a slight performance degradation on the Essay dataset. These results confirm the effectiveness of our modification to log PPL, and indicate that the impact of MSD is less pronounced for longer and more formal texts, and therefore its contribution should not be overly amplified in such settings.

7 Conclusion

We proposed VaryBalance, a black-box LLM-generated text detector based on the observation that the difference between the MSD of human text and machine text is significant. Extensive experiments show that VaryBalance surpasses SOTA detectors by 0.3 in terms of the AUROC score while keeping robustness against multiple genres, multiple generating models, and multiple languages.

Acknowledgments

This work was partly supported by the National Key Research and Development Program of China (2023YFB3107100).