Linguistic features of AI mis/disinformation and the detection limits of LLMs

Abstract

The persuasive capability of large language models (LLMs) in generating mis/disinformation is widely recognized, but the linguistic ambiguity of such content and inconsistent findings on LLM-based detection reveal unresolved risks in information governance. To address the lack of Chinese datasets, this study compiles two datasets of Chinese AI mis/disinformation generated by multi-lingual models involving deepfakes and cheapfakes. Through psycholinguistic and computational linguistic analyses, the quality modulation effects of eight language features (including sentiment, cognition, and personal concerns), along with toxicity scores and syntactic dependency distance differences, were discovered. Furthermore, key factors influencing zero-shot LLMs in comprehending and detecting AI mis/disinformation are examined. The results show that although implicit linguistic distinctions exist, the intrinsic detection capability of LLMs remains limited. Meanwhile, the quality modulation effects of AI mis/disinformation linguistic features may lead to the failure of AI mis/disinformation detectors. These findings highlight the major challenges of applying LLMs in information governance.

LLMs struggle to reliably identify mis/disinformation content, raising concerns about information governance. This study presents two Chinese datasets of AI-generated mis/disinformation and uncovers key linguistic features affecting their quality and detection.

Introduction

The widespread adoption of generative AI technology (GenAI) has provided new avenues for the creation and dissemination of mis/disinformation. Recent studies have demonstrated that LLM-based GenAI can produce deceptive content that is remarkably difficult to detect¹, presenting significant challenges to existing detection systems and identification models. While LLMs face limitations in fact-checking scenarios due to issues such as hallucination², lack of domain-specific knowledge³, and inaccurate citation retrieval^4,5, advanced models like GPT-4 and LLaMa are still seen as promising solutions for combating mis/disinformation, owing to their sophisticated language comprehension and analytical capabilities⁶. Some preliminary efforts, such as feature information fusion based on LLMs^7,8, prompt engineering^9,10, online retrieval-augmented generation (RAG)^11–13, and multi-agent collaboration^6,14,15, have been considered effective methods for addressing mis/disinformation in the GenAI era.

However, other studies have reached opposing conclusions^16–18. For instance, when LLMs are used to fabricate details or rewrite mis/disinformation, detecting AI-generated mis/disinformation becomes a significant challenge for both humans and existing methods^19,20. Meanwhile, Schlichtkrull et al.²¹ showed that online retrieval frameworks achieved a macro F1 of no more than 50% across various claim types. Tian et al.²², without accounting for data leakage, improved detection performance by approximately 20% using more advanced LLMs and RAG strategies, yet their F1 remained only around 71%. Moreover, Haupt et al.²³ revealed that prompts incorporating sociopolitical identities reduce ChatGPT’s detection accuracy from 68.1% to 29.3%, highlighting LLMs’ susceptibility to biases. These contradictions raise an unresolved question: given that conflicting conclusions are common across various LLM-based fact-checking approaches, what are the key factors and performance boundaries governing LLMs’ ability to detect mis/disinformation once framework differences are excluded? Therefore, building on the work of Augenstein et al.²⁴, this study aims to refocus attention on the LLMs themselves, thereby addressing the community’s concerns regarding the key challenges of involving LLMs in fact-checking tasks.

This work provides a comprehensive analysis of these contradictory phenomenon in existing research. It argues that examining the linguistic features of generative mis/disinformation and the capability of zero-shot offline detection is an effective way to explore the inherent limitations of LLMs and to explain the inconsistencies in prior conclusions. Research on linguistic feature fusion is similar to traditional fake news detection methods, where models are enhanced with features such as vocabulary, sentiment, and syntax in hopes of achieving better performance. The zero-shot prompting method aims to activate the mis/disinformation knowledge learned by LLMs and leverage their intrinsic understanding to identify fake content. However, these approaches implicitly rely on two critical assumptions: that the feature differences between AI-generated fake and authentic content resemble those observed in traditional human-crafted mis/disinformation, and that LLMs can understand and distinguish what constitutes mis/disinformation.

In this context, we argue that these assumptions need to be discussed more cautiously. Specifically, in the GenAI era, two fundamental research questions regarding LLM-based mis/disinformation need to be further investigation. The first research question (RQ1) seeks to understand why LLM-based linguistic feature fusion methods sometimes demonstrate limited effectiveness in detecting AI-generated mis/disinformation. To address this issue, it is essential to examine, first, the similarities and differences between AI-generated and human-authored real and fake content (RQ1a), and second, the latent linguistic and structural features that could potentially enhance the detection accuracy of AI-generated mis/disinformation (RQ1b). The second research question (RQ2) focuses on whether LLMs can reliably comprehend and distinguish AI-generated mis/disinformation under zero-shot prompting conditions. This inquiry involves exploring whether intrinsic differences exist in the language comprehension patterns of LLMs when generating or interpreting mis/disinformation (RQ2a), and further, identifying the critical boundaries and inherent limitations of LLMs’ capability to recognize and differentiate AI-generated mis/disinformation (RQ2b).

Similar to human-generated fake content, AI-generated mis/disinformation can likewise be categorized into two types based on its motivation²⁵. AI misinformation typically refers to erroneous content generated by using GenAI technology, which may arise from hallucination inherent to LLMs or from negligence in verifying the factual accuracy²⁶. AI disinformation refers to intentionally fabricated and manipulated fake content produced using GenAI, usually with subjective malicious intent²⁷. For researchers or internet users, the underlying intent behind erroneous content is often indeterminate, and therefore, these two concepts are often used interchangeably and collectively referred to as mis/disinformation. Referring to this broader conceptual category, fake content generated or processed by GenAI can be termed AI mis/disinformation, regardless of whether it involves subjective malice^28–30. This term will serve as the standard expression in this study. The scope of this work is restricted to textual data, namely, fake content generated by GenAI and content refined by GenAI on the basis of human-produced fake content. Specifically, two distinctly different Chinese corpora datasets were constructed in this study. From the perspective of content quality, fake and real text in AI-based deep-rewritten and low-quality content were studied separately from psycholinguistic and computational linguistic aspects. Furthermore, the limitations of LLMs’ recognition under zero-shot prompt conditions were further investigated using existing state-of-the-art (SOTA) open-source LLMs and closed-source GenAI services. The overall research workflow is illustrated in Fig. 1.

Fig. 1. The pipeline of this work.

Step 1. data compile, which includes four sub-steps from 1a to 1e, step 2. psycholinguistics, step 3. computational linguistics, and step 4. zero-shot experiments.

The main contributions of this work can be summarized as follows. First, two multi-source AI-generated mis/disinformation datasets, including text-based deepfakes and cheapfakes, were proposed, addressing the critical gap in Chinese data resources within existing research. Second, the distinguishing features between AI-generated fake and real content were revealed from psycholinguistic, toxicity-error patterns, syntactic dependency, and bigram part-of-speech (POS) sequences. Third, the quality modulation effects of over-standardization and lack of individuality in AI-generated content were identified for the first time, preliminarily revealing the mechanism by which differences between human creative language and AI-patterned language gradually emerge as text quality increases. Fourth, under zero-shot prompt conditions, the performance boundaries of offline LLMs in detecting multi-source AI-generated mis/disinformation were found to be related to fabrication quality, content quality, and text length.

Results

LIWC-based psycholinguistic study

Overall differences between datasets

Linguistic Inquiry and Word Count (LIWC) is an authoritative psycholinguistic lexicon widely used in NLP and misinformation research³¹. We employed the Chinese version of the LIWC as a psycholinguistic analysis tool. The average frequency differences across all psycholinguistic and additional features are presented in Table 1. Notably, AI and HM content exhibited significant differences in six dimensions: language style, affective processes, cognitive processes, personal concerns, swear words, and emoji usage. Overall, on MCFEND dataset, all psycholinguistic features showed higher mean frequencies in AI content than in HM content. This suggests that for cheaply fabricated or augmented texts, greater attention should be paid to features with smaller divergences from HM content, as they may be output by LLMs under stricter constraints.

Table 1.

Relative change rate of mean differences in LIWC features between AI and HM data

	TouTiao				MCFEND
Dimension	Diff_FK	P.	Diff_RL	P.	Diff_FK	P.	Diff_RL	P.
Language Styles	0.01	***	−3.00	***	438.75	***	338.15	***
negations	−11.46	***	−15.13	***	503.03	***	353.55	***
quantifier	15.27	***	7.85	***	575.64	***	436.86	***
number	−35.98	***	−40.40	***	20.88	***	28.48	***
tense	−0.31	***	−1.77	***	395.12	***	319.18	***
Social Processes	22.20	***	16.16	***	452.85	***	356.56	***
family	1.85	0.1874	0.42	***	226.76	***	197.16	***
friend	−13.35	0.3529	−19.37	***	171.11	***	213.23	***
humans	13.03	***	7.77	***	397.25	***	315.83	***
Affective Processes	50.55	***	43.87	***	523.24	***	444.74	***
positive emotions	54.76	***	55.32	***	572.24	***	507.97	***
negative emotions	32.22	***	22.54	***	408.35	***	320.95	***
Cognitive Processes	20.20	***	12.50	***	588.27	***	444.43	***
insight	18.45	***	9.85	***	565.16	***	437.47	***
causation	43.06	***	29.98	***	720.47	***	558.71	***
discrepancy	8.95	***	2.83	***	612.59	***	451.24	***
tentative	20.08	***	7.14	***	613.21	***	430.11	***
certainty	11.50	***	1.78	***	604.21	***	485.82	***
inclusive	34.56	***	28.60	***	611.91	***	469.14	***
exclusive	−26.99	***	−30.00	***	364.69	***	286.06	***
inhibition	50.75	***	30.21	***	415.88	***	283.84	***
Perceptual Processes	4.94	***	−0.83	***	330.83	***	339.04	***
seeing	11.40	***	5.12	***	182.62	***	236.23	***
hearing	−31.59	***	−30.75	***	329.78	***	247.01	***
feeling	4.88	0.3741	−7.17	0.1629	424.70	***	407.05	***
Biological Processes	4.70	0.0026	−9.66	***	250.59	***	214.44	***
body	−8.95	0.6155	−25.14	***	241.23	***	244.82	***
health	25.58	***	13.06	***	342.77	***	220.15	***
sexual	33.34	0.1609	10.10	0.6493	173.04	***	193.27	***
ingesting	−26.04	***	−33.44	***	218.09	***	208.60	***
Relativity	10.46	***	6.43	***	300.44	***	256.58	***
motion	1.06	***	0.79	***	383.48	***	276.13	***
space	18.71	***	13.79	***	395.78	***	312.22	***
time	1.31	***	−4.56	***	186.76	***	181.81	***
Personal Concerns	36.25	***	28.23	***	388.37	***	295.55	***
work	38.93	***	24.41	***	380.09	***	270.73	***
achievement	62.65	***	53.81	***	508.91	***	420.65	***
leisure	35.42	***	23.18	***	374.72	***	258.01	***
home	26.60	0.0308	14.86	0.0484	420.91	***	288.22	***
money	−5.82	0.0069	−0.34	0.1960	326.58	***	210.71	***
religion	24.80	***	32.29	***	219.85	***	194.23	***
death	−9.72	0.9943	−5.58	0.3196	176.30	***	147.05	***
Swear Word	−60.36	***	−62.14	***	70.53	0.4238	95.14	0.4835
Assent	−2.58	***	−8.85	***	461.58	***	433.35	***
Symbol	1.38	***	−8.06	***	260.60	***	194.61	***
Emoji	−86.15	0.8885	−85.03	0.0241	95.38	0.8444	14.03	0.6572

Where, P. refers to the results of two-tailed the Kolmogorov-Smirnov (K-S) test assessing the differences in $avg\_count{s}_t$ across different fact label groups. *** indicate significance levels of p < 0.001. Bold and underlined font colors highlight the top three largest and smallest AI-HM difference rates, respectively. On Toutiao, the sample sizes used by Diff_FK and Diff_RL are N = (633, 5083) and N = (1991, 14,368), while on MCFEND they are N = (9339, 9339) and N = (5387, 5387).

Differences in linguistic style primarily reflect the use of numerical terms. In Toutiao, the linguistic features of AI_FK and AI_RL texts are mostly lower than HM content, whereas the opposite trend is observed in MCFEND. This phenomenon stems from the AI-generated text augmentation process.

In affective processes, AI-generated content contains significantly more positive emotion words than human-generated content. Moreover, negative emotion words are more prevalent in mis/disinformation compared to real content, which aligns with prior research findings³². We believe that the elevated use of positive emotion words, even in fake content, may be a result of reinforcement learning from human feedback (RLHF). RLHF is used to reward models to output more favored and polite responses to humans, and existing research shows that it may lead to more positive emotional responses^33,34. However, it is noteworthy that AI still employs more negative emotion words in fake content than in real content, likely to attract readers and induce sharing. This behavior appears to be spontaneous, as it is not only observed in Toutiao, where AI content was rewritten or polished, but also evident in MCFEND.

In cognitive processes, substantial differences in cognitive word frequencies (e.g., causation, discrepancy, tentativeness, and certainty) are primarily found in MCFEND rather than Toutiao. Specifically, AI-generated fake short-texts contains significantly more causal, contradictory, and tentative language than the original human-written fake content. This likely reflects the cautious response style encouraged during AI training. Unfortunately, these divergences diminished in human-revised content.

In personal concerns, notable differences are mainly observed in Toutiao. It is worth noting that AI content has lower frequencies than human content only in terms related to money and death, while other positive or neutral topics occur more frequently. This aligns with LLMs’ safety alignment strategies and platform algorithms, which suppress socially contentious topics while promoting mainstream discourse. Thus, AI content (even factual) may reflect curated biases inherited from training data and amplified by platform dynamics, rather than objective deliberation.

Regarding swear words, AI content consistently contains fewer words than HM across all datasets and text types, in line with prior research³⁵. This aligns with LLMs’ design ethos of fostering polite, rational communication. Notably, in MCFEND, there is no statistically significant difference in the distribution of profanity between AI and HM. This may be partly due to the shorter text length in MCFEND, and partly because approximately 41.05% of its data originates from formal news websites in Supplementary Fig. 1.

In the additional category of symbol and emoji, the average frequency of symbol use shows little difference between AI and human content, although their distributional patterns differ substantially. Furthermore, AI tends to use emojis less frequently than humans in both datasets. However, we caution against overgeneralizing this trend, as platform type, prompts, or LLM may neutralize such differences.

Differences from quality perspective

The LIWC mean differences heatmaps and K-S test results across different quality groups were further calculated and visualized in Fig. 2.

Fig. 2. LIWC analysis and K-S test results across quality groups.

a The mean differences in LIWC linguistic features. These differences reflect the average-level distinctions in linguistic characteristics between AI and HM. The odd and even numbers on the x-axis correspond to FK and RL, respectively. b The differences in standard deviations (S.D.) of linguistic features. They indicate the stability of lexical usage patterns. The error bars on the scatter points represent the differences in S.D. from the MCFEND. c The distributional differences in psycholinguistic word usage between AI and HM across various quality and fact labels. It shows the distributional differences in psycholinguistic word usage between AI and HM across various quality and fact labels, capturing the overall patterns and variability of psycholinguistic characteristics. The horizontal and vertical axes represent the feature dimensions in two-tailed K-S testing and a simplified representation of p-values, respectively. NS means not significant.

As shown in Fig. 2a, differences in social processes, relativity, assent, and symbol are further exhibited quality-dependent variations. Since MCFEND consists of AI content based on short human-written texts, almost all differentiated linguistic features exhibit higher average usage frequencies in AI content than in HM texts. In social processes, AI content consistently emphasized human-related references (e.g., man, woman, and teenager) in both real and fake texts. For relativity, AI mis/disinformation contains motion, spatial, and temporal terms, which may be strategies to enhance emotional resonance and perceived authenticity. As for assent, it is more frequently used by AI in low- and average-quality FK content but is used even less than human content in high-quality Toutiao data. Notably, across all quality levels, AI tends to use a wider range of symbols than humans. In Toutiao, this may stem from AI content having a shorter average sentence length (ASL), while in MCFEND, it is likely a result of the text augmentation process. Equally important is the finding that in features exhibiting consistent differences across datasets and quality levels, the divergence between AI_FK and HM_FK is greater than that between their true counterparts. In other words, AI mis/disinformation is less human-like than AI_RL. This divergence cannot be explained by differences in ASL (Supplementary Fig. 8b), since HM_FK in Toutiao is actually longer than AI_FK.

This study aimed to obtain robust conclusions for RQ1 across datasets of diverse sources and styles. However, the prompts presented in Supplementary Table 1 may lead to a lack of representativeness in the cheaply generated MCFEND, potentially limiting the generalizability of the psycholinguistic findings. To mitigate this issue, a supplementary experiment was added. First, a basic prompt without any role or style constraints was presented in Supplementary Table 2. Second, considering that some independent creators might use research literature and open-source projects to obtain prompts to generate AI-real or AI-fake content, developer attack (Supplementary Table 3) and hybrid attack (Supplementary Table 4) prompts, based on the work of Li et al.³⁶ and Yu et al.³⁷, were incorporated. Following the same generation and processing procedure as the MCFEND, the MCFEND_base, MCFEND_dev, and MCFEND_hbd datasets were generated (Supplementary Table 5 and Supplementary Fig. 2) and were employed in the supplementary psycholinguistic experiments. Basic descriptions of the datasets and the experimental results are provided in Supplementary Table 6 and Supplementary Fig. 3, respectively. The integrated results of the supplementary experiments indicate that the results reported in Table 1 and Fig. 2 appear to be robust to prompt variations. This suggests that, whether the content is fake or real, the psycholinguistic features of AI-generated content may originate from the inherent linguistic patterns of LLMs.

Subsequently, Fig. 2c reveals a clear trend. As content quality increases, the K-S test results for all psycholinguistic features between AI and HM content (both RL and FK) become increasingly significant. This indicates widening distributional differences in LIWC patterns between AI and HM. This trend is observed consistently across the two different datasets and is particularly pronounced in Toutiao.

Based on the above analysis, three results can be summarized as follow:

• Fundamental Divergences. Eight core psycholinguistic dimensions (i.e., social, affective, cognitive, relativity, personal concern, assent, and symbol) showed significant AI and HM differences (both RL and FK).

• Mis/disinformation Distinctiveness. AI_FK demonstrated greater linguistic deviation from human patterns than AI_RL, suggesting GenAI amplifies distinctive psycholinguistic patterns in fake texts.

• Quality Modulation. As content quality increases, the lack of AI “individuality” becomes more pronounced. Regardless of whether the content is RL or FK, AI appears to aim toward producing high-quality outputs aligned with the average patterns in its training data. This is reflected in the fact that average linguistic differences between AI and HM (affected by the method of fabrication) decrease or increase with quality, while distributional differences (as measured by K-S) increase. In other words, in the low-quality group, both HM and AI texts demonstrate unstable and overlapping usage of LIWC features, leading to insignificant K-S results. As quality improves, humans exhibit unique creativity and diversity, while AI-generated texts increasingly reflect model-driven regularity and uniformity. This suggests that AI’s “high quality” be a self-consistent, standardized, and more predictable language.

Computational linguistic study

Content toxicity-error patterns, syntactic dependency distance (DD), and bigram POS analyses are discussed in detail in this section. The top 10 dependency parsing (DEP) and POS patterns across quality groups in each dataset are presented in Supplementary Figs. 4 and 5, respectively.

Linguistic toxicity-error patterns

Scatter plots of linguistic toxicity-error patterns across different quality groups for the two datasets, along with comparisons of average linguistic error counts and their distributional differences, are shown in Fig. 3.

Fig. 3. Language toxicity-error patterns of different quality groups on Toutiao and MCFEND.

a Toutiao AI content toxicity-error pattern. b Toutiao human content toxicity-error pattern. c Toutiao average error statistical difference. The sample sizes N of AI and HM on the three quality groups are (33, 2194), (44, 1209), and (2547, 16048), respectively. d MCFEND AI content toxicity-error pattern. e MCFEND human content toxicity-error pattern. f MCFEND average error statistical difference. The sample sizes N of AI and HM on the three quality groups are (586, 7332), (5281, 3178), and (8859, 4216), respectively. P. is the result of the K-S test for linguistic errors.

In summary, the findings suggest that linguistic errors can serve as a distinguishing feature for AI-generated content across diverse datasets. However, conclusions drawn by Wu et al.³⁸ and Mindner et al.³⁹ require more nuanced application contingent upon text type and quality levels. Specifically, while linguistic errors exhibit divergent patterns between AI and HM content across quality tiers, AI does not universally produce fewer linguistic errors than human authors in all text categories⁴⁰.

Syntactic dependency distance

Based on the DD data of all texts, the cumulative distribution function (CDF) curves for AI and HM content, along with their corresponding FK and RL subsets, are plotted in Fig. 4a, b. A steeper ascent in the CDF curve indicates that the dependency structures are more concentrated within shorter distances, reflecting more localized syntax.

Fig. 4. CDF curves and MDD differences across all datasets.

a The CDF curves on Toutiao. b The CDF curves on MCFEND. c The MDD differences on Toutiao. d The MDD differences on MCFEND.

First, due to the differences in text length between the two datasets, the CDF curve for the MCFEND exhibits a steeper ascent compared to Toutiao. This indicates that most content in MCFEND features shorter DD, corresponding to lower cognitive load during reading. Overall, when CDF ≤ 0.5 (i.e., below the median DD), the DD of all AI content is smaller than those of HM content. In other words, syntactic dependencies in AI texts tend to occur between adjacent words, reflecting that syntactic dependencies in AI texts predominantly occur between adjacent words. This phenomenon may stem from the next-token prediction objective of LLMs, which seeks to maximize linguistic probability and thus favors syntactically safer expressions with shorter dependencies. However, when CDF > 0.5, all CDF curves for AI content surpass those of HM content. This may result from LLMs learning word order probabilities through maximum likelihood estimation, whereas human language tends to minimize dependency distance. The sparsity of long-distance dependencies in LLM training data, hindering accurate distribution modeling. Furthermore, it is notable that the crossover point occurs at a DD value of approximately 63 in Toutiao, while in MCFEND, it occurs at a significantly higher DD of 122. This earlier crossover in Toutiao suggests that AI texts are more compact in syntactic structure and more similar to human writing. This finding further corroborates that AI content that performs AI rewriting and post-editing based on human text weakens its AI characteristics.

Second, it is observed that in both Toutiao and MCFEND, the HM_FK CDF is slightly higher than HM, while HM_RL is slightly lower than it. In contrast, no discernible difference between FK and RL curves versus the aggregate AI distribution. This suggests that during the generation process, LLMs do not differentiate between fake and real content in terms of dependency syntax. This finding may reflect an inherent reason why LLMs can be utilized to produce mis/disinformation, and it warrants attention from LLM developers.

Subsequently, the mean dependency distance (MDD) of text content across different quality groups in both datasets was plotted, as shown in Fig. 4c, d. Across the two datasets, the results reveal no consistent pattern in MDD frequencies between FK and RL content generated by AI and HM. For instance, HM content generally exhibits higher MDD values than AI content in the Toutiao, whereas this pattern is not observed in the MCFEND. This discrepancy stems from the imbalanced distributions of quality groups, fact labels, and AI and HM labels. Therefore, it is necessary to compare the trends in peak MDD frequencies across different quality groups and dataset types. Notably, as quality increases, the MDD peaks of all text categories consistently shift from 3 to 4, indicating that high-quality content universally features more complex syntactic dependencies and places higher cognitive demands on readers. Meanwhile, the fact that MDD peaks in high-quality texts never exceed 4 demonstrates that AI-generated content also conforms to established findings regarding human MDD characteristics⁴¹.

Bigram POS sequences

Based on the POS data of all texts, the bigram POS sequences across different quality groups of the Toutiao and MCFEND datasets are presented in Supplementary Figs. 6, 7, respectively.

Supplementary Fig. 6, it can be observed that the most frequent POS combinations in the Toutiao dataset are n-w (noun-punctuation) and d-v (adverb-verb). In AI content, n-w consistently occupies the highest average frequency across all quality groups, while in HM content, the frequency of n-w usage increases with quality. The n-w structure typically indicates higher information density and shorter sentence segmentation, as exemplified by phrases like “房价飙升“ (means house prices soar) and “AI终结产业“ (means AI disrupts the industry). This suggests that higher-quality human content is more adept at controlling the reading pace, creating suspense hooks, or evoking emotional resonance. Additionally, the u-n (auxiliary-noun) structure appears more frequently in higher-quality AI content. The u-n structure helps make noun expressions clearer and reduces colloquialism, indicating that superior AI content tends to adopt more formal written expression.

Supplementary Fig. 7 demonstrates that d-v and n-w remain the predominant POS combinations in MCFEND. However, the most frequent POS combinations in AI content across different quality groups almost mirror the opposite of what is seen in Toutiao. The frequency of n-w usage in HM content in low-quality groups differs from that of Toutiao. This suggests that the absolute value differences in bigram POS frequencies may not be sufficient to distinguish AI and HM. From the perspective of quality, u-n also appears more frequently in higher-quality AI content, consistent with the findings from Toutiao. Additionally, n-v (noun-verb) is more frequently used in higher-quality HM content, which typically represents natural subject-verb relations or instrumental actions. This indicates that short human texts tend to exhibit more refined logical chains and dynamic semantic progression.

Based on the above analysis, five key results can be drawn follow:

• Shorter DD and Lower Syntactic Complexity in AI Content. HM content demonstrates greater maximum syntactic span in complex sentences with long dependencies, while AI tends to employ simpler linguistic structures. This finding provides a potential explanation for the AI-style in machine-generated texts.

• Syntactic Indistinguishability of AI-Generated Fake and Real Content. Unlike human authors, LLMs show no fundamental difference in DD between FK and RL content generation.

• Syntactic Features Weakened by AI-based Rewriting and Human Post-editing. When AI-generated content is refined or rewritten based on human-written material, its distinct syntactic dependency distance features tend to diminish.

• Superior Discriminative Power of DD and Toxicity Features. Compared to MDD and linguistic error, DD and toxicity metrics prove more effective in distinguishing between AI and HM content.

• Formalization Trend in High-quality AI Content. As text quality improves, AI content shows a marked trend toward reduced colloquialism and increased clarity, logical coherence, and linguistic formality. These characteristics are typically associated with academic writing, legal and contractual documents, official communications, product manuals, technical documentation, and philosophical texts. This trend is consistent with findings from LIWC-based analyses.

Zero-shot prompt study of LLMs

To further validate the conclusions of existing research, as well as the findings related to psycholinguistics and computational linguistics in this work, an investigation into the identification performance of zero-shot prompt-based LLMs was conducted (Supplementary Table 7). Specifically, a total of 8 models (GPT-4o-Latest, Gemini-2.0-Exp, LLaMa-3.3-70B-Versatile, GLM-4-Flash, Qwen-2.5-72B, DeepSeek-V3, Grok-3, and Claude-3.7-Sonnet, denoted as M1 to M8) were employed to identify AI, FK, and AI_FK content. The recall (RE), precision (PR), and F1 for the recognition results of all models are presented in Table 2. All evaluation metrics were converted into percentage scores ranging from 0 to 1. Due to the imbalanced data samples within the Toutiao, the F1 score represents a weighted F1 (wF1) result, which may not fall between RE and PR. As shown in Table 2, DeepSeek-V3 performed best on the Toutiao, achieving the highest wF1 scores of 52.98% and 9.34% for FK and AI_FK, respectively. However, its wF1 for the AI label is 6.84 percentage points lower than the best-performing Claude-3.7-Sonnet, and also 20.75% below the average score across all models. On the MCFEND, Gemini-2.0-Exp delivered the best performance, attaining wF1 scores of 82.76% and 60.74% for AI and AI_FK, respectively. Similarly, its wF1 for the FK label is about 5.98 percentage points lower than the best-performing DeepSeek-V3. This suggests that under zero-shot prompting conditions, no single offline LLM can simultaneously handle the AI, FK, and AI_FK labels across different datasets.

Table 2.

Recognition results of AI and FK by existing SOTA LLMs

Dataset	LLMs	AI label			FK label			AI_FK
		RE	PR	(w)F1	RE	PR	(w)F1	RE	PR	(w)F1
Toutiao	GPT-4o-Latest	65.09	11.30	19.26	57.59	47.04	51.78	43.44	4.49	8.14
	Gemini-2.0-Exp	73.89	13.40	22.68	13.89	48.30	21.58	12.01	5.17	7.22
	LLaMa-3.3-70b-Versatile	62.08	13.25	21.84	43.19	50.76	46.67	28.59	4.71	8.09
	GLM-4-Falsh	85.10	11.62	20.45	61.04	45.96	52.44	54.98	4.95	9.08
	Qwen-2.5-72B	55.98	12.89	20.95	36.62	54.21	43.71	22.59	4.68	7.76
	DeepSeek-V3	31.90	11.81	17.24	58.21	48.62	52.98	41.07	5.27	9.34
	Grok-3	46.19	12.33	19.47	45.14	44.81	44.97	18.48	4.10	6.70
	Claude-3.7-Sonnet	93.06	13.83	24.08	69.61	41.14	51.72	71.88	4.82	9.04
MCFEND	GPT-4o-Latest	94.14	59.68	73.05	46.20	85.65	60.02	42.43	49.33	45.62
	Gemini-2.0-Exp	84.57	81.03	82.76	62.60	87.14	72.86	57.38	64.52	60.74
	LLaMa-3.3-70b-Versatile	68.46	83.78	75.35	53.59	85.22	65.80	45.09	65.28	53.34
	GLM-4-Falsh	91.53	62.18	74.05	55.59	80.49	65.77	36.15	42.67	39.14
	Qwen-2.5-72B	84.11	58.20	68.80	45.90	88.74	60.50	36.01	36.70	36.35
	DeepSeek-V3	82.51	63.40	71.70	75.82	82.11	78.84	72.30	44.53	55.12
	Grok-3	95.54	58.56	72.62	91.43	68.95	78.62	85.49	39.21	53.77
	Claude-3.7-Sonnet	78.72	54.42	64.36	72.96	78.58	75.67	72.83	44.09	54.93

Moreover, the average wF1 scores of all LLMs on Toutiao for AI, FK, and AI_FK are 20.75%, 45.73%, and 8.17%, respectively, all of which are lower than the wF1 scores of 72.84%, 69.76%, and 49.88% on MCFEND. Overall, current SOTA LLMs are nearly unusable on the more challenging Toutiao. It is important to note that by comparing the recognition performance of all models on AI, FK, and AI_FK, the critical factor influencing the accuracy of LLMs in identifying AI_FK content appears to be the dataset type. Specifically, for long-text and deepfake content in Toutiao, the challenge lies in identifying deepfake AI. Existing LLMs identify almost all human content as AI content, which results in their PR scores not exceeding 15% and indirectly reduces the overall recognition effect of AI_FK content. In contrast, for short-text and cheapfake in MCFEND, the challenge shifts to distinguishing FK content. The low recognition performance of LLMs on the short-text FK content in MCFEND indicates that it cannot be applied to practical mis/disinformation governance tasks.

Recognition results on different quality groups

The recognition results of existing SOTA LLMs for AI, FK and AI_FK content on different quality groups are shown in Fig. 5.

Fig. 5. Recognition results of all models.

a The recognition results of the model for Toutiao. b The results for MCFEND. From left to right for (a, b), are the results of AI, FK, and AI_FK, respectively.

The AI_FK identification results across the two datasets reveal that different models exhibit inconsistent performance in detecting AI_FK content of varying quality. However, the F1 scores for AI and FK content show that all models have a higher perception accuracy for high-quality AI content. For FK content, the above phenomenon is completely opposite; that is, low-quality FK content is more easily perceived by GenAI models. Based on the analysis of LIWC and bigrams, the underlying reason for this phenomenon is that higher-quality AI content, while having more complete contextual information, tends to lose individuality and lean toward formal, written expression. This makes high-quality AI content easier for LLMs to recognize. In contrast, low-quality FK content often lacks sufficient contextual information, has fewer supporting materials and arguments, and requires less cognitive effort. As a result, it is less deceptive than high-quality FK content and is easier to detect. Additionally, all models perform better in recognizing AI, FK, and AI_FK content in MCFEND than in long-text and deepfake in the Toutiao. This suggests that current LLM models, relying solely on zero-shot prompts, struggle to identify AI-generated fake content that has been rewritten or manually polished.

These findings highlight the challenges in leveraging LLMs’ inherent capabilities for AI_FK detection. However, these conclusions do not fully explain how specific linguistic features interfere with model performance. To address this gap, we further analyzed false detection samples across all datasets to identify the root causes of zero-shot detection failures in AI_FK cases. These insights will provide targeted directions for the improvement of feature fusion engineering.

DD of false detection samples

The CDF curves of DD for misclassified AI fake texts across different quality groups are plotted in Fig. 6. The CDF curves reveal that below the median, the DD of correctly predicted AI_FK content exhibits significant differences from HM content across all quality groups in both Toutiao and MCFEND. This is reflected in the wider separation between their CDF curves compared to misclassified AI_FK content, indicating that misclassified AI_FK demonstrates greater syntactic similarity to human writing in terms of DD. Above the median, however, the DD of misclassified AI_FK shows different trends across datasets.

Fig. 6. CDF curves of DD for correctly and incorrectly classified AI_FK on different quality groups.

a The CDF curves on Toutiao. b The CDF curves on MCFEND.

In the Toutiao, the proportion of DD that was misclassified in different quality groups was lower than that of AI text. And the DD is only slightly higher than those of human content. This indicates that, in AI-rewritten and human-polished long-text content with longer DD, the DD differences between AI and HM are reduced. In contrast, on MCFEND, the average-quality and high-quality misclassified and correctly classified AI_FK show CDF lower than HM, while other quality groups exhibit the opposite pattern. This implies that misclassified two quality groups AI_FK suffers from a more severe long-tail DD distribution than HM. Meanwhile, the DD distribution of high- and low-quality AI_FK was similar to that of humans. Therefore, it can be inferred that misclassified AI_FK content tends to exhibit syntactic dependency usage patterns more similar to human content (compared to AI content). Besides, a minority of syntactically complex long sentences potentially serving as one contributing factor influencing LLMs’ judgment of content authenticity.

Bigram POS of false detection samples

The top 5 bigram POS sequences of AI_FK texts are shown in Fig. 7. Across all datasets, there is virtually no difference in the top two most frequent POS combinations for AI_FK content. However, we observed that misclassified AI_FK content across different quality groups exhibits a higher occurrence of v-n (verb-noun) and v-w (verb-punctuation) combinations. In the Chinese, the v-n pattern is a fundamental syntactic structure, indicating clear sentence organization and a textual preference for directly expressing actions and their objects. The v-w pattern typically appears in short, colloquial, emotionally charged, or frequently segmented sentences. The increased frequency of these two POS combinations suggests that AI_FK texts consistently misclassified by all models tend to exhibit informal linguistic features that align more closely with human expression patterns.

Fig. 7. Top 5 bigram POS sequences of correctly and incorrectly classified AI_FK.

a The results on Toutiao. b The results on MCFEND. The vertical axis data of all subgraphs is the average frequency of the group of data.

Based on the above analysis, three findings were identified:

• Persistent Challenge of Short-Text Fake Content Detection. Analysis of MCFEND misclassified samples shows that identifying FK in short-text remains a key research challenge in the GenAI era.

• Detection Performance Dependent on Content Quality. LLMs demonstrate higher accuracy in detecting both high-quality AI content and low-quality FK text.

• Robustness of AI-Rewritten and Human Post-Edited Content. When AI mimics human expression patterns or when human-edited AI text adopts informal, cognitively economical language, current LLMs under zero-shot prompting fail to distinguish AI_FK from genuine content. This suggests the formality of AI_FK no longer serves as a reliable distinguishing feature.

Discussion

This work analyzes the language characteristics of Chinese AI mis/disinformation and the performance boundaries of LLMs in its identification through three experiments. The results, impacts, and limitations are as follows.

RQ1 results

The differences between AI-generated (both real and fake) content and human-authored content, as revealed by RQ1a, are summarized as follows. According to the results of LIWC analysis, AI and HM content exhibit significant differences across eight dimensions: linguistic, social, affective, cognitive, relativity, personal concerns, assent, and symbol features. Furthermore, a quality moderation effect has been identified in AI content, characterized by excessive standardization and lack of individuality. Specifically, as text quality improves, the psycholinguistic diversity and colloquial expression in AI content tend to decline in favor of greater normativity and logical coherence. In addition, compared to HM content, AI texts tend to have shorter syntactic dependency distances, more concentrated distributions of syntactic complexity, and fewer instances of extremely complex long-distance dependency structures.

Features that may enhance the detection accuracy of AI_FK content (i.e., RQ1b findings) can be categorized into two aspects. Regarding psycholinguistic features, the divergence between AI_FK and HM_FK content is substantially greater than that between AI_RL and AI_RL. This discrepancy remains statistically significant across both deepfake and cheapfake AI-generated inauthentic content. Consequently, the aforementioned eight significant LIWC features can be leveraged for AI mis/disinformation detection tasks. In computational linguistic features, toxicity scores and dependency distance demonstrate greater potential than grammatical errors and mean dependency distance in distinguishing AI_FK content.

Unfortunately, when AI mis/disinformation undergoes rewriting or paraphrasing based on human-authored content, the LIWC and dependency distance distinctions between AI and HM content become increasingly blurred. Additionally, influenced by training data and RLHF, high-quality AI mis/disinformation tends to approximate human content in an average sense. As a result, linguistic feature-based detection methods prove ineffective against deepfake or high-quality AI mis/disinformation. These two factors are proposed in this work as potential explanations for the limited efficacy of hybrid linguistic detection approaches.

RQ2 results

According to the above experimental results, there is no significant difference between AI_FK and AI_RL content in terms of bigram POS combinations or syntactic dependency patterns. Meanwhile, all tested LLMs achieved average F1 of only 45.73% and 69.76% for FK content detection across the two datasets. This concerning result may even include cases where parts of the data existed in the LLMs’ training corpora. This suggests that, for both input and output, current LLMs exhibit only limited capacity to understand mis/disinformation in offline and zero-shot prompt settings (i.e., RQ2a). This highlights deficiencies in the current training processes of LLMs and warrants attention from model developers.

Moreover, the study identifies text length and content quality as significant factors influencing LLMs’ perception of AI_FK. Specifically, high-quality AI_FK and low-quality HM_FK are more readily detectable by LLMs, while short-text mis/disinformation remains a persistent challenge in GenAI era deception research. Notably, when AI mimics human stylistic habits or when humans post-edit AI text into informal or cognitively economical expressions, current LLMs are unable to distinguish AI_FK from other texts under zero-shot prompting conditions. Therefore, text length and content quality are regarded as key performance boundaries that affect LLMs’ ability to detect AI mis/disinformation (i.e., RQ2b).

As a result, this study concludes that offline LLMs under zero-shot prompting conditions cannot reliably leverage their intrinsic knowledge to understand or identify AI mis/disinformation. At the current stage, in addition to challenges such as hallucinations and privacy concerns, insufficient understanding of mis/disinformation is also a critical issue that needs to be addressed in LLM training.

Research impact

From a quality perspective, this work explores the risks associated with employing LLMs for AI mis/disinformation governance through psycholinguistic and computational linguistic analyses, as well as zero-shot prompting experiments. The findings hold practical significance and implications for researchers, social media platforms, and governing authorities alike.

First, this work contributes Chinese corpus to the field of AI mis/disinformation research. Similar to the works of Wang et al.⁴², and Chen and Shu⁴³, we provide two AI mis/disinformation datasets created by multiple LLMs based on multi-source human-written fake content. However, unlike previous studies, this work uses human-generated mis/disinformation from the Chinese internet. Our findings reveal that under zero-shot prompting conditions, offline LLMs struggle to distinguish AI content from HM content, as well as AI_FK from other types of content at the linguistic level. This outcome helps explain the divergence in prior research, where some studies expressed optimism about feature-fusion-based detectors^30,38,44,45, while others demonstrate the opposite results^18,46. This contradictory evidence provides researchers with a more nuanced understanding of how to better leverage linguistic features to improve detection systems. Moreover, this work reinforces the conclusions of Wu et al.³⁸ and Mindner et al.³⁹, who found that linguistic errors could serve as cues for detecting AI content. However, we suggest these findings must be applied more cautiously, as their efficacy depends heavily on corpus-specific differences. This insight also contributes to mitigating the potential negative impact of feature fusion engineering on detector performance.

Second, this work is the first to identify a quality modulation effect in AI-generated content. Specifically, as content quality increases, several distinguishing characteristics between AI and human content become progressively more pronounced. These include creativity differentiation, content standardization, and diminished detectability. Although LLMs initially demonstrate a generalized capacity for language acquisition, the creativity-differentiation feature reflects a convergence trend in the psycholinguistic characteristics of AI-generated mis/disinformation as content quality improves. While the argument from the poverty of the stimulus (APS), which attributes language acquisition ability to human brain structure and innate endowment, remains inconclusive⁴⁷, this trend reinforces the view of Lan et al.⁴⁸ that neural networks are trained on linguistically insufficient corpora. It also indirectly supports the findings of Mischler et al.⁴⁹, who reported convergence in language processing strategies among high-performing LLMs, and aligns with Mahowald et al.'s⁵⁰ conclusions regarding the functional linguistic limitations of LLMs. The content-standardization feature captures the normalization trend in syntax and style of AI content as quality increases. This trend is observed in both AI-real and AI-fake texts. Exemplar theory in psycholinguistics posits that human language use relies on memory of specific instances, with high-frequency exemplars being more readily accessed⁵¹. Since LLMs learn conditional probabilities over large-scale datasets to approximate instance distributions, the high-quality written corpora they are trained on, such as filtered books, scholarly articles, and encyclopedias, likely act as potential factors driving linguistic normalization⁵². This trend also suggests, based on Biber’s register theory, that adjusting and optimizing LLM training at the level of language style could be beneficial⁵³. The diminished detectability feature further illustrates the complexity of AI mis/disinformation detection. As content quality improves, LLM-based detectors show performance gains consistent with trends in creativity differentiation and content standardization. At the same time, they reveal a lack of consistent robustness in distinguishing AI-real from AI-fake. Considering that this study evaluates information quality along dimensions of clarity, specificity, and coherence, processing fluency theory appears to apply not only to human readers but also to LLMs as simulated readers^50,54. In other words, as content quality increases, LLMs may exhibit a processing fluency effect similar to humans, making them more likely to accept generated claims as credible. Although the preliminary findings on quality modulation effects require validation across a broader range of models and corpora, they advance the integration of psycholinguistics and sociolinguistics into machine language research and provide new directions and early-warning signals for mis/disinformation governance.

Third, this study identifies critical limitations in using LLMs for AI mis/disinformation governance. While LLMs show psycholinguistic differences when generating both fake and real content, they exhibit no significant variation in syntactic dependency or bigram POS patterns. This suggests that producing mis/disinformation poses no fundamental distinction for LLMs, despite their awareness of content that could form false narratives or mislead audiences. This limitation goes beyond hallucinations post-training or pre-training data issues. It reflects flaws in the training objectives and evaluation strategies regarding LLMs’ fidelity to human input. Unfortunately, research on this benchmark is still in its early stages. Additionally, our findings on LLMs’ inability to perceive AI_FK under zero-shot prompting align with existing studies^39,55,56, indicating that current LLMs inherently struggle to AI-rewritten and human-post-edited AI mis/disinformation. This implies that the effectiveness of adding retrieval-based or agent-discussion frameworks to LLMs for this purpose may be inherently constrained⁵⁷. Our work highlights the pressing need for LLMs specifically trained for mis/disinformation detection tasks.

AI mis/disinformation governance is one of the most crucial responsibilities for authorities in the GenAI era. We believe this study helps address several key obstacles in current research, paving the way for more effective detection models tailored to specific languages and content quality levels. The conclusions not only offer valuable insights for researchers but also provide actionable evidence for governments to enhance cyber-environment management strategies.

Limitations and future research

This work contributes to the exploration of beneficial features and potential risks in the governance of AI mis/disinformation while inspiring future research directions. Nonetheless, this work still has certain limitations and requires further research in the future.

First, the study employs a binary true/false labeling scheme, but this inevitably simplifies the complexity of disinformation. In practice, many cases involve reasoning errors or logical fallacies, and fact-checkers often rely on situational judgments beyond factual correctness. Future work can explore more detailed labeling schemes to better capture potential differences in AI mis/disinformation. Second, this work does not account for mis/disinformation in image, video, or audio formats, which may result in some potential characteristics remaining unidentified. Future research should expand to include a wider range of content modalities. Meanwhile, the earliest data in the MCFEND dates back to December 2010, whereas the earliest data in Toutiao originates from September 2023. This implies that some models, particularly non-open-source models such as GPT-4o-Latest and Claude-3.7-Sonnet, may have training data that potentially overlaps with MCFEND or Toutiao, which could introduce slight biases in the zero-shot experimental results. Moreover, although the deepfake-based Toutiao is derived from real-world content, the cheaply generated datasets augmented from MCFEND using four prompt strategies are still subject to representativeness issues. Therefore, the robustness of the RQ1 conclusions should be more extensively validated across a wider variety of datasets and models. At last, although we identified key linguistic features and performance boundaries, we did not integrate these features to examine whether they could collectively lead to improved detection performance. This limitation arises from the high cost of fine-tuning LLMs, which constrained our experimental design and should be addressed in future work.

Methods

This study implemented three complementary experiments using Python. For RQ1a, a psycholinguistic analysis based on the LIWC lexicon was conducted; for RQ1a, RQ1b, and RQ2a, a computational linguistic analysis examining language errors, toxicity, syntactic DD, and bigram POS was performed; and for RQ2a and RQ2b, zero-shot recognition and error sample analysis using LLMs were carried out. The scientific methods employed in the three experiments are briefly described in this section.

Data acquisition and processing

Hotspot information typically attracts higher readership and broader dissemination due to its superior information and topic quality. However, if such content contains AI mis/disinformation, it may lead to more severe public opinion consequences. While lower-quality AI-generated cheapfake has a smaller propagation range, its production costs are significantly lower and it carries greater potential risks. It is undeniable that, driven by the principle of minimum effort for maximum benefit, both quality levels of AI mis/disinformation coexist on internet social platforms. Research focusing solely on one category would inevitably lead to cognitive bias among researchers regarding AI-generated real and fake content. Therefore, this study has constructed two datasets representative of both mis/disinformation types in the following subsection.

Fact-checking sources

As the first step to compile AI mis/disinformation, we identified authoritative fact-checking sources for Chinese mis/disinformation corpora by referring to existing research^58,59. These include the China Internet joint rumor-refuting platform (https://www.piyao.org.cn) and the science facts platform (https://piyao.kepuchina.cn), which are established by national authorities and news verification departments. As well as the Baidu rumor-refuting platform (https://author.baidu.com/home/15060) and Weibo rumor-refuting accounts (https://www.weibo.com/u/1866405545). A total of 17855 fact-checking records from January 2019 to January 2025 were collected using web crawlers. After data cleaning (i.e., removing duplicates, government policies, historical introductions, and invalid content), 8181 valid fact-checked entries were retained. In addition, for Chinese-language content related to international news, two international fact-checking platforms (https://www.factcheck.org and https://www.politifact.com) were also incorporated into the verification process. In summary, six credible fact-checking sources served as the primary basis for annotating Chinese text data in this work.

Data collection and generation

Toutiao, developed by ByteDance, is one of the largest self-media platforms in China. Compared with TikTok, its primary focus lies in long-form text-image articles content⁶⁰. In this study, 22102 high-profile news items from the trending section of the Toutiao platform, dated between September 2023 and January 2025, were collected by using web crawlers. After removing invalid entries caused by network anomalies, a total of 22075 valid long-text news articles were retained, referred to hereafter as the Toutiao dataset.

In addition, based on the MCFEND proposed by Li et al.⁶¹, a human misinformation dataset of different sources, including social platforms, instant messaging applications and traditional online news organizations, a multi-source short-text AI mis/disinformation dataset was constructed. Specifically, we selected non-empty entries from the news data and removed duplicates, resulting in 5448 real and 9339 fake text samples sourced from eight platforms (e.g., Weibo, NetEase, Tencent News, and so on). Five models were employed to generate AI content based on MCFEND: GPT-4o-Latest⁶², DeepSeek-V3⁶³, Moonshot-V1⁶⁴, Doubao-1.5-Pro-32k⁶⁵, and Claude-3.7-Sonnet⁶⁶. Due to safety restrictions in some LLMs and AIGC services, some entries in the MCFEND dataset failed to generate responses. After removing unresponsive and invalid outputs, a total of 29452 valid samples (10774 fake and 18678 real) were obtained from MCFEND. This dataset is called MCFEND in this work. Since Li et al.⁶¹ had already labeled human-generated rumors and genuine content, the AI-generated real and fake content obtained through LLMs inherits these labels.

In this work, AI-generated mis/disinformation and real content are labeled as AI_FK and AI_RL, respectively, while their human-generated counterparts are labeled as HM_FK and HM_RL. The following sections will briefly describe the annotation process of the Toutiao dataset.

Data labeling and quality grouping

First, the generation labels for the Toutiao Dataset were annotated. Specifically, a labeling team consisting of two researchers and one NLP expert manually classified all 22075 texts as either AI-generated or human-written, marked numerically as 1 and 0, respectively. All team members strictly adhered to a unified annotation guideline, which states that “any content with suspected AI-generated portions should be labeled as AI in its entirety.” Notably, during the process, the two researchers identified three common types of AI-generated content on the Toutiao platform: i) AI-rewritten versions of human-authored originals with original comments repurposed as supporting images, ii) AI-generated rumor-refuting content based on official government debunking reports, and iii) content featuring AI-generated attention-grabbing introductions and grandiose conclusions while maintaining human-written core text.

Second, the fact labels in the Toutiao dataset were assigned through a two-stage process. A CoT style prompt (Supplementary Table 8) was designed to instruct GPT-4o in extracting all claim statements from the texts and verifying each claim’s truthfulness using search engine queries. Similarly, fake and real content were labeled as 1 and 0, respectively. Unfortunately, we observed that despite GPT-4o’s ability to return claim statements from the texts, 3.91% of its judgments directly referenced example links from the prompt (Supplementary Table 9), and 27.10% of the claims remained unverifiable even after web-based retrieval. This issue further confirms the views of Venkit et al.⁵ and Augenstein et al.²⁴ that under external retrieval conditions, the current LLMs may encounter limitations such as hallucinations and erroneous citations when applied to fact-checking tasks. Consequently, the labeling team manually cross-verified each claim against results from the six authoritative fact-checking organizations, thereby refining the veracity labels of the Toutiao. Through these two processes, the Toutiao was equipped with comprehensive generation and fact labels.

Following the methodology of Xu et al.⁶⁷, this study evaluated content quality of both the Toutiao and MCFEND datasets across three dimensions: clarity, specificity, and coherence. Quality was initially rated on a five-level scale: very poor, poor, average, good, and excellent. However, since the proportions of very poor and excellent ratings did not exceed 5% across different content types in both datasets, the scale was consolidated into poor, average, and good. During this process, two foundational models (i.e., Qwen2.5-14B and DeepSeek-V3) were employed. On the Toutiao dataset, the Cohen’s Kappa coefficient between the two models was 0.88. For the MCFEND dataset, the coefficients were 0.86 for both HM_RL and AI_RL texts, and 0.89 and 0.86 for HM_FK and AI_FK texts, respectively. Discrepancies between model judgments were resolved by an NLP expert to establish definitive quality labels.

In summary, this study utilizes two highly distinct datasets. On one hand, the Toutiao comprises long-text self-media news from real-world trending rankings, where AI-generated content is typically based on human-authored texts and further modified by content publishers. On the other hand, the MCFEND is a short-text dataset from mixed media sources, with AI content directly generated and expanded through specialized prompts across multiple GenAI models. By investigating texts of varying types and quality, this study aims to uncover the common linguistic and structural features that distinguish AI_RL and AI_FK content from human-authored text.

Data statistics

The label distribution of the datasets used in this work is presented in Supplementary Table 11. In the Toutiao, the proportions of AI and FK are 11.89% and 25.89%, respectively, with average-quality texts accounting for the smallest proportion among all quality labels. In the MCFEND, AI and FK labels account for 50% and 63.42%, respectively, with low-quality content representing the smallest portion. The distribution of FK and RL content generated by both HM and AI is shown in Supplementary Fig. 8a. The distribution of AI_FK and AI_RL in Toutiao’s trending content reflects real-world sample imbalance.

The ASL of all texts was also calculated. Specifically, the Jieba library was used for Chinese word segmentation, and punctuation marks such as periods, exclamation points, and question marks were used to divide the sentences. The ASL of each text was computed by dividing the number of words by the number of sentences. The ASL results of the two datasets are shown in Supplementary Fig. 8b, and it can be seen that the long-text Toutiao content has longer ASL. Moreover, in the Toutiao, HM texts have higher ASL than AI ones, whereas the opposite trend is observed in MCFEND. This discrepancy may stem from the fact that, despite MCFEND being multi-sourced, its contributing platforms are primarily microblogging-based. This further reflects the potential difference between long-text AI mis/disinformation based on deepfakes and short-text AI mis/disinformation based on cheapfakes.

From the perspective of text quality, Fig. 8 illustrates the overall text length (including both the title and the main content) across different quality levels. Overall, regardless of whether the content is FK or RL, and regardless of whether it comes from the long-text Toutiao dataset or the short-text MCFEND dataset, higher-quality texts tend to be longer. This trend is particularly pronounced in MCFEND’s cheaply fabricated AI short-texts. In the Toutiao, both FK and RL content authored by humans appear more frequently across all quality levels than AI-generated texts. One possible explanation is that Toutiao collects content from widely circulated trending data, and the platform’s recommendation algorithm may still favor human-written content, leading to an imbalanced distribution of AI-labeled texts. Although it may be affected by the platform recommendation algorithm, it is worth mentioning that the underreporting rate of AI-generated content on Toutiao reached 47.90%, which is not a good signal for the governance of AI mis/disinformation.

Fig. 8. Text frequency and average content length in different quality groups of all datasets.

Both the original length and the generated length include the title and body text. The “chars” refers to a measure of character length.

Based on the above analysis, the Toutiao and MCFEND datasets constructed in this study respectively represent long-text deepfake and short-text cheapfake content, exhibiting clear distinctions and strong representativeness. This study aims to achieve consistent analytical results across two datasets with distinctly different styles. For each sample in both datasets, the title and content are concatenated into a single text using period symbols, and these integrated texts are utilized for psycholinguistic and computational linguistic study.

LIWC dictionary

This study considered 10 psycholinguistic features while incorporating two additional statistical measures. That is, language style, social, affective, cognitive, perceptual and biological processes, as well as relativity, personal concerns, swear words, assent, symbol, and emoji. The vocabulary categories and example Chinese words covered by all analyzed linguistic features can be found in Supplementary Table 10. For any $toke{n}_i$ in sentence $s_i$, we checked its inclusion in the 12 linguistic categories to derive the feature frequency $count{s}_{t}t\in \left[1,12\right]$ across all texts, along with the mean frequency $avg\_count{s}_t$ under each fact label. Based on the work of Zhou et al.⁴⁴, we further calculated the differences in average frequencies between AI and HM across the features, denoted as $Diff=AIavg\_count{s}_t-HMavg\_count{s}_t$. Subsequently, a two-tailed K-S test for continuous data was employed to analyze distributional differences of each linguistic feature $Diff$ within the same fact-label category, and the calculations were performed using the ks_2samp function from the SciPy library.

Linguistic toxicity and errors

Content toxicity was intended to assess the degree of aggressiveness, rudeness, and hatefulness in a text. In this study, it was measured using the Perspective API, a well-known text bias research product developed by the Google team. The score is defined as a probabilistic score ranging from 0 to 1, where a higher score indicates a higher overall level of toxicity in the text. State Cloud was a core NLP service provided by China Telecom, one of the largest telecommunication operators in China. It was employed to identify linguistic errors in the text, including grammatical mistakes, visually similar characters, phonetically similar characters, and missing characters. Linguistic error analysis yields a list containing original erroneous characters and their corrected counterparts, where the length of the list reflects the total number of linguistic errors across all sentences in the text. A two-tailed K-S test was used to analyze distributional differences in the number of linguistic errors between AI- and human-generated texts across different quality groups. In addition, the trend of linguistic toxicity and error with quality, i.e., linguistic toxicity-error patterns, will be studied in this work to discuss the views of Wu et al.³⁸ and Mindner et al.³⁹ regarding the role of grammatical errors in AI content identification.

DEP and bigram POS

DEP is a computational linguistics analysis method that represents the dependency relationships between words in a tree structure. DEP analysis was conducted using HanLP, an open-source project widely applied in the field of Chinese text analysis⁶⁸. DD the linear positional difference between two words in a dependency relation, while the MDD is the mean value of these linear positional differences⁴¹. It is important to note that punctuation marks and root tokens were excluded from the computation of DD and MDD. An n-gram is a sequence of n words, and when n is set to 2 it is called a bigram. HLP is also used to analyze the POS of all datasets, and the bigram POS sequences are counted using the CountVectorizer function in scikit-learn. The resulting DD, MDD, and bigram POS sequences were utilized in the subsequent analyses.

Zero-shot prompt

Zero-shot prompting has been widely used to elicit the intrinsic capabilities of LLMs for downstream tasks. In this study, a structured zero-shot prompt (Supplementary Table 1) was designed to generate Chinese AI-real and AI-fake content on existing datasets. The prompt guided the model to produce content in the persona of a social media influencer, using role definitions, writing skill constraints, and output format specifications, thereby ensuring that the generated text closely resembled authentic social media content in both style and structure. To ensure the generalizability of the conclusions in RQ1a, a basic prompt without role or contextual constraints, the developer attack prompt proposed by Li et al.³⁶, and the hybrid attack prompt introduced by Yu et al.³⁷ (shown in Supplementary Table 2 to 4) are also used for AI content generation and experimental control.

Subsequently, a structured CoT prompt (Supplementary Table 7) was proposed and applied to study the zero-shot capability of LLMs in identifying AI-generated mis/disinformation. This prompt guided the model to simultaneously judge both generation and factual labels through an explicit reasoning process and stepwise instructions. Specifically, the proposed prompt incorporated the following reasoning steps: i) identifying the text’s structure and type; ii) extracting and assessing the veracity of all claim statements within the text, whereby the presence of any false statement would result in the text being classified as mis/disinformation; and iii) combining the understanding of AI-generated content to produce a binary classification output in list format. To avoid ambiguity, the prompt defined boundaries for non-fake content (e.g., nine counterexamples such as fables and artistic creations) and required the model to maintain consistent judgments when encountering ambiguous texts. Through this structured CoT prompt, systematic and reproducible reasoning is ensured without relying on external examples, while minimizing interference from unnecessary free-form generation.

Author contributions

Y.M. is the first author, X.Z. is the second and corresponding author of this article. X.Z. and M.W. acquired funding for the studies. Y.M. and J.R. conceptualized the study. Y.M. drafted the article. Y.C. provided feedback for the experiment. X.Z., J.R., R.Z. and M.W. all contributed to the review and revision process, improving the writing, data interpretation, and literature analysis.

Peer review

Peer review information

Nature Communications thanks Antonio Moreno-Ortiz, Chun-Ming Lai and the other anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Data availability

The raw Toutiao and MCFEND data are available under restricted access because they may be used to train harmful artificial intelligence systems. Access can be obtained by submitting a handwritten ethical statement, as illustrated in Supplementary Table 12, to the corresponding author. The processed Toutiao and MCFEND data are available at https://github.com/GovAIx/QualityModulation.

Code availability

Code will be made available on request. Experiments in psycholinguistics and computational linguistics were conducted using open-source projects stated in the methods section, and the closed-source LLMs were accessed through provider-supplied APIs. The code required to reproduce the experiments can be obtained from https://github.com/GovAIx/QualityModulation.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41467-025-67145-1.