Measuring LLM lexical tics: excess vocabulary, baselines, drift
Parent: Writing and Documentation · Published reference · snapshot 2026-09-23 · skill writing-expert/references/tic-measurement.md
↓ Facts as markdownall context files
Source-anchored research for Measuring LLM lexical tics: excess vocabulary, baselines, drift. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- Wikipedia's guide groups AI vocabulary by era (2023-mid 2024 GPT-4; mid 2024-mid 2025 GPT-4o; mid 2025 onward GPT-5) and carries a banner saying parts about the most recent models need updating. (low confidence) (verified as of 2026-09-23) [source]
How it works
- Kobak et al. build a counterfactual baseline by linear extrapolation from earlier years (q = p_{Y-2} + 2*max{p_{Y-2}-p_{Y-3},0}) and deliberately exclude 2023 because it may already be LLM-affected. (medium confidence) [source]
- The paper's model lineup changed between revisions (v2 lists GPT-5.4, Claude Opus 4.7, Grok 4.2, DeepSeek V3.2; latest lists GPT-5.5, Claude Opus 4.8, Grok 4.3, DeepSeek V4 Pro), so tic measurements are tied to a specific model version and get re-run when versions ship. (medium confidence) (verified as of 2026-07-05) [source]
- Juzek and Ward (COLING 2025) found no evidence that overrepresented words like 'delve' stem from model architecture, algorithm choice or training data, and tested RLHF as a candidate cause, so the mechanism behind version-to-version drift is unresolved. (low confidence) [source]
Measurements and reference values
- Headline estimate: lower bound of at least 13.5% of 2024 PubMed abstracts processed with LLMs, ranging roughly 5-40% across fields and countries. Earlier preprint versions reported 10%; the figure is a lower bound and moved between versions. (low confidence) (verified as of 2025-07-02) [source]
- Control by word class: authors hand-annotated 900 excess words from 2013-2024 as content (about 51%) or style (about 45%); Covid-era excess was mostly content nouns, 2024 excess mostly style verbs and adjectives, which separates topical drift from LLM tics. (low confidence) [source]
- Antislop builds a model-specific slop profile automatically by comparing generated text to human baselines; some patterns appear over 1,000x more often than in human text. (medium confidence) [source]
- Antislop Sampler suppresses over 8,000 patterns, whereas plain token banning becomes unusable at about 2,000; FTPO reports ~90% slop reduction with benchmarks maintained. (low confidence) [source]
- A 2025 audit (Jabarian & Imas, Chicago Booth) found Pangram, GPTZero and Originality.ai each under 1% false positives on human text, with false negatives of 2-4% (Pangram), 0-2% (GPTZero), 10-40% (Originality.ai); open-source RoBERTa was near random. All detectors lost accuracy on passages under 50 words. (medium confidence) (verified as of 2025-12-02) [source]
Problems, failure modes and limitations
- Frequency-based measurement is valid at population level only; it cannot label individual texts, and results from PubMed/arXiv abstracts may not generalise to other genres. (medium confidence) [source]
- Excess-word counts are only a lower bound on LLM use, and cannot distinguish LLM use from authors independently adopting LLM-preferred vocabulary. (low confidence) [source]
Comparisons and alternatives
- Excess words in 2024 are predominantly style words (about 66% verbs, 14% adjectives, e.g. 'meticulously', 'intricate', 'pivotal'), whereas COVID-era excess was mostly content nouns; this style-vs-content split is the diagnostic that separates LLM shift from topic shift. (low confidence) [source]
- Model-side comparison of Llama Base vs Instruct on continuations of 9,853 PubMed abstracts finds large post-alignment increases (e.g. 'nuanced' +8,342%), and 400 raters preferred variants with more such words (52.4% vs 47.6%), linking part of the overuse to RLHF; scope is Llama only, English abstracts, with some false-positive words. (low confidence) [source]
- Instruction tuning with human feedback drives much overuse (e.g. 'nuanced' +8,342% vs base Llama), so lists should be regenerated after each tuning/RLHF change; the study covers only Llama and does not track drift across models or time. (low confidence) [source]
- On 135,389 original vs professionally edited non-native academic manuscript pairs, 13 detectors (token-statistics, zero-shot, classifier) gave false-positive rates from 0.0% to 100.0% on identical human-written text; edit ratio correlated with score shifts (Spearman rho -0.418 to +0.343). Detectors partly measure linguistic polish, not origin. (medium confidence) (verified as of 2026-08-27) [source]
- Detector accuracy collapses roughly 25-50 percentage points on paraphrased, humanized, hybrid or lightly edited text versus unmodified LLM output. (medium confidence) (verified as of 2026-08-21) [source]
Facts and statements
- The method projects each word's expected 2024 frequency by linear extrapolation from 2021-2022 (2023 excluded as possibly already LLM-influenced), then flags words whose observed frequency exceeds the projection; it needs no labelled LLM text. (high confidence) [source]
- Two complementary metrics are used: frequency ratio r (observed/expected, surfaces rare words) and frequency gap delta (observed minus expected, surfaces common words); thresholds such as delta > 0.01 or a log-scaled ratio criterion were chosen so pre-pandemic years show minimal excess. (low confidence) [source]
- Once a word list is publicised, frequencies change: 'delve' and 'intricate' dropped sharply on arXiv from March-April 2024, while less suspicious words like 'significant' and 'additionally' kept rising. Excess counts of famous tic words therefore understate later LLM use. (low confidence) (verified as of 2025-02-13) [source]
- Excess words are flagged using two metrics, frequency gap and frequency ratio; thresholds were chosen so most pre-Covid years fall well below them, giving conservative estimates. (low confidence) [source]
- Human baselines for model-output tic detection can be built from pre-2022 corpora (wordfreq for single words; Reddit creative writing and Project Gutenberg for n-grams in Antislop), with overuse expressed as a ratio of model frequency to baseline frequency. (medium confidence) [source]
- Post-2021 web text is contaminated by generative-AI output, so frequency resources built from it no longer measure human language; wordfreq's maintainer stopped updating it for this reason, forcing baselines to pre-2022 archives. (medium confidence) (verified as of 2024-09-15) [source]
- Confound check: the hypothesis that 'delve' reflects Nigerian English annotators was not supported by International Corpus of English data (small corpus); RLHF-based Llama experiments fit better as the source of overrepresentation. (medium confidence) [source]
- Overused vocabulary changes over time: 'delve' was heavily overused by ChatGPT in 2023-early 2024, declined later in 2024, and dropped sharply in 2025, so a static list decays. (medium confidence) (verified as of 2026-09-23) [source]
- Overuse of a word does not imply its synonyms are overused, so list entries should be exact terms, not thesaurus expansions. (low confidence) [source]
- Human writing is increasingly influenced by LLM patterns, so human-baseline checks for list entries also need refreshing. (low confidence) (verified as of 2026-09-23) [source]
- Detectors are better suited to population-level inference than to individual accusation; rescanning the same text can give inconsistent verdicts, and small per-document false-positive rates compound at scale. (medium confidence) [source]
- Detector performance is expected to drift as detectors, LLMs and humanizer tools compete; the Chicago Booth authors recommend regular re-audits. (medium confidence) [source]
- The Verbal Tic Index paper scores eight frontier models over 160,000 responses (10,000 prompts, English and Chinese); VTI ranged from 0.590 (Gemini 3.1 Pro, highest) to 0.295 (DeepSeek V4 Pro, lowest) in the latest revision. (low confidence) (verified as of 2026-07-05) [source]
- Rudnicka and Juzek find linguistic patterns shifted measurably between 2024 and 2026 model cohorts, and models of the same generation differ sharply (contraction rate from over 1,200 to over 30,000 per million words). (low confidence) (verified as of 2026-09-23) [source]
Children
- Baseline contamination by AI text (frontier)
- Content-word vs style-word annotation (frontier)
- Counterfactual frequency projection (frontier)
- Era-tagging tic entries (frontier)
- Excess-vocabulary frequency method
- Frequency ratio vs frequency gap (frontier)
- Human baselines and control words for tic detection
- Humanizer tools and paraphrase evasion (frontier)
- Maintaining a living tic list
- Non-native writer false positives (frontier)
- Per-model idiolect profiles (frontier)
- Perplexity and burstiness detectors (frontier)
- RLHF-induced lexical overrepresentation (frontier)
- Retiring decayed tics (frontier)
- Stylometry and AI-text detector limits
- Tic drift across model versions
Frontier under this node: Baseline contamination by AI text, Content-word vs style-word annotation, Counterfactual frequency projection, Era-tagging tic entries, Frequency ratio vs frequency gap, Humanizer tools and paraphrase evasion, Non-native writer false positives, Per-model idiolect profiles, Perplexity and burstiness detectors, RLHF-induced lexical overrepresentation, Retiring decayed tics