LLM tells in code, commit messages, comments and READMEs
Parent: Writing and Documentation · Published reference · snapshot 2026-09-23 · skill technical-writing-craft/references/code-and-commit-ai-tells.md
↓ Facts as markdownall context files
Source-anchored research for LLM tells in code, commit messages, comments and READMEs. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Definitions
- The characteristic narration tell is a comment that restates the adjacent code, adding maintenance burden and burying the few comments that matter. (low confidence) [source]
- Comment-based detection is a weak, proxy signal: the repository study calls its results proxy-based and warns detectors both under- and over-estimate LLM use, and other work reports commercial detectors are affected by comments and unreliable. (medium confidence) [source]
Structure and components
- Agent guidance for avoiding the tell: do not add comments by default; add one only for non-obvious why (rationale, constraints, invariants, trade-offs, surprising external behavior); prefer better names, types and structure over narration. (low confidence) (verified as of 2026-09-04) [source]
- In one study of security-related AI-generated PRs, 70.4% of 2,823 commit messages were high quality (contain both Why and What) and 29.6% omitted one or both. (low confidence) [source]
- Practitioner signature list of AI-grown bloat: verbose scaffolding layers, partial fixes, ceremonial state layers mirroring other layers, non-functional placeholder subsystems left wired in, debug harness code in shipping source, and defensive abstractions that accumulate each agent turn; one Flutter app shrank 31.7% with tests passing. (low confidence) (verified as of 2026-06-01) [source]
- The readme-ai generator emits a fixed skeleton: header with badges and logo, project introduction, features table, project structure, project index, getting started (install, usage, tests), community and contributing, license; it offers emoji theme packs prefixed to section titles. (medium confidence) (verified as of 2026-09-23) [source]
How it works
- Wikipedia's AI-cleanup editors state the listed patterns are only potential signs of a problem, not the problem itself, and that text with these features is not necessarily AI-generated because LLMs are trained on human writing. (high confidence) [source]
Measurements and reference values
- LLM comments are not uniformly poor: in a 415-comment Java evaluation, human experts judged 58.8% to match originals and 27.7% to surpass them; automated metrics such as BLEU correlated negatively with human judgment. (low confidence) [source]
- A classifier on AIDev PRs identified which of five agents (Codex, Copilot, Devin, Cursor, Claude Code) authored a PR at 97.2% F1; commit-message features dominated, with multiline commit ratio the strongest single signal (44.7% importance). (low confidence) [source]
- Per-agent signatures differ: Codex extensive multiline commits, Copilot longer PR descriptions, Cursor bullet points and hyperlinks, Devin multiline messages. Claude Code recall was only 57% in that study, so its tells are weaker. (low confidence) [source]
- Across four LLMs, generated code had on average 63.34% more code smells than reference solutions (73.35% implementation smells, 21.42% design smells), with larger increases on more complex tasks. (low confidence) [source]
- Detection tools for AI text score poorly in independent testing (all under 80% accuracy, five over 70%), so a formatting tell in a README is grounds to review, not proof of AI authorship. (medium confidence) [source]
Problems, failure modes and limitations
- Search-result snippets (not fetched) report opposite security-study findings: LLM code lacking defensive constructs, and one model with error handling in only 3.3% of scripts. Excess defensiveness is not established as a general LLM tell. (low confidence) [source]
- Population-level word statistics cannot identify an individual LLM-written text, and cannot separate direct LLM use from humans unconsciously adopting LLM-preferred vocabulary; a single flagged word is weak evidence. (medium confidence) [source]
Comparisons and alternatives
- GitClear's analysis of 211M changed lines (2020-2024) reports copy/pasted lines rising from 8.3% to 12.3% of changed lines while moved (refactored) lines fell from 25% to under 10%, read as assistants adding new code rather than reusing existing functions. (low confidence) (verified as of 2026-01-01) [source]
- Excess-vocabulary analysis of 15M+ PubMed abstracts found 2024 excess words were mostly style words (verbs ~66%, adjectives ~14%) such as delves, intricate, pivotal, unlike COVID-era excess words which were content nouns; it estimates at least 13.5% of 2024 abstracts were LLM-processed. (high confidence) (verified as of 2025-07-02) [source]
- A Stanford study of seven GPT detectors found about 61% of non-native English writers' essays flagged as AI; detectors penalise plain, predictable wording, so clean simple prose is a false-positive risk rather than a tell. (high confidence) [source]
Facts and statements
- A 2021-2025 study of eight GitHub repositories found comments flagged as likely LLM-generated concentrated in Meta and Explanation categories, and remained relatively stable over time. (medium confidence) (verified as of 2026-07-01) [source]
- Claude Code by default appends a Co-Authored-By: Claude trailer to commit messages and a 'Generated with Claude Code' line to PR bodies; cloud sessions also add a claude.ai link. The attribution.commit, attribution.pr and attribution.sessionUrl settings change or hide them. (high confidence) (verified as of 2026-09-23) [source]
- Trailer tells are easily removed (git commit --amend, force-push, commit-msg hook), so their absence does not show human authorship. (low confidence) [source]
- Agent-authored PRs show message-code inconsistency: descriptions can claim functionality with no corresponding implementation, or describe changes incompletely. A description's confident summary therefore needs checking against the diff. (medium confidence) [source]
- Agent-generated code shows procedural bloat (Long Method dominant) on simple tasks and architectural decay (God Class, superficial file separation) in complex systems; total lines of code correlates with architectural degradation at rho=0.94. (medium confidence) [source]
- More capable models can produce messier code as they try to handle every edge case, yielding excessively long methods; detailed prompts and few-shot examples did not reduce defects in that study. (low confidence) [source]
- Google's developer documentation style guide prescribes active voice, second person, conditions before instructions, and friendly-but-not-frivolous tone; these are human-authored norms that clean LLM output can also satisfy, so following them is not itself a tell. (medium confidence) [source]
- The Wikipedia guide's stronger tells are content-level: undue emphasis on significance and legacy, vague attribution ('experts argue'), 'not just X but Y' parallelism, copula avoidance, and rigid 'challenges and future prospects' sections; these signal padding, not correctness of grammar. (medium confidence) [source]
- Human judgement alone performs no better than chance at telling AI from human text per the Wikipedia guide, and writers who fear accusation may distort their style; human and LLM writing are also converging. (low confidence) [source]
- Wikipedia's AI-writing guide lists emoji used as formatting, Markdown syntax, excess boldface, vertical lists with inline headers, and skipped or heading-only-heading structures as signs of AI text, and says they are signs of a possible problem, not the problem itself. (high confidence) [source]
- LLM changelog generation can produce entries that are fluent, correctly formatted and false, such as inferring an extra change that never shipped or inventing a value like a cache TTL; the recommended control is human review of the draft. (low confidence) [source]
- The avoid-ai-writing skill (74 pattern categories, deterministic detector, CI and pre-commit integration) treats generic conclusions, chatbot artifacts and formatting inconsistencies as tells that can be linted in docs. (low confidence) (verified as of 2026-09-23) [source]
Children
- Agent attribution trailers (frontier)
- Agent commit-style fingerprinting (frontier)
- Badge walls and generated project-structure trees (frontier)
- Ceremonial layering (frontier)
- Code comment narration and over-explanation
- Commit message and PR description tells
- Defensive-code and over-engineering signatures
- Detector false positives on plain writing (frontier)
- Docstring over-completeness (frontier)
- Duplication over reuse (frontier)
- Good engineering prose versus a tell
- Invented changelog entries (frontier)
- Message-code inconsistency (frontier)
- Meta comments (e.g. 'added for X') (frontier)
- Placeholder and debug debris (frontier)
- README and changelog boilerplate tells
- Specificity test for a sentence (frontier)
- What-versus-why comment distinction (frontier)
Frontier under this node: Agent attribution trailers, Agent commit-style fingerprinting, Badge walls and generated project-structure trees, Ceremonial layering, Detector false positives on plain writing, Docstring over-completeness, Duplication over reuse, Invented changelog entries, Message-code inconsistency, Meta comments (e.g. 'added for X'), Placeholder and debug debris, Specificity test for a sentence, What-versus-why comment distinction