llms.txt: the spec and its grammars

verified-as-of: 2026-08-30

Contents

1. What it is, in one paragraph
2. The spec, v2 (verbatim structure and the rules that changed)
3. The file grammars in the wild — llms.txt, llms-full.txt (three variants), .md twins
4. Discovery: link relations, Accept: text/markdown, Lighthouse
5. How consumers actually use it (and who doesn’t)
6. Spec gaps, validators, security
7. Related files: robots.txt, sitemap.xml, ai.txt, Content Signals
8. References

Honesty note (carry into every recommendation). llms.txt is a proposal, not a ratified standard, now at v2 (modified 2026-08-10).¹ Google says it neither reads nor plans to read it;²³ server-log studies find 97% of files get zero AI requests.⁴ The one consumer class that demonstrably fetches it is agents that are pointed at it — coding agents, MCP doc tools, RAG pipelines: in Ahrefs’ 137k-domain log study the Claude-Code user agent out-fetched every AI retrieval bot bar two (statespace-indexer, GPTBot).⁴ Design for that use, not for search visibility.

1. What it is, in one paragraph

A markdown file — /llms.txt at a site root or at any subpath — that gives a language model a curated, priority-ordered map of a site’s LLM-friendly content: an H1, a blockquote summary, optional prose, then H2 sections of - [name](url): description links.¹ The links should point at clean markdown (a .md twin of each page), so the index stays small enough for context and the detail is fetched only when needed.¹ Proposed by Jeremy Howard (Answer.AI) on 2024-09-03; revised to v2 on 2026-08-10 after “thousands of sites” adopted it and documentation platforms began generating it automatically.¹⁵

2. The spec, v2

Structure, in order (verbatim from llmstxt.org):¹

• “An optional byte-order mark (BOM)”
• “An H1 with the name of the project or site. This is the only required section”
• “A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file”
• “Zero or more markdown sections (e.g. paragraphs, lists, etc) of any type except headings, containing more detailed information about the project and how to interpret the provided files”
• “Zero or more markdown sections delimited by H2 headers, containing ‘file lists’ of URLs where further detail is available” — each entry “a required markdown hyperlink [name](url), then optionally a : and notes about the file.”

The spec’s own mock example:¹

# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)

Placement and scope (new in v2). “The llms.txt file spec is for files named llms.txt, at the root path /llms.txt of a website or at any subpath (e.g. /docs/llms.txt). A file covers the URLs under its path, and where more than one file applies, agents should use the most specific one.”¹ This is what lets a project that only controls a path (a GitHub Pages site) participate, and it is the mechanism for hub-and-spoke indexes (see sibling recreation-and-aggregation). The spec explicitly rejects /.well-known/ (RFC 8615) because well-known URIs exist only at the origin root.¹ The .well-known request (issue #2) is still open and Mintlify serves both locations.⁶⁷

## Optional. v1 gave it mechanical meaning (“the URLs provided there can be skipped if a shorter context is needed”); v2 keeps it only as a convention “for secondary information” and states the context-expansion tooling “is no longer part of the proposal”.¹⁵ Do not build logic that depends on it.

Markdown twins (widened in v2). Provide a clean markdown version of each page at the same URL “either with .md appended (page.html.md) or with the extension replaced by .md (page.md). (URLs without file names should append index.html.md or index.md instead.)”¹

What v2 changed and why (llmstxt.org/changes):⁵ link-relation discovery added; both .md URL forms allowed; subpath semantics defined; llms_txt2ctx removed from the proposal and with it the special meaning of Optional; background rewritten around how agents actually use sites. Search Engine Journal’s coverage notes the syntax “might still change before everything is finalized”.⁸

Consumption expectation (v2). “Agents are expected to view or search llms.txt to find the information they need, then follow the relevant links … The file itself stays small enough to fit in context. The detail lives behind the links, and is fetched only when needed.”¹ Authoring guidance in the spec: concise language, informative link descriptions, no unexplained jargon, and “test your file by asking an agent questions about your content, giving it only your llms.txt as a starting point.”¹

3. The file grammars in the wild

3.1 llms.txt — three real shapes

Consequence: a parser must be lenient — treat the H1 as the only invariant and everything before the first H2 as “info”.

3.2 llms-full.txt — not in the spec, and three grammars

llms-full.txt (the whole docset inlined into one markdown file) appears nowhere in the v1/v2 spec text or the repo README.¹⁰ Mintlify says it “was developed by Mintlify in collaboration with customer Anthropic”;¹¹ Lab451 dates its popularisation to early 2025.¹² There is no single page-block grammar; three verified variants:

A robust splitter therefore needs at least: # Title immediately followed by Source: (Mintlify); a --- YAML block carrying url:/title:; frontmatter + View as Markdown link (Cloudflare); and explicit delimiters.

Size reality. Mintlify caps a generated index at 100,000 characters and splits overflow into /_llms/ sub-indexes but sets no cap on llms-full.txt;⁷ Fern dropped llms-full.txt because it “exceeded most model context windows, added heavy serving overhead, saw little use”;¹⁴ Nuxt sizes its files at ~5K vs ~1M+ tokens and gates the full file to “200K+ token” tools;¹⁵ Mantine replaced a 2.2 MB inline file with a 45 KB link list after complaints that it “clogs the AI’s context window”;¹⁶ Cursor’s moderators say indexed files above ~50–60k tokens become unstable.¹⁷ Anthropic’s docs.anthropic.com llms-full.txt is 41.6 MB.⁹

3.3 .md twins

Mintlify, Fern, GitBook and ReadMe all serve a .md twin per page and link them from llms.txt “so AI tools can fetch the Markdown version of each page directly”.⁷¹⁴ Mintlify’s twins begin with a blockquote — > ## Documentation Index / Fetch the complete documentation index at: …/llms.txt — that a consumer should strip before indexing.⁹

4. Discovery

• Link relations (spec v2). rel="alternate" type="text/markdown" → the page’s markdown twin; rel="describedby" → the llms.txt that covers it; as HTML <link> or an HTTP Link: header, which “also works for non-HTML resources … and can be added in web server or CDN configuration”. Example: Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby".¹
• Accept: text/markdown content negotiation is not in the spec. Vercel proposed it (2026-02-03) precisely because it “requires no site-specific knowledge”;¹⁸ Mintlify, GitBook and Fern honour it (Mintlify adds X-Robots-Tag: noindex, nofollow and prepends the llms.txt blockquote);⁷¹⁹ Cloudflare “Markdown for Agents” (2026-02-12, Pro/Business/Enterprise) converts any proxied HTML at the edge and returns Content-Type: text/markdown, x-markdown-tokens, x-original-tokens, Vary: Accept, dropping ETag/Last-Modified, with a 2 MB origin cap and no advance-discovery mechanism — clients just try.²⁰ Uptake is thin: one 44-day log saw 1,421 such requests, none from GPTBot/PerplexityBot/ClaudeBot.²¹ Origin implementations must add Vary: Accept and bypass full-page caches keyed without it.²²
• Chrome Lighthouse (13.3, May 2026; doc updated 2026-05-05) has an “Agentic browsing” category that fetches /llms.txt: a 404 is Not Applicable, a server error is flagged; sibling audits cover WebMCP, agent accessibility and layout stability.²³

5. How consumers actually use it

Budgeting in practice is producer-side splitting, not consumer-side truncation: Mintlify’s /_llms/ recursion, Starlight’s llms-small.txt, Nuxt’s two sizes, Firecrawl’s page delimiters, Jina Reader’s x-max-tokens, Cloudflare’s token-count headers.^7151320

6. Spec gaps, validators, security

• No official validator. Community validators grade A–F and are stricter than the spec (blockquote required, absolute URLs, Optional last);³⁰ the spec requires only the H1.¹ llms-txt-validator --check-links gives JSON for CI; stale links are a real failure (litellm’s file carried a deleted page, Aug 2026).³¹³²
• Open gaps (repo issues): H2 ordering carries no defined meaning; no version/provenance field (#132/#133); which language a root file is (#147); and behavioural steering — issue #152 (2026-08-29) found 42.3% of 100 sampled files try to shape model answers, with no security-considerations section in the spec.³³
• Prompt injection: a linked markdown file is untrusted input; treat everything fetched via llms.txt as data (OWASP LLM01).³⁴ Ahrefs flags the same risk; no incident data found.⁴
• Parser reference (core.py): header ^#\s*{title}\n+{summ}\n+{info}; sections on ^##\s*(.*?$); links -\s*\[{title}\]\({url}\){desc}.²⁴ JS ports: the spec page’s sample parseLLMsTxt() and npm llms-txt-parser (→ {title, overview, links[{title,url,description,section}]}); PHP llms-txt-php.¹³⁵

7. Related files

References