/llms.txt` per section once the root index crossed the 10 KB split threshold. | Docset | Root index (bytes) | Spoke indexes | Full (tokens) | Facts (tokens) | |---|---|---|---|---| | developers.cloudflare.com | 9,241 | 243 | 4,162,267 | 1,889,300 | | developer.paypal.com | 4,104 | 193 | 2,921,259 | 1,680,485 | | docs.claude.com | 1,977 | 73 | 7,493,540 | 768,209 | | docs.langchain.com | 1,508 | 15 | 1,552,458 | 738,488 | The small file is the same size everywhere by construction: `build_small` fills an exact 200,000-character budget (about 50k tokens, the ceiling at which editor agents stay stable) and asserts on it. The root index is under 10 KB on all four because the sections were pushed out into spokes; the spokes together are the real index — for Cloudflare, 243 files totalling about 587 KB, which is the honest size of a 1,943-page table of contents with a description per page. Every unit in the facts file is one line, `- [type] text — url#anchor`, and every anchor resolves to a heading in the mirror (see the anchors post for why that was not true a day earlier). ## What the lint found Before the split landed, all four docsets carried a High: `S1` (index over 100 KB — an index that is itself a site dump). After `build_split_index`, the estate gate reported: - Cloudflare, Claude, LangChain: 0 High. Spoke indexes between 10 and 17 KB (about sixty pages with long descriptions each) remain `S1` Medium and are accepted; splitting further would produce single-page indexes. - PayPal: one High left, on `validation-errors/llms.txt` — 3 of 5 links have no description, because those pages carry no definition unit the extractor can turn into one. This is a generator gap (a description fallback from the page's H1 and first sentence is the fix), not a lint false positive. - PayPal also trips `P5` (secrets) on a real-looking RSA private key printed in its own docs. The lint keeps that High on purpose: whether to publish a third party's key material in a facts file is a human decision, not a regex's. Facts files pass `P7` (every line typed from the twelve allowed types, every line sourced) and `R3` (anchors resolve against the mirror) on all four. ## Lessons - A site that publishes `llms-full.txt` can be refined in one pass with zero model tokens; the deterministic extractors alone yield 6–25 units per page on these four sites. - The 10 KB index rule is a split rule, not a truncation rule: no page is dropped, the root gets one line per section with page and token counts, and the spokes are spec-v2 indexes in their own right (most-specific-wins nesting). - "No description" Highs point at pages with no definition unit; fixing them is generator work (fallback text), and hand-editing the index would be erased on the next export. - A probe must require real page blocks, not a 200 status: PayPal's redirect-to-index would otherwise have been recorded as an `llms-full` host and produced a four-page docset. - Pool placement rules are part of correctness: a box without the acquisition ladder produces a different (worse) mirror for the same URL. - The small file's size is a budget, not a measurement; its token count is the same on every docset and tells you nothing about the docset. ## Reproduce The exports live in this repository under `outputs/exports/.llms/`; each `manifest.json` carries the byte and token counts quoted above (the blog's figures are regenerated from them at build time by `site/tools/gen_figures.py`). To rebuild from scratch, run the commands block on a hub checkout, then `llms_lint.py check .llms/ --mirror .md` — it exits 1 while a High remains. Recipe 02 in the examples cookbook walks a split root by hand.