The vocabulary file

“Cookie” is an HTTP state token in a web docset, a monster in one children’s canon, and a snack in a recipe corpus. An index does not settle which one a family means; a facts file mentions all three without noticing. The vocabulary file is the layer that pins the sense, lists the surface forms people actually type, and says what the term is not. It is neither index nor facts. It is what makes both findable and unambiguous.

What a vocabulary file is

llms-vocabulary.txt is one line per term of a family, each line carrying: the canonical name, a definition taken from a kept unit, the neighbours it is easy to confuse it with (not:) and how it differs, the words people say instead (aka:), and the URL of the unit the definition came from. The file is shaped like a spec-v2 llms file — H1, blockquote, generator banner, H2 sections — so any reader that opens an index can open this.

Three properties make it different from a glossary someone typed:

• Every definition is extractive. It comes from a definition unit or an “X is/are …” sentence in the pool, and the line ends with that unit’s anchor. A term the pool names but never defines goes to ## Named, not yet defined with its hit count, and never receives a model-invented definition.
• Every alias appeared. aka: entries are surface forms found in the pool — backticked tokens clustered by normalised spelling, with the most frequent surface as the canonical name and the rest as aliases. No synonym list is imported from outside.
• Contrast is first-class. not: comes from contrast cues in the units themselves — not, unlike, vs, rather than, instead of, not to be confused with — so the file says “small is not full” because a source said it, and can point at where.

Sources are ranked by trust, deterministic first and a model last:

1. concept-tree node names and their existing aliases (the subject and its children);
2. backticked tokens the pool keeps (file names, headers, paths, tools), seen at least twice, clustered by spelling;
3. definition units and “X is/are …” sentences → definitions; contrast cues → not:;
4. --llm: the local model writes a missing definition or differentiator from at most six units that mention the term — and every name it returns must appear in those units, or the line is dropped. Lines that reach the file this way are marked origin: llm with a grounding score, and below the floor they carry verify before citing.

The line grammar

The full grammar, with every optional field shown:

# <Family> — vocabulary
> <n> terms of <family>; canonical name, definition, how it differs (not:), what people say instead (aka:). Each line anchored to the unit it came from.
<!-- generated by docset_refine vocabulary vN · <date> · sources: … -->

## Terms
- **<term>** [<sense-id>] (<pos>): <definition> — <url#anchor> · aka: a, b · not: <neighbour> — <how it differs> · ant: <antonym> · broader: <term> · narrower: <term>, <term> · related: <term> · measure: <unit> · field: <family-slug> · verified-as-of: <date>

## Homonyms
- **cookie** [web.cookie] · [folklore.cookie-monster] · [food.cookie]: <one line per sense with its family> — the sense picker's data

## Named, not yet defined
- <term> — seen in <n> units, no definition unit found (evidence rule)

Field by field:

Two honest notes on the grammar. The builder that exists today writes definition, aka:, not: (with differs: for the how-clause), the source anchor, and the origin: llm marker; ant:, broader:, narrower:, related: and measure: are proposed extensions that the concept abstractor’s relation taxonomy is expected to supply. And contranyms — a word whose senses oppose each other (sanction, cleave, oversight) — are two sense lines under ## Homonyms marked contranym, because a sense picker that offered only one would be wrong half the time.

A line in the target grammar, with the proposed fields shown, for the term llms-small.txt:

- **llms-small.txt** [llms.small] (noun): the budgeted variant of a full file — reference-class pages first, within about 50k tokens — /reference/formatting/#3-the-budgeted-file--llms-smalltxt · aka: small, llms-small · not: llms-full.txt — full is every page with no budget; small is a selection that fits a consumer's stable window · broader: llms-full.txt · related: manifest.json · measure: tokens (chars/4) · field: llms-txt

And the line the builder actually wrote for the same term in the llms.txt family’s own llms-vocabulary.txt, abridged — no sense id, no broader:/related:/measure:, and the how-clause under differs: rather than after a dash:

- **llms-small.txt** — llms-small.txt is a small variant of a tokenized text file used to enforce size budgets on the producer-side. · not: /_llms/, x-markdown-tokens, llms.txt, x-max-tokens · differs: not consumer-side truncation … — https://www.mintlify.com/docs/ai/llmstxt · evidence: hub estate · origin: llm (grounded 0.64)

The gap between the two is the honest state of the builder: the required fields ship, the relation fields do not yet. Either line tells an agent that “small” in a query is this file and that “full” is the neighbour it is contrasted with; only the first tells it that size is counted in tokens at four characters each.

Senses across fields

A sense id is <family-slug>.<term-slug>. A term is disambiguated by the pair (term × family): cookie in the web family is web.cookie, in a folklore family folklore.cookie-monster, in a recipe family food.cookie. The three vocabularies are built independently; the cross-family pass finds homonyms by matching term slugs across them and writes a ## Homonyms line listing every sense with its family.

What a consumer does with that depends on its scope:

• scoped to a family — the query gets that family’s sense and the others are invisible; “cookie expiry” inside a web docset never sees the snack;
• unscoped — the query gets the sense picker: the homonym line, one sense per row, and the consumer (or the reader) chooses;
• crossing families on purpose — an agent that switches from the web docset to a recipe corpus reads the new family’s vocabulary first and stops guessing.

The model has a known seam. Keying senses by family may split a term that is really one sense across two families — Link header in an HTTP family and in an llms-txt family are the same header. A “same-as” link between senses is the obvious fix and is left open; until it exists, the homonym line simply lists both.

Where it feeds

The vocabulary was built because three consumers were weak without it:

A fourth consumer is the concept abstractor, which seeds its lexicon — synonyms, parts, sub-types, contrasts — from the family’s vocabulary before it harvests, and a fifth is the precedence ladder in the CLLMS essay, whose rung 4 is “agreement with the canonical definition” — which is a lookup in this file.

The acceptance bar for the keyword consumer is written down but not yet measured, because server-side expansion is not shipped: when it lands, on the P12 question bank an expanded query must raise exact-token recall by at least one hit per family without losing any. Until then a client can do the expansion itself — read the term’s aka: list out of the vocabulary file and send the surfaces as one mode="keyword" query.

Build one

The walkthrough below builds the llms.txt family’s own vocabulary — the terms are index, full, small, facts, twin, describedby, family, split root, unit, anchor and their neighbours. It is the same procedure for any field.

1. Assemble the pool. Any mix of units.jsonl, llms-facts.txt and hub spokes. The pool is what the file may cite; nothing outside it can become a definition.

   PYTHONPATH=scripts .venv/bin/python -m docset_refine vocabulary \
     --from outputs/exports/site.llms/llms-facts.txt \
     --from skills/llms-deep-optimizer/references/attributes.md \
     --subject "llms.txt" \
     --out llms-topical/llms-txt.llms/

2. Read the candidates. The builder proposes terms from the tree, then from backticked tokens seen at least twice, then from definitions and contrast cues. A term with no definition lands in Named, not yet defined — that list is the research gap, not an error.

3. Decide about --llm. Without it, every line is deterministic. With it, the local model writes the missing definitions from at most six evidence units each, and anything it names that the units do not contain is dropped. Lines it wrote are marked, and the floor (--floor) decides which ones say verify before citing.

4. Lint. llms_lint.py check llms-vocabulary.txt --kind vocabulary parses every line against the grammar and checks that every definition’s anchor resolves. The site’s CI runs this on its own file; 0 High is the bar.

5. Register. --register merges each term’s aka: into the matching concept-tree node’s aliases. From then on the topical builder’s keyword pass matches the synonyms, and the FTS5 layer can expand through them.

6. Serve. The file lands beside the family’s other files and is served at /t/<slug>/llms-vocabulary.txt with the same markdown headers as everything else.

The pilot bar for the llms.txt family is at least 40 terms, at least 5 not: contrasts, and the cookie-style homonym demo across at least two families. The glossary page is a hand page — the terms in the sense this site uses them — and the site’s generated llms-vocabulary.txt is its machine twin; the two are checked against each other, not derived one from the other.