llms.txt: What It Actually Does, and Why It Rots

When ChatGPT, Claude or Perplexity answers a question about your product, it is not consulting a decade of PageRank. It fetches a handful of pages and tries to work out what your site is. That is a very different retrieval problem from classic search, and most sites are accidentally hostile to it. Why sitemaps are the wrong mental model A sitemap optimises for coverage — every URL, every paginated archive, every tag page. That is correct for a crawler with a huge budget and a ranking model to sort the noise afterwards. An LLM landing on your site has neither. It has a limited context window and one shot. If the first thing it ingests is 400 URLs of ?page=17 and /tag/misc , your three genuinely useful guides are buried. llms.txt inverts this. It is a small markdown file at your root that optimises for priority : # Your Product > One-line description of what this actually does. ## Docs - [ Quickstart ]( https://example.com/docs/quickstart ) : Install and first request in 5 minutes - [ API Reference ]( https://example.com/docs/api ) : Every endpoint with request/response examples Two rules make or break it: Every link carries a description. The colon-suffix annotation is what lets a model decide whether to fetch a page. A bare link list is barely better than a sitemap. Omit aggressively. If a page does not answer a question someone would ask, it does not belong. llms-full.txt and the context tradeoff llms-full.txt inlines expanded content rather than linking out, so a model can ingest everything in one request. This is genuinely useful for compact docs — and actively harmful for large sites, where you will blow the context window and get truncated mid-document. Rough heuristic: if your docs exceed roughly 50k tokens, ship llms.txt alone and let models fetch selectively. The part nobody mentions: it rots This is where most implementations quietly fail. You write the file, ship it, and three months later half the descriptions describe features you renamed and two links 4