llms.txt, and who actually reads it

llms.txt is a single Markdown file you drop at the root of your site, /llms.txt, that hands a language model a curated map of your content. It takes an afternoon to add and almost nothing can go wrong. What most write-ups skip is that in 2026 barely anything reads it. The one payoff that holds up is the one it was designed for: feeding documentation to coding agents. This blog ships one, so I want to walk through what it is, how to add it, and where it helps versus where it is wishful thinking.

What llms.txt is

The idea came from Jeremy Howard of Answer.AI (also behind fast.ai), published on llmstxt.org on 3 September 2024. It targets two concrete problems. First, context windows are too small to swallow a whole website, so a model needs a shortlist of what matters. Second, HTML pages in the wild are wrapped in navigation, ads, cookie banners, and JavaScript, and turning that soup back into clean text is lossy and slow. A hand-written llms.txt sidesteps both: it is already plain text, and you decide what goes in it.

Two things trip people up.

It is a proposal, not a ratified standard, and the difference matters. robots.txt has a formal spec (RFC 9309) and sitemap.xml has multi-party backing; llms.txt is one party's suggestion with a GitHub repo and a Discord. Nothing wrong with that, but it means no provider is obligated to honor it.

And it was pitched for inference, not training and not SEO. The intended moment is when someone is actively working and pulls your docs into a model right then, for example loading a library's reference into a coding session. Most of the confusion around the file comes from the SEO world repurposing it as a ranking lever it was never meant to be.

The format, by example

The spec is deliberately small. One required H1 with the site name, an optional blockquote summary, optional prose, then any number of ## sections that are just Markdown link lists. Each link can carry a short note after a colon. A special ## Optional section marks links a model can skip when it needs a shorter context.

# Acme Docs

> Acme is a payments API for issuing virtual cards and moving money between accounts.

Use the quickstart first; the API reference assumes you already have an API key.

## Docs

- [Quickstart](https://acme.dev/docs/quickstart.md): install, authenticate, first request
- [Cards API](https://acme.dev/docs/cards.md): create, freeze, and top up virtual cards
- [Webhooks](https://acme.dev/docs/webhooks.md): event types and signature verification

## Optional

- [Changelog](https://acme.dev/changelog.md)
- [Brand assets](https://acme.dev/brand.md)

That is the whole grammar. The descriptions carry the weight here, so write them to disambiguate ("create, freeze, and top up virtual cards") rather than to sell ("our world-class card API"). The model is trying to find the page that answers a question, so tell it which one that is.

One companion convention goes with it. Pages meant for models should offer a clean Markdown twin at the same URL with .md appended, so cards.html becomes cards.html.md, which is why the links above use .md.

Two file names come up, and they do different jobs:

• llms.txt is the curated index: title, summary, and links with one-line notes. Keep it small, a couple of kilobytes, ten to thirty links.
• llms-full.txt is the full dump: every doc inlined into one file so a tool can ingest everything in a single fetch. These get enormous. Cloudflare's runs into the millions of tokens.

Ship the index for almost anything. Add the full export when you have substantial docs and want single-fetch ingestion.

Who actually reads it

This is the part to be straight about, because it should change how much effort you spend.

Coding agents do use it. This is the original use case and it still works. Cursor lets you add an llms-full.txt URL as a docs source, and Claude Code, Copilot, and MCP or RAG pipelines will fetch a docs llms.txt to orient before they write code against a library. If you maintain developer documentation, that audience exists and it is using the file today.

AI search, on the other hand, mostly ignores it. In June 2026 Ahrefs published a study of 137,210 domains and found that of roughly 38,000 with a valid llms.txt, 97% received zero requests for it in May 2026. Not "little traffic", none. And the sliver that did see traffic tells the same story from the other side: the named AI tools fetching these files were led by GPTBot and Claude-Code, which are coding and crawling agents, not the AI Overviews or chat answers people hope to rank in.

Google has been blunt about it. Gary Illyes said at a Search Central Live event in mid-2025 that Google does not support llms.txt and is not planning to, and John Mueller compared it to the old keywords meta tag, a self-declared signal that got abandoned because it was trivial to game. Google's own AI-optimization guidance, updated in June 2026, says plainly that Search does not use these files and that keeping one will neither help nor hurt your ranking.

The distinction that clears up most of the noise: publishing is common, consumption is rare. Thousands of docs sites have the file, largely because platforms like Mintlify auto-generated it for everyone at once. That number says nothing about whether AI systems read it. For coding agents, yes. For getting cited in an AI answer, the evidence says no.

Adding one to your site

For most stacks you do not write this by hand. Documentation platforms like Mintlify, GitBook, and Fern auto-generate llms.txt, llms-full.txt, and clean Markdown versions of every page for hosted docs, and there are community plugins for the popular site generators: vitepress-plugin-llms for VitePress, starlight-llms-txt for Astro Starlight, plus options for MkDocs and Docusaurus, and a generator built into the Yoast plugin for WordPress. This site publishes both files that way, built from its content, and you can read them live at /llms.txt and /llms-full.txt. For anything that changes often, generating the file from your content beats maintaining it by hand, because the usual failure mode is the file drifting out of sync with reality.

If you would rather hand-roll it, or your stack has no plugin, it is just a text file:

1. Write the Markdown by hand using the format above. Curate; do not paste your sitemap.
2. Save it as llms.txt and serve it from the web root so it resolves at https://yoursite.com/llms.txt.
3. Confirm it returns HTTP 200 and comes back as plain text, not an HTML error page.
4. Check robots.txt is not blocking the path or the AI user-agents you actually want reading it.

Doing it well

The good version of this file is short and current.

Curate ruthlessly. Ten to thirty high-value links beat a flattened sitemap. If you need more than that, llms-full.txt is where the bulk goes.

Keep the summary factual. Read it back and ask whether it sounds like the opening line of a Wikipedia article or the top of a landing page. You want the former: what the thing is, no adjectives.

Keep it in sync. A stale index full of dead links is worse than no index, because anything that does read it now distrusts it. Generate it from your content and it stays honest on its own.

Do not keyword-stuff. No model is extracting keyword density from this file, and it reads as junk to any human who opens it.

So should you bother?

It depends on what you run.

If you publish developer or API documentation, yes, and ship both files. Coding agents fetch them, structured docs make those agents more accurate and cheaper on tokens, and if you are on Mintlify, GitBook, or Nuxt Content it is nearly free. This is where the file earns the effort.

If you run a marketing or content site, treat it as a cheap hedge and cap the effort at half a day. Curate your fifteen-to-thirty canonical pages, write plain descriptions, ship it, move on. Do not promise anyone AI-search gains from it, because the 2026 data does not back that up.

Either way, the things that actually help a model understand your site are the unglamorous ones: clean semantic HTML, a correct robots.txt that lets in the AI user-agents you want, valid structured data, and content worth reading. llms.txt sits on top of those as a nice-to-have. Add it because it is cheap and points the right direction, not because you expect it to move a number. If you are already deep in the coding-agent workflow, it pairs well with the rest of that toolchain, and I have written more about getting more out of Claude Code and building your own Claude Code plugin if that is where you are headed.