Insight · August 30, 2026
What is
llms.txt
A curated markdown file that tells a model what matters on your site. In 2026 the open question is whether anything reading the web actually opens it.
01 · The idea
A clean map of your site, written for machines.
llms.txt is a single markdown file that lives at the root of a website, at yoursite.com/llms.txt. Inside it, a title, a one line summary, and a curated list of links point a language model to the pages that matter most. Jeremy Howard, co-founder of Answer.AI, proposed it in September 2024 as a sibling to two files the web already understands: robots.txt, which says what a crawler may touch, and the sitemap, which says where everything is.
llms.txt answers a different question. Not where everything is, but what a model should read first. It is a short, human curated index that treats the model as the reader and strips the site down to its signal.
02 · The problem it solves
A web page is built for people, not models.
Open any page and look at what surrounds the actual content. Navigation, menus, cookie banners, ads, scripts, and layout. People skip past all of it without thinking. A model cannot. It has to read the whole page, then spend effort turning that mess back into clean text, and every token it spends on markup is a token it does not spend on the answer.
Context windows are finite. When a model tries to understand a large site by reading raw HTML, it fills up fast and loses the thread. llms.txt is the proposed fix. Hand the model a clean version up front, in plain markdown, and point it only at the pages worth reading. The file does the curation a person would otherwise have to do by hand.
The one line to keep
“A sitemap tells a crawler where everything is. An llms.txt tells a model what matters.”
03 · The anatomy
Five parts, in a fixed order.
01
The title
One H1 with the name of the site or project. It is the only required line. Everything else is optional structure built around it.
02
The summary
A blockquote directly under the title. One short paragraph that says what the site is and what a model should know before it reads any further.
03
The detail
Optional prose sections with no headings. Room for context that does not fit inside a list of links.
04
The links
Sections marked with H2 headers, each holding a file list. Every entry is a markdown link to a clean page, followed by an optional note on what it covers.
05
The optional block
A section named Optional. Its links are the ones a model can skip when the context window is tight. A direct way to say what matters least.
04 · Does anyone read it
The idea is clean. The evidence is not.
Here is the honest read for 2026. Adoption is real and rising. SE Ranking studied roughly 300,000 domains in May 2026 and found 10.13 percent now publish an llms.txt, up from 0.4 percent a year earlier. Documentation platforms generate the file automatically. Google added a check for it to Chrome Lighthouse.
And almost nothing reads it. Ahrefs looked at 137,000 sites and found 97 percent of their llms.txt files received zero reads. The same study that measured adoption found no lift in AI citations from having the file, once site authority and content freshness were accounted for. John Mueller of Google put it plainly. No AI service has said it uses llms.txt, and you can tell from server logs that the crawlers do not even check for it. He compared it to the keywords meta tag, the classic example of metadata that owners fill in and no engine ever reads.
No major AI company, not OpenAI, Google, Anthropic, Meta, or Mistral, has committed to reading llms.txt in its production search systems. A standard is only as real as the readers who honor it, and the search crawlers are voting with their absence.
05 · What the file looks like
The whole spec fits on one screen.
There is no framework to install and no schema to validate. You write a markdown file by hand or generate it from your docs. The title names the project. The blockquote sets the context. Each link points at a clean page a model can read without wading through the site around it.
# Acme Docs > Acme is a payments API. These pages give an agent > everything it needs to integrate without reading the > marketing site. ## Docs - [Quickstart](https://acme.dev/docs/quickstart.md): five minute setup - [Authentication](https://acme.dev/docs/auth.md): keys and tokens - [API reference](https://acme.dev/docs/api.md): every endpoint ## Optional - [Changelog](https://acme.dev/docs/changelog.md): version history
One title · one summary · curated links · a skippable block
A companion file, llms-full.txt, takes the same idea further. It inlines every page in full, so an agent that prefers self-contained context can pull the entire documentation set in one request. The v2 revision of the spec, published in August 2026, added a formal way for an HTML page to point at its clean markdown twin, so an agent browsing the site can find the readable version without guessing.
06 · Where it actually works
The search crawlers skip it. The coding agents live on it.
The file did not fail. It found a narrower job than the one it was pitched for. The place llms.txt earns its keep in 2026 is inside coding tools. Point Cursor, Claude Code, GitHub Copilot, Cline, or Aider at a documentation site and they look for llms.txt first. When an agent needs to write code against a library it does not know, the clean file is the fastest path to the current API.
Even Mueller granted this. He called llms.txt a reasonable way to save tokens for AI coding tools reading developer docs, while dismissing it for search. That is the honest shape of it. Not a ranking signal, not a citation lever, but a real convenience for agents that already know your URL and want the readable version of your docs.
One condition matters. A stale file is worse than none, because an agent will trust it and generate code against endpoints that no longer exist. If you ship one, it has to stay current, which means generating it from the same source as the docs rather than maintaining it by hand.
07 · Should you ship one
Ship it for docs. Skip it for search.
If you run a developer product or a documentation site, ship an llms.txt and keep it current. The cost is close to zero when your docs tooling generates it, and the agents writing code against your product will use it today. That is a real audience with a real workflow.
If your goal is to be cited more often in AI answers, llms.txt is not the lever. The evidence says it does nothing for that, and the crawlers do not read it. The work that moves AI citations is the same work that moves classic search. Clean HTML that a model can parse without the file, real structure, current content, and authority earned elsewhere. Do that first.
The safe posture is small. Ship the file if it costs you almost nothing to keep accurate. Do not reorganize a content strategy around a standard the readers have not adopted. Watch whether the major engines ever announce support. Until they do, treat llms.txt as a convenience for agents, not a channel to search.
Closing
Draw the map. Just know who is reading it.
A standard is a promise between a writer and a reader. llms.txt kept its half. It is clean, simple, and honest about what your site contains. The other half, the readers who honor it, arrived in the coding tools and not yet in search. Write the file for the agents that use it now, and let the rest catch up on its own time.
llms.txt specification, Answer.AI, v2 · Ahrefs analysis of 137,000 sites, 2026 · SE Ranking study of roughly 300,000 domains, May 2026 · John Mueller of Google on llms.txt and server logs
Share this perspective
More insights
Adjacent perspectives.
Bttr. Field Brief
The brief Bttr. writes for senior buyers.
Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.