Signal Briefs

For eighteen months, llms.txt has been sold as the entity layer's missing declaration — a markdown file at the domain root telling large language models what a site is and which parts matter. It was a reasonable hypothesis. By mid-2026 it has been measured, and the measurement is null. This brief records the result, because a discipline that only publishes its confirmations is not a discipline, and because the reason it failed is more interesting than the failure.

The Facts

Adoption stalled rather than compounded. As of June 2026, 8.7% of the top 1,000 websites publish an llms.txt file on a conservative measure holding unreachable domains in the denominator; among reachable sites in that sample the rate is 15.8%. A separate SE Ranking study across 300,000 domains found 10.13% adoption. After eighteen months of near-continuous industry conversation, the file sits on roughly one site in ten — and the distribution is inverted from what an authority signal would predict: high-traffic sites adopt at 8.27%, mid-traffic sites at 10.54%. Large, authoritative domains are less likely to publish one than mid-tier domains.

The citation evidence is worse than flat. SE Ranking trained an XGBoost model on AI-citation data and tested whether llms.txt presence predicted citation frequency. Removing the llms.txt variable improved the model's accuracy. The file was not a weak signal. It was noise. Search Engine Land separately reported eight of nine sites seeing no measurable traffic change after implementation.

The mechanism behind the null result is the simplest possible one: nobody reads it. An analysis of more than 500 million LLM bot traffic events found that GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended overwhelmingly skip /llms.txt and crawl HTML directly. John Mueller has noted that no AI crawler has claimed to extract information via the file; Google has reportedly likened it to the keywords meta tag — a comparison that should be read as the epitaph it is. No IETF RFC had materialised as of April 2026; the convention remains community-maintained.

Why It Failed

The failure is instructive because the file is technically well-formed. The syntax is stable, the semantics are clear, the cost of publishing is half a day. None of that mattered, and the reason it did not matter is the reason the keywords meta tag died in 2009.

A self-declaration channel that only the declarer can write is worthless to the reader. The value of a signal to a machine is a function of what it costs to fake. Schema markup is checkable — it makes claims that cross-reference against other entities, other sites, and the model's existing graph, and inconsistency is detectable. An llms.txt file makes assertions about itself, to itself, with no verification surface and no cost to inflate. A rational consumer of that channel discounts it to zero, and the crawler logs show that consumers did exactly that, immediately, without announcement.

This is the distinction Entity Engineering has drawn from the start, and it is worth restating plainly: ontological presence is established through cross-validation, not through declaration. Identity Integrity, Structural Continuity, Signal Provenance, and the Validation Loop all describe a system in which claims are checkable against something external — schema linkage, temporal consistency, proof of execution, recognition stability across independent platforms. llms.txt has none of those properties. It is a claim with no counterparty. The discipline predicted this outcome, and should say so now rather than after it becomes consensus.

What It Is Actually For

The file is not worthless. It has been sold in the wrong market.

IDE and coding agents fetch llms.txt routinely — Cursor, Windsurf, Claude Code, GitHub Copilot, Cline, and Aider all look for /llms.txt and /llms-full.txt when pointed at a documentation site, and the established pattern is direct: the agent identifies which dependency owns a feature, fetches that library's llms.txt, and pulls only the linked pages it needs before writing code. LangChain shipped mcpdoc, an open-source MCP server that exposes llms.txt files to host applications. This is a live, load-bearing ecosystem.

The distinction is the consumer. Search and answer crawlers are trying to evaluate a source, and for evaluation an unverifiable self-description is inadmissible. A coding agent that has already been pointed at a documentation site is trying to navigate it, and for navigation an author-curated index is exactly right — the trust question was settled the moment the developer named the target. llms.txt is a routing surface, not a credibility surface. It works where trust is already established and fails where trust is the thing being decided.

That reframing gives the file a real, narrow role: a business-to-agent routing convenience for institutions whose surfaces agents are already being directed toward. It is optionality, priced at half a day, on the possibility that a major provider later decides to honour it. Ship it if the cost is genuinely half a day. Do not report it as an AI-visibility initiative, and do not let it appear in an audit rubric as a scored citation factor, because the only study to test that relationship found it made predictions worse.

The Generalisable Lesson

The llms.txt cycle is a template, and the template will run again — on the next proposed file, header, or manifest that promises to let an institution tell the models what it is. The tell is always the same, and it is available before the data arrives: ask what it costs to lie in the channel. If the answer is nothing, the channel will be discounted to nothing, and the only question is how many quarters the industry spends discovering it.

Entity clarity was never a file. It is the property of an institution whose claims about itself survive contact with everything else the model knows — the schema that resolves, the authorship that checks out, the temporal record that does not contradict itself, the third-party surfaces that corroborate rather than diverge. That property cannot be published at a root path. It has to be earned across time and cross-validated across systems, which is exactly why it remains scarce, and exactly why it remains defensible.

The institutions that spent 2025 and 2026 shipping llms.txt files and calling it AI readiness have a measurable result to show for it. The result is zero. That is a useful number, and it belongs in the record.

‍