llms.txt: What It Does, What It Doesn’t, and Whether Enterprise Sites Need One
The request usually arrives as a one-line ticket from the CMO’s office, sent the morning after someone senior read about it: do we have an llms.txt? A developer generates one that afternoon. It lists the product pages, the solutions pages and a few case studies, and the team deploys it to the root of the domain. Then a quarter passes and nobody can say whether it did anything.
Publishing the file alone is not evidence of improved AI visibility. llms.txt is a proposed Markdown guide that helps agents navigate a site’s content, but a request for the file does not establish that it influenced an answer or citation. For enterprise teams, the stronger priority is accurate content that machines can access and interpret. Contentstack implementation partner eight25 brings enterprise CMS experience to that work, including Sophos’s consolidation onto Contentstack. An llms.txt file can support the resulting architecture without becoming the strategy itself.
Ignoring it entirely would be its own mistake, though. There is one context where llms.txt earns its keep, and a useful exercise hiding inside it.
llms.txt is a guide for agents, not a shortcut to AI citations.
Jeremy Howard proposed the format in September 2024. The idea was reasonable: HTML pages carry navigation, scripts and layout, and a model working inside a limited context window would do better with a short, curated index of what a site contains and where the clean versions live.
The format is lightweight. Written in Markdown, llms.txt can sit at the site root or under a path such as /docs/. The only required section is an H1 naming the project or site; optional summaries, context and H2 link lists follow a specified order. The August 2026 revision clarifies path coverage and recommends HTML links or HTTP Link headers for discovery. It also permits Markdown page URLs with .md appended or the original extension replaced. An “Optional” section remains a convention for secondary links.
# Example Corp
> Example Corp builds payroll software for companies with 500 to 5,000
> employees in the US and Canada. It does not serve sole traders or
> companies outside North America.
## Products
- [Payroll](https://www.example.com/payroll.md): Core payroll engine, US and Canadian tax filing
- [Time tracking](https://www.example.com/time.md): Hourly and shift workers, integrates with Payroll
## Customers
- [Case studies](https://www.example.com/customers.md): Named customers with headcount and rollout timelines
## Optional
- [Press](https://www.example.com/press.md): Announcements since 2021The proposal primarily targets inference: an agent consulting information while helping a user. It does not exclude training use. Its practical value depends on a system discovering the file, fetching it and using the linked resources. Publication alone establishes none of those steps.
Fetching is limited, and a fetch does not prove influence.
Ahrefs’s June 2026 study examined 137,210 domains using its Web Analytics service that received traffic in May. Twenty-eight percent published llms.txt, and 97 percent of those files received no requests that month. The sample skews toward technical and SEO-aware sites, so its adoption rate should not be generalized to the whole web.
Among recorded requests, SEO audit tools accounted for 21.7 percent and general crawlers for 13.1 percent. AI retrieval bots represented 1.1 percent, while AI agents and agentic infrastructure accounted for 10.5 percent. The study measured fetching, not changes in citations or visibility. Its findings support caution about investing in the file, rather than a claim that no AI system uses it.
Google’s position is narrower and clearer: Google Search does not use llms.txt as a special signal, including for its generative AI features. Google may still crawl or index the file as an ordinary URL. Its guidance also says maintaining llms.txt for other systems neither helps nor harms a site’s visibility or rankings in Google Search. That statement should not be extended to every AI provider or agent workflow.
A curated index does not replace accessible pages, internal links or accurate on-page information. Its contents should agree with the resources it describes. Automated generation makes publication easier: documentation platforms and CMS plugins already offer it. But availability and adoption are different from demonstrated influence on AI answers. For an enterprise team, the useful question is whether a relevant agent workflow consumes the file and whether maintaining it adds meaningful effort.
The costs are small, and they surface later.
On a marketing site the file takes an afternoon to create. Three costs follow it.
The first is indexing. Google can discover and index a linked llms.txt file as an ordinary text resource. If you want to keep it out of Google Search, serve an X-Robots-Tag: googlebot: noindex response header and allow Googlebot to crawl it so the directive can be read. Before applying broader indexing restrictions, check the requirements of any systems expected to consume the file.
The second is drift. A manually maintained llms.txt can become outdated when teams rename pages, retire products or change URLs during a replatform. Broken links and obsolete descriptions reduce its usefulness. If you publish the file, give it an owner, generate it from maintained content where practical, and validate its links during migration QA.
The AI visibility checks your CMS migration QA needs
The third cost is displacement. Completing an llms.txt ticket can distract from access problems that deserve attention first: robots.txt rules blocking intended search bots, CDN challenges preventing retrieval, or essential product facts available only through client-side JavaScript. These issues can restrict a provider’s ability to retrieve and use your pages, although their effects vary by system. Publishing llms.txt does not resolve them. Test crawler access and the actual response content before treating the file as progress toward AI visibility.
Start with a representative period of CDN or edge access logs. Check requests to llms.txt and your key content pages, including status codes and user agents. Origin logs alone may miss requests served from cache, and an HTTP 200 response may still contain a challenge page or empty template. Logs establish access patterns; response inspection establishes what was delivered.
# Who is actually reading llms.txt?
grep '"GET /llms.txt' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn
# Are retrieval crawlers reaching your key pages, and with what status?
grep -E 'OAI-SearchBot|ChatGPT-User|Claude-SearchBot|PerplexityBot' access.log \
| awk '{print $9, $7}' | sort | uniq -c | sort -rn | head -20These commands are starting points for Apache-style combined logs; adapt them to your logging format and include path-specific files such as /docs/llms.txt. Add Claude-User when checking user-directed retrieval. User-agent labels alone do not verify a bot’s identity, and a successful request does not prove a citation. Track access and answer visibility separately.
The useful part is the writing exercise, not the file.
Drafting an llms.txt can expose gaps in how a company explains itself. Useful inputs include a factual company description, concise product summaries, clearly defined audiences and verified customer examples approved for publication. Where names or offerings are easily confused, explain the distinction directly.
These facts should be available on the pages buyers and machines already use. If essential information exists only in a sales deck or internal FAQ, bring it into maintained website content. An llms.txt can point to that material, but it should not become the sole public source of important product or company facts.
Publish those facts in accessible page content, supported by relevant structured data that matches what visitors can see. Schema markup and structured data, what still earns its place in 2026 Contentstack provides content types, delivery APIs and publishing webhooks that developers can use to reuse approved facts across pages and generate an llms.txt from the same source. That automation must be implemented, and the frontend must still deliver readable content. The enterprise advantage is consistent content management across outputs, with less duplicate maintenance as products, regions and languages change.
So does an enterprise site need one?
For documentation used by coding assistants, it is often worth generating. For other enterprise sites, the decision should depend on a defined consumer, maintenance effort and the readiness of the underlying content.
The one place it earns its keep: developer documentation
Coding assistants work against libraries and APIs they were not trained on, and developers routinely hand them a docs index so the agent can find the right reference page. A clean llms.txt with Markdown versions of each page is exactly what that workflow needs. Documentation platforms such as Mintlify generate llms.txt and llms-full.txt on every build for that reason.
A person can point an assistant directly to the file, and compatible tools can discover it through advertised links or response headers. That gives documentation teams a concrete consumption workflow to support and test.
The caution comes from the same place. Adyen has written publicly that llms.txt can speed up the spread of broken code snippets, because an agent treats whatever is in its context as fact. A docs index is only as good as the pages it lists, and an agent will reproduce a stale example from llms-full.txt with full confidence.
Everywhere else: it depends on who maintains the file
| Situation | Recommendation |
|---|---|
| Developer portal, API or SDK documentation used by coding assistants | Usually worthwhile. Generate the index and Markdown resources, then test them with the intended tools. |
| Marketing site using Contentstack or another structured CMS | Optional. Generate it from maintained content if there is a defined use case; prioritize accessible pages and accurate facts. |
| Marketing site without a maintenance owner | Establish ownership or automation before publishing, regardless of CMS architecture. |
| Site being replatformed | Plan generation during implementation and validate links against final production URLs before launch. |
| A relevant AI platform or partner explicitly requires the file | Implement and test its stated requirements. Confirm successful consumption rather than assuming publication is enough. |
A specific consumer with documented requirements is a stronger reason to publish llms.txt than a general promise of AI visibility. Where crawler access or essential page content is broken, address those issues first. Keeping approved facts in structured Contentstack entries makes it easier to add new outputs later, without assembling another disconnected document.
Frequently Asked Questions
Does llms.txt help with Google AI Overviews or AI Mode?
Google says llms.txt does not help visibility or rankings in Google Search, including its generative AI features. Follow Google’s existing Search guidance: keep important content accessible and indexable, publish useful information, and ensure any structured data matches the visible page. Meeting those requirements does not guarantee inclusion in an AI Overview or AI Mode answer.
Can llms.txt stop AI companies from training on our content?
No. llms.txt does not control access or opt content out of training. Providers document separate robots.txt controls for training crawlers such as GPTBot and ClaudeBot, and search crawlers such as OAI-SearchBot and Claude-SearchBot. Configure them according to your policy. User-directed retrieval can follow different rules, and robots.txt is not authentication or a guarantee that every third party will comply.
Should we also publish llms-full.txt?
Publish llms-full.txt when a specific workflow benefits from a consolidated text export. Documentation is a common example: Mintlify offers it alongside llms.txt. Check the intended tool’s context limits, exclude restricted material and automate updates. For a marketing site, use it only where there is a clear consumer and purpose.
How do we know if anything reads our llms.txt?
Check CDN or edge logs for requests to the root file and any path-specific llms.txt files, then inspect user agents, status codes and delivered content. Requests from audit tools show auditing activity, not use in AI answers. Identifiable agent requests show fetching, but logs alone cannot establish whether the file influenced an answer or citation.
Before you assign the llms.txt ticket, check what AI crawlers can already reach
Before assigning the llms.txt ticket, establish where approved company and product facts live, how pages deliver them, and which agents can access them. Contentstack provides the structured content foundation; implementation determines how consistently those facts reach the website and other outputs. eight25’s Sophos case study describes consolidating two Sitecore estates onto Contentstack and continuing with managed services after launch. For teams planning a replatform, that combination of content architecture and ongoing ownership is more relevant than publishing a standalone file.