llms.txt best practices

llms.txt is a proposal, not an engine requirement. The current text is v2 at llmstxt.org, updated 10 August 2026. It describes a Markdown file that tells an agent what a site is and which links to open. Google’s AI optimization guide says a special Markdown file neither helps nor harms visibility in Google Search. OpenAI and Perplexity do not list the file as a crawler control. CiteScore does not log into an answer engine.

Ten live files, and the one choice each gets right, are on llms.txt examples. The generator that copies a starter file into your browser is the llms.txt generator. What the file is not is on the llms.txt checker.

The order the spec describes

A file that follows the spec, at /llms.txt or at a subpath such as /docs/llms.txt, uses this order. A file covers the URLs under its path. Where more than one file applies, the spec says agents should use the most specific one.

1. An optional byte-order mark.
2. An H1 with the name of the project or site. This is the only required section.
3. A blockquote with a short summary. The spec calls this the key information for understanding the rest of the file.
4. Zero or more Markdown sections that are not headings: paragraphs or lists with details.
5. Zero or more H2 sections. Each one is a list of links, written [name](url), optionally followed by : and a note.

The heading “Optional” is a convention for links an agent can skip. The changes page says v2 removed the mechanical meaning that heading had for a context-expansion tool. A parser that drops everything under Optional is implementing the old tool, not the current spec.

# Example docs

> Example is a documentation site. This file lists the pages an agent should open first.

The H1 is the only required line. The blockquote is the short definition.

## Docs

- [Getting started](https://example.com/docs/start.md): Install and the first request
- [API reference](https://example.com/docs/api.md): Every public endpoint

## Optional

- [Changelog](https://example.com/changelog.md): Releases an agent can skip

The spec also asks that the links point at LLM-friendly content, such as a Markdown version of the page (page.md or page.html.md). It recommends rel="alternate" with type="text/markdown", and rel="describedby" pointing at the llms.txt that covers the page. Those can be HTML link elements or an HTTP Link header.

Mistakes that fail a read of the spec

• No H1. The spec calls the H1 the only required section. A file that starts with a paragraph has no name.
• Headings before the file lists. The free-form block may not contain headings. An H3 in that block breaks the order.
• An H2 that is not a link list. After the first file-list heading, each section is a Markdown list of [name](url) items. A bare URL, or a title with no link, is not that item.
• HTML at the URL. A soft 404, a redirect to the homepage, or an error page that returns 200 is not the Markdown file. Open /llms.txt and read the first line.
• A second sitemap. The spec says a sitemap lists pages for search engines, is often too large for a context window, and does not mark which URLs matter. Curate the list. Leave the full URL set in sitemap.xml.
• Links that do not resolve. The spec says agents view the file and then follow the relevant links. A 404, a login wall, or a tracking URL with a session parameter is a link that does not lead to the page you named.
• Treating Optional as a command. In v2 it is a name for secondary links. Keep the section if you want a short list and a long list. Do not expect clients to implement the retired tool.

llms-full.txt is not a section of this spec. The proposal does not require a second file that inlines every page.

How to validate

Read the file against the list above. Then request each URL in the H2 lists and confirm it returns the page the note describes. The generator builds a file with an H1, a blockquote, and absolute links. It does not check that those URLs are live. The llms.txt checker describes CiteScore’s fetch: /llms.txt returned a successful response and a body that is not empty. That fetch does not parse headings, and it does not follow the links. An HTML error page with a 200 can still count as found. Validate the Markdown yourself.

Robots.txt is a different file. A model that can read llms.txt still cannot fetch a page the search crawler is disallowed from crawling. Those tokens are on AI crawlers and robots.txt for AI crawlers. The cross-engine page is how to get cited by AI answer engines.

Check the public files

The free check records whether /llms.txt is a non-empty success. It does not grade the H1, the blockquote, or the links.

Run the free check on the host. It reads the homepage, robots.txt for GPTBot, PerplexityBot, and Google-Extended, /llms.txt, a sitemap, and JSON-LD on the homepage. It does not log into an answer engine. One URL continues at checkout.

Sources

• The /llms.txt file, v2. The section order, the H1 requirement, file lists, and where the file may live.
• Changes. Why Optional no longer has a mechanical meaning.
• Optimizing your website for generative AI features on Google Search. Google’s statement that a special Markdown file is not required for Search.