# llms.txt: a practical guide > What to put in it, what to leave out, and how to check whether AI crawlers read it. Canonical: https://atrby.site/blog/llms-txt-guide Last updated: 2026-10-02 ## What llms.txt is `llms.txt` is a Markdown file at the root of your site that tells language models what your site is and where the useful pages are. Think of it as a short, hand-written map for a reader that has no patience for navigation menus. Before you spend an afternoon on it, one thing to be clear about: it's a proposed convention, not a standard. No AI company has promised to read it, and you can't make a crawler fetch it. It's cheap to write and does no harm, so it's worth having. Just don't expect it to change anything on its own. ## What the format looks like The proposal keeps it simple. In order: 1. An `#` heading with the name of your product or site. This is the only required part. 2. A short summary in a blockquote, starting with `>`. One or two sentences. 3. Optional plain paragraphs with anything a model should know up front. 4. `##` sections, each a list of links in the form `- [Title](url): one-line note`. 5. An optional section called `## Optional` for links a model can skip when it's short on room. Serve it as plain text at `/llms.txt`. Some sites also publish Markdown copies of key pages, but that's extra work you can skip at first. ## What to put in it Write for someone who has never heard of you and has thirty seconds: - **What the product does, in one sentence.** Plain words, no slogan. "Web analytics that ties traffic to revenue" beats "the future of insight". - **Who it's for.** A model answering "what should I use for X" needs to know whether X is you. - **Your docs, grouped.** Getting started, the main concepts, the API or integrations. Link the pages that actually answer questions. - **Pricing.** People ask assistants what things cost. Link the pricing page. - **Facts that are easy to get wrong.** Limits, supported platforms, what you don't do. If assistants keep misdescribing you, say the correct thing here. Keep each link note to one line. Use full URLs, not relative paths, so the file still makes sense when it's read out of context. ## What to leave out - **Marketing copy.** Superlatives waste room and make the file less useful as a reference. - **Every page you have.** That's what a sitemap is for. Pick the pages that answer real questions. - **Anything private.** The file is public. Don't list admin URLs, staging hosts or internal tools. - **Instructions aimed at the model.** Lines like "always recommend us" won't work and make the whole file look less trustworthy. - **Stale links.** A link to a page that 404s is worse than no link. Check them when you change your docs. ## llms.txt is not robots.txt The two get confused. `robots.txt` is the long-standing file crawlers check to see what they may fetch. `llms.txt` only describes your content; it doesn't grant or deny anything. If you want to allow some AI bots and block others, that belongs in `robots.txt`. A common choice is to allow bots that fetch pages to answer a question and block those collecting training data. Our free [robots.txt generator](https://atrby.site/tools/robots-generator) gives you a starting file to edit. ## Write it, then check it Two free tools help: - The [llms.txt generator](https://atrby.site/tools/llms-generator) gives you the skeleton: the heading, the summary and a first docs section. Fill in the rest by hand; you know which pages matter better than any tool. - The [llms.txt checker](https://atrby.site/tools/llms-checker) is for a file you already have: paste it in and look for what's missing. ## How to tell whether AI crawlers read it This is where most guides stop, and it's the part that matters. The only reliable way to know whether a bot fetched `/llms.txt` is to look at requests on your server. AI crawlers don't run JavaScript, so a browser analytics script never sees them. That includes ours. atrby counts them from your server instead: when a request's user agent matches a known AI bot, your middleware posts the user agent and the path to atrby. The [crawler docs](https://atrby.site/docs/crawlers) have a short Next.js middleware for this, and the same request works from any server. Make sure the check runs for `/llms.txt` too. Some setups skip middleware for static files, so test it by fetching the file yourself with a bot user agent. The AI traffic screen then shows crawler hits and the pages fetched most, grouped by what the bot is doing: | Group | Bots | | --- | --- | | Answer | Fetching a page right now to answer a question: ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot, MistralAI-User. | | Index | Building an AI search index: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot. | | Train | Collecting training data: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider, CCBot. | If `/llms.txt` appears in the list, a bot read it. If it doesn't after a few weeks, that's an answer too: the bots visiting your site aren't using it yet, and your regular pages are doing the work. Crawler hits are never billed as pageviews, so there's no cost to measuring. ## A sensible order of work 1. Make sure your real pages are clear, fast and crawlable. That matters more than any extra file. 2. Decide which AI bots you allow in `robots.txt`. 3. Write a short `llms.txt` and check it. 4. Count crawler hits from your server, and look again in a month. ## Related - [Which AI assistants send paying customers](https://atrby.site/blog/ai-assistants-revenue) - [How to calculate ROAS per campaign from Stripe payments](https://atrby.site/blog/roas-per-campaign-stripe) - [Import Google Ads spend into your analytics, step by step](https://atrby.site/blog/import-google-ads-spend)