Back to blog

What Is llms.txt? How to Create One and Control AI Crawlers

GEO Course

What Is llms.txt? How to Create One and Control AI Crawlers

By Sanita | Generative Engine Optimization Specialist

What llms.txt is, how to create one, what it can and can't do, and how to allow or block AI crawlers like GPTBot, ClaudeBot and PerplexityBot in robots.txt.

Updated 31 October 2026 · Part of the GEO guide series

Short answer: llms.txt is a proposed markdown file, usually at yourdomain.com/llms.txt, that gives AI tools a short, clean summary of your site and links to the pages that matter most. It is a proposal, not an official web standard, and Google says you don't need any new AI text files to appear in its AI features. Use it as a low-cost helper for AI agents. For real AI visibility, focus on crawler access (robots.txt rules for each AI bot), content that is readable in plain HTML, and content worth citing.

What llms.txt is

The llms.txt proposal, written by Jeremy Howard in September 2024 and updated as v2 in August 2026, suggests adding a markdown file that:

  • Names the site or project and gives a one-paragraph summary.
  • Adds key background and guidance.
  • Lists links to the most useful pages, grouped by section, with a short note for each.

It can sit at the site root or under a path (for example /docs/llms.txt for documentation). The proposal also suggests offering clean markdown versions of important pages.

The author notes that thousands of sites now publish one, documentation platforms generate them automatically, and AI companies publish llms.txt files for their own developer docs. Adoption is real, especially for documentation; whether any AI system uses it when choosing what to cite is not publicly confirmed.

What llms.txt can and can't do

It can It can't
Give agents and tools a clean map of your key pages Guarantee that any AI system reads it
Help coding and docs assistants find the right documentation Make AI tools cite or recommend you
Act as a curated summary you control Replace robots.txt, sitemaps or good content

What Google says: for AI Overviews and AI Mode, "You don't need to create new machine readable files, AI text files, or markup to appear in these features." Normal SEO best practices apply.

How to create an llms.txt file

# AIRankMeter

> AIRankMeter is an AI visibility and SEO toolkit that checks how websites appear in Google and AI answer engines.

## Tools
- [AI Visibility Checker](https://airankmeter.com/ai-visibility-checker/): Test whether AI engines mention a brand
- [AI Citation Analyzer](https://airankmeter.com/ai-citation-analyzer/): See which sources AI engines cite

## Guides
- [What is GEO](https://airankmeter.com/blog/what-is-geo/): Generative engine optimization explained
- [What is a sitemap](https://airankmeter.com/blog/what-is-a-sitemap/): XML sitemaps for SEO

(Example structure.)

Steps:

  1. Write a short summary of what your site or business is.
  2. List 10 to 30 of your most useful, stable pages with one-line descriptions.
  3. Save as plain text/markdown at /llms.txt; make sure it returns status 200.
  4. Keep it updated when key pages change.
  5. Many CMS plugins can generate one automatically; review the output.

AI crawlers: training, search and user requests

Most AI companies now separate their bots by purpose. That lets you block training while staying visible in AI search. Bot names and behavior change, so confirm them in each company's current crawler documentation before you edit robots.txt.

Company Bot (robots.txt token) Purpose
OpenAI OAI-SearchBot ChatGPT search results. Opted-out sites won't appear in ChatGPT search answers.
OpenAI GPTBot Collecting content for training foundation models
OpenAI ChatGPT-User User-initiated visits; robots.txt may not apply
Anthropic Claude-SearchBot Improving Claude's search results
Anthropic ClaudeBot Collecting content that may be used for training
Anthropic Claude-User User-initiated visits
Perplexity PerplexityBot Perplexity search results (not used for training foundation models)
Perplexity Perplexity-User User-initiated visits; generally ignores robots.txt
Google Google-Extended Controls whether Google may use your content for Gemini training and grounding. Does not affect Google Search inclusion or ranking. No separate crawler; it is a robots.txt control token.

Google's AI Overviews and AI Mode use Google Search's normal crawling (Googlebot), so blocking Google-Extended does not remove you from them.

robots.txt recipes

Allow AI search, block training:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Allow everything (default): if robots.txt has no rules for these bots, they are allowed.

Slow a bot down: Anthropic supports Crawl-delay for ClaudeBot, for example Crawl-delay: 1.

Notes:

  • Rules apply per host: add them on every subdomain you want covered.
  • User-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt.
  • Check that your firewall or CDN bot protection isn't blocking the bots you want to allow. OpenAI and Perplexity publish IP ranges for their bots.

Common robots.txt Mistakes With AI Bots

  • Blocking everything by accident. A leftover User-agent: * followed by Disallow: / from staging blocks search and AI crawlers alike.
  • Blocking the search bot when you meant the training bot. Disallowing OAI-SearchBot removes you from ChatGPT search answers, while disallowing GPTBot only affects training collection.
  • Security plugins and CDNs. Some add bot blocks by default. Check the live file at /robots.txt after each plugin or CDN change, and review firewall rules too.
  • Rules on the wrong host. A rule on www.example.com does not cover blog.example.com.
  • Treating robots.txt as security. It is a request that well-behaved bots follow, not access control. Use authentication for private content.

What matters more than llms.txt

  1. Crawlable HTML: not every AI crawler is known to render JavaScript the way Google does, so the safest option is to serve main content in the HTML (server-side rendering or prerendering).
  2. Access: robots.txt and firewall rules that let the bots you want reach your pages.
  3. Clean URLs and canonicals: one version of each page, consistent internal links and a current XML sitemap.
  4. Citable content: clear answers, specific facts, original data and consistent brand information. See how to write content for AI retrieval.
  5. Google basics: indexed, snippet-eligible pages for AI Overviews and AI Mode. See technical SEO foundations.

How to test AI access

  • Fetch your pages with JavaScript disabled to see what non-rendering crawlers get.
  • Check server logs for the bot user agents above and their status codes.
  • Validate robots.txt rules for each bot token.
  • Ask AI tools about your brand and products repeatedly, and track whether you are mentioned and cited, for example with an AI visibility checker.

Example: Deciding What to Allow

(Illustrative.) A publisher wants readers to find its articles through AI search but does not want its archive used for model training. It allows OAI-SearchBot, Claude-SearchBot and PerplexityBot, disallows GPTBot and ClaudeBot, and decides separately about Google-Extended. It then runs the AI crawler access check to confirm that the live robots.txt and server do what it intended, and a free SEO audit to check that the pages themselves are crawlable and fast. A different publisher with a licensing business may choose to block more. It is a business decision, and neither choice is technically wrong.

FAQs

What is llms.txt?

A proposed markdown file at the root of a website (or under a path) that gives AI tools a short summary of the site and links to its most important pages. It is a proposal, not an official standard.

Does Google use llms.txt?

Google says you don't need to create new machine-readable files or AI text files to appear in AI Overviews or AI Mode; standard SEO practices apply.

Does llms.txt help you get cited by ChatGPT?

There is no public confirmation that ChatGPT or other assistants use llms.txt to choose citations. For ChatGPT search, allowing OAI-SearchBot in robots.txt matters, along with crawlable, useful content.

How do I block AI crawlers from training on my site?

Disallow the training bots in robots.txt, for example GPTBot (OpenAI), ClaudeBot (Anthropic) and Google-Extended (Google). You can still allow search bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot.

Should I Add llms.txt to My Site?

It is optional and cheap. It can help documentation tools and AI agents that look for it, but there is no public evidence that it improves citations in search or chat answers. Do it after the basics, such as crawlable content and correct robots.txt, and keep it accurate.

How Often Should I Check My robots.txt?

After every CMS, plugin, CDN or firewall change, and at least once a quarter. Open the live file, not a copy, and test the bots you care about.

Does blocking Google-Extended remove me from Google Search or AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. AI Overviews use Google Search's regular crawling.