A seemingly simple question often appears: Should we add llms.txt so artificial intelligence reads our website correctly—and at the same time does not take our content for training? There is no universal yes or no. Three different things have been mixed together: crawler access, search visibility and guidance for AI agents.
Separating them matters more than the file itself. Otherwise we may block traffic we want while failing to obtain the protection we expected.
First clarify what you want to achieve
- I want Google or ChatGPT Search to find the page. Address indexing, technical SEO and the particular search crawler's rules.
- I do not want to provide content for model training. Address the training crawler and the service's terms.
- I want to offer an agent a concise documentation map. This is where
llms.txtmay make sense. - I want to protect non-public content. No text file is enough. You need authentication, authorisation and real security.
These decisions are technically independent. A company's search crawler and training crawler may have different names and rules.
What llms.txt is—and is not
LLMs.txt is a proposal for a simple Markdown file at a website's root. It offers an agent a short description and links to important sources. For extensive documentation it can be a directory: product, API, terms and common procedures.
It is not an access policy. It does not authoritatively say who may use content for training, prevent downloading a page or guarantee citation in results. Google explicitly says no special AI file is needed for its AI features in Search and that llms.txt adds nothing to Google Search.
On a company website I would deploy it when we can maintain a short, accurate and useful source list—not as an empty SEO gesture. An outdated map is worse than none because it offers invalid prices, old instructions or removed pages.
Robots.txt controls crawling, not trust or security
The robots.txt file tells cooperating crawlers where they may or may not go. It is public, its rules depend on the user agent, and it is not a firewall against a disobedient robot.
Blocking in robots.txt is not a reliable way to remove a URL from a search engine. Google may know the address from other links even if it cannot read its content. For non-public data, use sign-in. For a page that should not appear in results, typically use noindex on a crawlable URL or the appropriate status code.
There is no single switch for all AI robots
OpenAI: search and training can be decided separately
OpenAI documents separate crawlers. OAI-SearchBot discovers content for ChatGPT search, while GPTBot concerns possible use to improve generative models. You may therefore allow the first and block the second:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /ChatGPT-User identifies a request triggered by a user and is not a mechanism for inclusion in search. Read the provider's current terms rather than copying an old user-agent list.
Google: Search and Google-Extended are not the same
According to Google Search documentation, AI Overviews and AI Mode use the search index and its systems. Googlebot access therefore affects ordinary search and eligibility for these features.
Google-Extended is a separate product token for using content to improve Gemini models and for grounding in some products. Google says blocking it does not affect inclusion or ranking in Search. It is not a substitute for Googlebot rules.
Other services have their own rules
Anthropic distinguishes ClaudeBot, Claude-SearchBot and user-initiated loading. Perplexity describes PerplexityBot for its search index. Names and purposes can change. Keep a list of important services and check it against their primary documentation.
How I would decide on a company website
- Write down the business goal. Do I want organic traffic, citations, protection of paid content, or all of these with different rules?
- Check the technical foundation. Canonical URLs, indexing, status codes, sitemap and internal links. No special file can save a missing foundation.
- Separate crawlers by purpose. Search, training and user-triggered loading are different categories.
- Actually secure non-public data. Client offers, administration and internal documents are not protected merely by instructing a robot.
- Add LLMs.txt only with a content owner. Someone must ensure links and descriptions remain accurate.
- Measure the result. Monitor logs, Search Console and visits from relevant sources; the file's existence is not completed optimisation.
If you want to become a source for AI answers, start with content and evidence rather than a file at the website root. I discuss this in How AI and Google choose sources. To review crawler settings without risking organic traffic, use the contact page.