AI crawlers and robots.txt: which bots should you allow?

Why the bot name matters

A robots.txt file contains rules for specific user-agent tokens. A broad rule can affect more crawlers than intended. Search visibility controls and model-training preferences may be separate, so do not assume that one setting controls every use of your content.

OpenAI: search, training and user-requested access

• OAI-SearchBot: used to surface websites in ChatGPT search. OpenAI recommends allowing it if you want your site to be eligible for ChatGPT search results.
• GPTBot: used to crawl content that may be used to improve OpenAI foundation models. Its robots.txt control is separate from OAI-SearchBot.
• ChatGPT-User: used for some user-initiated page fetches. OpenAI says it is not an automatic web crawler and robots.txt rules may not apply to these user actions.

Read the current OpenAI crawler documentation before making a policy decision.

Anthropic: Claude search, training and user fetches

Anthropic documents three separate agents: Claude-SearchBot for improving search result quality, ClaudeBot for collecting content that may contribute to model training, and Claude-User for user-directed page retrieval. Anthropic says its bots respect standard robots.txt directives and documents a non-standard Crawl-delay extension.

See Anthropic’s current crawler policy.

Perplexity: search crawler and user-triggered fetcher

Perplexity documents PerplexityBot for surfacing and linking websites in Perplexity search results, and Perplexity-User for requests initiated by users. The provider says the user-triggered fetcher generally ignores robots.txt because it responds to a direct user request. Review its official crawler documentation for current IP and firewall guidance.

Example: allow search crawlers but decide training separately

The following is an illustrative starting point, not a universal recommendation. It allows the named search crawlers while separately disallowing the listed training crawlers. Change it to match your own content policy and verify the final file before publishing.

User-agent: OAI-SearchBot
Allow: /
Disallow: /api/

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /
Disallow: /api/

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /
Disallow: /api/

User-agent: *
Allow: /
Disallow: /api/

Because a named user-agent group can take precedence over the wildcard group, repeat any important exclusions (such as /api/) inside each explicitly named group you allow. This example does that for the API path. Important: robots.txt is a cooperative crawler directive, not authentication or a security boundary. It cannot guarantee that every client will comply, and a user-triggered fetch can work differently from an automated crawler. Keep private data behind authentication and server authorization, never behind robots.txt alone.

How to test your policy safely

1. Back up your current robots.txt and identify any existing disallow rules.
2. Check the official documentation for every provider you care about; bot names and policies can change.
3. Publish the file at the root of each relevant hostname, including subdomains that need their own rules.
4. Fetch the public robots.txt and inspect server/CDN logs to see whether legitimate crawler requests are challenged.
5. Use Search Console and provider-specific analytics where available. A robots.txt allow rule is not proof of indexing, ranking or citation.

What AEOCheck can and cannot verify

AEOCheck can inspect the public robots.txt response and report rules that affect recognized crawler tokens. It cannot prove a provider has crawled your site, verify every bot IP, bypass your CDN’s firewall, or confirm that a model cited your content. Review the methodology for the full scan scope and use the complete check list to see related rules.