Quick answer
Should I block AI crawlers on my website?
How to decide which AI crawlers to allow, what each one does, and what blocking them changes.
The short answer
Decide separately about AI search, model training and user-requested fetches, because providers use different crawlers for each. Blocking a search crawler such as OAI-SearchBot can remove you from that assistant’s answers. Blocking a training crawler such as GPTBot or ClaudeBot doesn’t. Robots.txt isn’t access control, so protect private content with a login.
Answered by Stefan Velikov, Webflow Premium Partner since 2022
The detail
- 1OpenAI: OAI-SearchBot for ChatGPT search, GPTBot for training, ChatGPT-User for fetches a user asks for.
- 2Anthropic: Claude-SearchBot for search, ClaudeBot for training, Claude-User for fetches a user asks for.
- 3Google: Google-Extended controls certain uses of crawled content for Gemini training and grounding. It is a robots.txt token, not a separate crawler, and it doesn’t control inclusion or rankings in Google Search.
- 4User-requested fetches follow each provider’s own rules. OpenAI says robots.txt rules may not apply to ChatGPT-User requests; Anthropic says its bots honour robots.txt.
- 5Private information belongs behind authentication, not just a robots.txt rule.
If you want to be found in AI answers
Allow the search crawlers: OAI-SearchBot and Claude-SearchBot, and Googlebot for Google’s AI features. Block them and those assistants can’t cite your pages.
If you don’t want your content used for training
GPTBot and ClaudeBot provide separate controls for training-related crawling. Google-Extended also controls certain Gemini grounding uses, so review that separately. Blocking Google-Extended doesn’t affect inclusion or rankings in Google Search.
How to set it up
Add rules per user agent in robots.txt on every subdomain. Changes take effect after the crawler rereads the file; OpenAI says about a day for search.
Review the list once in a while: providers add and rename crawlers.
Related questions
Does blocking GPTBot remove my business from ChatGPT?
No. GPTBot is for training. ChatGPT search uses OAI-SearchBot.
Will every user-triggered fetch follow robots.txt?
Not always. OpenAI says robots.txt rules may not apply to user-initiated ChatGPT-User requests.
Should a business block AI crawlers?
Most businesses that want to be found allow the search crawlers and decide separately about training.
Does blocking AI crawlers hurt Google rankings?
No. Google-Extended doesn’t affect Google Search. Blocking Googlebot would.
Can I block AI crawlers by IP address?
Some providers, including OpenAI and Anthropic, publish IP ranges. For crawling preferences, follow each provider’s robots.txt guidance. IP blocking is a separate access restriction and shouldn’t replace those settings.
AI search
Show up when buyers ask an AI assistant who to hire.