Article
Which AI crawlers should your website allow?
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended: what each AI crawler does, what blocking it costs you, and which ones to let in.
Published 2 min readBy Visibility
Short answer
Allow the crawlers that fetch pages while an assistant is answering, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot: blocking them stops your pages being cited. Training crawlers such as GPTBot, ClaudeBot and Google-Extended are a business choice; most local businesses gain more from allowing them than from blocking.
On this page
What are the two kinds of AI crawler?
Training crawlers collect pages that may be used to train future models. They shape what an assistant knows about you from memory, months from now.
Live crawlers fetch pages at the moment an assistant is answering a question with web search, or when a person asks it to read a specific page. They decide whether you can be cited today.
What does each AI crawler do?
These are the crawlers our free AI crawler check looks for, and what blocking each one costs:
| Crawler | robots.txt name | What it does |
|---|---|---|
| GPTBot (OpenAI) | GPTBot | Collects pages that inform what ChatGPT knows about you over time. |
| OAI-SearchBot (ChatGPT search) | OAI-SearchBot | Fetches your pages while ChatGPT is answering. Blocking this one stops you being cited live. |
| ChatGPT-User | ChatGPT-User | Visits your site when a person asks ChatGPT to look at it directly. |
| ClaudeBot (Anthropic) | ClaudeBot | Collects pages that inform what Claude knows about you. |
| Claude-SearchBot | Claude-SearchBot | Fetches your pages while Claude is answering with web search. |
| PerplexityBot | PerplexityBot | Perplexity cites sources on every answer. Blocking this removes you from those citations. |
| Google-Extended (Gemini) | Google-Extended | Controls whether Google may use your pages to train Gemini and to ground answers in the Gemini app. It does not affect Google Search, where AI Overviews and AI Mode appear. |
| Applebot-Extended | Applebot-Extended | Controls whether Apple Intelligence can use your pages. |
Which ones should a local business allow?
For a business that wants customers to find it through AI assistants, the simplest good answer is to allow all of them. The live crawlers are essential. The training crawlers mean future models know you exist.
Blocking training crawlers is a reasonable choice for publishers whose content is the product, such as news or research sites. For a restaurant, a law firm or a shop, the website is there to be found.
How do I allow or block a crawler?
In robots.txt, each crawler is addressed by its name. This blocks GPTBot only and allows everything else:
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.yourdomain.gi/sitemap.xmlRemember that robots.txt is not the only gate. Firewall and bot-protection settings at your host or CDN can block AI crawlers too, whatever robots.txt says. See let AI crawlers read your website.
Common questions
Do AI crawlers obey robots.txt?
The crawlers listed here are run by the companies that publish them, and each company documents that its crawler follows robots.txt. robots.txt is a request, not a lock, so it does not stop crawlers that choose to ignore it.
If I block GPTBot, will ChatGPT stop mentioning me?
Not necessarily. GPTBot affects training. ChatGPT can still find and cite you through web search if OAI-SearchBot is allowed, and it may still know about you from other sites.
Does Google-Extended affect AI Overviews?
No. Google says Google-Extended controls use of your pages for training Gemini and grounding answers in the Gemini app, and does not affect Google Search, where AI Overviews and AI Mode appear.