Visibility
Waitlist

Guide · Chapter 2 of 6

Let AI crawlers read your website

Which AI crawlers need to reach your website, how robots.txt can block them without anyone noticing, and how to check yours in under a minute.

Published 3 min readBy Visibility

Short answer

AI assistants read websites with their own crawlers, such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. If your robots.txt file or a firewall setting blocks them, assistants cannot read or cite your pages, however good they are. Checking takes under a minute with a free tool.

On this page
  1. What is an AI crawler?
  2. How do AI crawlers get blocked without anyone noticing?
  3. How do I check whether AI crawlers can reach my site?
  4. Which crawlers should I allow?
  5. What should you do this week?
  6. Common questions

What is an AI crawler?

A crawler is a program that fetches web pages automatically. Google has Googlebot; AI companies have their own. Some collect pages that shape what a future model knows. Others fetch pages live, at the moment an assistant is answering someone's question.

The distinction matters. Blocking a training crawler affects what future models learn about you. Blocking a live search crawler stops you being cited in answers today. The crawler guide lists each one and what blocking it costs.

How do AI crawlers get blocked without anyone noticing?

In two common ways:

  • robots.txt. A small file at yourdomain.gi/robots.txt tells crawlers what they may read. Rules written years ago for Google, or copied from a template, can turn away crawlers that did not exist when they were written. A line such as User-agent: * followed by Disallow: / blocks everyone.
  • Firewall and bot protection. Hosting companies and content delivery networks, Cloudflare among them, offer settings that block AI crawlers. Depending on the service and when the account was set up, such a setting may already be on. robots.txt can say "welcome" while the firewall says "no".

Nobody gets an alert when this happens. The site keeps working for people and for Google; it just stops being readable by some assistants.

How do I check whether AI crawlers can reach my site?

Put your address into our free AI crawler check. It reads your robots.txt and tells you, crawler by crawler and in plain English, whether each one is allowed, partly allowed or blocked.

The check reads robots.txt only. If it says everything is allowed but you use Cloudflare or a similar service, also look in that service's dashboard for settings named along the lines of "AI bots", "AI crawlers" or "bot fight mode".

Which crawlers should I allow?

For a business that wants to be recommended, allow at least the crawlers that fetch pages while answering: OAI-SearchBot and ChatGPT-User for ChatGPT, Claude-SearchBot for Claude and PerplexityBot for Perplexity. Blocking those removes you from live citations.

Training crawlers such as GPTBot, ClaudeBot and Google-Extended are a business decision. Allowing them lets future models learn about you; some publishers block them for their own reasons. For most local businesses, being known is worth more than keeping the homepage out of a training set.

A robots.txt that allows everything, and names your sitemap, looks like this:

User-agent: *
Allow: /

Sitemap: https://www.yourdomain.gi/sitemap.xml

What should you do this week?

  1. 1Run the AI crawler check on your website.
  2. 2If anything shows as blocked, send the result to whoever manages your site and ask them to allow those crawlers.
  3. 3If you use Cloudflare or similar, check its AI bot settings as well.

Common questions

Will allowing AI crawlers slow my website down?

For a typical small business site, no. The crawlers fetch a small number of pages. If a particular crawler ever caused a problem, you can slow or block that one without blocking the rest.

Does blocking Google-Extended remove me from Google's AI Overviews?

No. Google says Google-Extended controls whether your pages can be used to train Gemini and to ground answers in the Gemini app, and does not affect inclusion in Google Search. AI Overviews and AI Mode are part of Google Search.

Where is my robots.txt?

At the root of your domain: add /robots.txt to your homepage address. If nothing loads, you do not have one, which means crawlers are allowed by default.

Read next