Is your site blocking AI crawlers?
Type a domain to see whether its robots.txt lets in GPTBot, ClaudeBot, PerplexityBot, and ten more crawlers. The bots that cite you in AI answers are kept apart from the ones that collect pages for training, since blocking one group is a choice and blocking the other hides you. No account needed.
Check a site
What the checker reads
It fetches /robots.txt once and applies its rules to your home page for each crawler, the same way the crawler would.
- AI search and answersOAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User. They read pages to answer a question and link to the source.
- AI trainingGPTBot, ClaudeBot, Google-Extended, Applebot-Extended, and CCBot. They collect pages to train models.
- Search enginesGooglebot and Bingbot, so you can see that a rule meant for AI did not lock out search too.
- Named or notWhether robots.txt has its own group for the crawler, or the crawler falls back to the rules for every bot (User-agent: *).
- Sitemap lineWhether robots.txt lists your sitemap, so every crawler can find all your pages.
What the result means, and how to fix it
Allowed means the crawler may read your home page. Blocked means a Disallow rule covers it. A blocked AI answer crawler or Googlebot is marked to fix, because it keeps you out of answers or results. A blocked training crawler is not, because that is a fair choice to make.
If there is no robots.txt at all, every crawler may read the site. That is fine, but a short file with a Sitemap line helps crawlers find every page.
To stay in AI answers while keeping your pages out of training, name each group in robots.txt. A crawler follows the most specific group that names it, so these lines win over any rule for *:
# Let AI search and answer tools read the site
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Keep pages out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlWant AI tools to train on your pages too? Drop the middle group. Robots.txt is a request, not a lock. The companies behind these crawlers say their crawlers follow it, though some treat a page a person asked for directly (ChatGPT-User, Perplexity-User) differently, and it does not stop a crawler that ignores it. Some hosts and CDNs also block AI crawlers with a firewall setting that robots.txt does not show, so check that setting too if a crawler is allowed here but still cannot read the site.
Robots.txt is one of many checks. The full free audit also reads your pages, sitemap, page speed, social previews, structured data, llms.txt, and email records. In LopeBase it runs again whenever you like, for every site you run.
AI crawler questions
Something else? Contact us.
Should I block GPTBot?
It depends on what you want. GPTBot collects pages to train OpenAI's models. Blocking it does not remove you from ChatGPT search answers, which use OAI-SearchBot and ChatGPT-User. Many sites block training and allow answers, with the robots.txt lines above.
What is the difference between ClaudeBot and Claude-User?
ClaudeBot collects pages for training. Claude-SearchBot reads pages for Claude's search, and Claude-User fetches a page when a person asks Claude about it. The checker shows each one, grouped by what it is for.
Why does a crawler say "Follows the * rules"?
Your robots.txt has no group that names it, so it uses the rules under User-agent: *. If those rules block everything, that crawler is blocked too, even though you never named it.
Does it check pages other than the home page?
This checker applies robots.txt to your home page. The SEO audit in LopeBase also checks your blog, and runs again whenever you like.
Do you store the domain I check?
No. The check runs once and the answer comes straight back to this page. Nothing is saved and no email is sent.
Get new guides by email
Practical guides and free tools for running several SaaS products, plus what is new in LopeBase. Unsubscribe any time.
Or skip the inbox: start LopeBase free and see every product on one screen.