Skip to content

Free tool · Free, no signup

AI crawler checker: is GPTBot blocked on your site?

Find out whether GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are allowed to read your site, or whether a line in your robots.txt has been keeping them out. The check fetches your robots.txt once, parses it the way a crawler does, and reports each AI user agent as allowed or blocked, including rules you never wrote yourself.

By Telman Gadimov · Updated

Reads one file, your /robots.txt. Free, no signup.

How it works

Free, no signup. One request, a second or two.

  1. Enter your domain

    The site, not a page: robots.txt lives at the root, so example.com is enough.

  2. We read your robots.txt once

    One request for /robots.txt, parsed the way a crawler parses it, including the wildcard group every bot falls back to.

  3. See each AI crawler allowed or blocked

    GPTBot, ClaudeBot, PerplexityBot and Google-Extended, each with the rule that blocks it, and how to fix it if the block was an accident.

A real run

Example result

theguardian.com, checked on 9 October 2026. Its robots.txt disallows ClaudeBot and PerplexityBot from the whole site. GPTBot and Google-Extended are not named, so they fall under the wildcard group, which allows them.

Result · https://theguardian.com

AI crawlers allowed in robots.txt

Blocking

2 of the 4 AI crawlers are blocked in robots.txt. They cannot read the pages you want them to quote.

  • ClaudeBot — Disallow: /
  • PerplexityBot — Disallow: /

How to fix it: Remove the Disallow rules for those user agents in /robots.txt, or drop the blanket block they inherit.

GPTBot, ClaudeBot, PerplexityBot and Google-Extended have to reach your pages before any of them can quote you.

Crawlers tested

  • GPTBot
  • ClaudeBot
  • PerplexityBot
  • Google-Extended

AI crawlability starts with robots.txt

Blocked AI crawlers are usually an accident. A staging site gets a blanket “Disallow: /” that follows it to production. A CMS plugin adds bot rules with a checkbox someone ticked in 2023, a common story on WordPress. A security vendor ships an AI-crawler blocklist as a default. None of that shows up anywhere you would normally look, and the symptom is silence: you simply never appear in AI answers, with nothing to tell you why.

The check costs one HTTP request, and if it comes back clean you can stop wondering about this particular explanation and go look at the other seven.

The user agents that matter

One trap worth knowing: OpenAI runs two. GPTBot is for training, OAI-SearchBot fetches pages for ChatGPT search. Plenty of sites block GPTBot on principle, keep OAI-SearchBot open without realising it, and are fine. Plenty of others block both and wonder why ChatGPT has never heard of them.

User agentRun byWhat it does
GPTBotOpenAICrawls pages for training. Blocking it keeps you out of the model, not out of ChatGPT search.
OAI-SearchBotOpenAIThe one that fetches pages for ChatGPT search results. Blocking this is what removes you from live ChatGPT answers.
ClaudeBotAnthropicFetches pages Claude reads and cites.
PerplexityBotPerplexityIndexes pages for Perplexity answers, which cite their sources by default.
Google-ExtendedGoogleControls whether Google uses your content for Gemini models and Gemini grounding. Separate from Googlebot: blocking it does not affect search rankings or AI Overviews, which are part of Search.

The checker tests the four that carry the most weight for being recommended: GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.

A robots.txt that lets the AI crawlers in

If the check came back blocked and you want the simplest fix, this is it. Explicit groups beat the wildcard, so these override a blanket disallow above them.

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap.xml

Swap the sitemap line for your own. If you want to keep a crawler out of one section rather than the whole site, replace “Allow: /” with the specific “Disallow” paths, since a group with no rules at all is treated as allowing everything.

Every day, not once

The GEO audit

The free check runs once, on one URL. The GEO audit keeps the results for your site, turns failures into To-dos, and rechecks a page after you fix it.

See the GEO audit Open the demo

Keep going

AI readiness checkAll eight checks: schema, llms.txt, server-rendered copy, about and pricing pages.AI visibility checkerAsk a real buying question on four engines and see which ones name you.llms.txt checkerOnce the crawlers can get in, check the file that tells them what to read.
All free tools

Questions

What is AI crawlability?

Whether the bots AI companies send can fetch your pages at all. It is decided first by robots.txt, then by anything in front of your site that refuses bots, such as a firewall or a CDN rule. A page an AI crawler cannot fetch is a page no AI answer can quote.

What does this AI crawler checker do?

It fetches your /robots.txt, parses it the way a crawler does, and tells you which AI user agents are disallowed. That includes rules you never wrote yourself, like a blanket "User-agent: * / Disallow: /" that every bot inherits.

Should I allow AI crawlers?

It depends on what you sell. If you want to be recommended in AI answers, they have to be able to read you, and blocking them is a guaranteed way to be left out. If your business is selling the content itself, blocking is a legitimate choice. What is not a choice worth making is blocking them by accident, which is the common case.

Is this a GPTBot checker?

Yes, and it checks three more at the same time: ClaudeBot, PerplexityBot and Google-Extended. Each is reported as allowed or blocked, with the robots.txt rule that blocks it.

How do I unblock GPTBot in robots.txt?

Remove or narrow the Disallow rule under its User-agent group. If you have no GPTBot group at all, it inherits whatever "User-agent: *" says, so a blanket disallow there blocks it too. Adding an explicit group with "User-agent: GPTBot" and "Allow: /" overrides the wildcard.

Is robots.txt the only thing that blocks AI crawlers?

No, and this is where people get caught out. A Cloudflare bot-fight rule, a WAF, a 403 on non-browser user agents, or an X-Robots-Tag header can all block a crawler that robots.txt permits. This tool reports what robots.txt says; it does not prove a crawler will get a 200.

Does blocking Google-Extended hurt my Google rankings?

No. Google-Extended is a separate token for Gemini models and Gemini grounding. Googlebot handles search crawling, and AI Overviews are part of Search, so neither is affected by blocking Google-Extended.

Does allowing crawlers mean AI will recommend me?

No. It means you are eligible. Engines recommend brands that independent sources corroborate, so crawler access is the floor. The full readiness check covers the rest of the floor, and the AI visibility checker tells you whether any of it is working.

Track it every day instead of once

CueScout runs these checks on a schedule and turns the gaps into To-dos.

See pricing All free tools