Free tool
Find out whether GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are allowed to read your site, or whether a line in your robots.txt has been keeping them out.
Blocked AI crawlers are usually an accident. A staging site gets a blanket “Disallow: /” that follows it to production. A CMS plugin adds bot rules with a checkbox someone ticked in 2023. A security vendor ships an AI-crawler blocklist as a default. None of that shows up anywhere you would normally look, and the symptom is silence: you simply never appear in AI answers, with nothing to tell you why.
The check costs one HTTP request, and if it comes back clean you can stop wondering about this particular explanation and go look at the other seven.
One trap worth knowing: OpenAI runs two. GPTBot is for training, OAI-SearchBot fetches pages for ChatGPT search. Plenty of sites block GPTBot on principle, keep OAI-SearchBot open without realising it, and are fine. Plenty of others block both and wonder why ChatGPT has never heard of them.
| User agent | Run by | What it does |
|---|---|---|
| GPTBot | OpenAI | Crawls pages for training. Blocking it keeps you out of the model, not out of ChatGPT search. |
| OAI-SearchBot | OpenAI | The one that fetches pages for ChatGPT search results. Blocking this is what removes you from live ChatGPT answers. |
| ClaudeBot | Anthropic | Fetches pages Claude reads and cites. |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers, which cite their sources by default. |
| Google-Extended | Controls Gemini and AI Overviews use of your content. Separate from Googlebot, so blocking it does not affect classic search rankings. |
The checker tests the four that carry the most weight for being recommended: GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.
If the check came back blocked and you want the simplest fix, this is it. Explicit groups beat the wildcard, so these override a blanket disallow above them.
User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / Sitemap: https://example.com/sitemap.xml
Swap the sitemap line for your own. If you want to keep a crawler out of one section rather than the whole site, replace “Allow: /” with the specific “Disallow” paths, since a group with no rules at all is treated as allowing everything.
Next
It fetches your /robots.txt, parses it the way a crawler does, and tells you which AI user agents are disallowed. That includes rules you never wrote yourself, like a blanket "User-agent: * / Disallow: /" that every bot inherits.
It depends on what you sell. If you want to be recommended in AI answers, they have to be able to read you, and blocking them is a guaranteed way to be left out. If your business is selling the content itself, blocking is a legitimate choice. What is not a choice worth making is blocking them by accident, which is the common case.
Remove or narrow the Disallow rule under its User-agent group. If you have no GPTBot group at all, it inherits whatever "User-agent: *" says, so a blanket disallow there blocks it too. Adding an explicit group with "User-agent: GPTBot" and "Allow: /" overrides the wildcard.
No, and this is where people get caught out. A Cloudflare bot-fight rule, a WAF, a 403 on non-browser user agents, or an X-Robots-Tag header can all block a crawler that robots.txt permits. This tool reports what robots.txt says; it does not prove a crawler will get a 200.
No. Google-Extended is a separate token that governs Gemini and AI Overviews. Googlebot handles classic search crawling and is unaffected.
No. It means you are eligible. Engines recommend brands that independent sources corroborate, so crawler access is the floor. The full readiness check covers the rest of the floor, and the AI visibility checker tells you whether any of it is working.