Free tool
Which crawlers do you want to block?
Everything is allowed by default, which is almost always what you want. Tick a box only if you have a reason.
# robots.txt # Generated with cuescout.com/tools/robots-txt-generator User-agent: * Disallow: /admin Disallow: /cart Disallow: /checkout
Upload this to the root of your domain so it is served at https://yourdomain.com/robots.txt. A robots.txt in a subdirectory does nothing.
This is the distinction the whole page exists for. When a publisher says “we blocked the AI bots”, they usually mean they do not want their archive used as training data. Fair enough. But the agent that trains and the agent that indexes for answers are separate, and they respect separate rules.
GPTBot, ClaudeBot, Google-Extended and Applebot-Extended are the training side. Block them and your pages stop feeding future model weights, while your presence in today's answers is untouched. OAI-SearchBot, PerplexityBot, ChatGPT-User and Claude-User are the answer side. Block those and you disappear from the citations, which for most businesses is the thing they were trying to get more of.
A publisher with a paywall and a licensing team has good reason to block both. A startup that would quite like ChatGPT to recommend it almost never does, and yet the blanket block is what most of them end up with, usually because it arrived in a template.
robots.txt matching is not additive across groups. A crawler obeys the single most specific group that names it, and ignores the rest. So a file with User-agent: * and Disallow: /blog, followed by User-agent: GPTBot with Disallow: /, does not give GPTBot both rules. GPTBot reads only its own group.
The practical consequence runs the other way too, and it is the one we see most: a site with a broad Disallow: / under the wildcard, added during a staging deploy years ago and never removed, is blocking every AI crawler without a single AI-specific line in the file. If you have never actually read your own robots.txt, read it before you generate a new one.
It cannot enforce anything. Crawlers that ignore robots.txt are not rare, and scrapers building datasets for resale are not known for their manners. It also cannot remove content already absorbed into a model that was trained last year. Blocking is forward-looking, and if something needs to be genuinely unreachable, that is a job for authentication.
Next
A plain text file at the root of your domain listing which crawlers may fetch which paths. It is a request, not a wall — well-behaved crawlers honour it and badly behaved ones ignore it entirely. If you need something actually blocked, block it at the server, not here.
It depends on what you are trying to achieve, and most people who block it are trying to achieve something else. GPTBot collects pages for training future models. It is not what makes you appear in ChatGPT answers today — that is OAI-SearchBot. So blocking GPTBot is a reasonable position if you object to your writing being training data, and it does not, on its own, cost you citations. Blocking OAI-SearchBot does.
GPTBot crawls for training. OAI-SearchBot builds the search index ChatGPT reads when answering with live results. ChatGPT-User fetches a specific page when a user pastes a link and asks about it. Three agents, three jobs, one company. People routinely block all three when they meant to block the first, and then wonder why they stopped being cited.
No. Google-Extended controls whether your content is used to train Gemini. Google Search crawling is Googlebot, and it is unaffected. AI Overviews are also unaffected, which surprises people — opting out of Google-Extended does not opt you out of appearing in AI Overviews.
The root of the domain, served at https://yourdomain.com/robots.txt. Not in a folder, not on a subdomain you also want covered — each subdomain needs its own. A robots.txt at /blog/robots.txt does nothing at all.
Our AI crawler access checker fetches your live robots.txt and tells you which AI agents are blocked, including by a blanket wildcard rule someone added years ago for a different reason. That is the more common failure than a deliberate block.
No. The generator is JavaScript running in your browser and makes no request. There is nothing on our side to log.