Free tool · Free, runs in your browser
Robots.txt generator for AI crawlers like GPTBot
Which crawlers do you want to block?
Everything is allowed by default, which is almost always what you want. Tick a box only if you have a reason.
Your robots.txt
# robots.txt # Generated with cuescout.com/tools/robots-txt-generator User-agent: * Disallow: /admin Disallow: /cart Disallow: /checkout
Upload this to the root of your domain so it is served at https://yourdomain.com/robots.txt. A robots.txt in a subdirectory does nothing.
How it works
Free, no signup. Built in your browser.
Choose who may crawl
Training crawlers and answer crawlers are listed separately, so you can keep your pages out of model training without dropping out of AI answers.
Add paths and your sitemap
Disallow the paths no crawler needs, such as admin or checkout, and point every crawler at your sitemap.
Publish it and test the live file
Put it at the root of your domain, then run the AI crawler checker to see which agents your live file actually blocks.
A real run
Example result
Our own robots.txt, as cuescout.com served it on 9 October 2026. It names no AI crawler at all: one wildcard group allows everyone, AI agents included, and keeps them out of the app’s private paths. When you want every crawler treated the same, that is all it takes.
User-Agent: * Allow: / Disallow: /admin Disallow: /api Disallow: /checkout Disallow: /citations Disallow: /dashboard Disallow: /demand Disallow: /feedback Disallow: /forgot-password Disallow: /inbox Disallow: /insights Disallow: /login Disallow: /onboarding Disallow: /opportunities Disallow: /outreach Disallow: /plan Disallow: /prompts Disallow: /published Disallow: /r/ Disallow: /readiness Disallow: /report Disallow: /reset-password Disallow: /s/ Disallow: /scout Disallow: /settings Disallow: /site-audit Disallow: /sources Disallow: /visibility Disallow: /warmup Host: https://cuescout.com Sitemap: https://cuescout.com/sitemap.xml
Training crawlers and answer crawlers are different things
This is the distinction the whole page exists for. When a publisher says “we blocked the AI bots”, they usually mean they do not want their archive used as training data. Fair enough. But the agent that trains and the agent that indexes for answers are separate, and they respect separate rules.
GPTBot, ClaudeBot, Google-Extended and Applebot-Extended are the training side. Block them and your pages stop feeding future model weights, while your presence in today's answers is untouched. OAI-SearchBot, PerplexityBot, ChatGPT-User and Claude-User are the answer side. Block those and you disappear from the citations, which for most businesses is the thing they were trying to get more of.
A publisher with a paywall and a licensing team has good reason to block both. A startup that would quite like ChatGPT to recommend it almost never does, and yet the blanket block is what most of them end up with, usually because it arrived in a template.
The rule that catches people out
robots.txt matching is not additive across groups. A crawler obeys the single most specific group that names it, and ignores the rest. So a file with User-agent: * and Disallow: /blog, followed by User-agent: GPTBot with Disallow: /, does not give GPTBot both rules. GPTBot reads only its own group.
The practical consequence runs the other way too, and it is the one we see most: a site with a broad Disallow: / under the wildcard, added during a staging deploy years ago and never removed, is blocking every AI crawler without a single AI-specific line in the file. If you have never actually read your own robots.txt, read it before you generate a new one.
What this file cannot do
It cannot enforce anything. Crawlers that ignore robots.txt are not rare, and scrapers building datasets for resale are not known for their manners. It also cannot remove content already absorbed into a model that was trained last year. Blocking is forward-looking, and if something needs to be genuinely unreachable, that is a job for authentication.
Every day, not once
The GEO audit
The free check runs once, on one URL. The GEO audit keeps the results for your site, turns failures into To-dos, and rechecks a page after you fix it.
Keep going
All free toolsQuestions
What is robots.txt?
A plain text file at the root of your domain listing which crawlers may fetch which paths. It is a request, not a wall — well-behaved crawlers honour it and badly behaved ones ignore it entirely. If you need something actually blocked, block it at the server, not here.
Should I block GPTBot?
It depends on what you are trying to achieve, and most people who block it are trying to achieve something else. GPTBot collects pages for training future models. It is not what makes you appear in ChatGPT answers today — that is OAI-SearchBot. So blocking GPTBot is a reasonable position if you object to your writing being training data, and it does not, on its own, cost you citations. Blocking OAI-SearchBot does.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot crawls for training. OAI-SearchBot builds the search index ChatGPT reads when answering with live results. ChatGPT-User fetches a specific page when a user pastes a link and asks about it. Three agents, three jobs, one company. People routinely block all three when they meant to block the first, and then wonder why they stopped being cited.
Does Google-Extended affect my Google rankings?
No. Google-Extended controls whether your content is used to train Gemini. Google Search crawling is Googlebot, and it is unaffected. AI Overviews are also unaffected, which surprises people — opting out of Google-Extended does not opt you out of appearing in AI Overviews.
Where does robots.txt go?
The root of the domain, served at https://yourdomain.com/robots.txt. Not in a folder, not on a subdomain you also want covered — each subdomain needs its own. A robots.txt at /blog/robots.txt does nothing at all.
How do I check my current robots.txt is not blocking anything by accident?
Our AI crawler access checker fetches your live robots.txt and tells you which AI agents are blocked, including by a blanket wildcard rule someone added years ago for a different reason. That is the more common failure than a deliberate block.
Is anything I type here sent to you?
Not while you choose. The file is built in your browser. When you copy or download it, we log that use with the agents you blocked, the paths you disallowed, your sitemap URL and any crawl delay, so we can see which tools get used.
Track it every day instead of once
CueScout runs these checks on a schedule and turns the gaps into To-dos.