AI crawler checker: does your robots.txt block ChatGPT?
Type your domain and we'll read your robots.txt the way each AI crawler does, then tell you which ones are allowed and which are blocked. A blocked crawler can't read your pages, so the assistant behind it has less to go on when someone asks it who to hire. The check is free, runs in a few seconds and stores nothing.
Which crawlers does it check?
The ten user agents below. Some collect pages for training, some index pages for an assistant's search results, and some fetch a page when a user asks about it. Blocking any of them takes away part of what that assistant can read about you.
- GPTBotOpenAI. Collects pages OpenAI may use to train its models.
- OAI-SearchBotOpenAI. Indexes pages for ChatGPT search results.
- ChatGPT-UserOpenAI. Fetches a page when a ChatGPT user asks about it.
- ClaudeBotAnthropic. Collects pages Anthropic may use to train Claude.
- Claude-SearchBotAnthropic. Indexes pages for Claude's web search.
- anthropic-aiAnthropic. An older Anthropic token some robots.txt files still use.
- PerplexityBotPerplexity. Indexes pages for Perplexity's answers.
- Google-ExtendedGoogle. Controls whether Google may use your pages for Gemini. It does not affect Google Search.
- CCBotCommon Crawl. Builds the open web archive many AI models are trained on.
- Applebot-ExtendedApple. Controls whether Apple may use your pages to train its AI models.
How do I unblock an AI crawler?
Open yourbusiness.com/robots.txt in a browser. If you see a group like User-agent: GPTBot followed by Disallow: /, delete it or change Disallow: / to Allow: /. If the block comes from a User-agent: * group, add a short group for each AI crawler with Allow: / on its own. A crawler that's named in its own group ignores the * group.
On WordPress, robots.txt is often written by an SEO or security plugin, so change it in the plugin settings or your edit may be overwritten. On Wix, Squarespace and Shopify, look for robots.txt or crawler settings in the SEO section. Then run this check again.
What does "some paths blocked" mean?
The crawler can read your homepage, but a Disallow rule shuts it out of part of the site. Blocking /wp-admin/ or a checkout page is normal. Blocking /services/ or your blog isn't, because those are the pages an assistant would quote.
More on this in our guide: Is your website blocking AI? How to check robots.txt for GPTBot, ClaudeBot and PerplexityBot.
Common questions
What is robots.txt?
A plain text file at the root of your site, like yourbusiness.com/robots.txt, that tells crawlers which pages they may read. It's grouped by User-agent: each crawler follows the group that names it, and only falls back to the User-agent: * group if no group names it.
Should I block GPTBot?
If you want ChatGPT to recommend you, we'd say no. GPTBot collects pages for training, and OAI-SearchBot indexes them for ChatGPT search, so blocking them leaves ChatGPT working from other sites' descriptions of you. Publishers sometimes block training crawlers on purpose. For a local service business that wants customers, there's little to gain from it.
Does blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended only controls whether your pages are used for its Gemini models. Googlebot, which ranks you in Search, is a separate crawler and isn't affected by a Google-Extended rule.
My robots.txt allows everything. Why am I still not recommended?
Because access is only the first step. Assistants also lean on your Google Business Profile, reviews, the directories in your industry, roundup articles and pages that answer the exact question asked. Our free report checks those and shows who's named instead of you.
Can a crawler be blocked even when robots.txt allows it?
Yes. Some hosts, CDNs and security plugins block AI crawlers at the firewall, and Cloudflare offers a setting that does this. That never shows up in robots.txt, so if this check passes and you still suspect a block, look at your host's or CDN's bot settings.
Do you store the domain I check?
No. We fetch one file, your /robots.txt, show you the result and keep nothing. The check doesn't call any AI model and doesn't start a report.