Is your website blocking AI? How to check robots.txt for GPTBot, ClaudeBot and PerplexityBot

A plain text file at the root of your website that tells crawlers which pages they may read. It's a request, not a lock, but the big AI companies say their crawlers follow it.

The file is split into groups. Each group starts with one or more User-agent lines naming a crawler, then the Allow and Disallow rules for it. A crawler follows the group that names it and ignores the rest. Only a crawler with no group of its own falls back to the User-agent: * group.

OpenAI runs GPTBot (training), OAI-SearchBot (ChatGPT search) and ChatGPT-User (fetching a page a user asked about). Anthropic runs ClaudeBot and Claude-SearchBot, and some files still use the older anthropic-ai token. Perplexity runs PerplexityBot.

Google-Extended and Applebot-Extended are tokens rather than separate crawlers: they control whether Google and Apple may use your pages for their AI models. Google says Google-Extended doesn't affect Google Search. CCBot belongs to Common Crawl, whose archive many AI models are trained on.

Look for each crawler's name. If it has its own group, it reads that group and nothing else. Disallow: / on its own means the whole site is off limits. An empty Disallow: means nothing is.

If a crawler has no group, read the User-agent: * group. Disallow: / there blocks every crawler that isn't named elsewhere, AI or not. Deeper rules like Disallow: /wp-admin/ only close part of the site, which is normal for admin and checkout pages.

We see the same few causes. A security or SEO plugin with an "block AI bots" option switched on. A staging setting left in place after a site went live. A snippet copied from a news article about publishers blocking AI. A host or CDN preset that blocks AI crawlers at the firewall, which doesn't appear in robots.txt at all.

None of these is unusual, and the owner rarely knows. That's why crawler access is the first check in every fix plan we write.

If a crawler is named with Disallow: /, delete that group or change the line to Allow: /. If the block comes from the * group, add a group for each AI crawler you want to let in, each with Allow: / under it. Named groups win over the * group, so this works without changing the rules for everyone else.

On WordPress, change it in the plugin that writes the file, or your edit may be overwritten. On Wix, Squarespace and Shopify, look in the SEO settings. Then check the live file again, and give the crawlers a few weeks to come back.

Some publishers do that: block GPTBot and ClaudeBot, which collect training data, and allow OAI-SearchBot and Claude-SearchBot, which fetch pages for search answers. It's a reasonable choice for a site whose writing is the product.

For a local service business that wants to be recommended, we'd allow both. Training data shapes what an assistant knows about you when it doesn't search, and there's little for you to protect by keeping it out.