Allow search and user-triggered AI bots

What the scan checks: it reads robots.txt and fails if any search-grounding or user-triggered AI bot is fully disallowed. Blocking training bots is a separate, softer warning (see below), because that can be a legitimate licensing choice.

Why it matters for AI search visibility

AI crawlers fall into three classes, and they are not interchangeable:

• Training crawlers (GPTBot, ClaudeBot, CCBot) bulk-fetch content for future model training. Blocking them costs long-term presence in the model’s memory, not today’s citations.
• Search-grounding crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, bingbot) build the retrieval index the assistant searches when answering. Block these and you vanish from retrieval-time citations.
• User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) fetch your page in real time when a user’s question references it. This is the single most consequential class to keep open, because it maps directly to the moment a prospect asks a model about your category.

The rule of thumb from the manual: block training only with a real licensing strategy; never block search-grounding; never block user-triggered.

How to fix it

In robots.txt, make sure none of these appear under a Disallow: / group. To explicitly welcome them:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

If you have a training-bot block in place (GPTBot, ClaudeBot), keep it only if you are a publisher or IP-heavy business pursuing a licensing deal. For most SaaS, services, e-commerce, and local businesses, that block is giving up the long lever to protect content with no commercial uniqueness. And blocking is not a kill-switch anyway: 70 to 92% of blocking sites still get cited from other sources.