Returns HTTP 200 to AI crawlers

What the scan checks: it fetches your page while pretending to be each major AI crawler (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, bingbot) and confirms every one gets an HTTP 200, not a 403, a 5xx, or a bot-management challenge page.

Why it matters for AI search visibility

If a crawler cannot fetch the page, none of your content, schema, or answer blocks exist as far as that engine is concerned. This is the very first gate: a page that 403s to GPTBot is invisible to ChatGPT no matter how good it is. The most consequential misconfiguration is blocking user-triggered fetchers (ChatGPT-User, Claude-User), because those fire at the exact moment a real person asks the model about your category.

How to fix it

The block almost always lives at the CDN or WAF, not in your app. Allow verified AI bots through:

Cloudflare: turn on the managed “Verified bots” allowance, or add a WAF custom rule that skips challenges when the user agent matches the AI bots and the request passes bot verification. Do not just allow the UA string alone, since that is spoofable.
Whitelist the user agents at your edge: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, bingbot.
Remove blanket rate-limit or JS-challenge rules that catch datacenter IP ranges, since that is where these bots egress from.

One caveat the scan itself flags: a CDN challenge is IP-keyed. Our fetcher spoofs the UA from one vantage point, so a challenge here does not always prove a verified bot from a published IP range is blocked. If T1 shows a challenge rather than a hard block, confirm against your server logs (Layer 2) before assuming you are shut out. A genuine non-200 with no challenge is a hard fail worth fixing today.