The bot-wall, when your WAF blocks bots robots.txt allows

What the scan checks: it cross-references two signals. Does robots.txt allow GPTBot? And does the homepage, fetched as GPTBot, return real content or a bot-management challenge? When robots.txt says allow but the edge serves a challenge or a 403, the check fails: your file makes a promise the WAF is breaking.

Why it matters for AI search visibility

This is the most common silent killer of AEO programs. Everyone checks robots.txt, almost nobody checks the WAF. A well-meaning bot-management rule (Cloudflare’s “Bot Fight Mode,” an aggressive rate limit, a JavaScript challenge on all traffic) catches AI crawlers because they come from datacenter IP ranges. robots.txt says the door is open; the security layer slams it shut. The result is the same as an outright block: the engine cannot read the page.

How to fix it

The fix lives at the CDN or WAF, not in robots.txt.

• Cloudflare: turn off “Bot Fight Mode” for the zone (it has no allowlist), and instead add a WAF custom rule that skips the challenge for verified bots. Use the cf.verified_bot_category or the verified-bots field so real GPTBot, ClaudeBot, and PerplexityBot from published IP ranges pass, while spoofers do not.
• Other WAFs: add the AI bot user agents to your bot allowlist, ideally combined with reverse-DNS or published-IP verification so the allowance is not spoofable.
• Exempt robots.txt, sitemap.xml, and llms.txt paths from challenges entirely. Crawlers must always be able to read those files.

The important caveat, which the scan states honestly: a CDN challenge is IP-keyed. Our scanner spoofs the bot UA from one location, so a challenge here does not prove a verified bot from an official IP range is blocked. Before concluding you are walled off, cross-reference your server logs (Layer 2 crawler analytics) for successful hits from verified bot IPs. If verified bots are getting 200s in your logs, the real-world situation is better than the spoofed-UA test suggests.