robots.txt sanity, the four never-dos
What the scan checks: it reads your robots.txt and flags four specific mistakes: a blanket Disallow: / for all bots, any AI bot blocked, and a Crawl-delay of 30 or more. If robots.txt is missing, that is treated as allow-all (a warning, not a fail).
Why it matters for AI search visibility
Section titled “Why it matters for AI search visibility”robots.txt is the first file every crawler reads, and one wrong line has outsized blast radius. The classic disaster is a staging robots.txt with User-agent: * / Disallow: / shipped to production, which removes the entire site from every crawler at once. Blocking AI bots forfeits both training-time presence (your content in future models) and retrieval-time citations, usually for no real benefit.
How to fix it
Section titled “How to fix it”Open https://yourdomain.com/robots.txt and check for each never-do.
1. No blanket disallow. This block, if it targets *, wipes you out:
User-agent: *Disallow: /Remove it, or scope disallows to the specific paths you actually want hidden (/admin/, /cart/).
2. Do not block search-grounding or user-triggered AI bots. See the R2 guide for the full bot taxonomy. Never disallow OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, or bingbot.
3. No Crawl-delay: 30 or higher. Googlebot ignores crawl-delay, but most AI bots honor it, so a high value caps a bot at under 3,000 pages a day and starves your own crawl budget. Remove the directive or keep it in single digits.
4. Publish one deliberately. An absent robots.txt defaults to allow-all, which is fine, but shipping a real file (with your sitemap reference) is the professional move.
A safe minimal robots.txt:
User-agent: *Allow: /Sitemap: https://example.com/sitemap.xml