Is your CDN or bot filter challenging AI crawlers?
No CDN or bot filter blocks AI crawlers. robots.txt can say "welcome" while the bot management layer in front of the site says "prove you are human".
What the check looks at
The check fetches your homepage with each AI crawler's real user-agent and looks at what comes back: a 403, a 503, a Cloudflare "just a moment" challenge page or a managed-challenge header means the crawler is being served a wall instead of the site.
Why it matters
robots.txt can say "welcome" while the bot management layer in front of the site says "prove you are human". AI crawlers cannot solve a challenge. To them the site is an error page, and retrieval moves on to a competitor that answers. This is invisible from a browser, which is why it goes unnoticed for months.
How to see it on your own site
Fetch your homepage with curl -A "<the bot's user-agent>" and compare the status and body to a normal request. The crawler check tool runs this for eight AI user-agents and flags challenge pages by their markers.
AI Crawler Access Check → — free, no signup.
How to fix it
- In your CDN's bot management, add the AI crawlers to the allow list — most vendors (Cloudflare, Akamai, Fastly, AWS WAF) have a verified-bots category that already includes OpenAI's, Google's and Perplexity's crawlers; check that it is on.
- Verify by IP, not only by user-agent: OpenAI, Google and Perplexity publish IP-range feeds, Anthropic documents its range — the bot verifier tool checks an address against them.
- Never serve a JavaScript challenge to a verified crawler: it cannot run it.
What the agent does: the bot filter sits in front of the site, outside anything on the site can change; the agent names the rule and the vendor, and you flip it in the CDN.
Related checks
Part of the 23 checks · group: Crawler access — robots.txt, CDN/bot filter, login walls, noindex, HTTP status, sitemap, canonical.
Questions people ask
My robots.txt allows everyone — can I still fail this check?
Yes, and it is the common case. robots.txt is read by the crawler; the challenge is served by the CDN before the crawler sees anything. The two are independent.
How do I know the request really came from OpenAI?
Check the source IP against the published ranges (openai.com/gptbot.json, searchbot.json, chatgpt-user.json). The bot verifier tool does this and adds reverse DNS with forward confirmation.
Will allowing verified bots let scrapers in?
No. The allow list is for crawlers that verify by IP range. A scraper spoofing the user-agent fails the IP check and is handled as before.
Start with the free check. Then tell the agent what you need.
The free check runs the 23 checks on your public pages. Nothing is installed, nothing changes until you press it.
Free check from outside in minutes · the report by email · nothing installed yet