Do your priority pages return HTTP 200?
All pages return HTTP 200. A crawler that is rate-limited or served an error does not retry patiently.
What the check looks at
The check requests each priority page and records the final status after redirects: 200 is a page, 404 is a missing one, 429 is rate limiting, 5xx is a server error. Anything but 200 is a page a crawler did not get.
Why it matters
A crawler that is rate-limited or served an error does not retry patiently. It records the failure and moves on; the page is absent from the index until the next visit, if there is one. Rate limiting (429) is the common one on sites that defend against scrapers.
How to see it on your own site
Fetch each page with curl -I and read the status line. Check the pages in your sitemap, not only the homepage.
How to fix it
- Fix 404s by restoring the page or redirecting it (301) to its replacement.
- For 429, raise or exempt the rate limit for verified crawlers — they fetch in bursts.
- For 5xx, read the server log; a page that errors for a crawler usually errors for someone else too.
What the agent does: the agent lists the failing pages and their status; the cause is in your server or host.
Related checks
- No CDN or bot filter blocks AI crawlers
- Canonical tags point to the page itself
- sitemap.xml is present and valid
Part of the 23 checks · group: Crawler access — robots.txt, CDN/bot filter, login walls, noindex, HTTP status, sitemap, canonical.
Questions people ask
Is a redirect a failure?
No — the check follows redirects and judges the final response. A chain of several redirects is slow but not failing.
Why "priority pages"?
The check reads the homepage and the handful of pages it fetches each run. The agent, once on the site, sees the status of every request.
Start with the free check. Then tell the agent what you need.
The free check runs the 23 checks on your public pages. Nothing is installed, nothing changes until you press it.
Free check from outside in minutes · the report by email · nothing installed yet