Are AI crawlers actually visiting your site?
AI crawler visits are being seen. Access that is allowed on paper and access that happens are different things.
What the check looks at
This one cannot be answered from outside. The page check reads your site; whether GPTBot, ClaudeBot or PerplexityBot ever arrive is only visible from inside — in the requests the site receives. The agent on the site records every such request and shows them page by page.
Why it matters
Access that is allowed on paper and access that happens are different things. A site can pass every door check and still never be fetched because nothing links to it, or because a layer nobody checked drops the requests. Observed visits are the proof; their absence is a question.
How to see it on your own site
Search your server or CDN logs for the AI user-agents, verify the source IPs against the vendors' published ranges, and count per page. Or install the agent: it does exactly this on every request and separates a page fetched to answer a live question from one crawled to learn.
AI bot verifier → — free, no signup.
How to fix it
- Fix the door checks first — robots.txt, the bot filter, walls.
- Make the pages reachable: in the sitemap, linked from other pages, returning 200.
- Then watch: AI traffic in the dashboard shows which engine opened which page, and whether it was to answer someone right then.
What the agent does: the agent on the site records every AI request as it happens — this is the counting the page check cannot do from outside.
Related checks
- robots.txt allows AI crawlers
- No CDN or bot filter blocks AI crawlers
- sitemap.xml is present and valid
Part of the 23 checks · group: Crawler access — robots.txt, CDN/bot filter, login walls, noindex, HTTP status, sitemap, canonical.
Questions people ask
Why is this "not checkable from outside"?
Because a visit is an event on your server. An outside check can only see your site the way a visitor does; it cannot see who else is visiting.
Does Google Analytics show AI visits?
Not the crawler fetches — those run no JavaScript, so no tag fires. Analytics may show a person who clicked through from an AI answer, which is a different and much smaller number.
Start with the free check. Then tell the agent what you need.
The free check runs the 23 checks on your public pages. Nothing is installed, nothing changes until you press it.
Free check from outside in minutes · the report by email · nothing installed yet

