AuditsAI crawler checker
Which AI crawlers may read this page?
Enter a page and MaxiMarket evaluates your robots.txt for that exact URL, the way each crawler would: AI search crawlers such as OAI-SearchBot and PerplexityBot, AI training crawlers such as GPTBot and Google-Extended, user-initiated AI fetchers and Googlebot. It also tells you whether the page could be reached at all.
What it checks
- robots.txt rules for one URL
- AI search crawlers
- AI training crawlers
- User-initiated AI fetchers
- Googlebot access
- Bot challenges and blocked access
Check AI crawler access for a page
Free, read-only and usually done in about a minute. No account needed for your first audit.
Crawlers checked
Search, training and assistant traffic are not the same.
Blocking an AI training crawler and blocking an AI search crawler have different consequences, so MaxiMarket reports them separately. The tokens below are the ones Audit Standard 2026.4 evaluates.
| Group | Purpose | robots.txt tokens | In the report |
|---|---|---|---|
| AI search crawlers | Fetch pages for AI search and answer products. | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Applebot |
Each disallowed crawler is listed in an Info finding. Never scored. |
| AI training crawlers | Collect content for training AI models, or control whether content may be used for it. | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent |
Allowed or disallowed, per token, as an observation. Never scored. |
| User-initiated AI fetchers | Fetch a page when a person asks an AI assistant about it. | ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Google-Agent, Google-Read-Aloud |
Which of these your robots.txt names, as an observation. Never scored. |
| Search engine crawler | Googlebot also fetches pages for Google Search features, including its AI features. | Googlebot |
A warning when the audited URL is disallowed. |
Example
Reading robots.txt for one URL.
robots.txt is evaluated as RFC 9309 defines it: a crawler follows the most specific group that names it, and the * group only when no group does. With the file below, an audit of a product page reports:
- GPTBot: disallowed, reported as your AI training policy.
- PerplexityBot: allowed for the product page (only
/checkout/is disallowed), so no finding. - OAI-SearchBot and the others: allowed through the
*group.
User-agent: GPTBot
Disallow: /
User-agent: PerplexityBot
Disallow: /checkout/
User-agent: *
Allow: /Before robots.txt
Can the page be reached at all?
Crawler rules only matter if requests get through. MaxiMarket reports when robots.txt itself fails, when the page answers with a block or challenge instead of content, and how that blocking response looked.
Limits
What an AI crawler checker can’t see.
robots.txt states your rules; it does not show whether a crawler visited, and several vendors say their user-initiated fetchers may not apply it. Allowing AI crawlers makes a page eligible to be fetched, not included or cited: that is up to each AI system. For the wider picture, see the AI readiness audit.
Free during the Public Beta
Check AI crawler access for a page
The crawler results appear in the AI Discovery view of every audit, next to the technical SEO findings for the same URL.
Read how a MaxiMarket page audit works or browse the published Audit Standard.