Resources Free tool
12 of 15 AI crawlers can fetch every path you listed; 2 are blocked from some; 1 from all.
The check runs the RFC 9309 parser from the AwesomeSEO crawler in your browser: a crawler’s own group wins over the * group, the longest matching rule decides, and Allow wins a tie. Nothing is uploaded.
Example data: saltolinen.se/robots.txt (demo workspace, example data).
5 paths · 15 crawlers that fetch pages · 2 opt-out tokens
Hover a verdict for the rule that decided it. Group: the user-agent group that applied, the crawler’s own token or *. Opt-out tokens (Google-Extended, Applebot-Extended) never fetch pages; they control what the operator’s search crawler may use for AI.
- Other user-agents named in the file
- anthropic-ai
- Not in the list of AI crawlers above: search crawlers, retired tokens or agents we have not verified.
- Sitemaps declared
- https://saltolinen.se/sitemap.xml
How the check decides
Every group that names the crawler’s product token (case-insensitive, version suffix ignored) is merged. If none does, the * groups apply. A crawler with its own group ignores the * rules entirely, which is how a site ends up blocking GPTBot from everything while meaning to block only the cart.
The matching rule with the longest path wins; when an Allow and a Disallow match with the same length, Allow wins. * matches any run of characters and a trailing $ anchors the end. No matching rule means allowed, and /robots.txt is always allowed.
Allowed means the operator’s documented crawler may fetch the path. It does not mean it has, which is what server logs show, and it does not mean the page is readable without JavaScript. Some user-initiated fetchers say they ignore robots.txt because a person asked for the page. The demo workspace shows bot visits from logs beside this access matrix.
Only tokens whose purpose the operator documents are labelled with a purpose; the rest say so. The list is the one the product uses on the Context readiness screen and is reviewed when operators change their documentation.
Related: the AI-visibility track and the practice “Decide which AI crawlers may read which pages” in the playbook.
See which of these crawlers actually visit, and what they fetch.
The demo workspace joins this access matrix with bot visits from server logs and AI referrals from GA4, for a fictional store with example data.