Robots.txt checker for search and AI crawlers
Enter your domain. We fetch your live robots.txt and show which search engines, AI answer engines, AI assistants and AI training bots can reach your site, the exact rule behind each answer, and the fixes to make first.
Enter a domain above to get a crawler access score, a per-crawler breakdown with the exact rule applied, a prioritized fix list and your AI answer mentions.
Four kinds of crawlers, four different decisions
Most robots.txt advice still assumes one crawler that matters: Googlebot. In 2026 your file is read by at least four different kinds of bot, and blocking the wrong one has very different consequences.
Block a training crawler like GPTBot and you opt out of future model training. That is a licensing decision and it does not affect whether AI answers can cite you. Block an answer engine crawler like OAI-SearchBot or PerplexityBot and you remove yourself from the pool of pages those engines can cite. That is a visibility decision, and usually an accidental one.
The most common mistake we see is a single "block AI" snippet copied from a blog post, which shuts out the search crawlers along with the training ones.
Learn how AI search picks sourcesWhat each group controls
- Search engines (Googlebot, Bingbot): rankings, and the index that AI Overviews and Copilot draw from.
- AI answer engines (OAI-SearchBot, PerplexityBot, Claude-SearchBot): whether you can be cited in AI answers.
- AI assistants (ChatGPT-User, Claude-User): live fetches when a user asks an assistant to read your page.
- AI training (GPTBot, ClaudeBot, Google-Extended, CCBot): whether your content is used to train models.
A robots.txt that stays visible in AI search
If your goal is to be cited in AI answers while opting out of model training, this is the pattern. Search and answer crawlers inherit the open wildcard group; training crawlers get their own named group.
# Everyone, including search and AI answer crawlers
User-agent: *
Disallow: /admin/
Content-Signal: search=yes, ai-input=yes, ai-train=no
# Opt out of AI model training only
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlTwo details trip people up. First, a named group replaces the wildcard group for that crawler, it does not add to it, so if GPTBot had a named group with only "Disallow: /private/", it would ignore your "Disallow: /admin/" rule. Second, Google-Extended and Applebot-Extended never fetch anything themselves: they are control tokens that tell Google and Apple whether crawled content may be used for AI training, so blocking them does not affect normal crawling.
Find out if AI actually cites you
This checker shows whether AI crawlers can reach you. DataWise tracks whether ChatGPT, Perplexity and Google AI Overviews mention your brand, which prompts you win, and which competitors get cited instead. Free with an AI Ranking membership.
Robots.txt checker FAQ
What does this robots.txt checker test?
It fetches your live robots.txt and runs 12 checks: whether the file is reachable, whether search engines, AI answer engines and AI assistants can crawl your homepage, how you treat AI training crawlers, whether a wildcard rule blocks everything, whether your sitemap is referenced and valid, crawl-delay, Content-Signal preferences, whether your homepage and a sample of sitemap URLs are crawlable by Googlebot, rule hygiene, and whether your AI policy is declared at all. It also shows how often ChatGPT and Google AI Overviews mention your domain.
Which crawlers does it check?
Search engines: Googlebot and Bingbot. AI answer engines: OAI-SearchBot, PerplexityBot, Claude-SearchBot, DuckAssistBot and Applebot. AI assistants: ChatGPT-User, Claude-User and Perplexity-User. AI training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, meta-externalagent and cohere-ai. For each one you see whether it is allowed, and which exact rule decided it.
If I block GPTBot, will I disappear from ChatGPT?
No. GPTBot collects training data. ChatGPT search uses a different crawler, OAI-SearchBot, and live page fetches when a user asks come from ChatGPT-User. You can block GPTBot to opt out of training and still allow OAI-SearchBot so you can be cited in answers. The same split exists at Anthropic (ClaudeBot vs Claude-SearchBot and Claude-User) and Perplexity (PerplexityBot vs Perplexity-User).
How does a crawler decide which rule applies?
A crawler looks for a group that names it in a User-agent line. If one exists, it follows only that group and ignores the "User-agent: *" group entirely. If none exists, it follows the wildcard group. Inside a group, the longest matching path wins, and if an Allow and a Disallow match with the same length, Allow wins. Applebot is a special case: if nothing names it, it follows your Googlebot rules. This tool applies exactly those rules, so it shows the rule line that decided each result.
What happens if my robots.txt is missing or returns an error?
A missing file (HTTP 404) is treated as "everything is allowed", which is fine for crawling but leaves you no place to state a policy or point to your sitemap. A server error (HTTP 5xx) or a timeout is much worse: Google treats an unreachable robots.txt as "do not crawl anything" until it can read the file again. Bot protection that challenges requests for robots.txt is a common cause.
Does robots.txt keep a page out of Google?
No. robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it, just without a description. To keep a page out of search, let it be crawled and add a noindex tag. To keep content private, put it behind a login. robots.txt is a public file that anyone can read, so never use it to hide anything sensitive.
What is Content-Signal?
Content-Signal is an optional robots.txt line from Cloudflare's Content Signals Policy, for example "Content-Signal: search=yes, ai-input=yes, ai-train=no". It states how you want content used: for search results, as input to AI answers, or for AI training. It is a declared preference, not access control, so it sits alongside your Allow and Disallow rules rather than replacing them.
What is the AI visibility panel?
It shows how many times your domain is mentioned in Google AI Overviews and ChatGPT answers in the US, and the combined AI search volume of the questions that trigger those mentions. The data comes from DataForSEO's LLM Mentions database. Crawler access is the first step to being cited; this panel shows whether it is actually happening.