Can AI tools read your site?
ChatGPT, Claude, Perplexity and Google can only recommend what their crawlers are allowed to read. Enter your website and see, crawler by crawler, what your robots.txt says.
What it checks
- Sixteen AI crawlers from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, Meta, Common Crawl and ByteDance
- Whether each one is for training, for search answers, or for fetching a page a user asked about
- The exact robots.txt rule that allows or blocks it, following the same rules the crawlers use
- Whether your site has an llms.txt
Questions
Should I block AI crawlers?
It depends on what you want. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) keeps your pages out of future model training. Blocking search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) keeps you out of AI answers and their links, which most businesses want to be in. Many sites block training and allow search.
Does blocking Google-Extended remove me from AI Overviews?
No. Google AI Overviews and AI Mode use the normal Googlebot. Google-Extended only controls use for Gemini training and grounding.
Do all AI crawlers follow robots.txt?
The ones listed here say they do. Some crawlers ignore it; blocking those takes a firewall or bot-management rule at your host or CDN.
Is anything stored?
Only a count of checks per network, for the rate limit. We read your public robots.txt and llms.txt and keep nothing else.