Check whether ChatGPT, Claude, Perplexity and Gemini are allowed to read your website — and get the exact robots.txt lines to fix it.
It reads your site’s robots.txt — the file crawlers check before visiting — and applies the same rules the crawlers do. A crawler follows the group with its own name if there is one, otherwise the User-agent: * group. “Blocked” means that group contains Disallow: /.
AI companies run separate crawlers. Search crawlers such as OAI-SearchBot and PerplexityBot find pages to cite in answers. Training crawlers such as GPTBot and CCBot collect pages to train models. You can block training and still be cited — what matters for showing up in ChatGPT or Perplexity answers is the search crawlers.
No. Google-Extended only controls whether Gemini may use your content. Google Search crawls with Googlebot, which this setting doesn’t affect.
Some hosts and CDNs block AI crawlers before robots.txt is ever read — for example a firewall or “block AI bots” setting at Cloudflare. This checker only reads robots.txt, so check your hosting or CDN settings too.