AI Crawlers Are Scanning Your Site Right Now - How to Check and Control Access
AI crawlers now appear in many server logs alongside traditional search bots. Some are used for search retrieval, some for training, and some for broader web indexing. If you care about AI search visibility, you need to know which ones can access your public pages. The most common accidental blocker is simple: a robots.txt rule or CDN bot setting that prevents AI crawlers from reaching the content you want discovered. The major AI crawler tokens to check Here are crawler tokens you may see in logs or robots.txt rules: Crawler token Company Notes GPTBot OpenAI Documented OpenAI crawler token OAI-SearchBot OpenAI Documented OpenAI search-related crawler token ChatGPT-User OpenAI Documented OpenAI user-triggered agent token ClaudeBot Anthropic Documented Anthropic crawler token Claude-SearchBot Anthropic Documented Anthropic search-related crawler token Google-Extended Google Google control token for Gemini Apps and Vertex AI use CCBot Common Crawl Web corpus crawler used by many downstream systems PerplexityBot Perplexity Commonly referenced Perplexity crawler token Crawler names and purposes change. Always confirm against official platform documentation before making sitewide access decisions. First, check what is actually happening Before you change anything, find out who is already crawling. If you have server logs: grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Google-Extended|CCBot|PerplexityBot" access.log If you use Cloudflare, check bot and security events and filter by user agent. Three quick diagnostic steps: Open https://yourdomain.com/robots.txt and look for broad Disallow: / rules. Confirm the sitemap is listed in robots.txt or discoverable at /sitemap.xml . Use our AEO Checker to validate robots.txt and flag restrictive AI crawler rules. The most common mistake The blunt rule that makes sites invisible to many crawlers: User-agent: * Disallow: / This blocks every well-behaved crawler that follows the wildcard rule. If you see it on