Cloudflare's AI block names eight crawlers. None is ChatGPT's search bot
Eight user agents, and the one that decides whether ChatGPT cites you is not among them. An r/SEO post from April, 53 points and 40 comments, says Cloudflare quietly cut the author's site off from ChatGPT, from Perplexity and from Google's AI Overviews. I went and read the block. It names ChatGPT's training crawler and not its search crawler, and it never mentions Perplexity at all. The Google agent it does name is one Google says does not affect inclusion or ranking in Search. Take just two of the eight agents. GPTBot governs inclusion in OpenAI's training data, and Google-Extended governs grounding in Gemini Apps. Those are real things to give up. They are not the three things the warning names. What is actually in the file Cloudflare prints the whole block on its managed robots.txt page : # BEGIN Cloudflare Managed content User - Agent : * Content - signal : search = yes , ai - train = no , use = reference Allow : / User - agent : Amazonbot Disallow : / User - agent : Applebot - Extended Disallow : / User - agent : Bytespider Disallow : / User - agent : CCBot Disallow : / User - agent : ClaudeBot Disallow : / User - agent : Google - Extended Disallow : / User - agent : GPTBot Disallow : / User - agent : meta - externalagent Disallow : / # END Cloudflare Managed Content Read that against Cloudflare's own crawler reference table and a pattern falls out. GPTBot is in, OAI-SearchBot is out. ClaudeBot is in, Claude-SearchBot is out. For both of those pairs the table calls the blocked agent an AI Crawler and the one left alone AI Search. Applebot-Extended is in and plain Applebot is not, the same split again. The table has no Applebot-Extended row, so it cannot tell you what Cloudflare calls that one. The block runs along the training and search seam, and that looks deliberate to me. OpenAI's side of it is one line: "Each setting is independent of the others". I worked through the three OpenAI bots and which one governs search visibility in an earlier post , so I will