Your firewall is your AI policy — I probed 18 major sites to read it

Everyone's arguing about AI search visibility. Almost nobody is measuring the boring part: whether AI crawlers can fetch your pages at all . So I built a small open-source tool — geo-crawl-audit — that probes any site with the user-agents of every AI crawler that matters (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, ChatGPT-User, and friends), measures how each is treated versus a normal browser, and checks the thing most people never think about: how many words exist in the raw HTML before any JavaScript runs . Because here's the detail the industry keeps missing — GPTBot, ClaudeBot, and PerplexityBot don't execute JavaScript. For most sites, Googlebot (feeding Gemini) and Applebot are the only AI-adjacent crawlers that render it. A site can rank #1 in Google and be a blank page to nearly everything else. I pointed it at 18 major sites on August 7. Five findings worth your time. 1. Access patterns line up with the business relationships The Guardian — which has a content deal with OpenAI — serves my simulated GPTBot, OAI-SearchBot, and ChatGPT-User a clean 200 . The same request wearing ClaudeBot, PerplexityBot, or CCBot : 403 , and those names are in its robots.txt disallow list too. Policy and enforcement agree. The New York Times — in litigation with OpenAI — 403s nearly everyone: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Common Crawl, meta-externalagent. Two user-agents got through: bingbot and Amazonbot. I want to be careful about the claim here: a status code tells you who is blocked , not why . Any individual 403 has mundane explanations. But when the allow/deny matrix maps this cleanly onto public deals and public lawsuits, the firewall config has become a business document — and it's readable by anyone with a terminal. 2. robots.txt is a polite sign. Some doors are unlocked anyway. Reddit's robots.txt blocks every AI bot in my list — fourteen tokens, no exceptions. Enforcement tells a different story: my GPTBot UA got a 403 and ClaudeBot and CCB