GPTBot in robots.txt: the hosting toggle developers need to check

Your robots.txt may express an AI policy you did not write. We checked the homepage and robots.txt of 9,037 live AI tools listed on directree on 6 and 7 September 2026. Of those, 945 explicitly disallow OpenAI’s GPTBot in its own user-agent group: 10.5% of the sample. Treat AI crawler rules as deployment configuration. Review them when you change hosting, enable a CDN feature, adopt a starter template, or hand site operations to someone else. Read the full research and methodology . GPTBot, search, and user browsing are separate A common configuration blocks model training while keeping a site available in AI-assisted search and browsing: User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / These are separate crawlers with separate purposes. In our sample, 839 of the 945 sites that block GPTBot, or 88.8%, still allow OAI-SearchBot. That is a deliberate and useful distinction if your goal is to opt out of training while remaining eligible to be cited in ChatGPT search. The same pattern appears across AI labs. ClaudeBot is explicitly blocked by 10.1% of the 9,037 tools, while Claude-SearchBot is blocked by just 0.1%. Google-Extended is blocked by 9.9%, but its purpose is also distinct from ordinary Google Search crawling. Do not assume a broad-looking rule has the result you want. Check the actual crawler names and decide which capabilities you want to permit. A safe way to review your file Start by opening the public URL: https://your-domain.example/robots.txt Then look for three things: A named crawler group, such as User-agent: GPTBot . A Disallow: / directly inside that group. A wildcard group, User-agent: * , that could affect all crawlers. Our measurement only counts a site as blocking GPTBot when the named GPTBot group itself contains Disallow: / . This matters because ordinary technical exclusions are widespread. Only 31 sites in the 9,037-site sample, or 0.3%, block every crawler outright. Meanwhile, 44% have a path-level Disallow rule in a wildc