今日已更新 339 条资讯 | 累计 19899 条内容
关于我们

Blocking AI crawlers earns you nothing. Here's how to price them instead

fernforge 2026年07月12日 23:44 3 次阅读 来源:Dev.to

Disallow: GPTBot is a wall. Walls don't pay rent, and the crawlers that matter most either ignore them or route around them. If your content is worth training on, the interesting question isn't "how do I keep the bots out" — it's "what do they owe me, and how do I say so in a way a machine can read." That's what RSL (Really Simple Licensing) is for. It shipped 1.0 in December 2025 with around 1,500 publishers behind it — Reddit, Yahoo, Quora, O'Reilly, Medium, Vox. This post is a from-scratch walkthrough of what the format actually is, the six places you can put it, the one mistake that makes crawlers silently ignore your terms, and where the declaration stops and enforcement begins. No tooling required to follow along — it's all plain XML and HTTP. The format is an XML vocabulary, not a config file An RSL document says: for this content, here's what's permitted, what's prohibited, and what it costs. Minimal example: <?xml version="1.0" encoding="UTF-8"?> <rsl xmlns= "https://rslstandard.org/rsl" max-age= "7" > <content url= "/" > <license> <permits type= "usage" > search </permits> <prohibits type= "usage" > ai-train </prohibits> <payment type= "crawl" > <amount currency= "USD" > 0.015 </amount> </payment> </license> </content> </rsl> Read it out loud: search engines may index this; training on it is prohibited; if you want to crawl it anyway, the rate is $0.015. usage tokens include search , ai-train , ai-use (inference/grounding), and a few more. You can scope rules by user and geo too. One rule that trips people up: prohibition wins . If the same token shows up under both permits and prohibits , the content is prohibited. Don't try to express "allowed except for X" by listing X in both — just prohibit X. The namespace is the thing crawlers actually key on The single most common way to publish RSL that quietly does nothing: getting the namespace wrong. It must be exactly: xmlns="https://rslstandard.org/rsl" http instead of https , a trailing slash, or a plausible

本文内容来源于互联网,版权归原作者所有
查看原文