A 50-capability map for governed web crawling and AI agents

Giving an agent “web access” sounds like one feature. In practice, it is a stack of separate decisions: How does the system discover URLs? Which destinations can it contact? Does it need a browser, or is static HTTP enough? What turns the response into agent-ready data? Where are request, byte, depth, and time limits enforced? What evidence comes back with the extracted content? Treating all of that as one unrestricted browser capability makes systems difficult to reason about. A better approach is to choose the smallest acquisition surface that completes the job, then make its authority explicit. This article maps 50 current Cockroach Crawler capabilities into seven jobs. It is also a practical checklist you can use with another crawler: if a capability matters to your workflow, identify its input contract, output contract, failure behavior, and authority boundary before an agent depends on it. Disclosure: I’m Ajnas N B, the developer of Cockroach Crawler. The project is open source under the MIT license. Start with a finite crawl contract The next channel currently contains the reviewed 0.7.0-rc.1 prerelease. A bounded documentation crawl can start like this: npm install cockroach-crawler@next import { crawlDetailed } from " cockroach-crawler " ; const result = await crawlDetailed ({ seeds : [ " https://docs.example.com " ], allowedOrigins : [ " https://docs.example.com " ], include : [ " /guides/ " , " /reference/ " ], exclude : [ " /archive/ " ], traversal : " bfs " , obeyRobots : true , maxPages : 25 , maxRequests : 120 , maxDepth : 4 , maxTotalBytes : 10 _000_000 , maxDurationMs : 60 _000 , concurrency : 4 }); for ( const page of result . pages ) { console . log ( page . url , page . contentHash , page . markdown . length ); } The important part is not the number of options. It is ownership: the creator of the agent sets the origins and ceilings. Model-facing input can narrow that contract, but it should not be able to expand it. 1. Crawl and discover — 15 cap