Introducing Crawlberg v1.0.0
We're upgrading Crawlberg to a new version: Crawlberg v1.0.0. It builds on the previous kreuzcrawl. It declares the public API frozen under the new project name. All technical features below shipped in v0.3.0 (2026-06-23); v1.0.0 is a stability declaration and rename, not a new feature release. The four production-facing changes most likely to require operational action: Package and env var rename - every artifact identifier has changed; see the migration table. SSRF defense is now on by default - internal crawl targets (localhost, RFC 1918, cloud metadata) will fail without CRAWLBERG_ALLOW_PRIVATE_NETWORK=1 . CrawlError::WafBlocked is now a struct variant - exhaustive match arms will not compile until updated. max_retries semantics changed - off-by-one fixed; max_retries=3 now produces exactly 3 retries. Precompiled binaries cover Linux (x86_64/aarch64), macOS (ARM64 and x86_64), and Windows x64. Homebrew bottles and Docker images on GHCR are also available. What Is Crawlberg? Crawlberg is a web crawling engine written primarily in Rust that exposes a single consistent API across 14 language runtimes. It handles HTTP transport, JavaScript rendering, robots.txt compliance, per-domain rate limiting, SSRF safety, and structured extraction. Extension points ( Frontier , RateLimiter , CrawlStore , EventEmitter , ContentFilter , WafClassifier , ProxyProvider ) are injectable traits; wire in your own frontier, storage backend, or proxy pool without forking the engine. A single scrape() call returns text, metadata, links, images, assets, JSON-LD, Open Graph tags, hreflang, favicons, headings, response headers, and clean HTML→Markdown. When a site requires JavaScript, the optional headless browser tier handles it transparently. v1.0.0 promotes v1.0.0-rc.2 and freezes the public API under the new project name. The features described in the sections below represent the platform that 1.0.0 declares stable; they shipped in v0.3.0. What v1.0.0 Declares Stable These capabilities