标签：#cloudflare

Canonical URL: https://blog.1001020.xyz/ Suggested cover image: use a recent image from https://blog.1001020.xyz/gallery I have been building a small publishing system called 1001020 , a serverless blog and AI gallery running on Cloudflare Workers. The live site is here: 1001020 — AI Gallery & Cloudflare Experiments The goal was not to build another static blog generator. I wanted something that could publish articles, serve an image gallery, manage uploaded assets, expose structured sitemaps, and stay operational without a traditional server. The basic architecture The whole public site runs on Cloudflare Workers. Articles, settings, comments, gallery metadata, and telemetry live in Cloudflare KV. Managed images are stored in R2 and served through a dedicated image domain. The main pieces are: Cloudflare Workers for request routing and rendering Cloudflare KV for article and site metadata Cloudflare R2 for managed image uploads A theme system for different frontend layouts XML sitemap and image sitemap generation A small local AI drafting tool for preparing and publishing content The gallery is a first-class part of the site, not just a media folder. You can browse it here: AI Gallery on 1001020 Why Workers instead of a conventional backend? For this project, Workers are a good fit because the workload is mostly request routing, HTML generation, metadata reads, and small API writes. A conventional server would work, but it would add deployment and maintenance overhead that I did not need. Cloudflare Workers also make it easy to keep the app close to the edge while still handling dynamic behavior. The blog can render pages server-side, expose APIs, and support admin operations without a separate Node or container deployment. KV as the content store The project stores persistent content in KV using explicit keys for articles, gallery records, settings, telemetry, comments, newsletter subscribers, and other small datasets. This shape works well for a personal publishi

A few months ago I moved a batch of WordPress sites off a shared LAMP host and onto Cloudflare Pages as static exports. The pitch is obvious: no PHP process to keep patched, no MySQL to babysit, effectively free hosting, and a CDN in front of everything by default. What the pitch doesn't tell you is how many small, boring things break on the way there. This post is a rundown of what actually went wrong migrating a set of ten WordPress sites — one of them is burningtribe.tokyo , which I'll use as the concrete example — and how I fixed each issue. The approach The migration itself is conceptually simple: crawl the live WordPress site, save every URL as a static HTML file plus its assets, and serve that tree from Cloudflare Pages. I used a combination of wget --mirror and a custom crawler for a couple of sites where wget choked on query-string-based pagination. The static output then gets pushed with wrangler pages deploy . No build step, no framework, just files. That simplicity is exactly why it seemed low-risk. It was not. Problem 1: relative canonical tags pointed everything at the homepage The first thing I noticed after deploying was that Google Search Console started reporting most inner pages as "duplicate, Google chose different canonical" — and the canonical it picked was the homepage. The cause was almost funny once I found it: the WordPress theme emitted <link rel="canonical" href="/"> as a relative path in a few cached page fragments, instead of an absolute URL like https://burningtribe.tokyo/some-post/ . On the original WordPress install this didn't matter because the page itself resolved the relative reference correctly at the point of caching. Once the HTML was frozen and served statically from a different origin structure (Pages serves everything from the apex), that relative canonical collapsed to the site root for every single page that had it. The fix was a straightforward but tedious pass: grep every exported HTML file for rel="canonical" , and rew

Cloudflare recently introduced Meerkat, an internal globally consistent control-plane service based on the QuePaxa consensus algorithm. Unlike Raft, it allows leaderless writes while preserving strong consistency, improving availability across Cloudflare's global network. By Renato Losio

Key takeaways A request is judged on many layers at once — IP reputation, TLS fingerprint, HTTP/2 shape, headers, and how the browser is driven — and failing any one is enough for a 403. Your laptop passes because every layer is consistent with a real home browser; a server changes one (usually the IP) and the inconsistency is the tell. A 403 with no challenge page almost always means you were blocked at the network layer (IP/ASN reputation or TLS/JA3 fingerprint) before any HTML was served — not a credentials or rate-limit bug, so 'add a User-Agent' or 'slow down' won't fix it. The fix order that actually works: get off datacenter IPs (residential/ISP proxies), match a real browser's TLS fingerprint, and spin up a real-browser stealth setup only for the pages that truly need JavaScript — escalate, don't lead with a browser. A proxy only changes your IP; a Linux VPS still leaks a Linux-shaped TLS/JS fingerprint, so 'residential IP + datacenter everything-else' is a contradiction a real home machine never makes — which is why a proxied server can get blocked harder than your laptop. Your scraper runs perfectly on your laptop. You deploy it to a VPS or a CI runner, change nothing in the code, and suddenly every request comes back 403 . It feels like a bug — the code is identical — but it usually isn't. Anti-bot systems judge a request on many signals at once, and moving from your home machine to a datacenter flips several of them at the same time. This post breaks down exactly which signals change, how to tell which one is blocking you, and how to fix it — for authorized access to public data (we'll keep that framing honest throughout; nothing here is about defeating a protection). A request is judged in two stages It helps to know that detection happens in two stages: Stage 1 — before any HTML is served. IP reputation, your TLS handshake, your HTTP/2 settings, and header order are all inspected on the connection itself, passively and cheaply, before your request is e

Cloudflare has launched its Internal DNS service, providing authoritative and recursive DNS for private networks. This service simplifies DNS management by consolidating private and public DNS operations on a single platform. By Gianmarco Nalin

The same parser, four times Over the last year I kept running into the same shape of problem: A docs site generator that wanted Markdown chapters turned into navigation JSON. A RAG ingestion script where each Markdown file needed to become a list of text chunks plus its frontmatter metadata. An n8n flow that took Markdown emails and extracted only the tasklists. A static-site backend that accepted user Markdown and needed to validate structure before persisting. Each one is small on its own. But every time I reached for a different library — remark here, gray-matter there, marked once, a hand-rolled regex once too many — and every time one of them broke on the same edge cases: Nested GFM tasklists where the checked state was silently lost YAML frontmatter that included quoted booleans (parsed as strings, not booleans) Tables whose headers contained spaces (regex parsers treated them as one key) Code blocks containing Markdown — re-parsed as Markdown instead of fenced code So I built one endpoint that does it once, properly. What it returns POST /v1/parse takes a Markdown body ( text/markdown ) or a JSON envelope ( application/json ) and returns one stable JSON shape: { "success" : true , "data" : { "title" : "Project Alpha" , "frontmatter" : { "title" : "Project Alpha" , "status" : "shipping" }, "headings" : [ { "level" : 1 , "text" : "Project Alpha" , "id" : "project-alpha" } ], "sections" : [ { "heading" : { ... }, "children" : [ ... ], "content" : [ ... ] } ], "lists" : [ { "ordered" : false , "items" : [ "ship MVP" , "write README" ] } ], "tasklists" : [ { "items" : [ { "text" : "ship MVP" , "checked" : true } ] } ], "tables" : [ { "headers" : [ "Module" , "Status" ], "rows" : [{ "Module" : "API" , "Status" : "Done" }] } ], "codeBlocks" :[ { "lang" : "js" , "value" : "..." } ], "links" : [ { "text" : "..." , "url" : "https://..." } ], "paragraphs" :[ "..." ], "ast" : null } } The sections tree is the part I care most about. It's not just a flat list of headings

I wanted two things: short links under my own brand (every link I share points traffic back to my site ), and a real excuse to run a complete system on the edge — DNS, distributed compute, storage, auth and a dashboard — in production, at zero infrastructure cost. The result is flino.link : a shortener that responds in under 10 ms from 300+ locations, runs entirely on Cloudflare's free tier, and whose only expense is the domain: $4.68 a year. This post covers the design decisions, which is where the interesting parts are. The architecture in 30 seconds A single Cloudflare Worker serves the whole domain: GET /<slug> — the hot path. One read from Workers KV (globally replicated) and a 302 redirect. Nothing else touches that path. /api/links — a REST API with Bearer auth to create, list and delete links. /admin — a single-page dashboard served as inline HTML from the Worker itself. No framework, no build step. A Durable Object with embedded SQLite keeps per-slug click counts. No servers, no containers, no database to manage. The whole Worker is three TypeScript files and zero runtime dependencies. Why a dedicated domain? My first idea was to hang the shortener off a route on flino.dev . Bad idea: shorteners attract abuse — spam, phishing — and their domains sooner or later end up on blocklists. If that happens, I don't want it dragging my main domain down with it. A separate domain isolates that reputation risk completely, and a short .link costs less than a coffee per year. KV for links, a Durable Object for counters This is the central design decision. Workers KV is perfect for a shortener's access pattern — read-heavy, write-light, reads served from the edge — but its writes are eventually consistent : two concurrent increments in different datacenters would clobber each other. For counting clicks, it's useless. A Durable Object solves exactly that: it's a single global instance with transactional SQLite storage. Every increment, no matter which datacenter it comes

I shipped a Next.js 16 app on Cloudflare Workers via OpenNext. Not a demo. A real product with streaming chat, server components, D1 at the edge, and anonymous user sessions. Here is what broke, what barely worked, and what turned out to be surprisingly fine. The stack Next.js 16.2 (App Router) @opennextjs/cloudflare 1.19 D1 for SQLite at the edge Streaming chat via the AI binding (DeepSeek-V3 through a Workers proxy) React 19 Tailwind CSS 4 No auth wall, no OAuth, no database on the origin The site runs a few thousand sessions a week across ~30 persona pages, blog posts, guides, and learning content. Most pages are statically generated. The chat interaction is server-rendered components with streaming responses. What worked surprisingly well Static generation and ISR Pages, blogs, guides, persona pages — everything that does not need user-specific rendering — runs as static HTML at deploy time. Next.js 16 with generateStaticParams and fetch caching worked without modification. OpenNext handles the Cloudflare output format. The build step produces something Workers can serve. Revalidations are limited to Workers' cache API, but since most content changes at deploy time, I never hit that limit in production. The one caveat: revalidateTag() does not work the same way in a Workers runtime. Tags are Node.js memory constructs, and Workers are stateless. If you depend on tag-based revalidation for content updates, you need to either trigger deploys or accept stale-while-revalidate behavior from the CDN. D1 at the edge D1 was the least surprising part of the stack. SQL queries from Next.js route handlers feel like calling a regular database. Sessions store in D1, messages store in D1, and the latency is low enough that restoring a full chat thread from 30 messages takes under 200ms cold. The only sharp edge: D1 connections count against your Worker's concurrent request limit in development. With Next.js making its own fetch calls for compilation, I hit the D1 connection ce

This started as a comment @wrencalloway left on my last post. It was sharp enough that it deserved more than a reply — so here's the full version. You did the work. The page is server-rendered. The JSON-LD is in the raw response. curl returns the whole article, headline and all. A crawler that fetches your URL gets everything it needs. Except the crawler never fetches your URL. It asks the CDN, and the CDN says 403. Your content is perfect and unreachable. This is the layer underneath the one everyone talks about — and it's invisible in every tool you'd normally reach for. Two different kinds of "no" robots.txt is a note taped to the door. It says "please don't come in." A polite crawler reads it and turns around. A rude one ignores it and walks straight past. Either way, the note never touches your bytes — it's a request, enforced entirely by the visitor's own manners. A WAF or CDN rule is the door. It answers the request itself, at the edge, before anything reaches your origin: 403 Forbidden , 429 Too Many Requests , or a JavaScript challenge the bot can't solve. The HTML behind that door could be a masterpiece or a blank page. The bot sees neither. It sees the status code. People spend weeks perfecting the note and never check whether the door is locked. This isn't hypothetical — here's the data Cloudflare Radar publishes what actually happens to AI-crawler traffic across its network. From the AI Insights dashboard (7-day view, as of 18 July 2026): Of all HTTP responses served to AI bots and crawlers: 200 OK — 73.6% 403 Forbidden — 5.2% 429 Too Many Requests — 1.3% 503 — 1.1% AI-crawler response codes. Source: Cloudflare Radar, AI Insights, 7-day view, 18 Jul 2026. Read that again. More than one in fifteen requests from AI crawlers is being actively refused at the edge — 403 or 429 — before it ever touches the content. Not deprioritised. Refused. And most of that isn't a decision. It's a default — a managed WAF ruleset, a "block AI bots" toggle flipped in 2024, a

Patreon is strengthening its defenses against AI scraping by working with Cloudflare to block bots that train AI models on creators’ content without permission. The move marks a shift away from relying on websites using robots.txt alone to actively block unauthorized AI training.

A scraper fails, someone swaps the User-Agent, someone else adds a proxy, then the job starts passing locally but fails again in CI. That usually happens because Cloudflare did not block “scraping” as one thing. It evaluated several signals, and each failure needs a different fix. This is about authorized automation: your own sites, customer-approved workflows, testing, monitoring, data access you are allowed to perform. If you do not have permission to automate against a site, changing fingerprints or rotating IPs does not make it okay. Start with the failure you actually see Cloudflare failures often get collapsed into “403”, but the page body matters. Common cases: Error 1020 : usually an access denied page from a Cloudflare rule or bot score decision. The HTTP status may still be 403, so inspect the HTML. 403 without a 1020 page : often IP reputation, firewall rules, geo restrictions, or an auth problem. 429 : rate limit exhaustion. Slowing down can help here, but it will not fix a fingerprint problem. Endless Just a moment... page : your client did not complete the browser-side challenge. CAPTCHA or Turnstile loop : Cloudflare still considers the session borderline after earlier checks. Add classification before you add workarounds. Even a basic classifier saves time: import time import requests CLOUDFLARE_MARKERS = { " 1020 " : " cloudflare_access_denied " , " Just a moment " : " cloudflare_js_challenge " , " cf-turnstile " : " cloudflare_turnstile " , " cf-error-code " : " cloudflare_error_page " , } def classify_response ( resp : requests . Response ) -> str : body = resp . text [: 5000 ] if resp . status_code == 429 : return " rate_limited " for marker , label in CLOUDFLARE_MARKERS . items (): if marker in body : return label if resp . status_code == 403 : return " forbidden_unknown " return " ok " if resp . ok else f " http_ { resp . status_code } " def get_with_backoff ( url : str , max_attempts = 4 ): for attempt in range ( max_attempts ): resp = request

Cloudflare has recently introduced temporary accounts that let AI agents deploy Cloudflare Workers immediately, without first creating or authenticating with a permanent account. If left unclaimed, the accounts and their deployments expire automatically after 60 minutes. By Renato Losio

I optimized my architecture for the wrong model. I used to think black-box infrastructure was the right abstraction for AI-driven development. Vercel, Supabase, Cloudflare Workers — a sharp contract in front, a managed backend behind — felt like the obvious fit. The less an agent had to reason about, the fewer places it could get lost. Give it a clean interface, hide the messy backend, move fast. I still think that was right for the agents we had last year. I don't think it's right for the agents we're starting to use now. The shift is not that AWS got simpler. It didn't. Setup still takes longer, CI/CD takes more work to wire, and cost control has real limits. The shift is that agents got better at reading complexity — and once an agent can actually use a large structured context, the things I treated as overhead (resources, provider schemas, IAM policies, explicit queues, explicit alarms, explicit networks) become the highest-signal context I can hand it. To keep this concrete, I'm holding the tool constant. This is HCL/Terraform on AWS vs HCL/Terraform on Cloudflare — same language, same workflow, two providers. The inversion Agents are… Best served by… Context-poor (last year's loops) Black boxes — shrink the surface, hide the backend Context-rich (now) Inspectable systems — describe everything as code The one-line version: The better agents get at reading, the more valuable explicit infrastructure becomes. I didn't become more pro-AWS because AWS got easier. I became more pro-AWS because agents got better at reading it. Same Terraform, two providers The interesting comparison isn't AWS-elegance vs Cloudflare-elegance. It's how much of the infrastructure topology and operational contract an agent can reconstruct from the HCL plus the provider schema alone. Terraform × AWS Terraform × Cloudflare Provider maturity AWS provider is about as battle-tested as IaC gets; enormous public corpus of modules/examples v5 is a ground-up, OpenAPI-generated rewrite — improving

Cloudflare and AWS both implemented x402 stablecoin micropayments at their edge networks within two weeks. The open protocol under the Linux Foundation revives HTTP 402 for agent-to-service payments with sub-cent transaction costs. Coinbase reports 169 million transactions in year one. Enterprise tax and invoicing gaps remain unresolved. By Steef-Jan Wiggers

Every developer has that one project. The passion build that sits in the back of your mind for months—or even years—before you finally sit down, crack your knuckles, and make it a reality. For me, that project was building a modern, open-access bilingual digital lexicon bridging English and Assamese: AssameseDictionary.org . While it started as a personal milestone dream, it quickly turned into a massive data engineering and architecture challenge. Here is how I tackled parsing a massive vocabulary database and serving it globally with near-zero latency. 🏗️ The Core Challenge: Scale vs. Speed A dictionary isn't like a standard SaaS app or landing page. It lives and dies by its database depth. To make this a truly definitive tool, I compiled, cleaned, and programmatically validated an extensive vocabulary index mapping over 293,000 words . The dataset doesn't just hold simple translations; it maps complex bidirectional lookups, phonetic transliterations, advanced English definitions, context usage examples, and cross-linked synonym tokens. If I threw this massive dataset into a traditional relational database hooked up to a standard server setup, I ran into immediate roadblocks: Latency: Heavy search queries on a dataset this size can cause noticeable lag. Cost/Overhead: Maintaining and scaling database servers for unpredictable public traffic gets expensive fast. I wanted the search utility to snap back instantly. To achieve that, I had to ditch traditional server paradigms entirely. ⚡ The Architecture: Serverless Edge Caching To keep things ultra-lightweight, highly cost-effective, and blazing fast, I built the platform around an edge-computing topology: The Runtime: I offloaded the backend logic entirely to Cloudflare Workers . Instead of routing traffic to a centralized origin server, queries are intercepted and executed at serverless edge locations physically closest to the user. The Data Layer: Instead of an active SQL database bottleneck, I mapped the data mat

Cloudflare is giving AI companies until September 15 to separate web crawlers used for search from those used for AI training and agents, or risk being blocked by default on many publisher sites.

Why your Cloudflare Turnstile token works in the browser but 403s from requests You solved the Turnstile widget. You can see the token in the page. You copy it into your script, POST the form from requests, and the server hands you back a 403 — or a JSON body with "success": false. The token clearly worked a second ago in the browser, so what changed? Short answer: a Turnstile token is not a password you can carry around. It's a one-time, short-lived proof bound to a very specific context, and replaying it from a different context is exactly what it's designed to reject. Below is what that context is, how to tell which constraint you're hitting, and the fix for each. The real scenario You're automating a flow on a Cloudflare-protected site. There's a cf-turnstile widget on the form. You get a token one of two ways: you render the page in a real browser (Playwright/Selenium) and read cf-turnstile-response, or you hand the sitekey + page URL to a solving service and get a token back. Either way, you then submit the form with a plain HTTP client requests, httpx, axios) and it fails. The frustrating part: it's intermittent-looking. The reason it feels random is that there are four separate constraints, and you're usually tripping a different one each time. The four things a Turnstile token is bound to 1. It's single-use Once Cloudflare validates a token server-side (the siteverify call your target makes), that token is spent. Submit twice, retry, or test it once by hand, and the second use returns false. You get a fresh one per submission. 2. It has a short TTL Turnstile tokens expire fast — a few minutes. Solve early, do other work, submit later, and the token can be dead on arrival. The widget auto-refreshes in the browser precisely because tokens go stale; a script that grabs the token and sits on it loses that refresh. 3. It's bound to the sitekey and the page URL Multiple widgets. Some pages embed more than one Turnstile (login + newsletter). Solving the wrong site

Cloudflare released the Cloudflare One stack, an open-source library of agent skills for planning, deploying, and managing Zero Trust environments. The skills include automated migration logic for Zscaler and Palo Alto Networks, the same logic used in Cloudflare's Descaler program that has moved enterprise customers in hours rather than months. By Steef-Jan Wiggers

Every AI developer has been here: your app is throwing 503s, users are pinging you, and you have 12 browser tabs open — OpenAI status page, Anthropic status page, the GitHub Copilot health page, three different Discord servers — trying to figure out is this me or is it them? That's the problem we set out to solve. Prismix aggregates status from 77 AI services in one place. Six weeks of running it in production taught us some things that might save you time. The problem is worse than you think AI APIs don't fail like traditional infrastructure. They fail in weird, partial ways: Degraded performance that passes your health checks but makes your product feel broken Regional outages — OpenAI US-East is down while EU is fine, so half your users are affected Silent rate-limit cascades — the API returns 429s but their status page says "operational" for another 20 minutes Incident lag — providers often post status updates 10–30 minutes after engineers are already aware The official status pages are optimistic by design. They're customer-facing communications tools, not real-time engineering dashboards. There's nothing wrong with this — but it means you need a different mental model for "is this service down?" What 77 status pages look like in aggregate When you watch 77 AI services simultaneously, patterns emerge fast. OpenAI is the most-watched service (and has the most incidents to watch). The pattern is almost always the same: investigating → identified → monitoring → resolved , typically in 45–90 minutes. The investigating phase is where most developers panic — it looks bad but usually resolves without action on your end. Anthropic runs noticeably clean compared to its API usage growth. Incidents are rarer and shorter. When they do happen, updates arrive faster than most providers. The long tail is interesting. Services like Replicate, Runway, ElevenLabs, and Suno have incident patterns that don't correlate with OpenAI at all. If you're routing across multiple providers

Last Tuesday at 3am, a multi-agent loop hit 12K KV writes/minute and froze. The loop was a one-line counter bug. That part was fixable. What I found while tracing it was worse. I had --dangerously-skip-permissions enabled on a Claude Code session that was running D1 migrations. I thought it was pointing at staging. It wasn't — I'd misconfigured my env file reference, loading .env.production instead of .dev.vars . Claude didn't ask. The flag told it not to. The migration was ADD COLUMN , not DROP COLUMN , so no data loss. Survivable. But only barely. The thing I got wrong: I treated --dangerously-skip-permissions as "skip the annoying confirmation popups." It's actually "remove the only moment a human sees what command is about to run." Those are very different things. Turning the flag back off helps, but it doesn't constrain what Claude attempts — it just adds a prompt you'll click through anyway at 3am. What actually worked was adding a deny rule in .claude/settings.json : { "permissions" : { "allow" : [ "Bash(wrangler d1 execute * --local*)" ], "deny" : [ "Bash(wrangler d1 execute *)" ] } } The allow rule is more specific than the deny, so --local calls go through and everything else is blocked before execution. Over 2 weeks post-fix, Claude attempted zero production DB commands. Three deny events were logged — all from ambiguous prompts I wrote during fast context-switches, not from Claude going rogue. I ended up running three layers: the settings.json allowlist, a separate git worktree for migration work that physically contains only staging credentials, and a CLAUDE.md that instructs Claude to ask before anything touching production. The CLAUDE.md approach has a real caveat though — in long sessions the instructions lose weight as context grows. Anything critical needs to be restated in the prompt itself. I wrote up the full breakdown — including the worktree setup, the exact CLAUDE.md wording, and why MCP tool permissions behave inconsistently with the deny ru