标签：#Caching

Cloudflare KV for Session Caching in Multi-Tenant FastAPI: Reducing PostgreSQL Load Without Redis Complexity Every SaaS I've built hits the same wall: session validation on every request hammers PostgreSQL. You add Redis, suddenly you're managing another service, debugging cache invalidation, and paying for redundancy you don't need. Then I discovered Cloudflare KV sits between your users and origin server. It's not a replacement for PostgreSQL—it's a read cache positioned at the edge that auto-syncs on writes. For multi-tenant session and permission data, this eliminates 60–80% of auth-related database queries without the operational complexity of Redis. This is the approach I use in CitizenApp. Here's why it works, how to implement it, and where I nearly broke production. Why Cloudflare KV Beats Redis for Session Caching Redis requires: A separate service deployment (Render, AWS ElastiCache) Connection pooling logic in your app Cache invalidation strategies you'll get wrong Monitoring for memory leaks and eviction Cost that scales with your hot data size Cloudflare KV requires: A binding in your edge worker (one line of config) Simple key-value storage at 200+ edge locations Automatic TTL expiration Zero operational overhead—Cloudflare manages it Here's my honest take: I prefer KV because I don't have to think about it. My workers validate JWT tokens and fetch session data from KV before even routing to my FastAPI origin. Cache misses flow to PostgreSQL and write back to KV. No connection pools. No eviction policies. No debugging Redis memory fragmentation at 3 AM. The tradeoff? KV is slower than in-memory Redis (ms vs microseconds), but for session lookups happening 200+ times per second per user at global scale, edge-cached responses beat origin-fetched ones every time. Architecture: Edge Validation + Origin Sync Your flow looks like this: Request hits Cloudflare Worker Worker checks KV for session + permissions (hit = serve immediately) KV miss → fetch from Fas

LLM APIs like Claude feel snappy—until latency spikes hit your users. By caching prompt‑response pairs right at the edge, you can cut round‑trip time to milliseconds. This post shows you how to make that happen with CloudFront Functions and a Lambda origin. Why Prompt Caching Matters for LLM‑Powered Apps When a user types a question, your front‑end sends the text to an LLM (large language model) API, waits for the model to generate a reply, and then shows the answer. The user experience is dominated by two things: Network latency – the time it takes for the request to travel from the user’s browser to the API endpoint and back. Model compute time – how long the LLM needs to think. Even if the model itself is fast, the network hop to the provider’s data center can add 100 ms – 300 ms, and sometimes more during traffic spikes. For a chat UI that refreshes every few seconds, those extra milliseconds feel like a noticeable lag. Prompt caching means storing the exact prompt (the user’s message) together with the response (the model’s answer) in a fast lookup table. If the same prompt arrives again within a short window, you can return the cached answer instantly, without touching the LLM provider at all. In plain English: Think of the cache as a “sticky note” on the receptionist’s desk. If someone asks the same question twice, the receptionist can hand them the note instead of calling the manager again. Freshness vs. Speed LLM responses are not immutable—new data, temperature settings, or model updates can change the answer. A short time‑to‑live (TTL) of a few minutes gives you a good trade‑off: most users repeat recent prompts, but you still get new answers after a reasonable window. Setting Up a CloudFront Distribution with an Edge Key‑Value Store The big picture Edge KV store – a tiny key‑value database that lives on every CloudFront edge node. CloudFront Function – a lightweight JavaScript snippet (max 2 MB) that runs on every request before it reaches the origin. It

Cloudflare recently introduced Cache Response Rules, a rules engine that operates after an origin server responds but before content is written to Cloudflare's cache. Previously, Cache Rules operated only on request attributes. Cache Response Rules add a response phase that evaluates origin responses before they are cached. By Renato Losio

Canva redesigned session revocation infrastructure to support 100M active sessions while reducing database lookups. The architecture uses Amazon S3 for durable revocation records and distributes compact, in-memory indexes to application gateways. Canva said the design improved deployment speed, reduced database infrastructure requirements, and cut the revocation cache memory footprint by 87.5%. By Leela Kumili

Magento 2's full page cache is one of its strongest performance features — when it works. But every week, we see stores where a simple product save triggers a 30-second Varnish flush and subsequent cache stampede. The culprit is almost never Varnish itself. It's cache tags. This post covers how Magento 2 cache tags work, why broad tags destroy performance, and exactly how to audit and fix them. How Cache Tags Work in Magento 2 Every cached page, block, and data fragment in Magento is tagged with identifiers. When a product changes, Magento invalidates all cache entries tagged with that product's ID. The tag system is hierarchical: cat_p_123 — specific product cat_p — all products cat_c_5 — specific category cat_c — all categories cms_b_about_us — a CMS block cms_p — all CMS pages These tags are stored alongside cached content and used during invalidation. When you call $cache->clean(["cat_p_123"]) , every cache entry tagged with cat_p_123 is removed. This is elegant until someone tags a global block with cat_p , and saving any product flushes half your store. The Invalidation Storm Problem Here's what happens during a storm: Admin saves a simple product update (price change) Magento generates the invalidation list: cat_p_456 , cat_c (because the product is in categories), cat_p (from a badly written block) cat_p is too broad — it matches the product list page, layered navigation, homepage widgets, and every product detail page Varnish receives 50,000 BAN requests Store goes from sub-100ms response times to 2-5 seconds for the next 10 minutes while the cache rebuilds We've seen this on a store with 80,000 SKUs. A single product save dropped cache hit rate from 94% to 12%. Diagnosing Bad Cache Tags Check Your Current Tags Add this to any block template to inspect what tags are being applied: $block -> getCacheKeyInfo (); // Or for the full page: $block -> getIdentities (); For a full audit, intercept cache writes in development: // In di.xml: < type name = "Magento\Fr

DeepSeek V4 Flash costs $0.14 per million input tokens and $0.28 per million output, with cache hits at $0.0028, and the retrained 0731 build that now ships under that name has a defect you need to route around: with thinking on (the default) and a strict json_schema , integer fields came back corrupted in 8 of 13 default-thinking runs across two independent request paths. Turning thinking off fixed every run and cut the extraction to a seventh of the tokens. We measured deepseek-v4-flash-0731 on day one: the corruption, the sharper off-switch cliff the retrain introduced, the budget floor that rescues it, the 1,024-token cache pages, and what still separates the preview build and V4 Pro . TL;DR With default thinking plus strict json_schema , deepseek-v4-flash-0731 corrupted integer fields in 8 of 13 runs on two request paths; V4 Pro corrupted 2 of 4, and only the preview stayed clean. The 0731 retrain sharpened the off-switch cliff: 2-hop math fell 6/6 to 0/6. The cache serves 1,024-token pages from roughly a 1.1K-token floor, hits 0.3 seconds after priming, and entries outlive 45 minutes. enable_thinking: false fixed every structured run at a seventh of the tokens; for 2-hop math the safe thinking budget is 256. How do the three V4 builds compare on paper and on the meter? Same tokenizer, same cache, same thinking machinery; different prices, different failure modes. Everything measured below comes from identical probes run against all three (a dash means we did not probe that cell); the day-one analysis threads focus on benchmarks, so this is the operational half of the comparison: Flash 0731 Flash preview V4 Pro List price, in / out per 1M $0.14 / $0.28 $0.14 / $0.28 $0.435 / $0.87 Cache-hit input per 1M $0.0028 $0.0028 $0.003625 Thinking default on on on Strict JSON with thinking on 5/5 corrupted (our path) 4/4 clean 2/4 corrupted 2-hop math with thinking off 0/6 2/6 4/4 thinking_budget token-exact token-exact honored (4/4 at 16) Cache pages 1,024 tokens, hit a

Next.js Sitemap Not Updating? Here's the Real Fix If your Next.js sitemap is not updating after you publish new content, you're dealing with a cache-coherence bug that almost nobody writes up. It has an exact symptom, a reproducible root cause, and a one-line fix. This is the guide you'll wish you had the moment you notice /sitemap.xml serving fewer entries than your real site. The symptom: your sitemap lags behind your published content The mismatch is impossible to miss once you look. On our own site, /lab lists 11 published posts, yet /sitemap.xml shows only 7. Same database, same deploy, two different answers. If that gap sounds familiar, you're in the right place. You might have checked your afterChange hook, verified that revalidateTag('posts') fires, and even confirmed that the tagged data refreshes — only to find the sitemap still frozen. That's because the problem lives between two cache layers, not inside the data fetch. Why revalidateTag doesn't fix a stale Next.js sitemap The answer lies in what sitemap.ts actually is. According to the Next.js Metadata Files: sitemap.xml documentation, it's a special Route Handler. And like any Route Handler, Next.js caches its rendered output by default. Here's what happened in our own repository (this bug is documented in a comment at the top of app/(frontend)/sitemap.ts because it cost real indexation time): We read content from Payload using unstable_cache , tagged with the collection slug posts . An afterChange hook called revalidateTag('posts') whenever a post was published. That call did work — it invalidated the inner unstable_cache data entry. But the route's statically-rendered outer XML output was never re-run. The frozen route output kept serving the old XML built from the old data, long after the inner cache was refreshed. Two cache layers. Tag-based revalidation busted the inner one, but the outer route handler cache was never told to re-execute. That's the missing piece. The one-line fix: route-level ISR o

GitHub redesigned GitHub Issues navigation using a client-side architecture that combines caching, predictive prefetching, and service workers to reduce perceived latency. The approach uses IndexedDB, in-memory caching, and background synchronization to serve data faster. GitHub reported instant navigation improvements from 4% to 22%, with latency reductions across multiple navigation By Leela Kumili

Momentic, the company behind an AI-driven software testing platform, recently rearchitected its caching system to handle over 2 million queries per day across 20 billion total entries, while maintaining an average response latency of around 250 ms. This improvement was made possible by transitioning from PostgreSQL to the column-oriented database ClickHouse. By Sergio De Simone

Throttling: vì sao in-memory rate limit "biến mất" sau khi scale ngang, và chọn token bucket hay sliding window Throttling là cơ chế giới hạn số request một client (user, IP, API key, tenant) được xử lý trong một khoảng thời gian, để chống abuse, bảo vệ downstream, và phân bổ công bằng dung lượng service. Định nghĩa nghe đơn giản, nhưng lý do dev gặp nó trong việc thật lại rất cụ thể: sau khi scale service từ 1 pod lên 8 pod, cùng cấu hình "100 req/min mỗi user" đột nhiên trở thành 800 req/min thực tế — vì mỗi pod đếm riêng trong RAM, và load balancer rải request đều tám hướng. Rate limit vẫn "chạy", log không có lỗi, nhưng downstream vẫn bị flood. Đó là failure mode dẫn tới việc phải chuyển counter sang store phân tán, và kèm theo là câu hỏi chọn algorithm nào — token bucket, sliding window, hay leaky bucket — mỗi cái đánh đổi khác nhau. Cơ chế hoạt động Bốn thuật toán phổ biến, khác nhau ở cách đếm và cách xử lý burst. Fixed window counter. Chia thời gian thành khung cố định (mỗi phút bắt đầu tại giây 0). Mỗi request INCR một key rl:{user}:{minute} , nếu counter vượt limit thì reject. Đơn giản nhất, một INCR + EXPIRE trên Redis là xong. Nhược điểm cứng: tại biên khung có thể chịu gấp đôi limit trong một cửa sổ trượt — user gửi 100 req vào giây 59 của phút 12:00, rồi 100 req vào giây 01 của phút 12:01, tức 200 req trong 2 giây thật, trong khi limit là 100/phút. Sliding window log. Lưu timestamp của từng request trong sorted set, mỗi request ZADD + ZREMRANGEBYSCORE xoá các entry cũ hơn now - window , rồi ZCARD để đếm. Chính xác tuyệt đối nhưng tốn bộ nhớ tuyến tính theo số request. Sliding window counter. Cách Cloudflare mô tả trên engineering blog: giữ counter của khung hiện tại và khung trước, ước lượng lượng request trong cửa sổ trượt bằng nội suy có trọng số theo phần trăm khung trước còn nằm trong window. Chỉ tốn hai counter, sai số rất nhỏ so với log thuần, và không có failure mode biên như fixed window. Token bucket. Bucket có capacity B token, refill với tốc

Stale data và cache stampede: vì sao TTL một mình không đủ và vì sao origin sập khi key hết hạn Stale data là dữ liệu trong cache đã lỗi thời so với source of truth. Nó xuất hiện vì cache và origin là hai bản sao, và bất kỳ cơ chế đồng bộ nào — TTL, event-based invalidation, versioning — đều có cửa sổ giữa lúc origin đổi và lúc cache biết chuyện. Cái giá phải trả trong production không chỉ là "user nhìn thấy giá cũ vài giây". Khi một key hot vừa hết hạn, hàng nghìn request cùng miss, cùng đâm xuống DB để tính lại — đó là cache stampede (dogpile, thundering herd), và nó đủ sức đưa origin xuống trong vài chục giây. Cơ chế hoạt động Có bốn cơ chế invalidation dùng thật: TTL (time-to-live). Mỗi entry gắn một hạn dùng. Hết hạn coi như miss, đọc lại từ origin. Đơn giản, không cần coordination giữa writer và cache. Nhược điểm: staleness bounded bởi TTL, và tất cả replica của cùng một key hết hạn cùng lúc. Event-based invalidation. Khi origin thay đổi, phát một event (thường qua pub/sub, CDC như Debezium, hoặc gọi trực tiếp DEL) để cache xoá hoặc cập nhật entry. Fresh gần như realtime, nhưng đòi hỏi coupling giữa write path và cache — writer phải biết mọi key phái sinh từ dữ liệu vừa đổi. Versioning (cache key có version). Key gắn version của dữ liệu, ví dụ user:123:v42 . Đổi dữ liệu thì tăng version, key cũ tự nhiên bị bỏ qua, không cần xoá gì. Kỹ thuật này còn được gọi là generational caching; Rails cache dùng cách tương tự với cache_key_with_version . Single-flight (request coalescing). Không phải invalidation, mà là cách xử lý miss: khi N request cùng miss cùng một key, chỉ một trong số đó được phép gọi origin, các request còn lại đợi kết quả của nó. Go có golang.org/x/sync/singleflight implement sẵn pattern này; Facebook memcache dùng "leases" (paper của Nishtala et al., NSDI 2013) cho cùng ý tưởng ở scale phân tán. Kết hợp điển hình: TTL để bounded staleness, single-flight để chặn stampede khi key hết hạn, event-based invalidation để cắt TTL sớm khi có write. Ví dụ si

AWS has recently introduced durability for Amazon ElastiCache for Valkey, enabling reliable data retention across failures and expanding support beyond caching to persistent workloads. The feature offers new options that prioritize either minimizing data loss or maintaining lower write latency, expanding the range of use cases supported by the Redis fork. By Renato Losio

🎬 This article is a companion to my YouTube video. Watch it here: Introduction In this video we are going to talk about Redis — what it is, what it does, and why it is an important part of my back-end stack. What is Redis? Redis is a free, open-source, in-memory data store. Unlike PostgreSQL which stores data on disk, Redis stores data entirely in memory — in RAM. This makes it extremely fast. Redis can handle millions of operations per second with sub-millisecond response times. Redis is most commonly used as a cache, a session store, a message broker, and a real-time data store. What is Caching? When your application queries a database, that query takes time — it reads from disk, processes the query, and returns the result. If the same query is made thousands of times per second, you are hitting the database thousands of times unnecessarily. Caching solves this by storing the result of a query in memory. The first request hits the database and the result is stored in Redis. Every subsequent request gets the result from Redis — which is in memory and therefore much faster — instead of hitting the database again. Think of it like a shortcut. Instead of driving the long route to the database every time, you take the shortcut through Redis. What Does Redis Do? Caching Store frequently accessed data in memory for fast retrieval. Database query results, API responses, computed values — anything that is expensive to compute and accessed frequently is a good candidate for caching. Session Storage Store user session data in Redis instead of the database. Since sessions are read on every request, having them in memory is significantly faster than a database lookup. Rate Limiting Track how many requests a user or IP address has made in a given time window. Redis's atomic increment operations make it perfect for implementing rate limiting. Message Queues and Pub/Sub Redis supports publish/subscribe messaging and message queues. Applications can publish messages to a channel a

Rocicorp has released Zero 1.0, a stable version of its sync engine after two years of development. This update introduces a schema change hook for Supabase and includes bug fixes. Zero operates by pairing a client library with a read-only Postgres cache. Community feedback highlights positive developer experience but raises concerns about production readiness and existing limitations. By Daniel Curtis

Your LLM bill is exploding because 70% of user queries are semantically identical, yet your traditional cache ignores them completely. Even worse, if you implement semantic caching poorly, a single bad actor can poison your entire AI model's knowledge base, leading to incorrect or malicious responses for legitimate users. The Cost of Redundancy in LLM Systems Imagine running an AI-powered customer support chatbot for an e-commerce platform. Users frequently ask things like, "What's your return policy?", "How can I send this item back?", or "Do you offer refunds if I'm not satisfied?". To an LLM, these are distinct prompts, each triggering an expensive API call to OpenAI or Anthropic, costing you dollars per thousand tokens. On the surface, it looks like individual requests. But structurally, they all ask the same question with a similar intent. Your traditional HTTP cache, which relies on exact string matches, sees "What's your return policy?" and "How can I send this item back?" as entirely different requests. It misses the semantic similarity. So, for every variation of the same question, you're making a full LLM inference call. If 50-70% of your user queries fall into these semantically redundant categories, your LLM costs skyrocket. For a system handling millions of requests daily, this can quickly turn a profitable product into a money pit, all while adding unnecessary latency for your users. Semantic Caching: The "Fast Path" for LLMs Semantic caching solves this by moving beyond exact string matches. Instead of looking for an identical prompt, it looks for prompts that mean the same thing. It works by converting incoming user prompts into numerical vector representations (embeddings) and then performing a similarity search against a cache of previously embedded prompts and their corresponding LLM responses. Here's the workflow: USER PROMPT | v [ EMBEDDING MODEL ] -- Transform Prompt to Vector (e.g., [0.1, 0.5, -0.2, ...]) | v [ VECTOR DATABASE / CACHE ] | +--