标签：#performance

Engineering posts often end with: The new design is correct, scalable, and fast. Fast compared with what? When we changed Podium so tied players rank by arrival time instead of player ID, we added: a Lua script; a per-leaderboard sequence; a public-ID mapping; a second sorted set for ascending order. That design is fairer. It is also impossible for it to be free. So we built two benchmark layers: direct Redis strategy benchmarks to isolate the data-model cost, and end-to-end HTTP benchmarks to show what users actually experience. We are publishing the results, including the regression, because performance claims are useful only when readers can inspect the workload and reproduce the measurement. TeneficGames / podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium provides ready-to-run HTTP and gRPC APIs for scores, ranks, seasons, and player-relative views. It is designed for backend teams operating large fleets of independent leaderboards without provisioning each leaderboard in advance. Fair, deterministic ordering when scores are equal. Single and bulk score updates, including multi-leaderboard fan-out. Standalone Redis and real Redis Cluster integration coverage. Deploy one multi-architecture OCI image with Docker, containerd, Kubernetes or another OCI-compatible runtime. Quickstart · Performance · API · Documentation · Helm chart · Docker Hub · GHCR Quickstart Start Redis 8.2 and the latest stable Podium image: docker network create podium docker run --detach --name podium-redis --network podium redis:8.2-alpine docker run --detach --rm --name podium \ --network podium \ --publish 8880:8880 \ --publish 8881:8881 \ --env PODIUM_REDIS_HOST=podium-redis \ --env PODIUM_REDIS_PORT=6379 \ trungdlp/podium:latest start Verify the service: curl http://localhost:8880/healthcheck WORKING Submit two equal scores: curl --request … View on Gi

When building modern Python applications—whether scraping web pages, fetching data from external APIs, or querying databases—IO-bound operations often slow down execution. Python’s concurrent.futures module provides a high-level, elegant interface for running tasks asynchronously. In this guide, we'll break down what Futures are, why you need them, and how to use them effectively using a practical e-commerce product service. What is a Future? A Future represents an eventual result of an asynchronous operation. When you launch an expensive, long-running task concurrently, your program doesn't pause to wait for the output. Instead, it instantly gets back a Future object —a low-cost proxy or standard "claim ticket." The Future acts as a placeholder for a result that hasn't been computed yet. It keeps track of the task's execution state ( PENDING , RUNNING , CANCELLED , or FINISHED ). Once the task finishes, the Future stores the return value or any exception thrown during execution. Why are Futures Needed? In standard synchronous Python execution, calling a function blocks your main thread until that function finishes: Task 1 (2s) ──> Task 2 (3s) ──> Task 3 (1s) = 6 seconds total When dealing with IO-bound operations (like waiting for network responses or reading disks), your CPU sits completely idle during those delays. By offloading tasks into background threads or processes via Futures, your application can run multiple IO operations simultaneously: Task 1 (2s) [████████] Task 2 (3s) [████████████] Task 3 (1s) [████] ----------------------------------------- Total Time: 3 seconds (time of longest task) When Should You Use Futures? IO-Bound Workloads: Scraping multiple web pages, batch-calling microservices, querying multiple databases, or fetching images concurrently ( ThreadPoolExecutor ). CPU-Bound Parallelism: Performing heavy mathematical operations or image processing across multiple CPU cores ( ProcessPoolExecutor ). Decoupled Workflows: When you want to trigg

Embedding a browser game looks simple: <iframe src= "https://games.example.net/my-game" ></iframe> That line lets a third party join the page lifecycle immediately. It can download a large bundle, establish connections, run scripts, request storage, display advertising, or fail before the visitor decides to play. AI-assistance disclosure: I used AI to help draft and edit this article, then reviewed its architecture, code, claims, and limitations before publication. For a game directory, that default is both expensive and surprising. A visitor may have opened the page to read the controls, compare games, or check whether the game works on a phone. Loading the player before that intent is known wastes bandwidth and collapses two separate decisions—visiting the guide and opening the third-party game—into one. While working on a browser-game portal, I treated the site and the embedded player as two different trust and performance boundaries. The page renders first-party information immediately. The third-party frame is created only after an explicit Play action. This article explains that pattern and the engineering details that made it useful rather than merely decorative. Start with a two-layer model The outer page should be a complete page without the game: A descriptive heading and summary Controls and gameplay tips Developer and platform information Related games and category navigation A poster or cover image A real button that starts the player The inner layer is a small launcher responsible for the game lifecycle: Validate the requested game. Wait for an intentional Play action. Create the provider iframe. Report loading state. Offer recovery when loading is slow or blocked. Remove the frame when the player resets it. Do not put the remote URL in the initial markup Native iframe lazy loading is helpful below the fold, but it is not an intent gate. Browsers decide when a loading="lazy" frame is close enough to fetch. If the goal is “no third-party game request be

When building local RAG (Retrieval-Augmented Generation) applications, edge agents, or serverless AI pipelines, developers usually hit a wall with standard vector stores: memory overhead. Running a dedicated vector database locally often demands hundreds of megabytes—or gigabytes—of RAM just to keep indices warm. On the flip side, lightweight local options like scanning raw JSON files or querying SQLite don't scale well when vector dimensions climb into the thousands (1536d+). We built Bitweave to solve this exact trade-off: a zero-copy, SIMD-accelerated hybrid retrieval engine in Rust (with Python bindings) that handles categorical filtering and vector search while locking its active heap footprint under 1.1 MB RSS. The Architecture: How Bitweave Achieves Sub-Millisecond Speed at <1.1 MB RAM Bitweave relies on a 3-part design to maximize search speed while keeping memory consumption negligible: [ Categorical Filters ] ---> Bit-Sliced Bitmaps │ ▼ [ Query Vector (1536d) ] --> 1-Bit SIMD Pre-Filtering (Hamming Distance) │ (Top K Candidates) ▼ [ Raw Embeddings Buffer ] -> Zero-Copy Float32 Rescoring (exact_rescore=True) │ ▼ Top-K Results Array (NumPy) Zero-Copy Memory Mapping (memmap2) Instead of deserializing index files into Python RAM or Rust heap space, Bitweave uses memory-mapped files (.bweave). The operating system's page cache handles lazy loading of index segments directly from disk into virtual address space. As a result, the active RSS memory footprint remains static around 1.1 MB, whether your index holds 5,000 or 200,000 records. 1-Bit Vector Quantization & SIMD Hamming Distance High-dimensional float32 vectors (1536d) are quantized down to 1-bit sign masks (where values > 0 map to 1 and <= 0 map to 0). During pre-ranking, Bitweave uses SIMD bitwise XOR and POPCNT operations to compute Hamming distances across candidate vectors in microseconds. Zero-Copy 2-Pass Float32 Rescoring (exact_rescore=True) Quantization speeds up initial candidate selection, but f

log.info("user logged in") looks free. It isn't. Behind that one line is a chain of decisions — buffer or not, flush or not, block or drop, same thread or another — and each one trades latency , throughput , and durability against the others. This post walks the whole chain, from the method call down to the bytes hitting the disk platter. If you've ever wondered why your p99 latency has a mysterious spike, why logs vanish after a crash, or what "async logging" actually buys you, this is for you. First, the map: facade vs. implementation Java logging is a two-layer cake, and mixing up the layers is the #1 source of confusion. The facade is the API your code calls. The implementation is what actually writes the bytes. your code │ log.info(...) ▼ ┌───────────────────────────────┐ │ Facade: SLF4J (or Log4j2 API)│ ← the interface you compile against └──────────────┬────────────────┘ │ bound at runtime ┌───────────┼────────────┬──────────────┐ ▼ ▼ ▼ ▼ Logback Log4j2 Core java.util.logging ... (the engine that buffers, formats, and flushes) SLF4J — the de-facto standard facade. Your app should log against this. Logback — the reference SLF4J implementation. Solid, widely deployed. Log4j2 — the performance-focused implementation, famous for its lock-free async loggers. java.util.logging (JUL) — built into the JDK, rarely chosen on purpose. Why the split? So you can swap engines without touching a single log. call. Everything interesting in this post — the buffering, the flushing, the async magic — happens in the implementation layer. The anatomy of a single log call Before we talk flushing, let's see what one log.info(...) actually does. There are five stages: 1. Level check → is INFO enabled for this logger? (cheap, often the fastest bail-out) 2. Build LogEvent → capture message, timestamp, thread, MDC context, maybe a stack trace 3. Filter → run any configured filters 4. Layout / encode → turn the event into bytes ("2026-07-28 12:00:01 INFO ...") 5. Append → write those by

I run HydraDNS, an open-source DNS security gateway in Go. Last month I sat down to find out what one box could actually handle before I put it on anyone else's network. The plan had a rule I'd written for myself: every number we discover becomes either a sales claim or a fix ticket. No number, no claim. I expected to find one bottleneck. I found two, stacked on top of each other, and a third thing I wasn't looking for: a data structure in our own documentation that had never existed in the code. Everything below was measured on a 22-core dev machine with load generated inside the container, using dnspyre, so docker-proxy and host networking stay out of the numbers. It's not appliance hardware and I'm not making appliance claims. The shapes are what matter. The first ceiling: ~500 QPS, and it didn't care what I threw at it The first redline run capped at roughly 500 queries per second. Fine, servers have limits. What made it interesting was that the cap didn't move. Blocked queries: ~500. Cached queries that never touch upstream: ~500. Two code paths that do completely different work, hitting the same wall, with the CPU sitting under 30% of 22 cores. That signature is worth memorizing. When two very different paths hit the same ceiling and the CPU is bored, the bottleneck isn't in either path. It's in something they share, or something upstream of both. Ours was in the blocklist check. IsBlocked ran a SQL COUNT against a 92k-row blocklist_entries table on every query . Not just candidate blocks, every query, because the check sits in front of the cache, so even cache hits paid for it. And all of those reads were serialized through a single SQLite connection, MaxOpenConns=1 , which was also absorbing the async write traffic from query logging. The engine's self-measured latency under load: p50 of 50ms, p99 of 5000ms. Five full seconds at the tail, for DNS, which is supposed to be the fast part of the internet. The part where I found out our docs were lying Here's the

How a Missing Database Index Turned an 8ms API into an 8-Second Nightmare Every developer...

600 MiB of headroom My RTX 4070 was running Qwen 35B beautifully after the --cpu-moe trick from a previous run. The tokens/sec were where I wanted them. VRAM sat at 11,714 MiB out of 12,281 — 95% full. That leaves 600 MiB. Not enough for a serious agent. The context window I was giving llama.cpp was -c 4096 . Fine for chat. Not fine when a Claude Code-style agent hands the model 12,000 tokens of tool definitions before it says hello. I wanted -c 32768 . That's an 8× jump. And the memory that grows with context length is the KV cache. Multiply the cache by 8 with 600 MiB free, and llama.cpp dies during warm-up. I know because I tried it first. What actually sits on the GPU After offloading the MoE experts to CPU (the previous chapter's trick), the GPU is holding two things: The attention weights and non-MoE parameters The KV cache — a running record of every token the model has already read The first is fixed. The second grows linearly with context length. Double the context, double the cache. -c 4096 → -c 32768 doesn't just want 8× more tokens processed, it wants 8× more cache resident in VRAM the whole time. There is no room. So the cache itself has to shrink. Two flags llama.cpp takes two flags for KV cache dtype: llama-server -m qwen35.gguf -ngl 99 --cpu-moe -c 32768 \ -ctk q8_0 -ctv q8_0 -ctk is the Key cache, -ctv is the Value cache. Default is f16 (16-bit). q8_0 cuts each in half. Halving both means the KV cache footprint drops by roughly 50%. That freed-up VRAM is exactly what I need to make the context 8× bigger without touching the model weights. The measurement Same prompt, same seed, two runs — one at f16 KV, one at q8_0 KV: KV dtype Max -c I could allocate Tokens/sec (decode) Perplexity delta f16 (default) 4096 ~34.6 baseline q8_0 32768 ~34.1 negligible in my tests The speed loss is inside noise. The context is 8× longer. The quality drop I could not tell apart from run-to-run variance. Community measurements agree: symmetric q8_0 KV lands somewhere unde

Performance optimization in React is often surrounded by myths. Developers routinely wrap every single component in React.memo , wrap every function in useCallback , and wonder why their application is still sluggish or memory-heavy. Premature optimization can actually degrade app performance and clutter your codebase. To build fast React applications, you need techniques that address actual bottlenecks: unnecessary re-renders, unoptimized state placement, oversized bundles, and main-thread blocking. Here are five practical React performance optimization techniques that deliver measurable results in production. 1. Push State Down (Fix Rerender Cascades) Before reaching for useMemo or React.memo , evaluate your state placement . When state lives too high up in the component tree, every state update forces the entire sub-tree to re-render. ❌ The Anti-Pattern: State at the Root // Changing `color` forces <HeavyChartComponent/> and <ComplexTable/> to re-render! export default function App () { const [ color , setColor ] = useState ( ' #6366f1 ' ); return ( < div > < input type = "color" value = { color } onChange = { ( e ) => setColor ( e . target . value ) } /> < p style = { { color } } > Sample Text </ p > < HeavyChartComponent /> < ComplexTable /> </ div > ); } ✅ The Fix: Component Isolation Move the isolated state and its control into its own dedicated child component: Javascript function ColorPicker () { const [ color , setColor ] = useState ( ' #6366f1 ' ); return ( < div > < input type = "color" value = { color } onChange = { ( e ) => setColor ( e . target . value ) } /> < p style = { { color } } > Sample Text </ p > </ div > ); } export default function App () { return ( < div > < ColorPicker /> { /* These components are no longer impacted by color state changes */ } < HeavyChartComponent /> < ComplexTable /> </ div > ); } 2. Pass Components as Children (Component Composition) Sometimes state must remain in a parent component, but you don't want child components

SQL is quite flexible, making it easy to write a single query that works for two situations: one without a parameter and a WHERE clause, and another with a parameter for filtering, all in the same SQL query. For example, I came across a benchmark comparing MongoDB and PostgreSQL that shows how to handle pagination effectively—by avoiding OFFSET and instead using the last value to fetch the next set of results. The first page includes a WHERE clause along with ORDER BY and LIMIT, while the following pages add an extra WHERE condition. In the MongoDB version of this benchmark, the filter is handled within the application, which leads to two separate queries for these scenarios. export async function getOrders ( cursor ) { const match = cursor ? { timestamp : { $gt : new Date ( cursor ) } } : {}; const rows = await orders . aggregate ([ { $match : match }, { $sort : { timestamp : 1 } }, { $limit : PAGE_SIZE }, ]) We can do the same in PostgreSQL using a single prepared statement. SQL is such a powerful language that it often feels tempting to write it this way: SELECT * FROM orders WHERE $ 1 :: timestamptz IS NULL OR "timestamp" > $ 1 ORDER BY "timestamp" ASC LIMIT $ { PAGE_SIZE } If $1 is NULL, it skips the second condition in the OR clause and retrieves all rows without filters, resulting in a broad fetch. When $1 has a value, it filters the results using that specific value, enabling a more targeted search. However, using a generic query can sometimes lead to a less-than-ideal execution plan that's not perfectly tailored for each specific situation. I gave it a try: drop table if exists orders ; create table orders ( order_id text primary key , "timestamp" timestamptz not null ); create index idx_orders_timestamp on orders ( "timestamp" ); insert into orders select 'ORD-' || g , '2025-01-01' :: timestamptz + g * interval '1 minute' from generate_series ( 1 , 5000000 ) as g ; analyze orders ; prepare getorders ( timestamptz , int ) as select * from orders where $ 1 :

Hello everyone, I am currently building a high-performance API gateway that integrates business logic components—implemented via WASI and running within Wasmtime—as HTTP/TCP/QUIC handlers. I am exploring the best design approach to achieve a zero-copy data path from the upstream network layer—specifically hyper for HTTP/1.x and HTTP/2, and h3-quinn (based on Quinn) for HTTP/3—to the wasi:http guest environment. Given that: hyper and h3-quinn each manage their own internal buffer pools (e.g., bytes::Bytes ), asynchronous read/write streams, and frame decoders. Wasmtime's wasmtime-wasi-http implements the wasi:http (WASIp2) specification, which relies on resource types such as InputStream and OutputStream . I aim to minimize memory copying and CPU overhead when passing large request bodies or streaming responses across the sandbox boundary. For those experienced with bridging I/O between the host and guest in Wasmtime, I have a few architectural questions: Buffer ownership and memory mapping: How can host-side bytes::Bytes (from hyper or h3-quinn ) be mapped or bridged into Wasm linear memory (and vice versa) without requiring a CPU-based memcpy ? Does Wasmtime's resource streaming support direct memory views, or are we essentially limited to copying data chunks via guest memory pointers? Adapting stream abstractions: hyper uses http_body_util::combinators / http_body::Body , h3 uses its own stream primitives, while wasi:http uses wasi:io/streams . What is the idiomatic way to efficiently adapt these asynchronous streams on the host side (i.e., within the wasmtime-wasi-http handler implementation) without blocking the tokio runtime? Backpressure propagation: How can backpressure signals be correctly propagated from the Wasm guest (e.g., when the guest's InputStream is consuming data slowly) all the way back to the Quinn congestion controller or Hyper connection pool, thereby avoiding unbounded buffering on the host side? If anyone has built similar high-performance ga

The Go pod is running in production. CPU limit set to 2, metrics look reasonable. But under load, P99 latencies spike intermittently with no obvious cause. No errors, no goroutine leaks, just latency blowing up on traffic bursts. The root cause is usually invisible: GOMAXPROCS equals the number of CPUs on the physical node, not the container limit. Your Go app thinks it has 32 CPUs when it only has 2. The Linux kernel handles the gap in its own way — CFS throttling. What GOMAXPROCS reads (and what it ignores) By default, the Go runtime computes GOMAXPROCS via runtime.NumCPU() , which reads the number of CPUs available at the OS level. On a 32-core Kubernetes node, that returns 32 — regardless of what resources.limits.cpu says in your pod spec. Kubernetes CPU limits are enforced through Linux cgroups (v1 or v2). Cgroups are transparent to processes: a pod with limits.cpu: "2" doesn't see two virtual CPUs, it sees all the node's CPUs and gets suspended when it consumes too much. The Go runtime, historically, never read cgroups. It trusted the physical core count. package main import ( "fmt" "runtime" ) func main () { // Inside a pod with limits.cpu: "2" on a 32-core node fmt . Println ( runtime . NumCPU ()) // → 32 fmt . Println ( runtime . GOMAXPROCS ( 0 )) // → 32 } CFS throttling: how the kernel slows you down The Linux CFS (Completely Fair Scheduler) enforces CPU limits via two cgroup parameters: cpu.cfs_quota_us (allowed CPU time) and cpu.cfs_period_us (measurement window, 100 ms by default). A pod limited to 2 CPUs gets at most 200 ms of CPU time per 100 ms window. When Go spawns 32 OS threads for 32 parallel goroutines, those threads compete for physical CPUs. Once their combined usage exceeds the cgroup quota within the current window, the kernel suspends all threads in the cgroup until the next window starts. That's throttling: a complete application freeze lasting anywhere from a few milliseconds to several tens of milliseconds. A handful of these per second

The problem with "just add more workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffle and skew , and no amount of extra nodes fixes a badly partitioned join. This post builds a realistic pipeline (order events joined against a small dimension table, aggregated, and written to Delta Lake) from the ground up, and uses it to work through: How Spark's shuffle actually behaves during a wide transformation Diagnosing and fixing data skew with salting and adaptive query execution (AQE) Laying out the resulting Delta table with Z-Ordering so downstream queries skip irrelevant files Governing access to the whole pipeline with Unity Catalog Architecture overview Pipeline shape — a batch job reading raw events, joining against a dimension table, aggregating, and writing to a governed Delta table: What happens inside a shuffle stage — this is the part most tutorials skip, and it's the key to understanding why skew hurts: Step 1 — Set up governed tables in Unity Catalog Everything downstream depends on tables being registered under Unity Catalog, which gives you centralized access control and lineage instead of per-workspace table grants. -- setup.sql, run in a Databricks SQL or notebook cell CREATE CATALOG IF NOT EXISTS retail_analytics ; CREATE SCHEMA IF NOT EXISTS retail_analytics . events ; CREATE TABLE IF NOT EXISTS retail_analytics . events . raw_orders ( order_id STRING , customer_id STRING , product_id STRING , quantity INT , event_ts TIMESTAMP ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/raw_orders' ; CREATE TABLE IF NOT EXISTS retail_analytics . events . dim_products ( product_id STRING , category STRING , unit_cost DOUBLE ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/dim_products' ; GRANT SELECT ON TABLE retail_analytics . events . raw_orders TO `analysts` ; Step 2 — Read and force a broadcast join for the small dimension table dim_products is s

Headline: Next.js Middleware (middleware.ts at the project root) runs before every matched request — before cache, before rendering, before the route. That position makes it right for auth redirects, A/B cookie bucketing, and locale detection. Wrong for database queries and heavy imports. In 2026, Middleware on Vercel runs on Fluid Compute (standard Node.js), so the constraint is latency budget, not API availability. Key takeaways Middleware runs before every matched request — before cache, rendering, or route handler — the right layer for auth, locale, and A/B bucketing. Middleware can read requests, set cookies, redirect, rewrite, or return early — without the route running. DB queries and large packages add latency to every request. On Vercel in 2026, Middleware runs on Fluid Compute (standard Node.js). The constraint is latency: every added millisecond is paid on every matched request. Use matcher to scope Middleware to only the routes that need it; without it Middleware runs on every static asset request. Auth in Middleware = verifying a self-contained JWT without a DB call. Full session validation belongs in the route. I spent a long time only using Middleware for locale redirects. After shipping auth-protected routes and an A/B test, the full shape became clear. What is Next.js Middleware and where does it run? Middleware is exported from middleware.ts at the project root. It intercepts matched requests before route resolution, cache lookup, and Server Component execution. Returns one of four types: pass through ( NextResponse.next() ), redirect, rewrite (serve different content while keeping original URL in address bar), or a direct response. export function middleware ( request : NextRequest ) { return NextResponse . next (); } export const config = { matcher : [ ' /((?!_next/static|_next/image|favicon.ico).*) ' ], }; Without matcher , Middleware runs on every request including static files. On Vercel in 2026, Middleware runs on Fluid Compute — standard Nod

Have you ever noticed that your GenAI applications are spending massive amounts of time and money re-reading the exact same setup text? Every time a user asks a short question in a chatbot, the Large Language Model (LLM) must re-read your entire 2,000-word corporate playbook, your agent's system rules, and the full chat history from scratch. This phase is called the pre-fill math phase, and it drives up both your cloud bill and your user latency (Time-to-First-Token).With Amazon Bedrock Prompt Caching for Claude 4.6 (both Sonnet 4.6 and Opus 4.6), this problem is completely solved. You can achieve up to a 90% cost reduction on input tokens and an 85% drop in latency by using a clever architectural shortcut. Here is exactly how it works under the hood, how AWS maintains it across API requests, and how to implement it using Python. The Secret Architecture: Model Inference vs. AWS Infrastructure Prompt caching is a beautiful team effort between the AI model hardware and the AWS cloud infrastructure. The Model Level (The Brains): Inside Claude 4.6, text is processed through mathematical matrices called KV (Key-Value) Caches. Instead of re-reading text, the GPUs calculate the meaning of your system instructions once and build a "mathematical profile." When a cache point is triggered, the model freezes this calculated KV state inside the GPU memory. 2.The AWS Bedrock Level (The Manager): Normally, an LLM wipes its memory the millisecond an API call finishes. AWS Bedrock changes this. It takes your static prompt, creates a secure, unique cryptographic hash (fingerprint), and pins that KV memory block alive. When your next API request comes in, AWS Bedrock instantly hashes the new incoming prompt text. If the top section matches a saved fingerprint, Bedrock's router bypasses the standard pre-fill setup and routes your request directly to the GPU holding your frozen mathematical profile. It is exactly like loading a "Save Game" file instead of restarting a video game from Le

"You're spawning a fresh codex exec for every single call? Just run the app-server and reuse a thread. The prompt cache alone will pay for it." That sentence sounds so obviously right that nobody ever benchmarks it. We run a small bazaar of headless AI daemons — designers, reviewers, code workers — and each one shells out to OpenAI's Codex CLI dozens of times per work item. A resident codex app-server --stdio with warm threads looked like free money. We measured it four times. The prompt cache eventually hit 86% on the turns that mattered. The pipeline got slower than the boring baseline, and 39% more expensive in raw tokens. This is the story of why, and of what app-server is actually for. The setup was deliberately minimal: no resident daemon, no pool. The server is spawned privately for one formation , runs its turns, and is killed by a context manager on the way out — success or crash. That keeps the comparison honest: same lifecycle as a subprocess, and adopting it stays a measurement problem instead of an architecture debate. Trusting the probe Before writing any integration, we cloned the actual openai/codex source and pointed a probe at a real app-server: handshake, per-turn sandbox overrides, interrupts, forks, kill-and-resume. Eight checks, all green, plus three findings you won't find in any docs: cwd and sandboxPolicy are sticky when omitted — every turn must re-state them explicitly, or turn 3 silently inherits turn 2's write access. Token usage arrives on thread/tokenUsage/updated , not on turn/completed . An empty thread that never ran a turn is never materialized — you cannot resume it later. Pre-warming a thread pool is a trap. And the seductive one: on the probe's trivial two-turn thread, turn 2 reported 15,104 cached input tokens . See? The cache works. Ship it. That number cost us three more measurement runs to un-believe. The four runs Same fixture repo, same greenfield request, one real four-stage formation each: A: codex exec (baseline) B: app

Most ad-hoc timing code in Python looks like this: start = time . perf_counter () result = do_work () elapsed = time . perf_counter () - start stats [ name ]. append ( elapsed ) It works, until do_work() raises. Then the line that records the timing never runs, the exception propagates, and the one call that was probably slowest — the one that failed — is silently missing from your stats. If you're using timing data to find what's expensive, the failing case is exactly the one you can least afford to lose. timerx is a small, dependency-free Python timing library — a decorator, a context manager, and named stopwatches, all backed by one stats store. The one rule that shapes the whole implementation: a timing gets recorded whether or not the timed code raised. Decision 1: finally , everywhere, no exceptions to the rule @functools.wraps ( target ) def wrapper ( * args : Any , ** kwargs : Any ) -> Any : started = self . _clock () try : return target ( * args , ** kwargs ) finally : elapsed = self . _clock () - started with self . _lock : self . _record ( label , elapsed ) return wrapper The async wrapper is the identical shape with await added. The context manager ( _Lap ) does the same thing structurally, just split across __enter__ / __exit__ instead of try / finally : def __exit__ ( self , * exc_info : object ) -> bool : if self . _started is None : raise RuntimeError ( " timerx lap exited before it was entered " ) elapsed = self . _timer . _clock () - self . _started with self . _timer . _lock : self . _timer . _record ( self . _name , elapsed ) return False Note the return False — __exit__ deliberately never swallows the exception. It records the timing and lets the exception continue propagating unchanged, because a timing library has exactly one job here: observe, not intervene. A version that suppressed exceptions to "clean up" would be actively dangerous to drop into someone else's codebase. Three entry points — decorator, context manager, stopwatch — and all t

The engineering team at Zalando recently described the design and implementation of an in-process, client-side load balancer for a high-throughput API handling around 1 million requests per second. The result was more predictable latency, a drop in infrastructure costs, and better visibility into where failures actually originate. By Renato Losio

Redis gets reached for reflexively, "just add Redis," as if it were a single fix for slowness. It's genuinely one of the most useful tools in a backend engineer's kit, but using it well starts with understanding what it actually is: an in-memory data structure store, not just a cache. Once you see it as a fast, versatile store of real data structures, the range of problems it solves cleanly (caching, rate limiting, queues, sessions, leaderboards, locks) stops looking like a grab bag and starts looking like one idea applied many ways. This is the opening article of the Redis Masterclass, and it builds on the PostgreSQL series : Redis usually sits alongside a primary database like Postgres, not instead of it. In-memory is the whole point Redis keeps its data in RAM. That single fact explains most of its character. Reading from memory is orders of magnitude faster than reading from disk, so Redis operations typically complete in well under a millisecond, and a single instance handles a very high request rate. That speed is why it's the default choice for anything on the hot path, where a database round trip would be too slow. The tradeoff is that RAM is smaller and more expensive than disk, and volatile. Redis addresses durability with persistence options we'll cover later, but the mental model to start with is: Redis is fast because it's in memory, and you use it for data that benefits from being fast to access, not as the permanent home for everything. It's a data structure store, not a key-value blob The common misconception is that Redis is a simple key-value store, strings in and strings out. It's much more. Redis stores real data structures as values, each with its own commands: Strings for simple values, counters, and cached blobs. Hashes for objects with fields, like a user record. Lists for ordered sequences and simple queues. Sets for unique collections and membership checks. Sorted sets for ranked data like leaderboards and priority queues. Plus streams, bit

You've probably seen this: Kubernetes probes are green, /health returns 200, CPU isn't on fire - and users are timing out. Process is up. Just not doing anything useful. Classic event loop freeze. Something sync and expensive (or a dumb busy loop) grabs the thread, and suddenly timers, HTTP, DB callbacks - all of it waits. Including your health endpoint, because that also needs the loop. monitorEventLoopDelay() is fine for dashboards. It won't yell at you while the process is stuck, and it definitely won't write a log line from outside the frozen loop. That's the part that annoyed me enough to build this. I made @js-ak/watchdog - tiny N-API addon. The monitor runs in C++ on its own thread and writes JSON lines to stderr/file even while the JS event loop is stuck. Your freeze / recovered handlers only run after the loop unblocks. Optional RSS/CPU, optional stack sample. npm install @js-ak/watchdog const watchdog = require("@js-ak/watchdog"); watchdog.on("freeze", (e) => console.log(e.event, e.duration_ms)); watchdog.on("recovered", (e) => console.log("back after", e.duration_ms, "ms")); watchdog.start({ freezeThresholdMs: 1000, logTarget: "stderr", // or "file" | "both" source: "payments-api", }); Real freezes usually aren't while (true) . More like giant JSON.parse, a cursed regex, sync crypto/compress, or some dependency doing sync I/O on a hot path. Stuff you only notice after people complain. Prebuilds for the usual platforms (Node 22+). MIT. Repo: https://github.com/JS-AK/watchdog Curious how other people catch loop stalls in prod - and what you'd actually want in the freeze payload. Roast the API if it's wrong.