标签：#redis

AWS has recently introduced durability for Amazon ElastiCache for Valkey, enabling reliable data retention across failures and expanding support beyond caching to persistent workloads. The feature offers new options that prioritize either minimizing data loss or maintaining lower write latency, expanding the range of use cases supported by the Redis fork. By Renato Losio

Observability isn't just a buzzword; it's a necessity, especially when diving into distributed systems. If you're using NestJS, you might want to take a look at RedisX. It's a modular toolkit that can boost the observability of your applications. A standout feature? The Metrics Plugin. It meshes well with Prometheus, delivering insights into Redis operations in your NestJS setup. Getting RedisX Metrics Rolling in NestJS So, first things first. To harness the power of RedisX Metrics, you need to set up your NestJS app with RedisX. This means installing some packages and configuring the RedisModule with the MetricsPlugin. Hit your terminal and run: npm install @nestjs-redisx/core @nestjs-redisx/metrics Now, let's tweak your AppModule . You want it to use RedisModule with MetricsPlugin: import { Module } from ' @nestjs/common ' ; import { ConfigModule , ConfigService } from ' @nestjs/config ' ; import { RedisModule } from ' @nestjs-redisx/core ' ; import { MetricsPlugin } from ' @nestjs-redisx/metrics ' ; @ Module ({ imports : [ ConfigModule . forRoot ({ isGlobal : true }), RedisModule . forRootAsync ({ imports : [ ConfigModule ], inject : [ ConfigService ], plugins : [ new MetricsPlugin ({ prefix : ' redisx_ ' , endpoint : ' /metrics ' , defaultLabels : { service : ' my-service ' } }) ], useFactory : ( config : ConfigService ) => ({ clients : { host : config . get ( ' REDIS_HOST ' , ' localhost ' ), port : config . get ( ' REDIS_PORT ' , 6379 ), }, }), }), ], }) export class AppModule {} Prometheus Metrics: What You Get With MetricsPlugin set up, your app now exposes a /metrics endpoint. Prometheus can scrape this endpoint, dishing out detailed metrics about your Redis operations. Here's a snapshot of what you get: redisx_cache_hits_total : Tracks total cache hits. redisx_lock_acquired_total : Total locks acquired. redisx_redis_commands_total : Total Redis commands run. Making Prometheus Work for You To get those insights, set up Prometheus to scrape your /metrics end

I was now done with the basic setup. However, during my time working at my startup, I have learnt to think about a project wearing multiple caps. One such aspect was - With Dhrishti running on a server that was already loaded, I did NOT want the tracking application itself to be heavy. I had to set some benchmarks to ensure that Dhrishti did not consume a tonne of space while tracking the metrics. I also had a problem with unresolved requests - in my mock_services, I had a client that was continuously hitting the API Gateway service. I had to fine-tune all the requests so that I could run tests under different loads, but the advantage was that my project was easily able to discern where the client request was coming from. However, in a production scenario, you can never know where a request is coming from - obviously, we cannot resolve different customer IPs to their respective customer names. This was the first problem. I had to specify what a customer was, and what an unknown request was. I came up with the following solution - Any unresolved IPs are going to be added to a table in the UI called unresolved IP table. This would help me with debugging later. Now, any unresolved IPs which also made requests to an ENTRY-POINT into my application could be added as the customers. For this, I very simply had to filter out the unknown IPs, and keep a configurable entry-point in dhrishti.json in which I would add a bunch of entry-points (in the case of my mock micro-service architecture, only 1) Now, I could differentiate between 2 types of unknown IPs - one which was potentially a customer, one which was a background network call, not important to the working system. The next problem was with the client service itself. It was difficult to simulate, say - a million users in my system. I had essentially built a service which was only being used by 1 customer, but how would Dhrishti behave if I added multiple client IPs? Using K6 k6 is a Grafana based application that helps

🎬 This article is a companion to my YouTube video. Watch it here: Introduction In this video we are going to talk about Redis — what it is, what it does, and why it is an important part of my back-end stack. What is Redis? Redis is a free, open-source, in-memory data store. Unlike PostgreSQL which stores data on disk, Redis stores data entirely in memory — in RAM. This makes it extremely fast. Redis can handle millions of operations per second with sub-millisecond response times. Redis is most commonly used as a cache, a session store, a message broker, and a real-time data store. What is Caching? When your application queries a database, that query takes time — it reads from disk, processes the query, and returns the result. If the same query is made thousands of times per second, you are hitting the database thousands of times unnecessarily. Caching solves this by storing the result of a query in memory. The first request hits the database and the result is stored in Redis. Every subsequent request gets the result from Redis — which is in memory and therefore much faster — instead of hitting the database again. Think of it like a shortcut. Instead of driving the long route to the database every time, you take the shortcut through Redis. What Does Redis Do? Caching Store frequently accessed data in memory for fast retrieval. Database query results, API responses, computed values — anything that is expensive to compute and accessed frequently is a good candidate for caching. Session Storage Store user session data in Redis instead of the database. Since sessions are read on every request, having them in memory is significantly faster than a database lookup. Rate Limiting Track how many requests a user or IP address has made in a given time window. Redis's atomic increment operations make it perfect for implementing rate limiting. Message Queues and Pub/Sub Redis supports publish/subscribe messaging and message queues. Applications can publish messages to a channel a

For two years, our Sponsored Placements service booked limited ad inventory through Redis: a counter in Redis, a Redlock around the decrement, and a TTL key per hold. It oversold. Not catastrophically — consistently. 40–60 double-booked placements a month , each one a manual refund and an apology email to an advertiser. The root cause was never one bug. It was the architecture: two sources of truth that could not be made atomic with each other. The count lived in Redis; the ownership lived in SQL. No transaction spans both. The Redlock only ever protected the Redis half. The one mental shift SKIP LOCKED turns a contended table into a concurrent work queue. Instead of every request fighting over one counter, each request grabs different rows and ignores the ones someone else is holding. FOR UPDATE alone serializes — that's the experience that scares people off SQL locking. FOR UPDATE SKIP LOCKED is the opposite: a transaction that would have blocked instead skips the locked row and takes the next free one. One row per reservable unit, then: START TRANSACTION ; SELECT id FROM inventory_unit WHERE placement_id = 42 AND ( status = 'available' OR ( status = 'held' AND hold_expires_at < NOW ( 3 ))) -- self-healing expiry ORDER BY id LIMIT 2 FOR UPDATE SKIP LOCKED ; -- the whole trick UPDATE inventory_unit SET status = 'held' , reservation_id = 'uuid' , hold_expires_at = NOW ( 3 ) + INTERVAL 10 MINUTE WHERE id IN ( 1107 , 1108 ); INSERT INTO reservation (...) VALUES (...); COMMIT ; Two concurrent requests for the same pool lock different rows. Neither waits. The claim, the hold, and the reservation are one transaction — there is nothing to reconcile because there is nothing else. The numbers (8 weeks before vs 8 weeks after) Metric Redis + Redlock MySQL SKIP LOCKED Oversells / month 40–60 0 Reservation p95 210 ms 34 ms Reservation p99 540 ms 61 ms Throughput / instance ~600 RPS 1,400 RPS Lock-wait timeouts / day ~900 <5 Nightly reconciliation 9–14 min deleted Redis cluster

Welcome to another post in the "Under the Hood" series. The power of Redis lies in its simplicity. One thread, one event loop, zero locks . Single-threaded execution eliminates the "lock contention" that slows down traditional databases. Limitation : A single process can only utilise one CPU core. On a 64-core server, 98% of your hardware sits idle. Redis Core Design To scale, Redis Enterprise doesn't make the engine "bigger"; it makes the fleet smarter. Key Design Decisions One Core to Many (Multi-Tenancy) Instead of one massive process, Enterprise runs multiple Redis Cores (shards) on a single node. From Gossip to Proxy Standard Redis Clusters use a Gossip Protocol. The client must "know" the cluster topology and handle redirections. Solution : The Zero-Latency Proxy acts as the "Front Desk". The client talks to one endpoint; the proxy handles the complexity. It is multi-threaded and uses cut-through routing to ensure the "hop" is sub-millisecond. Separation of Concerns (Control Plane) Distributed Cluster Watchdogs oversees failovers and promotions. By separating the Data Path (Redis shards) from the Control Plane (watchdogs), the database can heal itself without interrupting traffic. Note : In the diagram, it may seem the watchdogs are coupled with the Redis shards, but in reality, they just share the hardware space for resource efficiency. Redis Cluster Architecture

A survival guide for when everything goes wrong in production. Your Redis dashboard looks perfect. Hit ratio: 99.2%. Latency: sub-millisecond. Memory usage: 60% of available. Every metric says healthy. Then at 2:47 PM, your API starts returning 500s. Response times spike to 30 seconds. Users can't log in. The dashboard still shows 99% hit ratio because the cache is working — it's serving cached errors to everyone equally fast. Redis is doing exactly what you told it to do. The problem is what you told it to do. Why Single-Threaded Is Fast (Until It Isn't) Redis processes commands on a single thread. No locks. No context switching. No synchronization overhead. One CPU core, fully utilized, can handle 100K+ operations per second because it never waits for another thread to release a lock. The event loop model (similar to Node.js) multiplexes thousands of client connections on a single thread using non-blocking I/O. Read a request, process it, write the response, move to the next. When your commands are simple — GET, SET, INCR — each one takes microseconds. The trap: slow commands block everything. KEYS * on a million-key database? That's a full keyspace scan on the main thread. While it runs, every other client waits. SORT on a large set? Same. LRANGE on a list with 10 million elements? Same. Redis 6.0 introduced I/O threading ( io-threads config) for reading and writing network data on multiple threads, but command execution is still single-threaded. Redis 7.0 improved this further, but the fundamental model hasn't changed. Long-running commands on the main thread stall everything. Rules: Never use KEYS in production. Use SCAN instead — it's cursor-based and returns results incrementally. Watch out for O(N) commands on large data structures: LRANGE , SMEMBERS , HGETALL on million-element structures. Use SLOWLOG to find commands that are blocking the event loop. Pipelining: The Easiest 10x You'll Ever Get Every Redis command involves a network round trip: send request