标签：#Redis

Python Redis: Caching and Fast Data Structures Redis is an in-memory data store used for caching, session storage, pub/sub messaging, leaderboards, rate limiting, and more. With redis-py 's async client, it integrates cleanly into any asyncio application. Installation pip install redis[hiredis] # hiredis is a C parser — 2-5× faster protocol parsing Connect and Verify import asyncio import redis.asyncio as aioredis from datetime import timedelta import json REDIS_URL = " redis://localhost:6379/0 " async def get_redis () -> aioredis . Redis : client = aioredis . from_url ( REDIS_URL , encoding = " utf-8 " , decode_responses = True , socket_connect_timeout = 5 , socket_timeout = 5 , retry_on_timeout = True , ) pong = await client . ping () print ( f " Redis connected: { pong } " ) return client Strings — Basic Cache with TTL async def cache_set ( r : aioredis . Redis , key : str , value : str , ttl : int = 300 ) -> None : await r . set ( key , value , ex = ttl ) async def cache_get ( r : aioredis . Redis , key : str ) -> str | None : return await r . get ( key ) # Cache-aside pattern async def get_user_profile ( r : aioredis . Redis , user_id : int , db ) -> dict : cache_key = f " user:profile: { user_id } " cached = await r . get ( cache_key ) if cached : print ( f " Cache HIT for user { user_id } " ) return json . loads ( cached ) print ( f " Cache MISS for user { user_id } — querying DB " ) user = await db . fetch_user ( user_id ) # your DB call if user : await r . set ( cache_key , json . dumps ( user ), ex = 600 ) return user or {} # Atomic counter async def increment_page_views ( r : aioredis . Redis , page : str ) -> int : key = f " views: { page } " count = await r . incr ( key ) await r . expire ( key , 86400 ) # reset counter after 24 h return count Hashes — Structured Objects async def save_session ( r : aioredis . Redis , session_id : str , data : dict , ttl : int = 3600 ) -> None : key = f " session: { session_id } " await r . hset ( key , mapping = data )

RQ (Redis Queue) is a delightfully simple way to run background jobs in Python. That simplicity is also why teams under-monitor it: it just works, until a downstream API gets slow or a bad deploy ships, and jobs start failing in bulk — quietly. Here's what to watch and how to get alerted before a customer tells you. RQ failures don't announce themselves When a job raises, RQ moves it to the FailedJobRegistry and moves on. The worker keeps running; nothing crashes. If you're not looking at that registry, the failure is invisible — the same trap BullMQ, Celery, and every robust queue share. So the job is to reach into the queue's state and turn it into a signal. The four signals that matter for RQ Failure count / rate — jobs landing in the FailedJobRegistry over a window. Backlog — how many jobs are queued vs. being worked; is the worker keeping up? Latency — how long jobs take, and how long they wait before a worker picks them up. Worker liveness — are your workers actually alive and heartbeating? Where to read them RQ exposes queue and registry state directly: from redis import Redis from rq import Queue from rq.registry import FailedJobRegistry , StartedJobRegistry redis = Redis () q = Queue ( " default " , connection = redis ) queued = len ( q ) # backlog failed = FailedJobRegistry ( queue = q ) # failures started = StartedJobRegistry ( queue = q ) # in-flight print ( " queued: " , queued ) print ( " failed: " , len ( failed )) print ( " started: " , len ( started )) Poll this on an interval and store the series — a single snapshot hides the trend , which is the part that matters. For failures specifically, walk the registry to get the actual exceptions: for job_id in failed . get_job_ids (): job = q . fetch_job ( job_id ) print ( job . id , job . exc_info . splitlines ()[ - 1 ] if job . exc_info else "" ) Two gotchas: Group by exception, not by job. A thousand jobs failing with the same traceback is one incident. Normalize the message (strip IDs, timestamps, host

In-process cron ( node-cron , @nestjs/schedule , OS crontab) runs inside one Node process. That is fine for a single instance, but it does not survive restarts gracefully, deduplicate across replicas, or share infrastructure with your other background jobs. BullMQ stores queues and schedulers in Redis . Job Schedulers (BullMQ 5.16+) are the recommended way to enqueue recurring work on a cron pattern or fixed interval. The same workers that process one-off jobs also process scheduled ones, with retries, backoff, and concurrency you already get from BullMQ. This post covers Job Schedulers in plain Node.js, operations and pitfalls, a NestJS setup with @nestjs/bullmq , and a runnable demo with a fast cron heartbeat and a daily cleanup cron. Prerequisites Node.js version 26 Redis at redis://localhost:6379 (included in the demo docker-compose.yml , or use Postgres and Redis containers with Docker Compose ) npm i bullmq For the NestJS section: npm i @nestjs/bullmq bullmq BullMQ 2.0+ does not require a separate QueueScheduler instance. Use the Job Scheduler API ( upsertJobScheduler ), not the deprecated repeat option on queue.add() . Mental model Piece Role Queue Holds jobs waiting to run Worker Executes jobs Job Scheduler Factory that enqueues jobs on a schedule Scheduled job A job instance produced by a scheduler A scheduler id is stable across deploys. Calling upsertJobScheduler with the same id updates the schedule in place instead of creating duplicates. Queue and worker Share one Redis connection config between the queue and the worker: import { Queue , Worker } from ' bullmq ' ; const connection = { host : ' localhost ' , port : 6379 }; const queue = new Queue ( ' reports ' , { connection }); const worker = new Worker ( ' reports ' , async ( job ) => { console . log ( `[ ${ job . name } ]` , new Date (). toISOString (), job . data ); }, { connection }, ); worker . on ( ' failed ' , ( job , error ) => { console . error ( job ?. name , error . message ); }); Start the

If you ask any senior backend engineer or database administrator how to instantly make a slow, disk-bound application faster, their first answer is almost always: "Put it in memory." But why is memory so preferred, and how do modern architectures utilize RAM to achieve sub-millisecond latencies? We're seeing a massive revival of RAM disks and in-memory architectures. Let's explore why computer experts are increasingly treating RAM like a hard drive. 1. The Physics of Storage: Why RAM Wins To understand the shift towards in-memory architectures, we have to look at the numbers. Hard Disk Drives (HDDs): Mechanical spinning disks. Seek times are around 2-5 milliseconds . Solid State Drives (SSDs): Flash memory. Seek times are around 0.1 milliseconds (100 microseconds) . RAM (Random Access Memory): Volatile silicon. Access times are around 100 nanoseconds . RAM is roughly 1,000 times faster than an SSD and 10,000 to 50,000 times faster than an HDD. When you have a high-throughput system serving millions of requests per second, waiting for a disk to seek is an eternity. 2. In-Memory Databases: Redis and Memcached The most common implementation of this principle in modern backends is the In-Memory Database . How They Work Instead of writing every transaction to an SSD, systems like Redis and Memcached store the entire dataset directly in RAM. This bypasses the OS file system cache, disk I/O bottlenecks, and complex B-tree traversals required by traditional relational databases like PostgreSQL or MySQL. The Trade-off: Durability RAM is volatile. If the server loses power, all data is gone. So how do in-memory databases survive crashes? Snapshots (RDB in Redis): Periodically dumping the entire memory state to disk. Append-Only Files (AOF in Redis): Logging every write operation to a disk sequentially. Sequential writes to disk are significantly faster than random writes. This hybrid approach gives you the read/write speed of RAM with a "good enough" durability guarantee for

Event-driven architecture promises scalability, but in Java-based real-time systems the tradeoffs only surface in production. Drawing on a Java/Kafka contact center platform handling 80k BHCC across 10k agents, this article details where the design breaks down—state management, partition limits, deduplication, JVM tuning, cascading consumer failures—and the Redis-backed patterns that fixed each. By Sagar Deepak Joshi

Originally published on bckinfo.com Redis with Docker Compose: Persistence, Security, and Production-Ready Configuration Table of Contents Why Redis Containers Lose Data RDB vs AOF: Choosing a Persistence Strategy Basic Setup with Docker Compose Full Persistence Configuration Securing Redis in Docker Setting Resource Limits Redis in a Multi-Service Stack Backing Up and Restoring a Redis Volume Common Issues and Quick Fixes Closing Notes Redis is one of the most common services to run in Docker — it's fast to spin up, lightweight, and perfect for caching, session storage, and queues. But that same simplicity hides a trap: by default, Redis in Docker stores everything in memory, and the moment a container is removed, all of that data disappears . This guide walks through setting up Redis with Docker Compose the right way — covering persistence, authentication, resource limits, and the health checks you need before putting it anywhere near production. Why Redis Containers Lose Data A Docker container is meant to be disposable. That's a feature for stateless services, but it's a liability for a database like Redis. If you start a plain Redis container without a mounted volume, here's what happens: The container writes its dataset only inside its own writable layer. docker compose down or docker rm removes that layer entirely. The next time the container starts, Redis initializes with an empty dataset. This single oversight accounts for a large share of "we lost our session data" incidents in small teams running Redis in containers for the first time. The fix is straightforward once you understand the two persistence mechanisms Redis offers. RDB vs AOF: Choosing a Persistence Strategy Redis supports two persistence models, and production setups typically combine both: RDB (Redis Database snapshots) Point-in-time snapshots of the dataset, saved at intervals you define. Fast to restore, but you can lose any writes that happened after the last snapshot. AOF (Append Only Fil

I Built Commercial Redis CDC Source Drivers for Ruby — Here's What I Learned For the past couple of years I've been building a Change Data Capture (CDC) ecosystem for Ruby. Like many CDC projects, it started with PostgreSQL. PostgreSQL's Write-Ahead Log (WAL) is an excellent source of truth: durable, ordered, replayable, and well understood. It provides exactly the properties you want when you're building reliable event pipelines. But the deeper I went into distributed systems, the more I realized something important. Many systems don't observe change from PostgreSQL first. They observe it from Redis. Redis often sits at the front of modern architectures: Redis Streams carry application events. Pub/Sub distributes transient state changes. Keyspace notifications react to cache invalidation and key expiry. Redis Cluster routes events across multiple primaries. In many systems, Redis sees a change before PostgreSQL ever commits it. That raised an interesting question: Can Redis become a first-class Change Data Capture source? The obvious answer is "yes." The interesting answer is "yes—but not in the same way PostgreSQL does." That distinction eventually became cdc-redis-pro , a commercial Redis source driver for the Ruby CDC ecosystem. This article isn't a product announcement. It's an engineering write-up about the architectural decisions behind the project, the tradeoffs Redis forces you to make, and the execution model that ultimately emerged. Redis Doesn't Have One CDC Interface One misconception I frequently encounter is the assumption that Redis has an equivalent of PostgreSQL's WAL. It doesn't. Instead, Redis exposes several completely different mechanisms for observing change. Source Delivery Replay Streams At-least-once Yes Pub/Sub At-most-once No Sharded Pub/Sub At-most-once No Keyspace Notifications At-most-once No At first glance they all look like "events." Operationally they're completely different systems. Streams are durable. Pub/Sub isn't. Keyspace not

AWS has recently introduced durability for Amazon ElastiCache for Valkey, enabling reliable data retention across failures and expanding support beyond caching to persistent workloads. The feature offers new options that prioritize either minimizing data loss or maintaining lower write latency, expanding the range of use cases supported by the Redis fork. By Renato Losio

Observability isn't just a buzzword; it's a necessity, especially when diving into distributed systems. If you're using NestJS, you might want to take a look at RedisX. It's a modular toolkit that can boost the observability of your applications. A standout feature? The Metrics Plugin. It meshes well with Prometheus, delivering insights into Redis operations in your NestJS setup. Getting RedisX Metrics Rolling in NestJS So, first things first. To harness the power of RedisX Metrics, you need to set up your NestJS app with RedisX. This means installing some packages and configuring the RedisModule with the MetricsPlugin. Hit your terminal and run: npm install @nestjs-redisx/core @nestjs-redisx/metrics Now, let's tweak your AppModule . You want it to use RedisModule with MetricsPlugin: import { Module } from ' @nestjs/common ' ; import { ConfigModule , ConfigService } from ' @nestjs/config ' ; import { RedisModule } from ' @nestjs-redisx/core ' ; import { MetricsPlugin } from ' @nestjs-redisx/metrics ' ; @ Module ({ imports : [ ConfigModule . forRoot ({ isGlobal : true }), RedisModule . forRootAsync ({ imports : [ ConfigModule ], inject : [ ConfigService ], plugins : [ new MetricsPlugin ({ prefix : ' redisx_ ' , endpoint : ' /metrics ' , defaultLabels : { service : ' my-service ' } }) ], useFactory : ( config : ConfigService ) => ({ clients : { host : config . get ( ' REDIS_HOST ' , ' localhost ' ), port : config . get ( ' REDIS_PORT ' , 6379 ), }, }), }), ], }) export class AppModule {} Prometheus Metrics: What You Get With MetricsPlugin set up, your app now exposes a /metrics endpoint. Prometheus can scrape this endpoint, dishing out detailed metrics about your Redis operations. Here's a snapshot of what you get: redisx_cache_hits_total : Tracks total cache hits. redisx_lock_acquired_total : Total locks acquired. redisx_redis_commands_total : Total Redis commands run. Making Prometheus Work for You To get those insights, set up Prometheus to scrape your /metrics end

I was now done with the basic setup. However, during my time working at my startup, I have learnt to think about a project wearing multiple caps. One such aspect was - With Dhrishti running on a server that was already loaded, I did NOT want the tracking application itself to be heavy. I had to set some benchmarks to ensure that Dhrishti did not consume a tonne of space while tracking the metrics. I also had a problem with unresolved requests - in my mock_services, I had a client that was continuously hitting the API Gateway service. I had to fine-tune all the requests so that I could run tests under different loads, but the advantage was that my project was easily able to discern where the client request was coming from. However, in a production scenario, you can never know where a request is coming from - obviously, we cannot resolve different customer IPs to their respective customer names. This was the first problem. I had to specify what a customer was, and what an unknown request was. I came up with the following solution - Any unresolved IPs are going to be added to a table in the UI called unresolved IP table. This would help me with debugging later. Now, any unresolved IPs which also made requests to an ENTRY-POINT into my application could be added as the customers. For this, I very simply had to filter out the unknown IPs, and keep a configurable entry-point in dhrishti.json in which I would add a bunch of entry-points (in the case of my mock micro-service architecture, only 1) Now, I could differentiate between 2 types of unknown IPs - one which was potentially a customer, one which was a background network call, not important to the working system. The next problem was with the client service itself. It was difficult to simulate, say - a million users in my system. I had essentially built a service which was only being used by 1 customer, but how would Dhrishti behave if I added multiple client IPs? Using K6 k6 is a Grafana based application that helps

🎬 This article is a companion to my YouTube video. Watch it here: Introduction In this video we are going to talk about Redis — what it is, what it does, and why it is an important part of my back-end stack. What is Redis? Redis is a free, open-source, in-memory data store. Unlike PostgreSQL which stores data on disk, Redis stores data entirely in memory — in RAM. This makes it extremely fast. Redis can handle millions of operations per second with sub-millisecond response times. Redis is most commonly used as a cache, a session store, a message broker, and a real-time data store. What is Caching? When your application queries a database, that query takes time — it reads from disk, processes the query, and returns the result. If the same query is made thousands of times per second, you are hitting the database thousands of times unnecessarily. Caching solves this by storing the result of a query in memory. The first request hits the database and the result is stored in Redis. Every subsequent request gets the result from Redis — which is in memory and therefore much faster — instead of hitting the database again. Think of it like a shortcut. Instead of driving the long route to the database every time, you take the shortcut through Redis. What Does Redis Do? Caching Store frequently accessed data in memory for fast retrieval. Database query results, API responses, computed values — anything that is expensive to compute and accessed frequently is a good candidate for caching. Session Storage Store user session data in Redis instead of the database. Since sessions are read on every request, having them in memory is significantly faster than a database lookup. Rate Limiting Track how many requests a user or IP address has made in a given time window. Redis's atomic increment operations make it perfect for implementing rate limiting. Message Queues and Pub/Sub Redis supports publish/subscribe messaging and message queues. Applications can publish messages to a channel a

For two years, our Sponsored Placements service booked limited ad inventory through Redis: a counter in Redis, a Redlock around the decrement, and a TTL key per hold. It oversold. Not catastrophically — consistently. 40–60 double-booked placements a month , each one a manual refund and an apology email to an advertiser. The root cause was never one bug. It was the architecture: two sources of truth that could not be made atomic with each other. The count lived in Redis; the ownership lived in SQL. No transaction spans both. The Redlock only ever protected the Redis half. The one mental shift SKIP LOCKED turns a contended table into a concurrent work queue. Instead of every request fighting over one counter, each request grabs different rows and ignores the ones someone else is holding. FOR UPDATE alone serializes — that's the experience that scares people off SQL locking. FOR UPDATE SKIP LOCKED is the opposite: a transaction that would have blocked instead skips the locked row and takes the next free one. One row per reservable unit, then: START TRANSACTION ; SELECT id FROM inventory_unit WHERE placement_id = 42 AND ( status = 'available' OR ( status = 'held' AND hold_expires_at < NOW ( 3 ))) -- self-healing expiry ORDER BY id LIMIT 2 FOR UPDATE SKIP LOCKED ; -- the whole trick UPDATE inventory_unit SET status = 'held' , reservation_id = 'uuid' , hold_expires_at = NOW ( 3 ) + INTERVAL 10 MINUTE WHERE id IN ( 1107 , 1108 ); INSERT INTO reservation (...) VALUES (...); COMMIT ; Two concurrent requests for the same pool lock different rows. Neither waits. The claim, the hold, and the reservation are one transaction — there is nothing to reconcile because there is nothing else. The numbers (8 weeks before vs 8 weeks after) Metric Redis + Redlock MySQL SKIP LOCKED Oversells / month 40–60 0 Reservation p95 210 ms 34 ms Reservation p99 540 ms 61 ms Throughput / instance ~600 RPS 1,400 RPS Lock-wait timeouts / day ~900 <5 Nightly reconciliation 9–14 min deleted Redis cluster

Welcome to another post in the "Under the Hood" series. The power of Redis lies in its simplicity. One thread, one event loop, zero locks . Single-threaded execution eliminates the "lock contention" that slows down traditional databases. Limitation : A single process can only utilise one CPU core. On a 64-core server, 98% of your hardware sits idle. Redis Core Design To scale, Redis Enterprise doesn't make the engine "bigger"; it makes the fleet smarter. Key Design Decisions One Core to Many (Multi-Tenancy) Instead of one massive process, Enterprise runs multiple Redis Cores (shards) on a single node. From Gossip to Proxy Standard Redis Clusters use a Gossip Protocol. The client must "know" the cluster topology and handle redirections. Solution : The Zero-Latency Proxy acts as the "Front Desk". The client talks to one endpoint; the proxy handles the complexity. It is multi-threaded and uses cut-through routing to ensure the "hop" is sub-millisecond. Separation of Concerns (Control Plane) Distributed Cluster Watchdogs oversees failovers and promotions. By separating the Data Path (Redis shards) from the Control Plane (watchdogs), the database can heal itself without interrupting traffic. Note : In the diagram, it may seem the watchdogs are coupled with the Redis shards, but in reality, they just share the hardware space for resource efficiency. Redis Cluster Architecture

A survival guide for when everything goes wrong in production. Your Redis dashboard looks perfect. Hit ratio: 99.2%. Latency: sub-millisecond. Memory usage: 60% of available. Every metric says healthy. Then at 2:47 PM, your API starts returning 500s. Response times spike to 30 seconds. Users can't log in. The dashboard still shows 99% hit ratio because the cache is working — it's serving cached errors to everyone equally fast. Redis is doing exactly what you told it to do. The problem is what you told it to do. Why Single-Threaded Is Fast (Until It Isn't) Redis processes commands on a single thread. No locks. No context switching. No synchronization overhead. One CPU core, fully utilized, can handle 100K+ operations per second because it never waits for another thread to release a lock. The event loop model (similar to Node.js) multiplexes thousands of client connections on a single thread using non-blocking I/O. Read a request, process it, write the response, move to the next. When your commands are simple — GET, SET, INCR — each one takes microseconds. The trap: slow commands block everything. KEYS * on a million-key database? That's a full keyspace scan on the main thread. While it runs, every other client waits. SORT on a large set? Same. LRANGE on a list with 10 million elements? Same. Redis 6.0 introduced I/O threading ( io-threads config) for reading and writing network data on multiple threads, but command execution is still single-threaded. Redis 7.0 improved this further, but the fundamental model hasn't changed. Long-running commands on the main thread stall everything. Rules: Never use KEYS in production. Use SCAN instead — it's cursor-based and returns results incrementally. Watch out for O(N) commands on large data structures: LRANGE , SMEMBERS , HGETALL on million-element structures. Use SLOWLOG to find commands that are blocking the event loop. Pipelining: The Easiest 10x You'll Ever Get Every Redis command involves a network round trip: send request