标签：#Redis

There are already good Redis tools for NestJS. Most of them mainly help you connect NestJS to Redis and use the Redis client. That's useful, but as a project grows, you often need to build more things around Redis yourself: caching, TTL, cache invalidation, locks, rate limiting, sessions, monitoring, and more. That's why I built Redora. Redora adds a higher-level layer for using Redis in NestJS. Today it includes: Redis service Cache service Cache decorators "remember()" caching TTL and expiration policies Cache tags and eviction Distributed locks Redis diagnostics Logger and observability _The idea is simple: Redis gives you the primitives. Redora gives you the architecture._ What I'm working on next I want Redora to cover more of the common things developers use Redis for: Session management Rate limiting for OTP, login, and APIs Distributed locks Message queues Redis for AI applications Redis and Valkey support More monitoring and telemetry Redora is still early and I'm building it in public. 📦 "npm i redora" 🌐 https://redora-sdk.com If you use NestJS and Redis, try it and let me know what you think. I'd really like to hear what works, what's missing, and what you would change.

Canva redesigned session revocation infrastructure to support 100M active sessions while reducing database lookups. The architecture uses Amazon S3 for durable revocation records and distributes compact, in-memory indexes to application gateways. Canva said the design improved deployment speed, reduced database infrastructure requirements, and cut the revocation cache memory footprint by 87.5%. By Leela Kumili

When I was building my backend API, I realized a big problem: anyone could spam my endpoints. If a user repeatedly reloads a page or hits an endpoint calling an external AI service, it can crash the server or run up high API costs. To fix this, I added Rate Limiting . Here is why I used Redis for it and how I set it up. The Problem with Simple In-Memory Limiters At first, I thought about saving request counts in a simple JavaScript object: // ❌ Simple in-memory check (Not good for production) const requestCounts = {}; app . use (( req , res , next ) => { const ip = req . ip ; requestCounts [ ip ] = ( requestCounts [ ip ] || 0 ) + 1 ; if ( requestCounts [ ip ] > 100 ) { return res . status ( 429 ). json ({ error : " Too many requests " }); } next (); }); This works locally, but has two big flaws: 1)Memory Leaks: The requestCounts object keeps growing in memory forever. 2)Breaks when Scaling: If you deploy multiple instances of your app behind a load balancer, each server keeps its own count. A user can easily bypass the limit by hitting different servers. The Solution: Centralized Redis Store Redis stores data in RAM outside our Node.js app. Because it is centralized, all server instances share the exact same count. [ Incoming Client Requests ] │ ▼ [ Cloud Load Balancer ] │ ┌───────────────┼───────────────┐ ▼ ▼ ▼ [ Express Node 1 ] [ Express Node 2 ] [ Express Node 3 ] │ │ │ └───────────────┼───────────────┘ ▼ [ Central Redis Store ] (Checks Request Limits) How I Configured It in My Project In my app, I use two levels of protection: Global Limit: 100 requests per 15 minutes for normal routes. Strict Limit: 5 requests per 10 minutes for heavy routes (like AI generation or OTP emails). 1 . Redis Connection ( config / redis . js ) import { createClient } from ' redis ' ; const redisClient = createClient ({ url : process . env . REDIS_URL || ' redis://localhost:6379 ' }); redisClient . on ( ' error ' , ( err ) => console . error ( ' Redis Error: ' , err )); redisClient .

This is Part 2 of the Directus + Coolify series. If you're new here, start with "Secure Your VPS Before Hackers Do" and the first Directus + Coolify post — the bundled, single-Compose-file setup — before following along with this one. Introduction In the first method, we coupled all of the services into one stack using a single Docker Compose file. The network between all services was created automatically, and we didn't have to start them up individually — which removes the risk of a race condition if service startup isn't handled properly. If you're running a single app, that's genuinely the recommended way to set up Directus on a Coolify-managed VPS. Going in, I assumed there were several good reasons to split the services apart instead — more control over backups, monitoring, restarts, that kind of thing. So before recommending decoupling, I actually tested each of those assumptions on a live Coolify instance. Most of them turned out to be wrong. Myth 1: Restarting Directus Restarts the Whole Stack I expected that restarting Directus inside the bundled Compose file would restart Redis and Postgres along with it. It doesn't. Coolify lets you restart each service in the stack independently — Directus, Database, and Cache each get their own Restart button, right there in the same view. No decoupling needed for this one. Myth 2: You Need a Separate Database Resource for S3 Backups Same story. Even with Postgres bundled inside the Directus Compose file, Coolify still gives it its own dedicated Backups option, S3 included. This isn't a separate-resource-only feature. Myth 3: Scheduled Tasks Require Separate Services Also not true. Coolify exposes a Scheduled Tasks tab per service, even inside a single bundled stack — complete with a Container name dropdown letting you target the cron job at just the database, or just Directus, without splitting anything apart. What Actually Holds Up Two things survived testing. First: metrics. This one's confirmed directly in Coolify'

User segmentation requirement Imagine you need to send a push notification to users who satisfy all of the following conditions: Push notification is enabled User is a VIP Active within the last 30 days Following the Voucher Hot category The traditional approach is to query multiple tables: SELECT DISTINCT u . id FROM users u JOIN user_configs c ON c . user_id = u . id JOIN devices d ON d . user_id = u . id JOIN follows f ON f . user_id = u . id WHERE c . push_optin = 1 AND c . mute = 0 AND d . fcm_token IS NOT NULL AND f . category = 'voucher_hot' AND u . last_active >= NOW () - INTERVAL 30 DAY ; As your user base grows into the millions, every campaign requires joining multiple large tables, filtering millions of records, and repeatedly computing the same audience. Query latency increases significantly, making real-time segmentation increasingly difficult. A Different Approach Instead of querying the database every time, we precompute each boolean attribute as a bitmap. Think of a bitmap as a huge array containing only 0 and 1, where the index corresponds to the user ID. For example, a bitmap representing whether a user has enabled push notifications: User ID : 0 1 2 3 4 5 6 7 Bitmap : 1 0 1 1 0 0 1 1 To check whether user 123 has enabled notifications, simply read bit 123. 1 → enabled 0 → disabled Each bitmap represents exactly one boolean property: bitmap:push_optin bitmap:vip bitmap:active30 bitmap:follow:voucher_hot Memory Usage Bitmap is extremely memory efficient. Each user requires only one bit. For 1 million users: 1.000.000 bits ~ 125.000 bytes ~ 122 KB That means every segment only consumes about 122 KB of Redis memory. Even 100 different segments require only around 12 MB . Finding Intersections Suppose you want all users that are: VIP AND Push Opt-in AND Active30 AND Following Voucher Hot Redis can calculate the result with a single command: BITOP AND result vip push_optin active30 follow_voucher_hot Need the number of matched users? BITCOUNT result No

Engineering posts often end with: The new design is correct, scalable, and fast. Fast compared with what? When we changed Podium so tied players rank by arrival time instead of player ID, we added: a Lua script; a per-leaderboard sequence; a public-ID mapping; a second sorted set for ascending order. That design is fairer. It is also impossible for it to be free. So we built two benchmark layers: direct Redis strategy benchmarks to isolate the data-model cost, and end-to-end HTTP benchmarks to show what users actually experience. We are publishing the results, including the regression, because performance claims are useful only when readers can inspect the workload and reproduce the measurement. TeneficGames / podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium provides ready-to-run HTTP and gRPC APIs for scores, ranks, seasons, and player-relative views. It is designed for backend teams operating large fleets of independent leaderboards without provisioning each leaderboard in advance. Fair, deterministic ordering when scores are equal. Single and bulk score updates, including multi-leaderboard fan-out. Standalone Redis and real Redis Cluster integration coverage. Deploy one multi-architecture OCI image with Docker, containerd, Kubernetes or another OCI-compatible runtime. Quickstart · Performance · API · Documentation · Helm chart · Docker Hub · GHCR Quickstart Start Redis 8.2 and the latest stable Podium image: docker network create podium docker run --detach --name podium-redis --network podium redis:8.2-alpine docker run --detach --rm --name podium \ --network podium \ --publish 8880:8880 \ --publish 8881:8881 \ --env PODIUM_REDIS_HOST=podium-redis \ --env PODIUM_REDIS_PORT=6379 \ trungdlp/podium:latest start Verify the service: curl http://localhost:8880/healthcheck WORKING Submit two equal scores: curl --request … View on Gi

"We use Redis Cluster" can mean two very different things: Our dataset is distributed across Redis nodes. Every individual data structure is distributed across Redis nodes. The first can be true while the second is false. That distinction matters for leaderboards. In Podium , each leaderboard uses several Redis keys and atomic Lua scripts. Redis Cluster helps us scale a large fleet of independent leaderboards, but it cannot split one giant sorted set across primaries. We are sharing this architecture because "Redis Cluster scales horizontally" is true only after you define what the system actually shards. TeneficGames / podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium provides ready-to-run HTTP and gRPC APIs for scores, ranks, seasons, and player-relative views. It is designed for backend teams operating large fleets of independent leaderboards without provisioning each leaderboard in advance. Fair, deterministic ordering when scores are equal. Single and bulk score updates, including multi-leaderboard fan-out. Standalone Redis and real Redis Cluster integration coverage. Deploy one multi-architecture OCI image with Docker, containerd, Kubernetes or another OCI-compatible runtime. Quickstart · Performance · API · Documentation · Helm chart · Docker Hub · GHCR Quickstart Start Redis 8.2 and the latest stable Podium image: docker network create podium docker run --detach --name podium-redis --network podium redis:8.2-alpine docker run --detach --rm --name podium \ --network podium \ --publish 8880:8880 \ --publish 8881:8881 \ --env PODIUM_REDIS_HOST=podium-redis \ --env PODIUM_REDIS_PORT=6379 \ trungdlp/podium:latest start Verify the service: curl http://localhost:8880/healthcheck WORKING Submit two equal scores: curl --request … View on GitHub Here is how the design works, why hash tags are necessary, and where the scaling bounda

A leaderboard looks like a one-command problem: ZADD weekly 100 alice ZADD weekly 100 bob ZREVRANGE weekly 0 -1 WITHSCORES While building Podium , an open-source Redis-backed leaderboard service, we discovered that the difficult part begins when two players have the same score. We are sharing the design because this edge case can silently turn player IDs into ranking rules. TeneficGames / podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium provides ready-to-run HTTP and gRPC APIs for scores, ranks, seasons, and player-relative views. It is designed for backend teams operating large fleets of independent leaderboards without provisioning each leaderboard in advance. Fair, deterministic ordering when scores are equal. Single and bulk score updates, including multi-leaderboard fan-out. Standalone Redis and real Redis Cluster integration coverage. Deploy one multi-architecture OCI image with Docker, containerd, Kubernetes or another OCI-compatible runtime. Quickstart · Performance · API · Documentation · Helm chart · Docker Hub · GHCR Quickstart Start Redis 8.2 and the latest stable Podium image: docker network create podium docker run --detach --name podium-redis --network podium redis:8.2-alpine docker run --detach --rm --name podium \ --network podium \ --publish 8880:8880 \ --publish 8881:8881 \ --env PODIUM_REDIS_HOST=podium-redis \ --env PODIUM_REDIS_PORT=6379 \ trungdlp/podium:latest start Verify the service: curl http://localhost:8880/healthcheck WORKING Submit two equal scores: curl --request … View on GitHub Both players have 100 points. Alice arrived first, so most game designers would expect: 1. alice: 100 2. bob: 100 But that is not what the data model says. Redis sorted sets order members with equal scores lexicographically. With a reverse range, that secondary ordering is reversed too. Your "fair" tie may therefore be de

Hello everyone! After publishing my Machine Learning with ML.NET series, I decided to turn the recommendation model into a complete application. In this new series, we build RockPlayer, a rock music player that combines modern software architecture, ASP.NET Core, Angular 22, Redis, and YouTube integration. Each article focuses on a different part of the project: 🎵 1. Introducing RockPlayer An overview of the project, its goals, and the overall architecture. https://devfullstack.net/blog/introducing-rockplayer 🔌 2. Adapters: Isolating the YouTube Provider Using the Adapter pattern to decouple the application from the YouTube integration. https://devfullstack.net/blog/adapters-isolating-the-youtube-provider ⚡ 3. No Database: Caching Lookups with Redis Using Redis to cache search results instead of storing external data in a database. https://devfullstack.net/blog/no-database-caching-lookups-with-redis 🅰️ 4. Angular 22 in Practice Applying modern Angular 22 features to build the user interface. https://devfullstack.net/blog/angular-22-in-practice 🚀 5. Building the RockPlayer API Building the API that orchestrates the application. https://devfullstack.net/blog/building-the-rockplayer-api ▶️ 6. The YouTube Adapter: Finding and Playing the Song Implementing the YouTube integration to search for and play songs. https://devfullstack.net/blog/the-youtube-adapter-finding-and-playing-the-song 🎧 7. RockPlayer in Angular 22: Onboarding Setting up the Angular application and organizing the project structure. https://devfullstack.net/blog/rockplayer-in-angular-22-onboarding 🎸 8. RockPlayer: Putting It All Together Bringing all the components together into a complete application. https://devfullstack.net/blog/rockplayer-putting-it-all-together I hope this series is useful for developers interested in software architecture, .NET, and Angular. See you there!

Redis gets reached for reflexively, "just add Redis," as if it were a single fix for slowness. It's genuinely one of the most useful tools in a backend engineer's kit, but using it well starts with understanding what it actually is: an in-memory data structure store, not just a cache. Once you see it as a fast, versatile store of real data structures, the range of problems it solves cleanly (caching, rate limiting, queues, sessions, leaderboards, locks) stops looking like a grab bag and starts looking like one idea applied many ways. This is the opening article of the Redis Masterclass, and it builds on the PostgreSQL series : Redis usually sits alongside a primary database like Postgres, not instead of it. In-memory is the whole point Redis keeps its data in RAM. That single fact explains most of its character. Reading from memory is orders of magnitude faster than reading from disk, so Redis operations typically complete in well under a millisecond, and a single instance handles a very high request rate. That speed is why it's the default choice for anything on the hot path, where a database round trip would be too slow. The tradeoff is that RAM is smaller and more expensive than disk, and volatile. Redis addresses durability with persistence options we'll cover later, but the mental model to start with is: Redis is fast because it's in memory, and you use it for data that benefits from being fast to access, not as the permanent home for everything. It's a data structure store, not a key-value blob The common misconception is that Redis is a simple key-value store, strings in and strings out. It's much more. Redis stores real data structures as values, each with its own commands: Strings for simple values, counters, and cached blobs. Hashes for objects with fields, like a user record. Lists for ordered sequences and simple queues. Sets for unique collections and membership checks. Sorted sets for ranked data like leaderboards and priority queues. Plus streams, bit

Expedia Group has introduced STAR, an internal AI-assisted observability platform that helps engineers investigate production incidents using service telemetry and LLMs. Built with FastAPI, Datadog, Celery, Redis, and Langfuse, STAR follows structured workflows to analyze telemetry, generate root cause assessments, and support incident response while keeping engineers in the loop. By Leela Kumili

Originally published on tamiz.pro . Real-time systems are at the heart of modern interactive applications, from chat platforms to collaborative editing tools and financial dashboards. Delivering a seamless, low-latency experience while ensuring high availability and fault tolerance presents significant architectural challenges. This deep-dive explores how WebSockets, for persistent client-server communication, and Redis, for state management and pub/sub, can be combined with strategic design patterns to build resilient real-time systems. The Core Challenge of Real-Time Resilience The primary challenge in real-time systems is maintaining continuous connectivity and data flow despite inevitable network issues, server failures, or application restarts. A single point of failure can disrupt numerous active user sessions, leading to a poor user experience. Resilience, in this context, means the system's ability to recover gracefully from failures and continue operating, even if in a degraded state. WebSockets: The Foundation for Real-Time Communication WebSockets provide a full-duplex communication channel over a single TCP connection, enabling persistent, low-latency message exchange between client and server. Unlike traditional HTTP requests, WebSockets keep the connection open, eliminating the overhead of connection setup for each message. However, managing a large number of concurrent WebSocket connections and ensuring their availability across a distributed system requires careful design. WebSocket Server Scalability and Load Balancing Directly load balancing WebSocket connections using standard HTTP load balancers can be tricky because WebSockets are long-lived. Sticky sessions are often employed to ensure a client consistently connects to the same backend server. While this works, it can lead to uneven server load and complicates server replacement during failures. A more robust approach involves a dedicated WebSocket gateway layer that can manage connections and

Over the past few months I built a side project that taught me more about production system design than any course. A Slack bot for deploy queue management. This isn't about the business side of it, it's a technical breakdown of the architecture decisions, the real problems I hit building a Slack app with NestJS, and what I learned solving each one. The problem Every engineering team has lived this. Two people deploy at the same time, one overwrites the other, and it turns into "who's touching prod right now?" shouted into a Slack channel. Sounds simple. Solving it properly across multiple teams, multiple environments, with timeouts, without ever locking anyone out, is not. Stack and why NestJS + TypeScript on the backend, PostgreSQL via Prisma, Redis + BullMQ for background jobs, @slack/bolt for the Slack integration. Choosing NestJS wasn't just preference. Its modular architecture (modules, providers, guards, interceptors) mapped really well to the domain. Every entity (Workspace, Project, Environment, Queue) became its own module, with the repository pattern keeping Prisma out of the business logic layer. The hardest problem: dynamic modals in Slack Block Kit Slack's Block Kit doesn't natively support a select input that reloads its options based on another select's value, within the same form. If you want "pick a project, then load that project's environments," there's no built-in prop for that. The solution combines two Bolt event types. A block_actions listener on the project select fetches the environments, then re-renders the whole modal via views.update : app . action ( ' project_select ' , async ({ ack , body , client }) => { await ack (); const selectedProjectId = body . actions [ 0 ]. selected_option . value ; const environments = await environmentService . findByProject ( selectedProjectId ); await client . views . update ({ view_id : body . view . id , view : buildModalWithEnvironments ( environments ), }); }); The detail that tripped me up the most: t

Python Redis: Caching and Fast Data Structures Redis is an in-memory data store used for caching, session storage, pub/sub messaging, leaderboards, rate limiting, and more. With redis-py 's async client, it integrates cleanly into any asyncio application. Installation pip install redis[hiredis] # hiredis is a C parser — 2-5× faster protocol parsing Connect and Verify import asyncio import redis.asyncio as aioredis from datetime import timedelta import json REDIS_URL = " redis://localhost:6379/0 " async def get_redis () -> aioredis . Redis : client = aioredis . from_url ( REDIS_URL , encoding = " utf-8 " , decode_responses = True , socket_connect_timeout = 5 , socket_timeout = 5 , retry_on_timeout = True , ) pong = await client . ping () print ( f " Redis connected: { pong } " ) return client Strings — Basic Cache with TTL async def cache_set ( r : aioredis . Redis , key : str , value : str , ttl : int = 300 ) -> None : await r . set ( key , value , ex = ttl ) async def cache_get ( r : aioredis . Redis , key : str ) -> str | None : return await r . get ( key ) # Cache-aside pattern async def get_user_profile ( r : aioredis . Redis , user_id : int , db ) -> dict : cache_key = f " user:profile: { user_id } " cached = await r . get ( cache_key ) if cached : print ( f " Cache HIT for user { user_id } " ) return json . loads ( cached ) print ( f " Cache MISS for user { user_id } — querying DB " ) user = await db . fetch_user ( user_id ) # your DB call if user : await r . set ( cache_key , json . dumps ( user ), ex = 600 ) return user or {} # Atomic counter async def increment_page_views ( r : aioredis . Redis , page : str ) -> int : key = f " views: { page } " count = await r . incr ( key ) await r . expire ( key , 86400 ) # reset counter after 24 h return count Hashes — Structured Objects async def save_session ( r : aioredis . Redis , session_id : str , data : dict , ttl : int = 3600 ) -> None : key = f " session: { session_id } " await r . hset ( key , mapping = data )

RQ (Redis Queue) is a delightfully simple way to run background jobs in Python. That simplicity is also why teams under-monitor it: it just works, until a downstream API gets slow or a bad deploy ships, and jobs start failing in bulk — quietly. Here's what to watch and how to get alerted before a customer tells you. RQ failures don't announce themselves When a job raises, RQ moves it to the FailedJobRegistry and moves on. The worker keeps running; nothing crashes. If you're not looking at that registry, the failure is invisible — the same trap BullMQ, Celery, and every robust queue share. So the job is to reach into the queue's state and turn it into a signal. The four signals that matter for RQ Failure count / rate — jobs landing in the FailedJobRegistry over a window. Backlog — how many jobs are queued vs. being worked; is the worker keeping up? Latency — how long jobs take, and how long they wait before a worker picks them up. Worker liveness — are your workers actually alive and heartbeating? Where to read them RQ exposes queue and registry state directly: from redis import Redis from rq import Queue from rq.registry import FailedJobRegistry , StartedJobRegistry redis = Redis () q = Queue ( " default " , connection = redis ) queued = len ( q ) # backlog failed = FailedJobRegistry ( queue = q ) # failures started = StartedJobRegistry ( queue = q ) # in-flight print ( " queued: " , queued ) print ( " failed: " , len ( failed )) print ( " started: " , len ( started )) Poll this on an interval and store the series — a single snapshot hides the trend , which is the part that matters. For failures specifically, walk the registry to get the actual exceptions: for job_id in failed . get_job_ids (): job = q . fetch_job ( job_id ) print ( job . id , job . exc_info . splitlines ()[ - 1 ] if job . exc_info else "" ) Two gotchas: Group by exception, not by job. A thousand jobs failing with the same traceback is one incident. Normalize the message (strip IDs, timestamps, host

In-process cron ( node-cron , @nestjs/schedule , OS crontab) runs inside one Node process. That is fine for a single instance, but it does not survive restarts gracefully, deduplicate across replicas, or share infrastructure with your other background jobs. BullMQ stores queues and schedulers in Redis . Job Schedulers (BullMQ 5.16+) are the recommended way to enqueue recurring work on a cron pattern or fixed interval. The same workers that process one-off jobs also process scheduled ones, with retries, backoff, and concurrency you already get from BullMQ. This post covers Job Schedulers in plain Node.js, operations and pitfalls, a NestJS setup with @nestjs/bullmq , and a runnable demo with a fast cron heartbeat and a daily cleanup cron. Prerequisites Node.js version 26 Redis at redis://localhost:6379 (included in the demo docker-compose.yml , or use Postgres and Redis containers with Docker Compose ) npm i bullmq For the NestJS section: npm i @nestjs/bullmq bullmq BullMQ 2.0+ does not require a separate QueueScheduler instance. Use the Job Scheduler API ( upsertJobScheduler ), not the deprecated repeat option on queue.add() . Mental model Piece Role Queue Holds jobs waiting to run Worker Executes jobs Job Scheduler Factory that enqueues jobs on a schedule Scheduled job A job instance produced by a scheduler A scheduler id is stable across deploys. Calling upsertJobScheduler with the same id updates the schedule in place instead of creating duplicates. Queue and worker Share one Redis connection config between the queue and the worker: import { Queue , Worker } from ' bullmq ' ; const connection = { host : ' localhost ' , port : 6379 }; const queue = new Queue ( ' reports ' , { connection }); const worker = new Worker ( ' reports ' , async ( job ) => { console . log ( `[ ${ job . name } ]` , new Date (). toISOString (), job . data ); }, { connection }, ); worker . on ( ' failed ' , ( job , error ) => { console . error ( job ?. name , error . message ); }); Start the

If you ask any senior backend engineer or database administrator how to instantly make a slow, disk-bound application faster, their first answer is almost always: "Put it in memory." But why is memory so preferred, and how do modern architectures utilize RAM to achieve sub-millisecond latencies? We're seeing a massive revival of RAM disks and in-memory architectures. Let's explore why computer experts are increasingly treating RAM like a hard drive. 1. The Physics of Storage: Why RAM Wins To understand the shift towards in-memory architectures, we have to look at the numbers. Hard Disk Drives (HDDs): Mechanical spinning disks. Seek times are around 2-5 milliseconds . Solid State Drives (SSDs): Flash memory. Seek times are around 0.1 milliseconds (100 microseconds) . RAM (Random Access Memory): Volatile silicon. Access times are around 100 nanoseconds . RAM is roughly 1,000 times faster than an SSD and 10,000 to 50,000 times faster than an HDD. When you have a high-throughput system serving millions of requests per second, waiting for a disk to seek is an eternity. 2. In-Memory Databases: Redis and Memcached The most common implementation of this principle in modern backends is the In-Memory Database . How They Work Instead of writing every transaction to an SSD, systems like Redis and Memcached store the entire dataset directly in RAM. This bypasses the OS file system cache, disk I/O bottlenecks, and complex B-tree traversals required by traditional relational databases like PostgreSQL or MySQL. The Trade-off: Durability RAM is volatile. If the server loses power, all data is gone. So how do in-memory databases survive crashes? Snapshots (RDB in Redis): Periodically dumping the entire memory state to disk. Append-Only Files (AOF in Redis): Logging every write operation to a disk sequentially. Sequential writes to disk are significantly faster than random writes. This hybrid approach gives you the read/write speed of RAM with a "good enough" durability guarantee for

Event-driven architecture promises scalability, but in Java-based real-time systems the tradeoffs only surface in production. Drawing on a Java/Kafka contact center platform handling 80k BHCC across 10k agents, this article details where the design breaks down—state management, partition limits, deduplication, JVM tuning, cascading consumer failures—and the Redis-backed patterns that fixed each. By Sagar Deepak Joshi

Originally published on bckinfo.com Redis with Docker Compose: Persistence, Security, and Production-Ready Configuration Table of Contents Why Redis Containers Lose Data RDB vs AOF: Choosing a Persistence Strategy Basic Setup with Docker Compose Full Persistence Configuration Securing Redis in Docker Setting Resource Limits Redis in a Multi-Service Stack Backing Up and Restoring a Redis Volume Common Issues and Quick Fixes Closing Notes Redis is one of the most common services to run in Docker — it's fast to spin up, lightweight, and perfect for caching, session storage, and queues. But that same simplicity hides a trap: by default, Redis in Docker stores everything in memory, and the moment a container is removed, all of that data disappears . This guide walks through setting up Redis with Docker Compose the right way — covering persistence, authentication, resource limits, and the health checks you need before putting it anywhere near production. Why Redis Containers Lose Data A Docker container is meant to be disposable. That's a feature for stateless services, but it's a liability for a database like Redis. If you start a plain Redis container without a mounted volume, here's what happens: The container writes its dataset only inside its own writable layer. docker compose down or docker rm removes that layer entirely. The next time the container starts, Redis initializes with an empty dataset. This single oversight accounts for a large share of "we lost our session data" incidents in small teams running Redis in containers for the first time. The fix is straightforward once you understand the two persistence mechanisms Redis offers. RDB vs AOF: Choosing a Persistence Strategy Redis supports two persistence models, and production setups typically combine both: RDB (Redis Database snapshots) Point-in-time snapshots of the dataset, saved at intervals you define. Fast to restore, but you can lose any writes that happened after the last snapshot. AOF (Append Only Fil

I Built Commercial Redis CDC Source Drivers for Ruby — Here's What I Learned For the past couple of years I've been building a Change Data Capture (CDC) ecosystem for Ruby. Like many CDC projects, it started with PostgreSQL. PostgreSQL's Write-Ahead Log (WAL) is an excellent source of truth: durable, ordered, replayable, and well understood. It provides exactly the properties you want when you're building reliable event pipelines. But the deeper I went into distributed systems, the more I realized something important. Many systems don't observe change from PostgreSQL first. They observe it from Redis. Redis often sits at the front of modern architectures: Redis Streams carry application events. Pub/Sub distributes transient state changes. Keyspace notifications react to cache invalidation and key expiry. Redis Cluster routes events across multiple primaries. In many systems, Redis sees a change before PostgreSQL ever commits it. That raised an interesting question: Can Redis become a first-class Change Data Capture source? The obvious answer is "yes." The interesting answer is "yes—but not in the same way PostgreSQL does." That distinction eventually became cdc-redis-pro , a commercial Redis source driver for the Ruby CDC ecosystem. This article isn't a product announcement. It's an engineering write-up about the architectural decisions behind the project, the tradeoffs Redis forces you to make, and the execution model that ultimately emerged. Redis Doesn't Have One CDC Interface One misconception I frequently encounter is the assumption that Redis has an equivalent of PostgreSQL's WAL. It doesn't. Instead, Redis exposes several completely different mechanisms for observing change. Source Delivery Replay Streams At-least-once Yes Pub/Sub At-most-once No Sharded Pub/Sub At-most-once No Keyspace Notifications At-most-once No At first glance they all look like "events." Operationally they're completely different systems. Streams are durable. Pub/Sub isn't. Keyspace not