标签：#devops

With shiftbloom studio. I build tools and projects about a variety of experimental approaches to real-world problems. The issue for such use-case often was how most small media systems start out: one big always-on recorder that keeps costing money even when nothing is happening. For live capture you obviously need to stay ready at all times — sometimes you can’t risk losing the first minutes. But for everything else it’s complete overkill. The Core Problem Backfills, VOD downloads, clip imports, repairs and re-encodes are queue work. They can wait a few seconds, run on burst capacity, or even on a regular VPS or laptop. They don’t need the same always-hot infrastructure as the live recorder. That’s why I split the system. Instead of one large monolith, I deployed: Observer cells — only for live streams (time-critical) Harvest cells — for all queue processing (can be delayed) The Three Roles 1. Mothership A small control-plane cron job. It checks queue sizes, currently live channels and running observer tasks, then decides: how many harvest cells should exist right now which channels need an observer cell It’s intentionally simple. The database remains the single source of truth. 2. Observer Cells Each observer cell records exactly one live channel. It receives its assignment through environment variables: +++env OBSERVER_VOD_ID OBSERVER_CHANNEL_ID OBSERVER_CHANNEL_LOGIN OBSERVER_CHANNEL_NAME +++ It starts recording immediately, writes HLS segments to object storage, sends heartbeats, and waits a short standby window after the stream goes offline. This window is important because streams sometimes drop and reconnect quickly. Without it you end up with many small broken VOD fragments. 3. Harvest Cells These handle all background work: downloading VODs, re-encoding, recovering broken files, etc. They can run anywhere Docker is available — AWS tasks, a small VPS, or even a spare laptop. They only need outbound access to Postgres and object storage. What Changed Previous

This is the technical, reproducible version of a fix I shipped on my own homelab. If you want the narrative version, that's on Medium. This one is the recipe: the measurements, the math, the Modelfile, and the exact prompt I gave Claude Code to generate it. Copy-paste friendly. Repo for the dashboard used throughout: https://github.com/SikamikanikoBG/homelab-monitor TL;DR One 24GB RTX 3090, two GPU services: WhisperX large-v3 (STT, 7.7GB peak) and a Devstral Small 24B email-triage LLM (Q4_K_M, ~18.3GB). 18.3 + 7.7 = 26GB → CUDA OOM whenever they overlapped. The LLM was loaded with a 40k context window but the triage job never needed more than ~5–8k tokens. Capped num_ctx to 8192 → KV cache drops from ~6.1GB to ~1.25GB → model footprint ~18.3GB → ~14.2GB . 14.2 + 7.7 = 21.9GB → both resident, zero OOM, no quality loss. The setup Host : openSUSE, Xeon (56 threads), 125GB RAM, 1x RTX 3090 (24GB) GPU svc : WhisperX large-v3 (speech-to-text) GPU svc : Ollama -> devstral-small-2 (24B, Q4_K_M) for background email triage Both services run all the time. The OOM only happened when I dictated to my assistant (WhisperX) while the triage loop was active. Step 1 — Make the contention measurable nvidia-smi shows instantaneous VRAM. It can't show you which service spiked or when two of them overlapped — and an intermittent OOM is a timing problem. You need per-service VRAM history. I use my own dashboard (homelab-monitor) for this. The relevant view is "AI Models", which attributes VRAM per model server and per loaded model, over a time range, with OOM markers and a capacity ceiling line. What the history showed at the overlap window: Service Peak VRAM Devstral 24B (triage) ~18.3 GB WhisperX large-v3 7.7 GB Total ~26 GB on a 24 GB card If you want to reproduce the measurement, the dashboard runs as a single container: git clone https://github.com/SikamikanikoBG/homelab-monitor cd homelab-monitor docker compose up -d --build # open http://<host>:9800 -> AI Models / GPU views (NVIDI

Just released an open-source bash checker for CIFSwitch (CVE-2026-46243) — the 19-year-old Linux kernel LPE disclosed last week that lets any unprivileged local user get root by abusing the CIFS/SPNEGO upcall path. The script runs on bare-metal, VMs, and inside containers, and is CI/CD-friendly with JSON output and clean exit codes. It checks: ✅ Kernel version against patched thresholds (6.18.22 / 6.19.12 / 7.0+) ✅ cifs-utils presence and exploitable version ✅ CIFS kernel module load state and blacklist status ✅ Unprivileged user namespace sysctl (the pivot point for the exploit) ✅ Active request-key cifs.spnego rules ✅ SELinux / AppArmor enforcement ✅ Container capabilities (CAP_SYS_ADMIN) ✅ Kernel symbol verification for the fix commit Outputs human-readable or JSON for SIEM ingestion. Exit 0 = safe, exit 1 = action needed — drop it straight into a pipeline. CIFSwitch is the fourth Linux LPE in under six weeks (after Copy Fail, Dirty Frag, and Fragnesia). If you're running multi-tenant Linux, CI runners, or container build farms, now is a good time to audit. I have also updated the cve_checks.conf in my my K8s-container_escape_audit toolkit to detect this issue.

The first time I deployed a Next.js app to production, it took me three days. Not because the app was complicated — it was a straightforward portfolio site. It took three days because I had no idea what I was doing with AWS, I'd never written a GitHub Actions workflow, and every tutorial I found either skipped the hard parts or assumed I already knew them. By the time I was done, I had a deployment pipeline I was genuinely proud of: push to main, GitHub Actions runs the build, tests pass, the app deploys to an EC2 instance behind CloudFront. Zero manual steps. Zero downtime deploys. Total cost: about $5/month. This guide is the one I wish had existed. We're going to deploy a Next.js app to AWS from scratch — EC2 for compute, CloudFront for CDN, GitHub Actions for CI/CD — with every step explained so you understand what you're building, not just copying commands. Why AWS Instead of Vercel? This is a fair question. Vercel is genuinely excellent for Next.js, and for most projects it's the right call. You push, it deploys. Done. AWS makes sense when: You need to control the infrastructure (compliance, data residency, custom VPC configuration) You're running other services (databases, queues, lambdas) in AWS and want everything in the same network You want to learn infrastructure skills that transfer to enterprise environments Your app has specific performance requirements that benefit from custom CloudFront configuration You're a freelancer or consultant who wants to bill separately for infrastructure If none of those apply to you, use Vercel. This guide is for when they do. The Architecture Here's what we're building: ┌────────────────────────────────────────────────────────┐ │ GITHUB ACTIONS CI/CD │ │ │ │ Push to main → Build → Test → Deploy to EC2 │ └──────────────────────┬─────────────────────────────────┘ │ SSH deploy ▼ ┌────────────────────────────────────────────────────────┐ │ AWS EC2 INSTANCE │ │ │ │ Ubuntu 22.04 LTS │ │ Node.js 20 + PM2 (process manager) │ │ N

Migrar 100 usuários manualmente no console da AWS é lento, suscetível a erros e impossível de auditar com precisão. Neste artigo você vai ver como automatizar esse processo usando AWS CLI e Shell Script direto no AWS CloudShell — sem instalar nada localmente. O resultado final: usuários criados, alocados nos grupos corretos e com MFA obrigatório, tudo em minutos. O que é o IAM? O AWS Identity and Access Management (IAM) é o serviço que controla quem pode acessar os recursos da sua conta AWS e o que cada pessoa ou serviço pode fazer. Com o IAM você gerencia: Conceito Descrição Usuário Identidade individual com credenciais próprias Grupo Conjunto de usuários que compartilham as mesmas permissões Política Documento JSON que define o que é permitido ou negado Role Identidade temporária assumida por serviços ou usuários A boa prática é nunca conceder permissões diretamente a um usuário — sempre use grupos. Visão geral da solução O fluxo é simples: Criar os grupos IAM no console Montar um arquivo CSV com os dados dos usuários Rodar um shell script no CloudShell que lê o CSV e cria tudo automaticamente Aplicar a política de MFA obrigatório nos grupos Passo 1 — Criar os Grupos IAM Antes de importar os usuários, os grupos precisam existir. No AWS Console , acesse IAM → User groups → Create group e crie um grupo para cada perfil do seu ambiente. Neste exemplo usaremos: RedesAdmin — administradores de rede LinuxAdmin — administradores de servidores Linux DBA — administradores de banco de dados Estagiarios — acesso limitado para estagiários Nomes de grupos suportam até 128 caracteres (letras, números e + = , . @ _ - ), são únicos por conta e não diferenciam maiúsculas de minúsculas. Passo 2 — Montar o arquivo CSV Crie uma planilha com os dados dos usuários e salve como CSV separado por vírgula (UTF-8) . O arquivo deve ter exatamente três colunas: Username , Group e Password . Username , Group , Password joao . silva , LinuxAdmin , Senha @2024 ! maria . souza , DBA , Senha @2024

The Scenario: The Help Desk Bottleneck From 2019 to 2021, while serving as Lead Backend Software Engineer at a fast-growing company, I occasionally support our Linux System Administration tasks. When the DevOps team encountered a critical bottleneck during an initiative to scale dozens of new server deployments, I stepped in to streamline the infrastructure processes. The DevOps team was being hampered by constant, fragmented requests from the help desk to manually create new Linux accounts for recruits testing the latest application. These interruptions were not only time-consuming but were directly preventing the team from focusing on the high-priority infrastructure deployments that define their core responsibilities. I realized that we weren't just struggling with a task; we were struggling with a scaling bottleneck. To regain the team's focus and ensure we hit our project deadlines, I decided to automate this workflow. The First Step: The Interactive Script My first objective was to develop a robust, automated shell script to efficiently create new Linux user accounts. I started with an interactive Bash script (create-user-interactive.sh) that prompted for input. This was a good educational exercise for learning the fundamentals of Bash—like useradd, passwd, and shell variables. However, I quickly learned that while interactive scripts are great for learning, they are rarely used in professional DevOps environments. Why Manual Scripts Don’t Scale As I transitioned into a more infrastructure-focused role, I realized that manual scripts fail for three key reasons: Lack of Automation: DevOps is about "Infrastructure as Code" (IaC). Asking an engineer to sit at a terminal and type prompts is slow, error-prone, and destroys the ability to automate. Lack of Centralization: In a real team, we aren't creating users on individual local machines. We manage identity across hundreds of servers. Security Risks: Hardcoding passwords or piping them through echo is a major red

Kyle Lexmond explains how to handle the high-pressure environment of severe production outages. He discusses the critical distinction between mitigation and root-cause resolution, sharing personal experiences from harrowing incident rooms. He shares valuable operational strategies on overcoming cognitive overload, establishing blameless cultures, and optimizing systems for faster recovery. By Kyle Lexmond

OpenTelemetry has introduced a new "Blueprints" initiative aimed at reducing the growing complexity of deploying and operating observability systems at scale. By Craig Risi

Introduction Managed Kubernetes platforms such as Amazon EKS, Google Kubernetes Engine (GKE), and Azure Kubernetes Service (AKS) abstract away much of the operational complexity involved in running Kubernetes clusters. While this significantly improves developer productivity, it also hides many of the internal systems responsible for cluster orchestration, networking, node registration, and workload scheduling. As a result, many engineers interact with Kubernetes daily without fully understanding the components that keep a cluster operational behind the scenes. To better understand Kubernetes from an operational perspective, I set out to build and operate a self-managed Kubernetes platform on AWS using kubeadm. Unlike lightweight local environments such as Minikube or kind, kubeadm bootstraps Kubernetes in a way that closely resembles how real-world self-managed clusters are provisioned and operated. The objective of this project was not simply to install Kubernetes, but to explore: How the control plane components interact. How worker nodes register with the cluster. How Kubernetes networking behaves. How cloud integrations work. How traffic reaches workloads running inside the cluster. How operational failures surface during deployment and runtime. How production-style systems behave beneath managed abstractions. This article documents the architecture, implementation process, engineering decisions, operational lessons, and troubleshooting insights encountered during the effort to bring the platform to a healthy operational state. Project Objectives The primary objectives of this project were to: Provision infrastructure on AWS using Terraform. Bootstrap a self-managed Kubernetes cluster using kubeadm. Configure Kubernetes networking using Calico. Integrate Gateway API with AWS Load Balancer Controller. Expose workloads externally using AWS Application Load Balancers. Validate cluster functionality through application deployment. Understand the operational mechani

Stanford Just Published Rules for AI Coding Agents — What Devs Should Know Stanford dropped a document last week that every developer using AI coding tools should read. It's called CLAUDE.md , it's part of CS336 (Language Modeling from Scratch), and it's a brutally honest set of rules for how AI agents should — and shouldn't — help students write code. The document hit #1 on Hacker News for good reason. It doesn't just apply to students. If you use Claude Code, Cursor, Copilot, or any AI coding assistant, these rules expose the uncomfortable gap between what these tools can do and what they should do. GitHub just rolled out token-based billing for Copilot, and developers are furious. The tension is the same: when does AI assistance stop helping and start hurting? The Core Principle: Teaching Assistant, Not Solution Generator Stanford's position is unambiguous: "AI agents should function as teaching aids that help students learn through explanation, guidance, and feedback — not by completing assignments for them." This isn't academic hand-wringing. It's a design constraint that maps directly to professional development. The same agent that writes your PR in 30 seconds is also the one that leaves you unable to debug it when it breaks at 2 AM. The AI agent role framework from Stanford's CS336 guidelines: teaching assistant vs solution generator The document draws a hard line: What agents SHOULD do: Explain concepts by guiding toward understanding Review your code and point out areas for improvement Ask guiding questions instead of giving fixes Reference documentation, lectures, and debugging tools Suggest sanity checks, assertions, and profiler investigations What agents SHOULD NOT do: Write any Python or pseudocode Complete TODO sections in assignments Give solutions to problems Edit code in the student repo Convert requirements directly into working code Point to third-party implementations If you're a professional developer, the "SHOULD NOT" list probably looks extr

The dominant production failure mode for LLM agents in 2026 isn't bad reasoning — it's capacity. Here's what the data shows, why nobody demos it, and the capacity-engineering patterns that actually keep agents alive under load.

Last week I built a little dashboard with Claude. Took maybe ten minutes. Then I spent the next hour trying to get it online. ssh in, install docker, write a Dockerfile, set up nginx, run certbot, certbot fails, read the log, oh the DNS hasn't propagated, wait, run it again, open port 443, realize ufw was blocking it the whole time. By the time it was live I'd forgotten what the app even did. I've done that maybe a few hundred times by now. I'm a backend guy, I'm fast at it. But fast at something boring still means doing the boring thing. So at some point I just thought: the AI already wrote the app. Why does it stop right when the annoying part starts? Why doesn't it just deploy the thing itself? The reason is it has no hands. The model can write you a perfect docker-compose file. It can't ssh into your box and run it. No connection to your server, nowhere to hold your key. So I gave it hands. It's an MCP server, vibe-deploy. You hook it up once to a VPS you own, and then you just say "deploy this to notes.mydomain.com" and the agent containerizes it, ships it over ssh, sets up nginx, gets a real Let's Encrypt cert. Node, Python, Go, plain static. It figures out the stack and writes the Dockerfile. No PaaS, no per-seat pricing, no free tier you'll outgrow. A $5 box runs a dozen of my projects and I own the whole thing. The "you gave an AI root on your server??" reaction is fair, so: it runs locally, your key never leaves your laptop. I used a separate ssh key scoped to deploys, not my real one, and you should too. It checks the server host key before connecting and validates everything you pass it, because a deploy tool that pastes your input straight into a shell is a horror story waiting to happen. I had someone audit the security before I put it out. They found two real bugs. I fixed them. It's free and MIT, on GitHub and npm as @cgnguyen/vibe-deploy . I built it because I wanted it. If you live in the same gap between "it works on localhost" and "it's online",

The test passed. The runbook completed. Infrastructure came back online inside the RTO window. None of that means the organization can recover from an actual disaster. Disaster recovery testing is designed to succeed. Clean environments, pre-staged dependencies, known failure modes, available staff — each design decision is operationally reasonable. Collectively they remove the conditions that make real recovery hard. What the test validates is test completion, not recovery capability. The Test Is Designed to Pass Every design decision in a standard DR test tilts toward a successful outcome. The test window is pre-announced, so the right engineers are available. The scope is pre-defined, so unexpected systems don't surface mid-exercise. The environment is either isolated or pre-staged, so competing failures don't complicate the recovery sequence. The data state is known and clean, so integrity issues don't slow the restore. The declaration point is assumed, so nobody has to make an ambiguous call under pressure. A test designed to remove the variables that make recovery hard cannot produce evidence about what happens when those variables are present. What Disaster Recovery Testing Actually Excludes Declaration threshold. In a DR test, recovery starts at a pre-agreed time. In a real incident, recovery starts when someone decides the situation has crossed the threshold for declaration — a decision that is rarely clean and routinely delayed 45 minutes to several hours. That delay is inside the real outage window and outside the test clock. Dependency assumptions. DR tests run against known, pre-cleared dependencies. Real incidents surface undocumented dependencies that were never in scope — a configuration service that hasn't been touched in two years, an authentication endpoint that wasn't in the architecture diagram. Data state. Test environments use clean or pre-staged data. Real recovery requires handling whatever state the data was in at the moment of failure — pa

The DORA research programme is the most rigorous empirical study of software delivery performance ever conducted. Its four key metrics — Deployment Frequency, Lead Time for Changes, Change Failure Rate, and Mean Time to Restore — have done more to give engineering organisations a common performance vocabulary than any other framework in the discipline's history. If you work in software and you have not read the State of DevOps Report, stop and read it before finishing this paragraph. Now: the DORA Four were derived primarily from organisations with cloud-native architectures, on-demand deployment infrastructure, and relatively unconstrained ability to release software when it is ready. The research cohort skews toward technology companies that have already made the cultural and architectural investments that make high-frequency, low-risk deployment possible. This is not a criticism of the research. It is an observation about its generalisability — and it has a specific consequence for practitioners who work in regulated enterprises: banks, healthcare systems, utilities, insurance carriers, government agencies. In these environments, the DORA Four are necessary but structurally insufficient. They measure the delivery pipeline accurately. They do not measure the operational sustainability of the team running that pipeline — and in regulated enterprises, operational sustainability is where SRE programmes go to die quietly, years before anyone realises the damage is permanent. This post proposes a fifth metric. Not to replace the DORA Four, but to complete them — to close the measurement gap that leaves regulated enterprise SRE teams flying blind on the dimension that most reliably predicts long-term programme failure. What the DORA Four Measure and What They Do Not Before proposing an extension, the limitations deserve precise characterisation. Imprecise criticism of a well-validated framework is noise. The limitations described here are structural — arising from the d

We've all been there. You're 45 minutes into a Docker tutorial, feeling great about yourself, and then someone casually drops: "Just pull the image and spin up a container." And you think: "...wait, aren't those the same thing?" First - this has happened to a good number of us if we are to be honest. Even almost every single DevOps engineer, cloud architect, and platform wizard you admire has typed the wrong term in a sentence at least once in their career. It's practically a rite of initiation. There should be a badge for it if you ask me. Why Does This Trip Everyone Up? Here's the sneaky truth: Docker commands blur the line constantly. You type docker run nginx and something called a "container" starts — but wait, didn't you just use an "image" called nginx ? Where did one end and the other begin? The confusion lives in the fact that they are deeply related — one literally gives birth to the other. But they are fundamentally, completely different things. Getting this distinction straight is your official rite of passage into DevOps. Once it clicks, the rest of Docker feels like cheating. Basically, A Docker Image is the blueprint : a frozen, static snapshot of everything your app needs - the OS layer, the dependencies, the config files, your actual code. It just sits there on disk, completely inert. You can't run a blueprint. A Docker Container is the house : the live, running instance that was built from that blueprint. It has processes running, files potentially being written, network ports being listened on. It's alive. And now, just like one blueprint can produce 10 identical houses on different streets - one Image can launch 10 identical Containers simultaneously; and that's where Docker's scaling magic comes from. # The image just sits here, unchanging docker pull nginx # Now we BUILD a house (container) from the blueprint docker run nginx # Build THREE houses from the same single blueprint docker run nginx docker run nginx docker run nginx Here is an exampl

Take Aways Performance and Scalability : Pinecone is a managed machine-learning database that provides exceptional levels of performance and scaling capability due to its cloud-based design. Because of its distributed architecture and ability to do near-neighbor searches, Pinecone handles such tasks as similarity searching and anomaly detection on very large datasets efficiently. Easy to Integrate : One of the standout benefits of Pinecone is how easily it integrates through a high-level API and SDKs across several programming languages. This gives developers a real productivity boost by making vector storage, indexing and querying for machine learning applications far less complicated to implement. Strategic Factors : Pinecone brings advanced features and managed services that genuinely enhance machine learning workflows, though it does come with considerations like recurring costs and vendor lock-in. Organizations should think carefully about these factors alongside the benefits of streamlined database management and optimized performance before committing to adoption. The importance of storing and accessing information properly to build the best possible machine learning model really cannot be overstated. Pinecone addresses this directly by offering a Vector Database built specifically for ML queries, creating a strong opportunity to tap into the power of cloud databases. Designed from the ground up as a cloud-native application, Pinecone makes it straightforward to index and search complex, high-dimensional vector data — which in turn makes building state-of-the-art machine learning applications much more approachable and helps software development companies deliver more value to their clients through custom software development. What is Pinecone? Pinecone is a fully managed Vector Database that lets you store, index, and query complex vector data quickly and efficiently. Because of its vector-native design, the primary use cases for Pinecone fall within similar

Mañana cambia el billing de Copilot: las premium requests dan paso a AI Credits calculados por tokens. Esto es lo que debe revisar un equipo técnico. El 1 de junio de 2026 GitHub Copilot empieza a migrar desde el modelo de premium requests hacia billing por uso con GitHub AI Credits. La unidad deja de ser una petición premium más o menos abstracta y pasa a reflejar consumo de tokens: entrada, salida y tokens cacheados, con precios vinculados al modelo usado. Decisión rápida Qué cambia mañana La idea de GitHub es alinear precio con coste real. Una pregunta rápida a un modelo ligero y una sesión larga de agente sobre varios archivos ya no son equivalentes. Para equipos técnicos, eso obliga a tratar Copilot como infraestructura de IA, no como una extensión de editor de coste fijo. Este artículo complementa la guía previa de AI Credits, pero se centra en el cambio operativo inmediato: qué mirar antes de que el modelo entre en vigor mañana. Briefing Qué es un AI Credit GitHub define AI Credits como una unidad de billing donde 1 AI Credit equivale a 0,01 USD. Cada interacción que usa modelos consume tokens. Esos tokens se valoran según el modelo y se convierten a créditos. En planes individuales, Copilot Pro, Pro+ y Max incluyen asignaciones mensuales de AI Credits. En organizaciones y empresas, cada licencia aporta créditos que se agrupan en un pool compartido a nivel de billing entity. La diferencia clave con el sistema anterior es que el consumo puede variar mucho dentro de una misma función. Dos sesiones de chat no cuestan igual si una es una pregunta corta y otra arrastra contexto de repositorio, varias iteraciones y generación de código extensa. Lectura práctica Qué consume créditos y qué no GitHub documenta que consumen AI Credits funciones como Copilot Chat, Copilot CLI, Copilot cloud agent, Copilot Spaces, Spark y agentes de terceros. Las code completions y Next Edit suggestions no se facturan en AI Credits y siguen incluidas en planes de pago. Esta distinción es

Most agent frameworks assume one agent, one app, one bill. The moment you run agents for many clients, two problems appear that no runtime solves for you: you can't prove which client burned which tokens , and nothing stops one client's workspace from leaking into another's . I built Octorato to fix exactly that. What Octorato is Octorato is an open-source AI agent operating system: one file-native "brain" — rules, 190+ skills, 180+ specialist agents, all plain markdown under git — that a single operator runs across many sealed client "arms," with per-client token attribution and opt-in budget caps. It's not a runtime you import. It's the agent's self as files you can read, diff, fork, and own — runtime-agnostic (it runs on Claude Code today). The octopus model One brain , many arms . The brain holds the shared self: rules (the constitution), skills (HOW to do things), agents (WHO does them). Each arm is a sealed deployment serving exactly one client. Knowledge flows down (generic skills cascade to every arm) and lessons flow up (anonymized patterns get distilled back into the brain). Like a real octopus, most of the neurons live in the arms, not the head. Why "file-native" matters Your agent's identity, skills, and memory normally live trapped inside vendor code and a cloud console — you can't read the whole self, diff a change, or move it. Octorato keeps all of it as plain markdown under version control. Identity becomes diffable, reviewable, portable, and ownable . Text outlives runtimes. The part nobody else does: FinOps and isolation are the same wall Because each arm is a sealed cell that no other arm can see, every token an arm spends is attributable to exactly one client by construction. Cellular isolation is per-client FinOps — the wall that seals a client is the wall that meters it. Concretely: per-arm USD rollup (estimated from local session logs at list price), cost-spike alerts, and an opt-in PreToolUse budget gate — wire the hook and set a client's cap

We were running a cloud-hosted platform on AWS EKS , with EC2 worker nodes managed by us, MongoDB Atlas for NoSQL workloads, AWS RDS for relational databases, and Amazon ElastiCache for Redis for caching and temporary data. Over time, the infrastructure had grown the way most real systems grow: more services, more data, more backups, more images, more snapshots, and more “temporary” resources that were no longer temporary. The platform worked, but the cloud bill was higher than it needed to be. So we started cutting waste, improving the application, and resizing the infrastructure around how the system actually behaved. The result: around $120,000 in annual savings , without sacrificing reliability. The Problem Was Not One Big Thing When we started reviewing the infrastructure, it was clear that there was no single expensive resource causing the entire problem. The cost came from many places at once. Some services were using more CPU and memory than they needed. Some microservices did not really need to be separate anymore. Some databases were oversized for their actual usage. Some storage had accumulated over time. Some backups and snapshots were kept longer than necessary. Some resources were simply unused. That is usually how cloud costs grow. Not because of one bad decision, but because of hundreds of small decisions that were reasonable at the time and never revisited later. So instead of looking for one magic fix, we approached the problem from multiple angles: application code, architecture, databases, Kubernetes resources, storage, backups, caching, and non-production environments. The Optimizations 1. Making the Application Use Fewer Resources One of the most important parts of the optimization was improving the application itself. It is easy to look at cloud cost as an infrastructure problem only, but inefficient code directly affects infrastructure cost. If the application uses too much CPU or memory, the platform needs more pods, larger nodes, bigger ins

A survival guide for when everything goes wrong in production. Your Redis dashboard looks perfect. Hit ratio: 99.2%. Latency: sub-millisecond. Memory usage: 60% of available. Every metric says healthy. Then at 2:47 PM, your API starts returning 500s. Response times spike to 30 seconds. Users can't log in. The dashboard still shows 99% hit ratio because the cache is working — it's serving cached errors to everyone equally fast. Redis is doing exactly what you told it to do. The problem is what you told it to do. Why Single-Threaded Is Fast (Until It Isn't) Redis processes commands on a single thread. No locks. No context switching. No synchronization overhead. One CPU core, fully utilized, can handle 100K+ operations per second because it never waits for another thread to release a lock. The event loop model (similar to Node.js) multiplexes thousands of client connections on a single thread using non-blocking I/O. Read a request, process it, write the response, move to the next. When your commands are simple — GET, SET, INCR — each one takes microseconds. The trap: slow commands block everything. KEYS * on a million-key database? That's a full keyspace scan on the main thread. While it runs, every other client waits. SORT on a large set? Same. LRANGE on a list with 10 million elements? Same. Redis 6.0 introduced I/O threading ( io-threads config) for reading and writing network data on multiple threads, but command execution is still single-threaded. Redis 7.0 improved this further, but the fundamental model hasn't changed. Long-running commands on the main thread stall everything. Rules: Never use KEYS in production. Use SCAN instead — it's cursor-based and returns results incrementally. Watch out for O(N) commands on large data structures: LRANGE , SMEMBERS , HGETALL on million-element structures. Use SLOWLOG to find commands that are blocking the event loop. Pipelining: The Easiest 10x You'll Ever Get Every Redis command involves a network round trip: send request