标签：#Kubernetes

I'm fairly new to SRE/DevOps, and one of the topics I recently spent time studying properly was EKS cluster upgrades . My first instinct, like most people starting out, was: "it's just a version bump, click upgrade in the console, done." That's basically what most beginner blog posts say too. But the more I read and the more I dug into real-world postmortems and discussions, the more I realized — the actual Kubernetes control plane upgrade is the easy part. Almost everything that can go wrong seems to happen around it, not because of it. Sharing what I learned here, mainly for my own notes, but hoping it's useful for anyone else early in their journey too. Learning #1: There's No "Undo" Button This was the first thing that surprised me. I assumed upgrades work like most software — if something breaks, you roll back. But with EKS, you cannot downgrade the control plane version once you upgrade it. So the plan can't be "upgrade, and if it breaks, revert." It has to be "test enough beforehand that breaking isn't really an option," and if something does go wrong, the fix is always moving forward, not backward. That single fact changes how you're supposed to approach the whole thing — testing has to happen before the button is clicked, not after. Learning #2: APIs Get Deprecated, and It's Usually Not Your Own Code That Breaks Kubernetes removes old API versions on a schedule. I already knew this conceptually, but what I didn't realize is that the risk usually isn't your own YAML files — it's the Helm charts and third-party tools you installed a while back and forgot about , which might still be using an older API version internally. There are tools built exactly for catching this before it becomes a problem: pluto detect-helm -owide pluto detect-files -d ./manifests kubent (kube-no-trouble) does something similar. I hadn't heard of either tool before researching this, and it made me realize how much of "being good at Kubernetes" is really just knowing which small tools e

Microsoft is placing greater emphasis on controlling disruption in Azure Kubernetes Service (AKS) Node Auto-Provisioning (NAP), publishing new guidance to help platform teams balance the efficiency benefits of automated node consolidation with application availability. By Craig Risi

Scalable Guardrail Service ASP.NET Core Kubernetes: Architecture, Code, and Ops Quick Answer Scalable Guardrail Service ASP.NET Core Kubernetes: A dedicated ASP.NET Core guardrail microservice on Kubernetes validates LLM requests, enables instant policy updates via Redis, and scales with custom HPA for high‑throughput. Scalable Guardrail Service ASP.NET Core Kubernetes: Why a Dedicated Guardrail Microservice Matters When you expose an LLM‑powered API to the world, every request is a potential compliance risk. A single malformed prompt can surface PII, trigger a policy violation, or even cause a brand‑damaging output. In my experience, the first version of such a system is a set of ad‑hoc filters sprinkled across controllers. Under load, those filters become latency bottlenecks, policy updates race, and audit trails vanish. The root cause is a missing architectural layer that treats guardrails as a first‑class microservice that can scale horizontally, be updated live, and be observed independently. Guardrail Layer Requirements We need a guardrail layer that: Validates every request before it hits the LLM engine. Can be updated without redeploying the entire API surface. Provides per‑tenant isolation and versioning. Logs every decision for compliance and red‑team analysis. Runs at the same scale as the LLM inference service. When This Fails in Production Policy updates are applied via a shared ConfigMap and the pods do not reload, so new rules are never enforced. The guardrail service is single‑instance; a spike in requests triggers a queue that exceeds the LLM engine’s rate limit, causing a cascading failure. Audit logs are written to local disk; a pod crash loses events. Latency spikes because each request performs a synchronous Redis lookup for every policy. Common Mistakes Engineers Make Embedding guardrail logic inside the API controller rather than a dedicated middleware. Using in‑memory policy caches without a TTL, leading to stale rules. Ignoring the fact that

Upgrading Kubernetes clusters across a large enterprise fleet is often a balancing act between staying current with security patches and avoiding outages. By default, Google Kubernetes Engine (GKE) rolls out automatic upgrades progressively according to Google Cloud regional timelines. While regional rollout works well for standalone clusters, it does not understand your organization's business topology. If you run staging clusters in us-central1 and critical production clusters in us-east1 , a standard regional rollout could upgrade your production environment before your pre-production validation completes. The General Availability (GA) release of GKE rollout sequencing with custom stages solves this challenge. It provides platform teams with declarative control to sequence cluster upgrades across fleets, environments, and even distinct Google Cloud organizations according to business criticality rather than cloud geography. How rollout sequencing works Rollout sequencing builds on GKE fleet management. Fleets serve as logical boundaries for environments such as development, staging, and production. With rollout sequencing, you define an ordered pipeline of upgrade stages managed by a central resource called RolloutSequence . When GKE publishes a new automatic upgrade target for a release channel, or when you explicitly trigger a target version, the system creates a Rollout object. This rollout progresses through your defined stages sequentially: Control plane upgrades start in the first stage. Once all control planes in that stage reach the target version, a stage soak timer begins. Node upgrades run in parallel with control plane upgrades, respecting node pool upgrade strategies such as surge or blue-green. When both control planes and nodes complete their upgrade and satisfy the configured soak duration, the rollout advances to the next stage in the sequence. If an individual stage contains clusters that take longer than 30 days to finish upgrading—due to restr

Short answer: for a Node.js app in Docker or Kubernetes, give startup, readiness, and liveness probes separate meanings, keep routine health traffic out of application logging, and measure state transitions instead of counting every successful check. For a property-management API rolling out a new pricing rule, this preserves useful metrics: whether an instance can calculate rent correctly and accept traffic, without turning each kubelet poll into noise. Which health signal should control each container decision? Start with the decision, not the endpoint name. Signal Question it answers Include Exclude Action Startup Has initialization completed? Configuration parsing, pricing-rule compilation, required local warm-up Long-term dependency health Allow the process more time before other probes apply Readiness Can this instance safely receive a new pricing request now? Ability to serve the active rule version and any required dependency state Optional analytics and background exports Remove the pod from Service endpoints Liveness Is the process stuck beyond local recovery? Event-loop progress or another narrow process invariant Database, cache, and third-party availability Restart the container This split is the main noise filter. A downstream dependency becoming unavailable can make a pod unready, but restarting the same healthy process usually doesn't repair that dependency. If the dependency is placed in liveness anyway, every pod can restart together. The health response has then amplified one problem into two: lost capacity plus a restart storm. The pricing rollout makes readiness more demanding than “the port is open.” Imagine rule version rent-2026-08 is enabled for one building cohort. A newly started instance has loaded configuration but hasn't compiled that version yet. It is alive. It isn't ready. Its startup check should hold back liveness and readiness until initialization finishes; afterward, readiness should stay false until the active rule can be evalua

By combining ProgressivePerGroup with Placement decision groups, you can roll out add-on configuration changes in the order dev → stg → prod. In this article, I use cluster-proxy as an example to explain the required configuration and how the rollout actually works. Overview flowchart TB Upgrade["helm upgrade<br/>change tag to vX.Y.Z"] subgraph Hub["Hub cluster"] direction TB Manager["cluster-proxy-addon-manager<br/>update Deployment"] Config["ManagedProxyConfiguration<br/>update spec"] ProxyServer["proxy-server<br/>update Deployment"] Hash["proxyAgent config<br/>update spec hash"] Rollout["OCM add-on manager<br/>ProgressivePerGroup"] Groups["progress through decision groups<br/>dev → stg → prod<br/>success + minSuccessTime before next group"] AddOn["ManagedClusterAddOn in current group<br/>Configured=True"] Render["cluster-proxy manager<br/>render agent chart"] Work["update ManifestWork"] end subgraph Spoke["spoke clusters in the current group"] direction TB WorkAgent["work-agent"] ProxyAgent["proxy-agent<br/>update Deployment"] end Upgrade -->|Helm updates directly| Manager Upgrade -->|Helm updates directly| Config Config -->|proxyServer.image<br/>not part of rollout| ProxyServer Config -->|proxyAgent.image<br/>part of rollout| Hash Hash --> Rollout Rollout --> Groups Groups -->|current group only| AddOn AddOn --> Render Render --> Work Work --> WorkAgent WorkAgent --> ProxyAgent WorkAgent -.->|Applied / Available| Work Work -.->|hash matches + Ready| Rollout classDef immediate fill:#fff3cd,stroke:#a66b00,color:#332200; classDef staged fill:#e8f3ff,stroke:#2563a6,color:#102a43; classDef spoke fill:#eaf7ed,stroke:#2f855a,color:#173d2a; class Manager,Config,ProxyServer immediate; class Hash,Rollout,Groups,AddOn,Render,Work staged; class WorkAgent,ProxyAgent spoke; Yellow indicates updates that happen immediately on the Hub. Blue indicates updates controlled by ProgressivePerGroup , and green indicates processing on the spoke clusters. Dashed lines represent status r

We have a web server container running the nginx image. The access and error logs generated by the web server are not critical enough to be placed on a persistent volume. However, Nautilus developers need access to the last 24 hours of logs so that they can trace issues and bugs. Therefore, we need to ship the access and error logs for the web server to a log-aggregation service. Following the separation of concerns principle, we implement the Sidecar pattern by deploying a second container that ships the error and access logs from nginx. Nginx does one thing, and it does it well - serving web pages. The second container also specializes in its task - shipping logs. Since containers are running on the same Pod, we can use a shared emptyDir volume to read and write logs. Create a pod named webserver . Create an emptyDir volume named shared-logs . Create a regular container in the webserver pod from the nginx:latest image named nginx-container , and an init container from the ubuntu:latest image named sidecar-container . Add the following command to the sidecar-container "sh","-c","while true; do cat /var/log/nginx/access.log /var/log/nginx/error.log; sleep 30; done" Mount the shared-logs volume in both containers at /var/log/nginx . Ensure all containers are in a running state. What is a Sidecar Container? Think of a sidecar like a motorcycle sidecar – it's attached to the main vehicle and extends its capabilities without changing the main vehicle itself. ┌─────────────────────────────────────────────────────────────────────────────┐ │ The Sidecar Analogy │ │ │ │ ┌────────────────────────────────────────────────────────────────────────┐ │ │ │ Motorcycle: The Main Vehicle │ │ │ │ - Does its primary job (serving web pages) │ │ │ │ - Doesn't worry about extra tasks │ │ │ └────────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────────────────────────────────────────────────────┐ │ │ │ Sidecar: Adds Extra Functional

Control Plane (Master) & Worker Nodes Control Plane components: API Server Scheduler Control Manager etcd Worker Node components: Container Runtime Kubelet Kube-proxy Node Processes Each node has multiple Pods on it. 3 processes must be installed on every node — used to schedule and manage those Pods. Nodes are cluster services that actually do the work. Container Runtime Examples: Docker, containerd, CRI-O. containerd is used in worker nodes — it's lightweight in nature. This should be installed on every node because application Pods need to run containers inside the node. Kubelet The process which schedules the Pods and containers underneath is Kubelet. Kubelet interacts with both the container and the node. Kubelet starts the Pod with the container inside. Communication between two nodes is because of Services. Creation of Pod: Kubelet insures the Pod is always running — if not, it will inform etcd. Kube-proxy Kube-proxy forwards the request from Pod to Service. Makes use of the communication, with load balancing. Provides networking (container ID, IP address). Load balancing — basically using IP tables. It makes sure to send the request to the same machine instead of sending it to others (from same node communications). So, how do you interact with this cluster? Schedule the Pod Monitor Re-schedule/restart the Pod Join a new node Managing processes are done by master nodes (the control plane). API Server When you, as a user, want to deploy a new application in a Kubernetes cluster, you interact with the API server using some client — could be UI or CLI. It's a cluster gateway — it gets the initial request of any update into the cluster, even the queries from the cluster. It also acts as gatekeeper for authentication. It means when you want to schedule new Pods, deploy new applications, create new services, or any other components — you have to talk to it first. Flow: Some request → API server → Validates request → Other processes → Pods Only one entry point to t

At LinkedIn's scale, relying solely on human reviewers or simply putting an off-the-shelf AI reviewer in front of GitHub is not an effective way to manage PRs. To address this, LinkedIn engineers built a multi-agent AI code review platform that understands the organization’s coding context, treats code review as production infrastructure, and minimizes hallucinations and low-signal feedback. By Sergio De Simone

Introduction: Kubernetes can feel overwhelming when you first hear terms like Pods, Services, and Deployments thrown around. In this first post of my Kubernetes series, I’ll break down the fundamentals — what Kubernetes actually solves, and the core building blocks you need to understand before going further. What is Kubernetes? Kubernetes is an open-source container orchestration tool , originally developed by Google. It helps manage containerized applications across different environments — physical machines, virtual machines, and cloud environments — which makes it a great fit for hybrid deployment setups. Why Kubernetes? The Problem It Solves To understand why Kubernetes exists, look at the trend that led to it: Applications moved from monolith to microservices. That shift drastically increased the number of containers teams had to manage. Managing hundreds of containers by hand became unsustainable — teams needed a proper way to orchestrate them. Key Features High Availability — no downtime Scalability — scale up or down based on load and performance needs Disaster Recovery — backup and restore built into the ecosystem Main Kubernetes Components Pods: Abstraction over containers Services: Stable networking & communication Ingress: Routes external traffic into the cluster ConfigMaps & Secrets: External configuration Volumes : Data persistence Deployments & StatefulSets: Replication (stateless vs. stateful) DaemonSets: One Pod per node, auto-scaled with the cluster

Iran Doesn't Need to Mine Hormuz — Your requirements.txt Is Already Rigged Every headline you've read this week is a diversion. The Strait of Hormuz is not the target. You are. And you have been for months, possibly years, while you retweeted tanker tracking maps and debated whether Brent crude would touch $150. Iranian state-sponsored groups — OilRig, APT33, MuddyWater, Agrius — did not spend the last decade pivoting to cloud infrastructure so they could watch you panic about a waterway. They did it so they could own your build pipeline while you were distracted. And they have. This is not speculation. CISA Advisory AA24-038A explicitly maps Iranian APT campaigns against U.S. and allied critical infrastructure to cloud identity, Kubernetes targets, and software supply chains. Not SCADA. Not PLCs. Your kubectl binary. Your Helm charts. That FastAPI microservice running payment webhooks that you deployed on a Friday and haven't touched since March. The Revolutionary Guard does not need a mine. They need a maintainer who hasn't updated python-jose in fourteen months. The Theater and the Operation You watched the Strait. They watched your CI/CD. Geopolitical analysis is a spectator sport for infrastructure engineers, and Iranian cyber command is the bookie. While your LinkedIn feed filled with satellite imagery and retired admirals explained chokepoint logistics, the actual operation ran silently against: Public Helm charts with hardcoded cluster-admin ServiceAccounts FastAPI services with python-multipart handling unbounded file uploads on single-threaded Uvicorn workers .kube/config files exfiltrated from developer laptops in a dev-legacy namespace that predates your current CTO Terraform state stored in a single S3 bucket with versioning disabled and a policy written by someone who left in 2021 The Hormuz closure narrative is Information Operations . The closure of your API gateway due to an unpatched ASGI memory exhaustion vulnerability is the kinetic effect. You a

Dan Finneran discusses the risks of unowned AI-generated code in production and demonstrates how eBPF can intercept and control AI API traffic in Kubernetes. He explains how kernel-level socket hooks enable transparent prompt filtering, model swapping, token limits, and syscall restrictions to secure AI agents without modifying application source code or restarting containers. By Dan Finneran

Flux has introduced Flux Mirror, a CLI plugin that mirrors container images, Helm charts and OCI artifacts between registries from a declarative configuration. The plugin is part of the Flux v2.9 CLI plugin system and is presented as a way to keep Kubernetes clusters reconciling only from registries that teams operate themselves. By Matt Saunders

I Deliberately Destroyed My Kubernetes Cluster at 2 AM. Here's What Died First. Chaos engineering is not about breaking things. It's about discovering that your "production-grade" homelab is held together by hope and a single etcd snapshot before someone else finds out for you. The Setup I was lying in bed at 1:47 AM, staring at the ceiling, unable to sleep. Not because of caffeine. Because of a thought that had been gnawing at me for weeks: If one of my nodes died right now, would my cluster actually survive? I run a 4-node bare-metal Kubernetes cluster on Talos Linux. Dell OptiPlex control plane. Three Raspberry Pi workers. Cilium eBPF. ArgoCD. Longhorn distributed storage. Prometheus. Grafana. The whole cloud-native stack, shoehorned into $220 of scrap hardware and stubbornness. From the outside, it looks solid. ArgoCD syncs green. Cilium status shows healthy. Longhorn volumes are replicated across three nodes. I have etcd snapshots every 6 hours to S3. On paper, I'm resilient. But I had never actually tested it. Not a controlled test. Not a graceful node drain. I mean chaos . Sudden death. The kind of failure that happens at 3 AM when a power supply dies, or a kernel panics, or a neighbor's construction crew hits the wrong breaker. So I got out of bed, walked to my desk, and installed Chaos Mesh. Why Chaos Engineering on a Homelab? Professionally, I design AWS infrastructure with multi-AZ failover, auto-scaling groups, and managed services that abstract failure away. At Siemens, if an EKS node dies, the managed node group replaces it before I finish reading the alert. But my homelab has no managed control plane. No AWS SLA. No auto-repair. If a Pi's USB boot drive corrupts, that node is gone until I physically fix it. I needed to know: What dies first when a worker vanishes? Not "what should die" — what actually dies. Does Longhorn really failover? Three replicas sound great until you realize two of them were on the same node. Does Cilium handle network partitio

For years, Kubernetes has been marketed as the platform that solves infrastructure at scale. It automates deployments, recovers from failures, scales applications, and provides a consistent environment regardless of where workloads run. Yet talk to enough engineering teams, and you'll hear a very different story. "Our cloud bill doubled." "We're running twice as many worker nodes as expected." "Our platform team spends more time maintaining Kubernetes than improving it." The obvious conclusion is that Kubernetes is expensive. The more accurate conclusion is that most organizations are running Kubernetes inefficiently. After working with production environments across different industries, a pattern starts to emerge. Clusters rarely become expensive because of Kubernetes itself. They become expensive because of operational decisions that seem harmless in isolation but compound over time. Oversized resource requests. Poor workload scheduling. Underutilized nodes. Too many clusters. Autoscaling without proper observability. None of these are platform limitations. They're operational challenges. Kubernetes Is Surprisingly Efficient One misconception still persists: Kubernetes consumes too many resources. In reality, Kubernetes itself has a relatively small footprint. The real cost comes from the applications running inside it and, more importantly, from how those applications are configured. Consider a typical deployment: resources: requests: cpu: "2" memory: "4Gi" Nothing looks unusual here. The application starts, deployments succeed, and everything appears healthy. Then someone opens Grafana. Average CPU usage? 0.18 cores. Memory consumption? Less than 1 GB. The scheduler doesn't know that. It only knows what you've told it. If a pod requests two CPU cores, Kubernetes reserves two CPU cores when placing that workload. Even if the application spends most of its life almost idle, those resources remain unavailable for other workloads. Multiply that across hundreds of s

With the Azure Kubernetes Services (AKS) platform, containerized workloads can be efficiently managed and scaled. However, the full potential of AKS is only realized when it is seamlessly integrated with other Azure services. This enables a complete cloud-native environment that is scalable, secure, and automatable, while providing maximum flexibility. In this final post of the series, we will show you how AKS can be integrated with other Azure services to create a robust and holistic platform for your applications. Why Integrating AKS into the Azure Cloud Is Crucial AKS provides a highly available and scalable infrastructure for managing containerized applications. However, integrating it with other Azure services like Azure DevOps, Azure Monitor, Azure Active Directory (Entra ID), and Azure Storage extends functionality and optimizes workload management. By leveraging Azure services alongside AKS, companies can: Ensure enhanced security for their containerized applications. Build robust monitoring and logging solutions to monitor the state of applications at all times. Set up automated pipelines for deployment and scaling. Seamlessly exchange data and status information across various Azure services. Key Azure Services to Integrate with Your AKS Platform Azure Active Directory (AAD) for Authentication and Security Azure Active Directory (AAD) provides comprehensive identity and access management that can be directly integrated with AKS. This ensures that only authorized users and services can access your Kubernetes clusters. With Azure RBAC (Role-Based Access Control), you can define granular access permissions for different users and teams, increasing the security of your environment. AAD Pod Managed Identities enable your AKS applications to securely access Azure resources like Azure Key Vault or Azure Storage without the need to manually manage sensitive credentials. Azure DevOps for CI/CD Pipelines Azure DevOps is one of the best solutions for automating CI/CD

The Quest Begins (The “Why”) Honestly, I used to stare at my CI/CD yaml files like they were ancient runes. Every push felt like a gamble: “Will the build pass this time?” I’d spend Friday nights hunting down a missing node_modules cache in Jenkins, only to realize the agent had run out of disk space because I’d forgotten to add a cleanup step. The pain was real, and the feedback loop was slower than a dial‑up modem. I kept asking myself: Why does this feel like wrestling a dragon every time I want to ship a feature? The answer was simple—I hadn’t yet found a pipeline that just worked out of the box. I wanted something that gave me confidence, not anxiety. So I embarked on a quest to compare the three big contenders: GitHub Actions, GitLab CI, and good ol’ Jenkins. Spoiler: the treasure wasn’t in the tool itself, but in how you shape the pipeline around your team’s flow. The Revelation (The Insight) The big “aha!” moment came when I stopped treating CI/CD as a one‑size‑fits‑all script and started seeing it as a contract between my code and my environment. The contract says: Every commit gets a clean slate. Dependencies are restored, not guessed. Tests run in parallel, not sequentially. Artifacts are published only if the gate passes. When I wrote that contract down, the yaml stopped looking like magic incantations and started looking like a checklist. The tools differ in syntax, but the underlying principles are the same. Here’s the secret: cache wisely, fail fast, and keep the pipeline short enough to give you feedback before you’ve even finished your coffee. Wielding the Power (Code & Examples) Below are three pipelines—one for each platform—that embody the contract above. I’ll first show a “struggle” version (the common pitfalls) and then the victorious version. 1. GitHub Actions – The Struggle name : CI on : [ push , pull_request ] jobs : build : runs-on : ubuntu-latest steps : - uses : actions/checkout@v3 - name : Install deps run : npm install # <-- no cache,

The Quest Begins (The "Why") I remember the first time I tried to take a weekend side‑project from my laptop to something that felt “real”. I had a cute Express API that talked to Postman, a PostgreSQL container spun up with docker-compose up , and a React front‑end that lived in its own dev server. Everything worked beautifully … until I hit Ctrl+C on my laptop and the whole thing vanished. I needed a way to say, “Hey, keep this running even if I close my laptop, and if something crashes, bring it back up automatically.” I started poking at Docker Swarm, then Nomad, but the docs felt like reading ancient runes. That’s when a coworker slid over a Slack message: “Just try a Kind cluster. It’s K8s locally, and you’ll see why everyone talks about it.” Spoiler: it felt like discovering the secret level in a classic arcade game. Suddenly I could describe what I wanted my system to look like, and the cluster would make it happen — no more babysitting containers. The Revelation (The Insight) Kubernetes isn’t a mystical black box; it’s a declarative orchestrator . You tell it the desired state of your application (how many replicas, which image, what ports to expose) and it works relentlessly to match reality to that state. If a pod dies, Kubernetes spins up a new one. If you ask for three replicas and only two are running, it creates the missing pod. If you update the image tag, it rolls out the change pod‑by‑pod, keeping traffic flowing. Think of it like the save‑game system in a RPG: you define the story you want to experience, and the engine handles the gritty details of loading, saving, and recovering from crashes. The core objects you’ll meet early on are: Pod – the smallest deployable unit (one or more tightly coupled containers). Deployment – manages a set of identical pods, handles updates and rollbacks. Service – a stable network endpoint that load‑balances traffic to a set of pods. Ingress (optional) – exposes HTTP/HTTPS routes from outside the cluster to service

This is Level 5 of our Kubernetes networking series. So far, we've built up a solid foundation: LEVEL 1 — Pod networking LEVEL 2 — Pod-to-Pod communication LEVEL 3 — Service (a stable internal endpoint) LEVEL 4 — DNS (service name → Service IP) But we still have a glaring gap: how does a real user on the internet actually reach your Kubernetes application? That's exactly what this article covers — Ingress, Ingress Controllers, and the newer Gateway API. Table of Contents The Problem: The Internet Can't Reach a ClusterIP The Basic Solution: Ingress and Gateway API What Is Ingress? A Routing Example Ingress Is Not the Actual Proxy A Simple Analogy: Traffic Police A Basic Ingress YAML Example Breaking Down the Key Fields Host-Based Routing Path-Based Routing Why Not Just Use a LoadBalancer Service for Everything? The Complete Traffic Flow Where Does DNS Fit In? The Ingress Controller A Typical Architecture Ingress vs Service Ingress vs LoadBalancer Service HTTPS and TLS Termination Why Terminate TLS at the Edge? Referencing a TLS Certificate Routing Multiple Domains The Gateway API GatewayClass, Gateway, and HTTPRoute Ingress vs Gateway API Important Distinctions: Ingress Is Not CNI or Service Troubleshooting Ingress Layer by Layer Common Ingress Mistakes The Complete Kubernetes Networking Picture (Levels 1–5) The Mental Model to Memorize Level 5 Checkpoint What's Next: NetworkPolicy The Problem: The Internet Can't Reach a ClusterIP Suppose you want users to reach your application at myapp.example.com . Inside your cluster, you have: Service : frontend ClusterIP : 10.96.20.10 frontend Service ├── Pod 1 ├── Pod 2 └── Pod 3 A user on the internet can't simply visit http://10.96.20.10 — that's a private Kubernetes Service IP, invisible outside the cluster. We need something sitting at the edge of the cluster to bridge that gap. The Basic Solution: Ingress and Gateway API Historically, Kubernetes solved this with Ingress . More recently, Kubernetes introduced a more expres

How we turned always-on Kafka sinks into on-demand workers that shrug off nightly bombardments — by scaling on the right signal, tuning per-pod drain rate, and keeping autoscaling from sabotaging itself. Every number in this post is measured from a local lab you can run yourself — the full code is on GitHub , and the Appendix has the commands. The problem We run a fleet of Kafka sinks — consumer services that read change events from Kafka, apply business logic, and write the result into a service-local database as a query-friendly materialized view. It keeps reads fast and independent from upstream systems, and it's a great pattern. But the workload has an awkward shape. Most sinks are idle most of the day, then buried in minutes. Traffic isn't steady: changes arrive in bursts, usually from nightly imports or CDC jobs. The rest of the day the topic is quiet. topic activity over 24h msgs ▲ │ ██ nightly import / CDC burst │ ██ │______________██______________ flat, idle ~22h/day └───────────────────────────────▶ time That shape creates two problems at once : Idle waste. When the topic is quiet, each sink still runs — it polls Kafka, holds connections, emits metrics, and occupies CPU and memory. Multiply one "small" sink across dozens of them and several regions, and you're paying around the clock for work that happens for a couple of hours a night. Spike lag. When the burst lands, a backlog builds fast. If consumers can't drain it quickly enough, consumer lag — the gap between what's been produced and what's been processed — climbs, and downstream reads start serving stale data. We want two things that sound contradictory: cost almost nothing when idle , and absorb the spike fast when it hits. Why the obvious autoscaler doesn't help The reflex is a Kubernetes Horizontal Pod Autoscaler (HPA) on CPU or memory. For sinks, that's the wrong signal. Sink work is I/O-bound : the consumer spends its time waiting on Kafka polls and database writes, not burning CPU. So when a ba