标签：#keda

How we turned always-on Kafka sinks into on-demand workers that shrug off nightly bombardments — by scaling on the right signal, tuning per-pod drain rate, and keeping autoscaling from sabotaging itself. Every number in this post is measured from a local lab you can run yourself — the full code is on GitHub , and the Appendix has the commands. The problem We run a fleet of Kafka sinks — consumer services that read change events from Kafka, apply business logic, and write the result into a service-local database as a query-friendly materialized view. It keeps reads fast and independent from upstream systems, and it's a great pattern. But the workload has an awkward shape. Most sinks are idle most of the day, then buried in minutes. Traffic isn't steady: changes arrive in bursts, usually from nightly imports or CDC jobs. The rest of the day the topic is quiet. topic activity over 24h msgs ▲ │ ██ nightly import / CDC burst │ ██ │______________██______________ flat, idle ~22h/day └───────────────────────────────▶ time That shape creates two problems at once : Idle waste. When the topic is quiet, each sink still runs — it polls Kafka, holds connections, emits metrics, and occupies CPU and memory. Multiply one "small" sink across dozens of them and several regions, and you're paying around the clock for work that happens for a couple of hours a night. Spike lag. When the burst lands, a backlog builds fast. If consumers can't drain it quickly enough, consumer lag — the gap between what's been produced and what's been processed — climbs, and downstream reads start serving stale data. We want two things that sound contradictory: cost almost nothing when idle , and absorb the spike fast when it hits. Why the obvious autoscaler doesn't help The reflex is a Kubernetes Horizontal Pod Autoscaler (HPA) on CPU or memory. For sinks, that's the wrong signal. Sink work is I/O-bound : the consumer spends its time waiting on Kafka polls and database writes, not burning CPU. So when a ba