标签：#kafka

A survival guide for when everything goes wrong in production. There's a moment every engineer who works with Kafka experiences. You check the producer. Messages are sending. You check the consumer. Nothing. The consumer group shows zero lag because there's nothing to lag behind — as far as the consumer knows, the topic is empty. But it's not empty. The messages are there. Somewhere. In some partition, at some offset, behind some configuration you set six months ago and forgot about. Kafka doesn't lose messages. But it's very good at hiding them from you. Consumer Lag: The Number Everyone Watches Wrong Consumer lag is the difference between the latest offset in a partition and the offset your consumer group has committed. Simple concept. Dangerous in practice. The mistake: treating lag as a single number. Lag is per-partition. If you have 30 partitions and one consumer is stuck on partition 17 while the others are healthy, the total lag looks manageable. But partition 17's data is hours behind, and whatever downstream system depends on that data is serving stale results. Monitor lag per partition. Tools like Burrow, Kafka Exporter for Prometheus, or even kafka-consumer-groups.sh --describe break it down. If one partition's lag is growing while others are stable, you have a stuck consumer, a hot partition, or a poison message. A poison message is a record your consumer can't process — malformed data, unexpected schema, null where it shouldn't be null. The consumer throws an exception, the offset doesn't commit, and it retries the same message forever. Lag grows. The consumer looks "alive" because it's processing — just not making progress. The fix: dead letter queues. After N retries, move the message to a separate topic, commit the offset, and move on. Alert on the dead letter topic. Investigate later. Don't let one bad record block millions of good ones. Rebalance Storms: The Silent Killer Consumer rebalancing is Kafka's mechanism for redistributing partitions acro

This article explores Kafka's transition toward a cloud-native architecture, examining how tiered storage, FinOps telemetry, elastic consumer scaling, virtual clusters, and Share Groups reshape the operational and economic model of event streaming platforms. It also analyzes emerging diskless-storage proposals and their architectural trade-offs. By Viquar Khan

This week's Java roundup for May 18th, 2026, features news highlighting: GA releases of WildFly 40, Micronaut 5.0, Maven Embedded GlassFish Plugin 8.0 and Apache Fory 1.0; the May 2026 edition of Open Liberty; point releases of Gatherers4j, Apache and Kafka; and the seventh milestone release of Spring AI 2.0. By Michael Redlich

Schema proliferation builds slowly and gets expensive fast. One schema per event type feels right until there are ten tables, union queries spanning all of them, and a single field rename touching every schema. Discriminator-based schema consolidation collapses that to two tables, turning multi-table unions into a single query, while new variants are additive and don't break existing consumers. By Spoorthi Basu