标签：#kafka

TL;DR Retailers process thousands of inventory transactions every second across physical stores, eCommerce platforms, warehouses, suppliers, and fulfillment centers. Yet many inventory systems still rely on scheduled synchronization, causing stock levels to become outdated within minutes. The result is overselling, delayed replenishment, inaccurate inventory visibility, and avoidable stockouts. Apache Kafka enables real-time inventory management by treating every inventory movement as an event that is streamed the moment it occurs. Sales, returns, warehouse transfers, supplier deliveries, and IoT sensor updates are continuously processed to maintain a consistent inventory view across all retail systems. This event-driven approach helps retailers improve inventory accuracy, automate replenishment, detect stockouts before they occur, and respond to changing demand in near real time. In this guide, you'll learn how Apache Kafka powers real-time inventory management, explore a production-ready reference architecture, understand how inventory events are processed across retail systems, and discover implementation best practices for building scalable, resilient inventory streaming applications. Introduction Retail inventory management has evolved far beyond tracking products on warehouse shelves. Today's retailers operate across physical stores, eCommerce platforms, online marketplaces, distribution centers, and supplier networks, where inventory levels change continuously throughout the day. Every sale, return, warehouse transfer, supplier delivery, and inventory adjustment impacts product availability, making accurate inventory visibility essential for delivering a seamless customer experience. However, many retailers still rely on scheduled synchronization between Point-of-Sale (POS) systems, Warehouse Management Systems (WMS), Enterprise Resource Planning (ERP) platforms, and online storefronts. While these systems perform different functions, they all depend on accur

Redpanda is a Kafka-API-compatible streaming platform written in C++ with no JVM and no ZooKeeper. This guide installs Redpanda on Ubuntu 24.04, secures it with a Let's Encrypt certificate and SASL/SCRAM authentication, tunes the kernel for production, verifies with a producer/consumer test, and exposes Redpanda Console behind Nginx basic auth. By the end, you'll have a secured, production-tuned single-node Redpanda cluster with a web console. Prerequisite: Ubuntu 24.04 server sized per Redpanda's CPU/memory requirements , non-root sudo user, and a domain A record (e.g. redpanda.example.com ). Install Redpanda $ sudo apt update $ curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash Warning: Only run vendor setup scripts you trust — piped curl | sudo bash runs with root privileges. $ sudo apt install redpanda -y $ rpk --version Open the Firewall Port Service Purpose 9092 Kafka API Producer/consumer traffic 8082 Pandaproxy (HTTP) REST access for non-Kafka clients 8081 Schema Registry Avro/Protobuf schema versioning 9644 Admin API Monitoring, config, health checks 33145 Internal RPC Inter-node communication $ sudo ufw allow 9092,8082,8081,9644,33145/tcp $ sudo ufw allow 80/tcp $ sudo ufw allow 443/tcp $ sudo ufw reload Issue a Let's Encrypt Certificate Redpanda ships with plaintext networking by default, fine for a lab, not for anything else. $ sudo apt install certbot -y $ DOMAIN = redpanda.example.com $ EMAIL = admin@example.com $ sudo certbot certonly --standalone -d $DOMAIN --non-interactive --email $EMAIL Certbot stores certs under /etc/letsencrypt/live , readable only by root. Redpanda runs as its own redpanda user, so copy the certs into a dedicated directory: $ sudo mkdir /etc/redpanda/certs $ sudo cp /etc/letsencrypt/live/ $DOMAIN /fullchain.pem /etc/redpanda/certs/node.crt $ sudo cp /etc/letsencrypt/live/ $DOMAIN /privkey.pem /etc/redpanda/certs/node.key $ sudo cp /etc/letsencrypt/live/ $DOMAIN /chain.pem /etc/re

Event-driven architecture promises scalability, but in Java-based real-time systems the tradeoffs only surface in production. Drawing on a Java/Kafka contact center platform handling 80k BHCC across 10k agents, this article details where the design breaks down—state management, partition limits, deduplication, JVM tuning, cascading consumer failures—and the Redis-backed patterns that fixed each. By Sagar Deepak Joshi

Most engineers don't think seriously about Kafka partitioning until something breaks in production. A topic that worked fine at low volume starts falling behind. Events that should be in order aren't. All of it traces back to a partitioning decision that was made quickly and never revisited. Why Partitioning Actually Matters Partitions are the unit of parallelism in Kafka. Every consumer in a group is assigned one or more partitions, and it processes those partitions alone. No two consumers in the same group share a partition. That means your partition count sets a hard ceiling on how many consumers can work in parallel: if you have 6 partitions, the 7th consumer in your group sits idle no matter how much load you're under. Partitioning also controls ordering. Within a single partition, events are strictly ordered. Across partitions, there are no guarantees. So how you distribute events across partitions determines what ordering guarantees your consumers can actually rely on. Get this wrong and you'll spend a long time debugging why events from the same user are being processed out of sequence. The partition key controls both of these things. It determines which partition an event lands in, and that decision has consequences that are expensive to reverse. Partitioning Strategies Partition by Key This is the most common strategy and the right default when ordering matters. You supply a key when producing an event, Kafka hashes it using the murmur2 algorithm, and takes the modulo against the partition count to decide where it lands. producer . send ( ' orders ' , key = b ' user_4821 ' , value = event ) Every event with the same key always lands in the same partition. That's what guarantees ordering within a key. All events for user_4821 go to partition 3 (or wherever the hash resolves), and your consumer reads them in the exact sequence they were produced. I default to this for almost everything I build now and only go keyless when I have a specific reason to. Use key

Atlassian details the Forge billing platform built for usage-based pricing across its cloud ecosystem. It processes large-scale usage events with correct attribution, deduplication, and aggregation using a streaming pipeline, idempotent processing, and layered storage to enable accurate billing, near real-time visibility, and reliable reconciliation across distributed services. By Leela Kumili

When a Spring Boot service needs to talk to another service without waiting on a synchronous HTTP call, message queues are the usual answer. Apache Kafka has become the default choice for this in most backend teams, but a lot of tutorials stop at a "hello world" producer and consumer that would never survive a real production load. Things like consumer retries, error handling, serialization of real objects, and graceful shutdown get skipped, and those are exactly the parts that page you at 2 a.m. In this tutorial, you will build a Spring Boot application that produces and consumes JSON messages over Kafka. You will configure a producer and a consumer, send a typed object instead of a plain string, handle deserialization errors so one bad message does not block your whole consumer group, and verify the whole thing works end to end. By the end, you will have a small but realistic messaging setup you can build on. Prerequisites To follow along, you will need: Java 17 or later installed. You can check your version by running java -version . A Spring Boot 3.x project. You can generate one at start.spring.io with the Spring for Apache Kafka dependency added. A running Kafka broker. The quickest way to get one locally is Docker, which the first step covers. Basic familiarity with Spring Boot, including how @Component and application.yml work. Step 1 — Running Kafka Locally with Docker Before writing any code, you need a broker to talk to. Running Kafka by hand involves Zookeeper, broker configuration, and a fair amount of setup, so you will use Docker Compose to bring up a single-broker cluster instead. Create a file named docker-compose.yml in your project root: services : kafka : image : apache/kafka:3.7.0 container_name : kafka ports : - " 9092:9092" environment : KAFKA_NODE_ID : 1 KAFKA_PROCESS_ROLES : broker,controller KAFKA_LISTENERS : PLAINTEXT://:9092,CONTROLLER://:9093 KAFKA_ADVERTISED_LISTENERS : PLAINTEXT://localhost:9092 KAFKA_CONTROLLER_LISTENER_NAMES : CONTRO

Last year I was on a team that pushed 40 million events per day through Kafka. We had consumer lag alerts, rebalancing incidents, and a whole runbook for when the broker got behind. I understood how to operate Kafka. But I did not understand how Kafka works. So I built a tiny one. No dependencies. No Zookeeper. No JVM. Just Python and the core ideas. Here is what I learned. The Three Things Kafka Actually Does People say "Kafka is a message queue." That is not quite right. Kafka is a distributed commit log . It has three jobs: Accept writes from producers and append them to a log Let consumers read from any offset in that log Remember where each consumer group is up to That third one is the thing that makes Kafka different from a traditional queue. A queue forgets a message once it is consumed. Kafka remembers. You can replay. You can have 10 different consumer groups reading the same topic at different speeds. The code to implement this is smaller than you think. brokelite: A Message Broker in 120 Lines import threading import time from collections import defaultdict from typing import Dict , List , Tuple class Partition : """ Append-only log for one partition of a topic. """ def __init__ ( self ): self . _log : List [ Tuple [ int , bytes ]] = [] # (offset, message) self . _lock = threading . Lock () self . _next_offset = 0 def append ( self , message : bytes ) -> int : with self . _lock : offset = self . _next_offset self . _log . append (( offset , message )) self . _next_offset += 1 return offset def read_from ( self , offset : int , max_count : int = 100 ) -> List [ Tuple [ int , bytes ]]: with self . _lock : return [ ( off , msg ) for off , msg in self . _log if off >= offset ][: max_count ] def __len__ ( self ): return self . _next_offset class Topic : """ A topic is just N partitions. """ def __init__ ( self , name : str , num_partitions : int = 3 ): self . name = name self . partitions = [ Partition () for _ in range ( num_partitions )] def route ( self , k

A conversation between Uncle (a backend architect) and Nephew (a curious developer) about events, publishers, subscribers, Redis Pub/Sub, RabbitMQ, and Kafka The Beginning: What is an Event? 👦 Nephew: Uncle, I see words like "Redis Pub/Sub", "RabbitMQ", "Kafka" everywhere in job descriptions. What do they all do? Are they the same thing? 👨‍🦳 Uncle: (smiles) No, they're different. But before I confuse you with names, let me ask you something. Have you ever watched the news on TV? 👦 Nephew: Yes uncle, every morning! 👨‍🦳 Uncle: Perfect! So when the news channel says "Breaking News: India won the cricket match" - what happened? 👦 Nephew: Something important occurred... and they announced it! 👨‍🦳 Uncle: Exactly! That "something important occurred" is called an Event . In software, when something happens - like a user placing an order, a payment succeeding, or a file uploading - that's an event. 👦 Nephew: So event = something that happened? 👨‍🦳 Uncle: Yes! Think of it as news. When you place an order at Zomato, that's an event. When you get a payment notification from Google Pay, that's an event. When someone follows you on Instagram, that's an event. 👦 Nephew: Okay, I get it. But uncle, why is this "event" concept important? I can just write code directly, right? 👨‍🦳 Uncle: Ah! That's where the real story begins... The Problem: Spaghetti Code 👨‍🦳 Uncle: Imagine you own a food delivery company. A customer places an order. Now, what all needs to happen? 👦 Nephew: Umm... save the order, send email confirmation, alert the restaurant? 👨‍🦳 Uncle: Good! Let me write the code without events: function placeOrder ( orderId ) { saveOrder ( orderId ); sendEmailConfirmation ( orderId ); sendSMSNotification ( orderId ); notifyRestaurant ( orderId ); updateAnalyticsDashboard ( orderId ); addLoyaltyPoints ( orderId ); updateInventory ( orderId ); } Now, one day your boss says "Also send WhatsApp notification". What do you do? 👦 Nephew: Add another function call in the same code? 👨‍🦳 Unc

İlk yazıda Event Driven Architecture’ın temel kavramlarını, Kafka üzerinde topic/channel tasarımını, event-command ayrımını, schema contract’ları ve producer-consumer ilişkisini ele aldık. Bu yazıda odağı bir adım ileri taşıyıp event’in platform içindeki yaşam döngüsüne bakacağız. Çünkü EDA tasarımında asıl zorluk yalnızca event üretmek değildir. Asıl mesele, üretilen event’in güvenilir, izlenebilir, tekrar işlenebilir, zenginleştirilebilir ve farklı tüketiciler tarafından kullanılabilir hale gelmesidir. Bu yazıda şu sorulara odaklanacağız: Ham event platforma geldiğinde ne olur? Event nasıl doğrulanır, zenginleştirilir ve tüketilebilir hale gelir? Raw, validated, enriched ve curated topic’ler nasıl konumlandırılmalıdır? Bu yapı modern lakehouse mimarilerindeki Medallion yaklaşımıyla nasıl ilişkilendirilebilir? DLQ ve alert topic’leri ne zaman devreye girer? Replay, idempotency, monitoring, security ve governance nasıl düşünülmelidir? Event Pipeline Nedir? EDA mimarilerinde özellikle data platform projelerinde event’ler genellikle bir yaşam döngüsünden geçer. Bu yaşam döngüsü şöyle modellenebilir: raw -> validated -> enriched -> curated | | v v dlq alert Bu yapı, veri akışının aşama aşama olgunlaşmasını sağlar. Raw topic kaynaktan gelen ham event’i taşır. Validated topic schema ve temel kalite kontrollerinden geçmiş event’leri içerir. Enriched topic event’in referans veriler veya başka veri kaynaklarıyla zenginleştirilmiş halidir. Curated topic ise tüketiciler için güvenilir, normalize edilmiş ve iş anlamı netleşmiş event’leri temsil eder. Event Pipeline ve Medallion Architecture İlişkisi Bu yapı, modern lakehouse mimarilerinde sık kullanılan Medallion yaklaşımıyla doğal bir benzerlik taşır. Lakehouse tarafında Bronze katmanı ham veriyi, Silver katmanı temizlenmiş ve zenginleştirilmiş veriyi, Gold katmanı ise iş tüketimine hazır veri ürünlerini temsil eder. Kafka üzerindeki raw, validated, enriched ve curated topic’leri de benzer bir olgunlaşma mantığını akan veri ü

A survival guide for when everything goes wrong in production. There's a moment every engineer who works with Kafka experiences. You check the producer. Messages are sending. You check the consumer. Nothing. The consumer group shows zero lag because there's nothing to lag behind — as far as the consumer knows, the topic is empty. But it's not empty. The messages are there. Somewhere. In some partition, at some offset, behind some configuration you set six months ago and forgot about. Kafka doesn't lose messages. But it's very good at hiding them from you. Consumer Lag: The Number Everyone Watches Wrong Consumer lag is the difference between the latest offset in a partition and the offset your consumer group has committed. Simple concept. Dangerous in practice. The mistake: treating lag as a single number. Lag is per-partition. If you have 30 partitions and one consumer is stuck on partition 17 while the others are healthy, the total lag looks manageable. But partition 17's data is hours behind, and whatever downstream system depends on that data is serving stale results. Monitor lag per partition. Tools like Burrow, Kafka Exporter for Prometheus, or even kafka-consumer-groups.sh --describe break it down. If one partition's lag is growing while others are stable, you have a stuck consumer, a hot partition, or a poison message. A poison message is a record your consumer can't process — malformed data, unexpected schema, null where it shouldn't be null. The consumer throws an exception, the offset doesn't commit, and it retries the same message forever. Lag grows. The consumer looks "alive" because it's processing — just not making progress. The fix: dead letter queues. After N retries, move the message to a separate topic, commit the offset, and move on. Alert on the dead letter topic. Investigate later. Don't let one bad record block millions of good ones. Rebalance Storms: The Silent Killer Consumer rebalancing is Kafka's mechanism for redistributing partitions acro

This article explores Kafka's transition toward a cloud-native architecture, examining how tiered storage, FinOps telemetry, elastic consumer scaling, virtual clusters, and Share Groups reshape the operational and economic model of event streaming platforms. It also analyzes emerging diskless-storage proposals and their architectural trade-offs. By Viquar Khan

This week's Java roundup for May 18th, 2026, features news highlighting: GA releases of WildFly 40, Micronaut 5.0, Maven Embedded GlassFish Plugin 8.0 and Apache Fory 1.0; the May 2026 edition of Open Liberty; point releases of Gatherers4j, Apache and Kafka; and the seventh milestone release of Spring AI 2.0. By Michael Redlich

Schema proliferation builds slowly and gets expensive fast. One schema per event type feels right until there are ten tables, union queries spanning all of them, and a single field rename touching every schema. Discriminator-based schema consolidation collapses that to two tables, turning multi-table unions into a single query, while new variants are additive and don't break existing consumers. By Spoorthi Basu