标签：#kafka

In this article, author Srikanth Mamidala discusses the data lake architecture used for analytics, reporting, and machine learning and shows how to manage the consumer lag metrics when using Kafka and Apache Hudi. By Srikanth Mamidala

A practical guide to building reliable event-driven systems in Go using Apache Kafka. Learn how to implement tiered retry strategies with delayed reprocessing, route permanently failed messages to dead letter queues in Golang with Sarama. Prerequisites What do you need to follow along? Working knowledge of Golang. Go & Docker installed on your PC. What is an Event-Driven Architecture? An Event-Driven Architecture (EDA) is a design approach where services communicate by producing and responding to events. Each service operates independently, producing or reacting to events as they happen. What are Events? An event is a record of something that has happened in a system, typically representing a state change or a significant action. An event contains data (payload) describing what happened. An example of an event could be: A user signing up for a service. A user placing an order in your system. Components of an Event-Driven Architecture To understand how events flow through a system, we need to know three key players: Event Producers : They are the sources of events. They generate and publish events like signup events, order placed events, etc. Producers generate events and transmit them to the rest of the system. They do not know who is listening for or handling the events. Event Brokers : They sit between producers and consumers, decoupling them so neither needs a direct connection to the other. Brokers receive event messages, maintain their chronological order, make them available for consumption, and route them to the right consumers. Apache Kafka is an example of an event broker, and it's the one we'll use throughout this guide. Event Consumers : They handle the processing tasks. They listen on event channels and react when an event they are subscribed to is published, then they process the event, which can include making API calls, updating a database, triggering other events, or logging information. The Complete Flow With those three pieces in place, the flow of

The textbook version of the transactional outbox is tight. You save the domain entity and an outbox row in one local transaction. A background scheduler picks up PENDING rows and publishes them to Kafka. You never publish inside the request thread — no dual-write, no atomicity breach. The pattern closes the consistency gap. Then you load-test it. I ran 1,000 authenticated requests through my event-driven platform in 70 seconds. The gateway returned 201 for every one of them. The outbox absorbed every row. The consumer drained everything. By every visible metric the system looked healthy. Underneath that health, I found three production-grade problems the textbook never mentioned. What a correct implementation looks like Before the problems, the shape of the solution. The outbox publisher runs on a @Scheduled virtual-thread worker: @Scheduled ( fixedDelay = 5000 ) @Transactional public void publishPendingEvents () { List < OutboxEvent > batch = outboxRepository . findTop20ByStatusOrderByCreatedAtAsc ( OutboxStatus . PENDING ); for ( OutboxEvent event : batch ) { event . setStatus ( OutboxStatus . PROCESSING ); outboxRepository . save ( event ); try { kafkaTemplate . send ( event . getTopic (), event . getPayload ()). get (); event . setStatus ( OutboxStatus . PUBLISHED ); } catch ( Exception e ) { event . incrementRetryCount (); if ( event . getRetryCount () >= MAX_RETRIES ) { event . setStatus ( OutboxStatus . FAILED ); } else { event . setStatus ( OutboxStatus . PENDING ); } } outboxRepository . save ( event ); } } This is correct. The PROCESSING state prevents another scheduler instance from claiming the same row. The retry cap prevents infinite cycling. The PENDING fallback on transient errors gives the event another chance. The dual-write problem is genuinely closed. Here is what that correctness does not cover. Gap 1: Your throughput ceiling is a config line fixedDelay = 5000 means the scheduler runs every 5 seconds. findTop20 means it picks up 20 rows per cycl

How we turned always-on Kafka sinks into on-demand workers that shrug off nightly bombardments — by scaling on the right signal, tuning per-pod drain rate, and keeping autoscaling from sabotaging itself. Every number in this post is measured from a local lab you can run yourself — the full code is on GitHub , and the Appendix has the commands. The problem We run a fleet of Kafka sinks — consumer services that read change events from Kafka, apply business logic, and write the result into a service-local database as a query-friendly materialized view. It keeps reads fast and independent from upstream systems, and it's a great pattern. But the workload has an awkward shape. Most sinks are idle most of the day, then buried in minutes. Traffic isn't steady: changes arrive in bursts, usually from nightly imports or CDC jobs. The rest of the day the topic is quiet. topic activity over 24h msgs ▲ │ ██ nightly import / CDC burst │ ██ │______________██______________ flat, idle ~22h/day └───────────────────────────────▶ time That shape creates two problems at once : Idle waste. When the topic is quiet, each sink still runs — it polls Kafka, holds connections, emits metrics, and occupies CPU and memory. Multiply one "small" sink across dozens of them and several regions, and you're paying around the clock for work that happens for a couple of hours a night. Spike lag. When the burst lands, a backlog builds fast. If consumers can't drain it quickly enough, consumer lag — the gap between what's been produced and what's been processed — climbs, and downstream reads start serving stale data. We want two things that sound contradictory: cost almost nothing when idle , and absorb the spike fast when it hits. Why the obvious autoscaler doesn't help The reflex is a Kubernetes Horizontal Pod Autoscaler (HPA) on CPU or memory. For sinks, that's the wrong signal. Sink work is I/O-bound : the consumer spends its time waiting on Kafka polls and database writes, not burning CPU. So when a ba

I thought this would be a simple task. I already had a Python Kafka producer running. Kafka was up in Docker. The topic existed, and I could send a message into it successfully. The next step sounded straightforward: Python Producer ↓ Kafka ↓ Spark Structured Streaming All I wanted Spark to do was read a JSON message from a Kafka topic. Instead, I ran into one error after another. At first, it looked like one problem: Spark cannot read Kafka. It was not one problem. It turned into a chain of failures across several different layers: Python / PySpark ↓ Spark runtime ↓ Kafka connector ↓ Hadoop / Windows ↓ Docker ↓ Kafka networking ↓ Ivy dependency resolution The useful part of this experience was not any single fix. It was learning how to separate the layers and stop treating every error as a problem in my Python code. This is the full debugging path. What I Was Building This was part of an financial data engineering project. The batch side of the project already looked roughly like this: Financial Data Source ↓ Python ingestion ↓ AWS S3 ↓ Snowflake ↓ dbt ↓ Financial anomaly models I wanted to add a streaming extension for newly arriving financial events. For the first version, I kept it intentionally simple: Python Kafka Producer ↓ Kafka topic: financial_events ↓ Spark Structured Streaming The producer sent a simulated financial event: { "company_id" : "COMPANY_001" , "company_name" : "Sample Company" , "report_type" : "quarterly_report" , "reporting_date" : "2026-08-08" , "event_id" : "FIN-20260808-001" , "source" : "simulated_financial_event" } Kafka accepted the message successfully. I could even read it with Kafka's console consumer. So Kafka itself was working. Then Spark entered the picture. Failure #1: PySpark Worked, but spark-submit Didn't I installed PySpark: pip install pyspark Then I installed Java 17 and verified it: java -version After reopening my terminal, Java was available. I tested Spark directly through Python: python -c "from pyspark.sql import S

You put a lot of work into authentication. A gateway validates the Keycloak JWT, maps realm roles to authorities, checks that the caller is allowed. By the time a request reaches your service, you know exactly who is calling. Then the request crosses into async land, and all of that evaporates. This is the story of the point where identity quietly disappears in an event-driven system, why the dead-letter queue is the worst possible place for it to disappear, and how I made the acting user as durable and replay-safe as the event itself. The flow everyone believes is fine The platform is a set of Spring Boot services: an API gateway in front, a user-service on MySQL, a notification-service on PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent. Authentication is handled at the edge. The gateway is an OAuth2 Resource Server; it validates the token once and propagates the caller's identity downstream as headers: // api-gateway — IdentityPropagationFilter (@Order(2), after security) IdentityContext identity = identityContextExtractor . extract ( jwt ); // Always set all three headers (empty when absent) to mask any spoofed values. enrichedRequest . putHeader ( IdentityHeaders . USER_NAME , nullToEmpty ( identity . username ())); enrichedRequest . putHeader ( IdentityHeaders . USER_EMAIL , nullToEmpty ( identity . email ())); enrichedRequest . putHeader ( IdentityHeaders . USER_ROLES , identity . rolesAsString ( DELIM )); One detail here matters more than it looks. The headers are always overwritten , even when a claim is absent. If a client tries to inject X-User-Name: admin on the inbound request, the gateway stomps it with the validated value (or empty). Downstream trust in those headers is only safe because the perimeter guarantees they cannot be forged. Miss that, and you've built an impersonation API. So far, so good. The synchronous hop carries identity. The problem starts one line later. The hidden f

You attach the OpenTelemetry Java agent, point it at a collector, and within minutes Grafana is drawing a service map you never drew. A box for each service, arrows between them, latency on every edge. It feels like magic, and — more dangerously — it feels complete . "The agent traces everything" is the sentence repeated in every onboarding doc. This is the story of the moment that sentence stopped being true on my platform, why I'm glad it did, and the difference between a system that is working and a system you can actually see . The flow everyone trusts The platform is an event-driven set of Spring Boot services: an API gateway in front, a user-service backed by MySQL, a notification-service backed by PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent. I didn't want to draw that topology. A hand-drawn architecture diagram is documentation that drifts — true the day you commit it, slightly wrong a month later, actively misleading after a quarter. I wanted the dependency graph generated from live traffic , so it would always reflect what the system actually does. Grafana Tempo does exactly this. Its service-graphs processor reads matched client/server span pairs out of trace data and emits a metric — traces_service_graph_request_total — that Grafana renders as a node graph. No edge is ever wired by hand. The topology is derived, continuously, from real spans. The edge that wasn't there I generated the graph and the synchronous edges lit up immediately: api-gateway → user-service user-service → MySQL notification-service → PostgreSQL Then I looked for the one edge I actually cared about — user-service → notification-service , the asynchronous hop over Kafka. It wasn't there. The naive conclusion (and why it's wrong) The tempting read is immediate and obvious: the async hop is broken. The event isn't getting across. Go debug the consumer. So I checked. And the consumer was completely fine. notification

In modern Data Engineering, there is an unspoken fetish for "Real-Time." If you ask any business stakeholder how fast they need their dashboard to update, the default answer will always be: "As fast as possible." This drives well-intentioned engineers to design incredibly complex architectures. We spin up Kafka clusters, implement Flink, and wrestle with latency, late-arriving data, and tumbling windows. All to have data flowing in milliseconds. But the harsh reality is that the vast majority of companies are building Ferraris just to sit in rush-hour traffic. 1. The Actionability Gap (The Golden Question) The biggest mistake when choosing a streaming architecture isn't technical; it's a business mistake. Before implementing real-time pipelines, the only question that matters is: "Does the company have the operational capacity to make a decision in milliseconds?" If you are building a credit card fraud detection system or a live e-commerce recommendation engine, yes, every millisecond counts. But if the data is feeding a financial dashboard that the executive board only reviews during their Monday morning meeting, updating that screen every second is a colossal waste of money and effort. Real-time data has zero value if the human action is batch. 2. The Hidden Complexity and the Cloud Bill Batch processing is forgiving. If a pipeline fails at 3 AM, you trigger a rerun, and by 8 AM, everything is fine. Batch is cheap, predictable, and easy to debug. Streaming, on the other hand, is unforgiving. Handling application state, event duplication (exactly-once semantics), out-of-order events, and sudden traffic spikes requires a senior engineering team dedicated solely to keeping the infrastructure alive. Furthermore, the cloud bill for 24/7 continuous processing is orders of magnitude higher than spinning up your compute clusters on a schedule. 3. "Micro-Batch" Solves 99% of Your Problems There is a perfect middle ground that the hype industry tries to ignore: the micro-ba

By Swetha Golla · 8 min read · Senior Application Architect 🔗 This post has a live interactive version with a clickable per-service verdict and the full comparison matrix: read it here TL;DR Need strict per-key ordering and replay? That's a log, not a queue — Kinesis or MSK. Pick MSK if you need real Kafka wire-protocol compatibility (existing clients, Kafka Streams, ksqlDB, Debezium); pick Kinesis if you'd rather AWS own shard mechanics and you're fine with its API. Need routing logic based on event content, not raw throughput? EventBridge — pattern-matching rules to many differently-interested targets, not identical delivery to everyone. Need a simple durable buffer between one producer and one consumer group? SQS. Need the same message fanned out to many independent subscribers? SNS — often paired with SQS underneath. Already running RabbitMQ, or need AMQP-specific routing? Amazon MQ for RabbitMQ is a lift-and-shift, not a rearchitecture. The expensive mistake isn't picking a slightly-suboptimal service — it's picking a queue when you needed a log, or the reverse. That's a rewrite, not a config change. The setup Scope note: this is a decision matrix for AWS's own catalog, not a survey of every messaging technology that exists. Self-hosted Kafka, Google Pub/Sub, Azure Service Bus, NATS, Pulsar, and plenty of others solve overlapping problems outside AWS's walls — worth knowing about, out of scope here. A platform team is replacing a single overloaded RabbitMQ broker that has become the answer to every "how do services talk to each other" question for three years running: order events, fraud signals, audit trails, third-party webhooks, and a slow-growing analytics pipeline all queue through it. It works, until it doesn't — a queue depth spike during a promotion in 2025 backed up every consumer behind it, including ones that had nothing to do with the promotion. The team's instinct is to "move it all to AWS-native," as one service. That instinct is the mistake. Thes

Video streaming has become a normal part of everyday life. Whether you are watching a live sports event, attending an online class, listening to internet radio, or enjoying a movie on a streaming platform, a complex technology system is working behind the scenes to deliver content smoothly. Most viewers simply press Play and start watching. They do not see the technology that makes videos load quickly, reduce buffering, and automatically adjust quality when internet conditions change. One of the most important technologies behind modern streaming is HTTP Live Streaming (HLS) . HLS is a widely used video streaming protocol that delivers high-quality audio and video across different devices and network conditions. Instead of sending one large video file, HLS divides content into smaller pieces called media segments and delivers them continuously while the viewer watches. For example, when a video automatically changes from 1080p to 720p during a slow internet connection without stopping completely, that experience is powered by Adaptive Bitrate Streaming (ABR) , one of the main features of HLS. In this guide, you will learn: What HLS Streaming is How HTTP Live Streaming works Why Apple created the HLS protocol How M3U8 playlists control video delivery How media segments are created How Adaptive Bitrate Streaming improves playback Where HLS is commonly used How HLS compares with other streaming technologies Whether you are a beginner learning about video technology or a developer exploring streaming protocols, this guide explains HLS step by step. What Is HLS Streaming? HTTP Live Streaming (HLS) is a video streaming protocol created by Apple that delivers audio and video content through standard HTTP and HTTPS connections. Unlike traditional video downloads, HLS does not send a complete video file at once. Instead, it breaks the content into many smaller parts called media segments and sends them one by one while the viewer is watching. This approach provides several a

Installing Apache Kafka 4.2 on Ubuntu 24.04 (WSL2) Using KRaft Mode: A Complete Step-by-Step Guide Learn how to install Apache Kafka 4.2 in KRaft mode, understand its architecture, create topics, produce and consume messages, and troubleshoot common configuration issues—all without ZooKeeper. 🚀 Introduction Apache Kafka has become the de facto standard for building event-driven , real-time , and high-throughput applications. Whether you're processing millions of financial transactions, collecting application logs, streaming IoT sensor data, or connecting microservices, Kafka provides a scalable and reliable messaging platform. Until recently, setting up Kafka required running Apache ZooKeeper alongside Kafka brokers. While powerful, ZooKeeper added operational complexity and introduced another distributed system that administrators had to manage. Beginning with recent Kafka releases, KRaft (Kafka Raft Metadata mode) removes this dependency by allowing Kafka to manage its own metadata internally. This makes installation simpler, reduces operational overhead, and improves scalability. In this guide, we'll install Apache Kafka 4.2 on Ubuntu 24.04.4 LTS (WSL2) , configure a single-node KRaft cluster, and walk through the complete lifecycle: Installing Kafka Understanding the Kafka architecture Configuring KRaft mode Starting the broker Creating topics Producing and consuming messages Troubleshooting common issues Understanding the purpose of each configuration parameter Rather than simply listing commands, I'll explain why each step is necessary so that you understand how Kafka works under the hood. What is Apache Kafka? Apache Kafka is a distributed event streaming platform designed to move data reliably and efficiently between applications. Instead of applications communicating directly with each other, they communicate through Kafka. A producing application writes messages to Kafka. Kafka stores those messages reliably. One or more consuming applications read those m

TL;DR Retailers process thousands of inventory transactions every second across physical stores, eCommerce platforms, warehouses, suppliers, and fulfillment centers. Yet many inventory systems still rely on scheduled synchronization, causing stock levels to become outdated within minutes. The result is overselling, delayed replenishment, inaccurate inventory visibility, and avoidable stockouts. Apache Kafka enables real-time inventory management by treating every inventory movement as an event that is streamed the moment it occurs. Sales, returns, warehouse transfers, supplier deliveries, and IoT sensor updates are continuously processed to maintain a consistent inventory view across all retail systems. This event-driven approach helps retailers improve inventory accuracy, automate replenishment, detect stockouts before they occur, and respond to changing demand in near real time. In this guide, you'll learn how Apache Kafka powers real-time inventory management, explore a production-ready reference architecture, understand how inventory events are processed across retail systems, and discover implementation best practices for building scalable, resilient inventory streaming applications. Introduction Retail inventory management has evolved far beyond tracking products on warehouse shelves. Today's retailers operate across physical stores, eCommerce platforms, online marketplaces, distribution centers, and supplier networks, where inventory levels change continuously throughout the day. Every sale, return, warehouse transfer, supplier delivery, and inventory adjustment impacts product availability, making accurate inventory visibility essential for delivering a seamless customer experience. However, many retailers still rely on scheduled synchronization between Point-of-Sale (POS) systems, Warehouse Management Systems (WMS), Enterprise Resource Planning (ERP) platforms, and online storefronts. While these systems perform different functions, they all depend on accur

Redpanda is a Kafka-API-compatible streaming platform written in C++ with no JVM and no ZooKeeper. This guide installs Redpanda on Ubuntu 24.04, secures it with a Let's Encrypt certificate and SASL/SCRAM authentication, tunes the kernel for production, verifies with a producer/consumer test, and exposes Redpanda Console behind Nginx basic auth. By the end, you'll have a secured, production-tuned single-node Redpanda cluster with a web console. Prerequisite: Ubuntu 24.04 server sized per Redpanda's CPU/memory requirements , non-root sudo user, and a domain A record (e.g. redpanda.example.com ). Install Redpanda $ sudo apt update $ curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash Warning: Only run vendor setup scripts you trust — piped curl | sudo bash runs with root privileges. $ sudo apt install redpanda -y $ rpk --version Open the Firewall Port Service Purpose 9092 Kafka API Producer/consumer traffic 8082 Pandaproxy (HTTP) REST access for non-Kafka clients 8081 Schema Registry Avro/Protobuf schema versioning 9644 Admin API Monitoring, config, health checks 33145 Internal RPC Inter-node communication $ sudo ufw allow 9092,8082,8081,9644,33145/tcp $ sudo ufw allow 80/tcp $ sudo ufw allow 443/tcp $ sudo ufw reload Issue a Let's Encrypt Certificate Redpanda ships with plaintext networking by default, fine for a lab, not for anything else. $ sudo apt install certbot -y $ DOMAIN = redpanda.example.com $ EMAIL = admin@example.com $ sudo certbot certonly --standalone -d $DOMAIN --non-interactive --email $EMAIL Certbot stores certs under /etc/letsencrypt/live , readable only by root. Redpanda runs as its own redpanda user, so copy the certs into a dedicated directory: $ sudo mkdir /etc/redpanda/certs $ sudo cp /etc/letsencrypt/live/ $DOMAIN /fullchain.pem /etc/redpanda/certs/node.crt $ sudo cp /etc/letsencrypt/live/ $DOMAIN /privkey.pem /etc/redpanda/certs/node.key $ sudo cp /etc/letsencrypt/live/ $DOMAIN /chain.pem /etc/re

Event-driven architecture promises scalability, but in Java-based real-time systems the tradeoffs only surface in production. Drawing on a Java/Kafka contact center platform handling 80k BHCC across 10k agents, this article details where the design breaks down—state management, partition limits, deduplication, JVM tuning, cascading consumer failures—and the Redis-backed patterns that fixed each. By Sagar Deepak Joshi

Most engineers don't think seriously about Kafka partitioning until something breaks in production. A topic that worked fine at low volume starts falling behind. Events that should be in order aren't. All of it traces back to a partitioning decision that was made quickly and never revisited. Why Partitioning Actually Matters Partitions are the unit of parallelism in Kafka. Every consumer in a group is assigned one or more partitions, and it processes those partitions alone. No two consumers in the same group share a partition. That means your partition count sets a hard ceiling on how many consumers can work in parallel: if you have 6 partitions, the 7th consumer in your group sits idle no matter how much load you're under. Partitioning also controls ordering. Within a single partition, events are strictly ordered. Across partitions, there are no guarantees. So how you distribute events across partitions determines what ordering guarantees your consumers can actually rely on. Get this wrong and you'll spend a long time debugging why events from the same user are being processed out of sequence. The partition key controls both of these things. It determines which partition an event lands in, and that decision has consequences that are expensive to reverse. Partitioning Strategies Partition by Key This is the most common strategy and the right default when ordering matters. You supply a key when producing an event, Kafka hashes it using the murmur2 algorithm, and takes the modulo against the partition count to decide where it lands. producer . send ( ' orders ' , key = b ' user_4821 ' , value = event ) Every event with the same key always lands in the same partition. That's what guarantees ordering within a key. All events for user_4821 go to partition 3 (or wherever the hash resolves), and your consumer reads them in the exact sequence they were produced. I default to this for almost everything I build now and only go keyless when I have a specific reason to. Use key

Atlassian details the Forge billing platform built for usage-based pricing across its cloud ecosystem. It processes large-scale usage events with correct attribution, deduplication, and aggregation using a streaming pipeline, idempotent processing, and layered storage to enable accurate billing, near real-time visibility, and reliable reconciliation across distributed services. By Leela Kumili

When a Spring Boot service needs to talk to another service without waiting on a synchronous HTTP call, message queues are the usual answer. Apache Kafka has become the default choice for this in most backend teams, but a lot of tutorials stop at a "hello world" producer and consumer that would never survive a real production load. Things like consumer retries, error handling, serialization of real objects, and graceful shutdown get skipped, and those are exactly the parts that page you at 2 a.m. In this tutorial, you will build a Spring Boot application that produces and consumes JSON messages over Kafka. You will configure a producer and a consumer, send a typed object instead of a plain string, handle deserialization errors so one bad message does not block your whole consumer group, and verify the whole thing works end to end. By the end, you will have a small but realistic messaging setup you can build on. Prerequisites To follow along, you will need: Java 17 or later installed. You can check your version by running java -version . A Spring Boot 3.x project. You can generate one at start.spring.io with the Spring for Apache Kafka dependency added. A running Kafka broker. The quickest way to get one locally is Docker, which the first step covers. Basic familiarity with Spring Boot, including how @Component and application.yml work. Step 1 — Running Kafka Locally with Docker Before writing any code, you need a broker to talk to. Running Kafka by hand involves Zookeeper, broker configuration, and a fair amount of setup, so you will use Docker Compose to bring up a single-broker cluster instead. Create a file named docker-compose.yml in your project root: services : kafka : image : apache/kafka:3.7.0 container_name : kafka ports : - " 9092:9092" environment : KAFKA_NODE_ID : 1 KAFKA_PROCESS_ROLES : broker,controller KAFKA_LISTENERS : PLAINTEXT://:9092,CONTROLLER://:9093 KAFKA_ADVERTISED_LISTENERS : PLAINTEXT://localhost:9092 KAFKA_CONTROLLER_LISTENER_NAMES : CONTRO

Last year I was on a team that pushed 40 million events per day through Kafka. We had consumer lag alerts, rebalancing incidents, and a whole runbook for when the broker got behind. I understood how to operate Kafka. But I did not understand how Kafka works. So I built a tiny one. No dependencies. No Zookeeper. No JVM. Just Python and the core ideas. Here is what I learned. The Three Things Kafka Actually Does People say "Kafka is a message queue." That is not quite right. Kafka is a distributed commit log . It has three jobs: Accept writes from producers and append them to a log Let consumers read from any offset in that log Remember where each consumer group is up to That third one is the thing that makes Kafka different from a traditional queue. A queue forgets a message once it is consumed. Kafka remembers. You can replay. You can have 10 different consumer groups reading the same topic at different speeds. The code to implement this is smaller than you think. brokelite: A Message Broker in 120 Lines import threading import time from collections import defaultdict from typing import Dict , List , Tuple class Partition : """ Append-only log for one partition of a topic. """ def __init__ ( self ): self . _log : List [ Tuple [ int , bytes ]] = [] # (offset, message) self . _lock = threading . Lock () self . _next_offset = 0 def append ( self , message : bytes ) -> int : with self . _lock : offset = self . _next_offset self . _log . append (( offset , message )) self . _next_offset += 1 return offset def read_from ( self , offset : int , max_count : int = 100 ) -> List [ Tuple [ int , bytes ]]: with self . _lock : return [ ( off , msg ) for off , msg in self . _log if off >= offset ][: max_count ] def __len__ ( self ): return self . _next_offset class Topic : """ A topic is just N partitions. """ def __init__ ( self , name : str , num_partitions : int = 3 ): self . name = name self . partitions = [ Partition () for _ in range ( num_partitions )] def route ( self , k

A conversation between Uncle (a backend architect) and Nephew (a curious developer) about events, publishers, subscribers, Redis Pub/Sub, RabbitMQ, and Kafka The Beginning: What is an Event? 👦 Nephew: Uncle, I see words like "Redis Pub/Sub", "RabbitMQ", "Kafka" everywhere in job descriptions. What do they all do? Are they the same thing? 👨‍🦳 Uncle: (smiles) No, they're different. But before I confuse you with names, let me ask you something. Have you ever watched the news on TV? 👦 Nephew: Yes uncle, every morning! 👨‍🦳 Uncle: Perfect! So when the news channel says "Breaking News: India won the cricket match" - what happened? 👦 Nephew: Something important occurred... and they announced it! 👨‍🦳 Uncle: Exactly! That "something important occurred" is called an Event . In software, when something happens - like a user placing an order, a payment succeeding, or a file uploading - that's an event. 👦 Nephew: So event = something that happened? 👨‍🦳 Uncle: Yes! Think of it as news. When you place an order at Zomato, that's an event. When you get a payment notification from Google Pay, that's an event. When someone follows you on Instagram, that's an event. 👦 Nephew: Okay, I get it. But uncle, why is this "event" concept important? I can just write code directly, right? 👨‍🦳 Uncle: Ah! That's where the real story begins... The Problem: Spaghetti Code 👨‍🦳 Uncle: Imagine you own a food delivery company. A customer places an order. Now, what all needs to happen? 👦 Nephew: Umm... save the order, send email confirmation, alert the restaurant? 👨‍🦳 Uncle: Good! Let me write the code without events: function placeOrder ( orderId ) { saveOrder ( orderId ); sendEmailConfirmation ( orderId ); sendSMSNotification ( orderId ); notifyRestaurant ( orderId ); updateAnalyticsDashboard ( orderId ); addLoyaltyPoints ( orderId ); updateInventory ( orderId ); } Now, one day your boss says "Also send WhatsApp notification". What do you do? 👦 Nephew: Add another function call in the same code? 👨‍🦳 Unc

İlk yazıda Event Driven Architecture’ın temel kavramlarını, Kafka üzerinde topic/channel tasarımını, event-command ayrımını, schema contract’ları ve producer-consumer ilişkisini ele aldık. Bu yazıda odağı bir adım ileri taşıyıp event’in platform içindeki yaşam döngüsüne bakacağız. Çünkü EDA tasarımında asıl zorluk yalnızca event üretmek değildir. Asıl mesele, üretilen event’in güvenilir, izlenebilir, tekrar işlenebilir, zenginleştirilebilir ve farklı tüketiciler tarafından kullanılabilir hale gelmesidir. Bu yazıda şu sorulara odaklanacağız: Ham event platforma geldiğinde ne olur? Event nasıl doğrulanır, zenginleştirilir ve tüketilebilir hale gelir? Raw, validated, enriched ve curated topic’ler nasıl konumlandırılmalıdır? Bu yapı modern lakehouse mimarilerindeki Medallion yaklaşımıyla nasıl ilişkilendirilebilir? DLQ ve alert topic’leri ne zaman devreye girer? Replay, idempotency, monitoring, security ve governance nasıl düşünülmelidir? Event Pipeline Nedir? EDA mimarilerinde özellikle data platform projelerinde event’ler genellikle bir yaşam döngüsünden geçer. Bu yaşam döngüsü şöyle modellenebilir: raw -> validated -> enriched -> curated | | v v dlq alert Bu yapı, veri akışının aşama aşama olgunlaşmasını sağlar. Raw topic kaynaktan gelen ham event’i taşır. Validated topic schema ve temel kalite kontrollerinden geçmiş event’leri içerir. Enriched topic event’in referans veriler veya başka veri kaynaklarıyla zenginleştirilmiş halidir. Curated topic ise tüketiciler için güvenilir, normalize edilmiş ve iş anlamı netleşmiş event’leri temsil eder. Event Pipeline ve Medallion Architecture İlişkisi Bu yapı, modern lakehouse mimarilerinde sık kullanılan Medallion yaklaşımıyla doğal bir benzerlik taşır. Lakehouse tarafında Bronze katmanı ham veriyi, Silver katmanı temizlenmiş ve zenginleştirilmiş veriyi, Gold katmanı ise iş tüketimine hazır veri ürünlerini temsil eder. Kafka üzerindeki raw, validated, enriched ve curated topic’leri de benzer bir olgunlaşma mantığını akan veri ü