标签：#distributedsystems

Last year I was on a team that pushed 40 million events per day through Kafka. We had consumer lag alerts, rebalancing incidents, and a whole runbook for when the broker got behind. I understood how to operate Kafka. But I did not understand how Kafka works. So I built a tiny one. No dependencies. No Zookeeper. No JVM. Just Python and the core ideas. Here is what I learned. The Three Things Kafka Actually Does People say "Kafka is a message queue." That is not quite right. Kafka is a distributed commit log . It has three jobs: Accept writes from producers and append them to a log Let consumers read from any offset in that log Remember where each consumer group is up to That third one is the thing that makes Kafka different from a traditional queue. A queue forgets a message once it is consumed. Kafka remembers. You can replay. You can have 10 different consumer groups reading the same topic at different speeds. The code to implement this is smaller than you think. brokelite: A Message Broker in 120 Lines import threading import time from collections import defaultdict from typing import Dict , List , Tuple class Partition : """ Append-only log for one partition of a topic. """ def __init__ ( self ): self . _log : List [ Tuple [ int , bytes ]] = [] # (offset, message) self . _lock = threading . Lock () self . _next_offset = 0 def append ( self , message : bytes ) -> int : with self . _lock : offset = self . _next_offset self . _log . append (( offset , message )) self . _next_offset += 1 return offset def read_from ( self , offset : int , max_count : int = 100 ) -> List [ Tuple [ int , bytes ]]: with self . _lock : return [ ( off , msg ) for off , msg in self . _log if off >= offset ][: max_count ] def __len__ ( self ): return self . _next_offset class Topic : """ A topic is just N partitions. """ def __init__ ( self , name : str , num_partitions : int = 3 ): self . name = name self . partitions = [ Partition () for _ in range ( num_partitions )] def route ( self , k

Quorum is an incident command plane built on Amazon Aurora DSQL. The failover story lives in another post. This one is about a narrower question that turned out to be the foundation: when several responders write to the same incident at the same moment, across regions, during the worst minutes of an outage, how do you guarantee the record never forks into two conflicting truths. The answer is two design choices that are really one choice seen from two angles: event sourcing, and DSQL's optimistic concurrency control. The data model is append-only Quorum is event-sourced across four tables. Every state change is an immutable event appended to a log, not an in-place update. The current state of an incident is a fold over its events. There is no UPDATE incidents SET status = ... ; there is an acknowledged event, a note event, a resolved event, and the status you render is computed from them. The event's UUID is its primary key and its idempotency key at the same time. A retried write carrying the same UUID cannot double-apply: the insert collides on the primary key and becomes a no-op. That property sounds minor until you remember what kind of system this is. A tool designed to survive network failure retries writes constantly, and "the responder tapped resolve twice because the first response was slow" must not produce two resolutions. Append-only also suits the domain directly. For an incident system the audit trail is the product, not a side effect. "Who acknowledged this, at what time, and what did the timeline look like at 02:14" is a first-class question for the post-incident review and a compliance requirement in regulated environments. Event sourcing gives you that for free. It also gives DSQL a write pattern it likes, which matters more than you would expect. The stack, briefly TypeScript end to end. Kysely as a typed query builder rather than an ORM, because I wanted type safety without surrendering control of the SQL: on a distributed database the exact shap

I Built a Consistent Hashing Ring in Pure Python and Finally Understood How Cassandra Distributes Data I've been using Cassandra and Redis Cluster for years. I knew consistent hashing was "how they work." But I never truly got it until I built one myself from scratch, in pure Python, with zero dependencies. This post is about what I learned doing that. The Problem Consistent Hashing Solves Imagine you have 3 servers and 1 million keys. The naive approach: server = hash(key) % 3 . It works great until you add or remove a server. Change 3 to 4, and almost every key remaps to a different server. In a caching layer, that means near 100% cache miss. In a database, it means massive data movement. That's the problem consistent hashing solves. When you add or remove a node, only a fraction of keys move. Specifically, 1/n of the keys, where n is the number of nodes. Building the Ring The core idea: place both nodes and keys on a circular number line from 0 to 2^32 (or any large integer). To find which node owns a key, walk clockwise until you hit a node. Here's the minimal version: import hashlib import bisect class ConsistentHashRing : def __init__ ( self , replicas = 150 ): self . replicas = replicas self . ring = {} # hash -> node name self . sorted_keys = [] # sorted hash positions def _hash ( self , key : str ) -> int : return int ( hashlib . md5 ( key . encode ()). hexdigest (), 16 ) def add_node ( self , node : str ): for i in range ( self . replicas ): virtual_key = f " { node } :vnode: { i } " h = self . _hash ( virtual_key ) self . ring [ h ] = node bisect . insort ( self . sorted_keys , h ) def remove_node ( self , node : str ): for i in range ( self . replicas ): virtual_key = f " { node } :vnode: { i } " h = self . _hash ( virtual_key ) del self . ring [ h ] idx = bisect . bisect_left ( self . sorted_keys , h ) self . sorted_keys . pop ( idx ) def get_node ( self , key : str ) -> str : if not self . ring : raise ValueError ( " Ring is empty " ) h = self . _hash

If you're building a modern data stack that requires either high-throughput transaction processing or large-scale analytical workloads, you've likely come across both Vertica and VoltDB (now rebranded as Volt Active Data). While both are distributed relational database management systems (RDBMS), they are architected for completely opposite use cases — choosing the wrong one can lead to 10x higher costs, missed latency SLAs, and poor application performance. In this guide, we break down every key difference between OpenText Vertica and Volt Active Data, with practical examples, real-world use cases, and best practices to help you make the right choice for your team. Table of Contents What is OpenText Vertica? What is Volt Active Data (Formerly VoltDB)? Core Differences Between Vertica and VoltDB Real-World Use Cases: When to Pick Which Best Practices & Common Mistakes Conclusion & Key Takeaways References What is OpenText Vertica? OpenText Vertica (formerly Micro Focus Vertica) is a columnar relational DBMS built exclusively for analytical (OLAP) workloads, first launched in 2005. As of 2026, the latest stable version is 26.1, with native lakehouse and Apache Iceberg export support for modern data ecosystems. Core Vertica Architecture Vertica's design is optimized for fast queries across massive datasets: Columnar storage : Data is stored by column instead of row, enabling significantly higher compression ratios and faster aggregation queries that only access a small subset of columns Massively Parallel Processing (MPP) : Query execution and data are distributed across hundreds of nodes for parallel processing Dual deployment modes : Enterprise Mode : Shared-nothing architecture with data stored locally on nodes for maximum performance Eon Mode : Compute and storage separated, using shared object storage (S3, GCS, ADLS) to scale compute independently of storage for cloud workloads Projections : Physical, sorted copies of data optimized for common query patterns (ins

Most developers never have to design a network protocol from scratch. You use HTTP, gRPC, WebSockets, or something else that already exists and has been debugged by thousands of people over many years. That is the right call for most situations. I did not take that path when building Vaylix, a key-value database engine. I designed a custom binary protocol called VTP2, and the process taught me things about networking that I would not have picked up any other way. This is not an argument that you should also build a custom protocol. For most things, you should not. This is an honest account of what I ran into. Why not HTTP The first question anyone reasonably asks is: why not just use HTTP? HTTP is everywhere. The tooling is excellent. Every language has a client. Debugging with curl is trivial. If I had used HTTP, I would have had working client libraries in a dozen languages before writing a single line of server code. The problem is that HTTP is stateless by design. Every request is independent. Every request carries headers. Every response carries headers. The model assumes that each round trip is a fresh conversation with no memory of what came before. A database session is the opposite of that. A client connects, authenticates, and then issues many commands over the same connection. The authentication should happen once. The session should carry state. Pipelining requests without waiting for each response to return should be natural, not something you fight the protocol to achieve. HTTP/2 closes some of this gap. But using HTTP/2 correctly for a stateful session model involves working against the grain of what HTTP was designed for. I would have been spending a lot of time on infrastructure that exists to make HTTP behave less like HTTP. The other issue is overhead. HTTP headers are verbose. For small key-value operations, the headers can easily exceed the payload. That felt wrong for something designed to be a tight operational data store. So I went with TCP d

I built a distributed compute grid where your idle laptop runs ML jobs — the orchestrator behind it The pitch: a single FastAPI hub takes compute jobs from ML researchers, and a fleet of home PCs and gaming rigs (RTX 4090s, M2 MacBooks, anything with a GPU and a Python interpreter) polls in, picks up work, and ships results back. A 20% platform fee funds the hub. An interactive dashboard shows the mesh in real time. I have been living inside this codebase for a few weeks. This post is about the part that actually determines whether the thing works or does not — the orchestrator . No frontend, no marketing — just the brain. Live dashboard: man44.zo.space/compute-pool Repo: github.com/AmSach/ComputePool-Grid The problem with "dumb" schedulers The first version of ComputeOrchestrator had a one-line bug that took down a 12-node stress test. Two jobs hit the hub at the same millisecond. Both saw the same node as idle . Both wrote busy to the same row. One node ended up double-allocated, the other starved, and the test logs looked like a hostage negotiation. The fix had to be three things at once: A scoring function that picks the right node, not just the first idle one. An async lock so concurrent submissions cannot race on a single node. A heartbeat monitor that reclaims nodes that ghosted. Here is what it looks like now. The scoring algorithm def _calculate_score ( self , capacity : Dict [ str , Any ], requirements : Dict [ str , Any ]) -> float : """ Heuristic for node-task matching. """ score = 0.0 if capacity . get ( " gpu_vram " , 0 ) >= requirements . get ( " min_vram " , 0 ): score += 10.0 if capacity . get ( " cpu_cores " , 0 ) >= requirements . get ( " min_cores " , 0 ): score += 5.0 return score The weights are deliberately lopsided. A node that satisfies a job's VRAM requirement gets a 2x bonus over a node that just barely has enough cores. The intuition: GPU work is the long pole. If you cannot fit the model in VRAM, nothing else matters, no matter how many

Introduction Modern cloud-native systems generate an enormous amount of telemetry data every second. Applications, containers, Kubernetes clusters, APIs, databases, and infrastructure components continuously emit metrics, logs, and traces to help engineering teams understand system behavior and troubleshoot issues. While observability has become essential for operating distributed systems reliably, it has also introduced a new challenge: managing the scale, cost, and quality of telemetry. OpenTelemetry (OTel) has emerged as the industry standard for collecting and processing observability data. It provides a vendor-neutral framework for instrumenting applications and exporting telemetry to different observability backends. However, simply adopting OpenTelemetry is not enough. Without proper optimization strategies, organizations often face excessive telemetry ingestion costs, noisy dashboards, high-cardinality metrics, trace overload, and inefficient debugging workflows. This article explores practical approaches for optimizing observability using OpenTelemetry. It focuses on metrics, logs, and traces individually while also discussing broader optimization strategies across the telemetry pipeline. Understanding the OpenTelemetry observability pipeline OpenTelemetry provides a unified framework for generating, collecting, processing, and exporting telemetry data. At its core, the OTel ecosystem consists of SDKs, instrumentation libraries, collectors, processors, and exporters. Applications generate telemetry using OpenTelemetry SDKs or auto-instrumentation agents. This telemetry is then sent to the OpenTelemetry Collector, which acts as a centralized telemetry processing layer. The collector can receive telemetry from multiple sources, enrich it with metadata, apply filtering or sampling, and export it to one or more observability backends. The observability pipeline typically follows this flow: Application → OTel SDK → OTel Collector → Observability Backend The Open

Have you ever wondered how Discord keeps your channel messages available even when a server goes down? Or how Amazon DynamoDB serves petabytes of data with single-digit millisecond latency? The unsung hero powering almost all these distributed systems is hashing — a simple but powerful technique that makes even load distribution, fast lookups, and seamless scaling possible. As more applications move to distributed cloud architectures, understanding hashing for distributed systems is no longer optional for developers. Choosing the wrong hashing algorithm can lead to cascading failures, cache stampedes, and expensive downtime. This guide breaks down every core hashing technique, real-world use cases, best practices, and common pitfalls to avoid in 2026. Table of Contents What is Hashing in Distributed Systems? Core Hashing Algorithms Explained Traditional Modulo Hashing Consistent Hashing Virtual Nodes (VNodes) Rendezvous Hashing (HRW) Jump Consistent Hash Maglev Hashing Multi-Probe Consistent Hashing Consistent Hashing with Bounded Loads Real-World Applications of Distributed Hashing Head-to-Head Algorithm Comparison Best Practices for Distributed Hashing Common Pitfalls to Avoid Conclusion References What is Hashing in Distributed Systems? Hashing in distributed systems is the practice of mapping data keys (e.g., user IDs, object keys, channel IDs) to server nodes using a deterministic hash function. The core goals are: Distribute load evenly across all nodes to avoid hotspots Enable fast lookups (O(1) or O(log N)) without a central coordinator Minimize data movement when nodes are added or removed during scaling Support fault tolerance by simplifying replication across nodes The simplest implementation is modulo-based hashing , where node_id = hash(key) % N and N is the total number of nodes. While trivial to implement, it suffers from a fatal flaw: the rehashing problem. When N changes (a node is added or removed), nearly all keys are remapped to new nodes, causin

Imagine you are building a cloud-native backend for a high-frequency trading platform or a core banking ledger. To ensure mathematical immutability and prevent silent data tampering, compliance mandates that every transaction for a specific financial account must be cryptographically chained. This means the signature of Transaction #50 must explicitly include the cryptographic hash of Transaction #49. You cannot sign them out of order, and the backend is strictly responsible for generating and validating this chain. This introduces a massive distributed systems headache: How do you enforce strict, sequential ordering while maintaining the concurrency required to scale a modern cloud architecture? Let's walk through the evolution of this system, deconstruct exactly how the standard event-driven approach fails in production, and examine the Staff-level architecture required to fix it. Phase 1: The MVP & The Database Bottleneck In the early days, traffic is low. An account might see one transaction every few minutes. The "Happy Path" is simple: The API receives a deposit request for Account A. The API queries Postgres for the last_signature_hash . The API computes the new hash in memory: SHA(last_hash + new_transaction_data) . The API writes the new transaction and updates the state. The Pitfall: The Thundering Herd To prevent two concurrent requests from reading the same previous hash, you wrap the database operation in a pessimistic lock: SELECT ... FOR UPDATE . This forces the database to serialize requests at the row level. When a massive partner bank initiates a bulk sync, dumping 5,000 transactions for a single corporate account onto the API in two seconds, 4,999 concurrent threads immediately hit the FOR UPDATE lock and block. The database connection pool is instantly exhausted, latency spikes platform-wide, and the MVP dies. The Insight: The database must be your last line of defense, not your primary queueing mechanism. Contention must be solved upstream in me

Most engineers build distributed systems hoping nothing breaks. Amazon S3 was engineered under the opposite assumption: that something is already broken, right now, and the system needs to be fine with that. That one mindset shift explains almost everything about how S3 works — and why it's one of the most reliable pieces of infrastructure on the planet. I went through a deep-dive conversation with Mai-Lan Tomsen Bukovec, VP of Data and Analytics at AWS, and extracted the engineering philosophy underneath the product. Not the marketing version. The real one. Here's what actually matters. 1. Hardware failure is not an emergency. It's Tuesday. S3 manages hundreds of exabytes of data across tens of millions of hard drives, spread across 120 Availability Zones in 38 AWS Regions. It currently stores over 500 trillion objects. At that scale, something is always failing. A disk here. A rack there. An availability zone every now and then. The math is unforgiving. So the S3 team made a deliberate architectural decision early: stop treating failure as an exception. Design it into the system as the baseline state. This means dedicated auditor and repair microservices run continuously in the background — not when something goes wrong, but always. They scan the entire fleet, inspect every byte of data, detect discrepancies, and trigger repairs automatically. No human in the loop. No incident ticket. No war room. There's also a specific property they engineer for called crash consistency — the system is designed so that after any fail-stop event, it automatically returns to a valid state without manual intervention. The failure happens. The system continues. Those two things are not in conflict. The system heals itself because it was designed to assume it's already sick. If you're building distributed systems and your failure handling is reactive — you only respond after something breaks — you've already lost. Design the repair loop as a first-class citizen, not an afterthought.

Blockchain performance is determined by consensus rules. There is no acceleration layer within the protocol. One of the most common misconceptions about blockchain technology is the belief that transaction speed can be dramatically increased through special tools, hidden settings, or external services. While applications can improve user experience and optimize how information is presented, they cannot change the fundamental rules that govern how blockchain networks process transactions. At the core of every blockchain is a consensus mechanism. Consensus is responsible for ensuring that independent participants agree on the validity and order of transactions before they become part of the permanent ledger. Whether a network uses Proof of Work, Proof of Stake, or another consensus model, transaction processing remains tied to the protocol rules that all participants follow. Every transaction moves through a structured lifecycle: submit → validate → confirm Submission introduces the transaction to the network. Validation ensures that the transaction complies with protocol requirements and contains legitimate data. Confirmation establishes agreement across the network and records the transaction as part of the blockchain. These stages are not optional. They are essential to maintaining consistency and trust within decentralized systems. Because blockchain performance is governed by consensus, there is no protocol-level acceleration layer that can bypass validation or force immediate finality. No application can override consensus. No service can remove verification requirements. No external process can alter the execution sequence established by the protocol. What users often interpret as slow performance is usually the result of network conditions such as congestion, validator workload, transaction prioritization, or fee market activity. These factors can influence confirmation times, but they do not change the underlying rules of the system. Blockchain networks are d

The Theorem That Changed How We Think About Databases In 2000, Eric Brewer stood at a conference and proposed a conjecture that would reshape distributed systems forever: "You can only guarantee two of these three properties at the same time: Consistency, Availability, and Partition Tolerance." Two years later, Seth Gilbert and Nancy Lynch proved it mathematically. It became known as the CAP Theorem — and every distributed system architect since has had to wrestle with it. It sounds abstract. But once you understand it, you'll never look at a database choice the same way again. You'll understand why Amazon DynamoDB and Google Spanner make opposite architectural choices. You'll know why your bank uses PostgreSQL while Twitter uses Cassandra. Let's break it down from first principles. The Three Properties C — Consistency Every read receives the most recent write, or an error. There's only one version of the truth — all nodes agree. Not the same consistency as ACID . CAP consistency (linearizability) means every read reflects the latest write across all nodes. ACID consistency means transactions don't violate database constraints. Different concepts, same confusing word. A — Availability Every request receives a non-error response — though it might not be the most recent data. The system is always up and answering. Note: "Available" in CAP doesn't mean "fast." It means "responds without error." A system that always returns a (possibly stale) answer is Available. P — Partition Tolerance The system continues operating even when network messages between nodes are lost or delayed. A partition is when part of your distributed system can't communicate with another part. The Unavoidable Truth: P Is Not Optional Here's the insight that makes CAP actually useful: In any real distributed system, partitions will happen. Networks fail. Cables get cut. Data centers lose connectivity. AWS regions go down. Since you must tolerate partitions (or have a single-server system, which does

E-food delivery is a trillion-dollar market . And most of that trillion is not going to farmers, store owners, or the people who actually move food around. It's going to the infrastructure layer sitting between them — the platform tax, the per-order cut, the SaaS subscription that charges you to exist inside someone else's garden. The walled garden isn't accidental. It's the product. What if food delivery was a protocol, not a platform? Not an app. Not a marketplace. A protocol — like HTTP, like SMTP — that any node can speak, that no single company owns, and that costs near zero to run. That's what DIFP is. The Djowda Interconnected Food Protocol. An open wire format for connecting food ecosystem participants — farms, stores, restaurants, wholesalers, delivery nodes, end users — directly to each other, without a platform in the middle extracting rent at every step. The spec covers: Presence & discovery — participants announce themselves to their spatial cell, others find them by location Orders, asks, and donations — not just commerce, but demand signals and surplus distribution in the same protocol Spatial routing — the MinMax99 grid maps the entire planet into ~500m cells; every message knows where it's going Decentralized registry — nodes find each other through a federated lobby system, no central server required Version 0.4 of the spec dropped a few weeks ago. Today we're publishing the first working implementation. The gRPC preview — what we built DIFP-gRPC is a skull implementation of the full protocol stack over gRPC. Thin, end-to-end, every domain wired — nothing production-hardened yet, everything clearly marked for what it is. What's inside difp.proto — the entire DIFP v0.4 spec as a single protobuf file. Two services, 30+ message types, the full DifpEnvelope wrapper with a oneof payload that covers every domain: message DifpEnvelope { string id = 1 ; string type = 2 ; // "trade.ask" | "presence.announce" | "node.ping" | … string version = 3 ; MessageSen

In part 1 , the single-document case was easy. In part 2 , two documents brought Write Skew, and we saw that even a native ACID transaction — snapshot isolation — lets it through. So teams reach for the reflex fix: a distributed lock — Redis-based, often a Redlock-style implementation. Acquire a lock on a key, do your Read → Modify → Write, release. On paper, you've finally serialized the critical section — operationally, at least. In practice, you've stepped on three mines. 1. Network latency Every guarded transaction now makes extra round-trips to Redis — before and after hitting your NoSQL store. You've doubled your coordination surface and taken a hard dependency on a second system being up, reachable, and fast on the hot path of every write. The "fast" database is now gated by the lock service. And the coupling bites harder than the average latency suggests: every Redis tail-latency spike becomes your write-latency spike — your p99 inherits Redis's p99 — and if Redis fails over mid-transaction, the lock you think you're holding can effectively vanish on the new primary, dropping you straight into the corruption case below. 2. Deadlock You can dodge deadlock entirely with a single coarse lock — but then every writer serializes on it, and you've thrown away the very concurrency you reached for NoSQL to get. So to keep throughput you go fine-grained, one lock per resource — and the moment an invariant touches more than one key (across this series, it always does), deadlock is back on the table: Transaction A locks key X, then needs Y. Transaction B locks Y, then needs X. Both block until timeout or intervention. The textbook cure — real deadlock detection, maintaining a wait-for graph across every lock holder and breaking cycles as they form — is a distributed-systems project in its own right: not something you bolt onto a cache you reached for precisely to save engineering time. So nobody builds it. Instead teams impose a standing discipline: always acquire locks

The argument sounds reasonable: fewer lines of code mean fewer bugs. Simpler to review, easier to...