标签：#systemdesign

"Finding something quickly and finding the best thing quickly are two completely different engineering problems." So far in this series, we've explored one of the most common behaviours in software systems: Fast lookup. Whenever a system already knows what it's looking for—a User ID, Product ID, Order ID or Session ID—a HashMap becomes an excellent choice. But not every software problem works this way. Imagine you're building a ride-sharing application. A rider requests a cab. The system doesn't already know which driver to assign. Instead, it must answer a different question: "Out of all available drivers, who is the best choice?" Now consider a task scheduler. Hundreds of jobs are waiting to run. The scheduler doesn't ask: "Find Job #123." It asks: "Which job should run next?" Or imagine a gaming platform. Thousands of players are competing. Nobody asks: "Find Player ID 1057." Instead, users ask: "Who are the top 10 players?" These problems are fundamentally different from fast lookup. They're not about finding a specific object . They're about finding the best object according to some priority. This shift in thinking introduces another important design behaviour. Fast Lookup vs Best Selection Let's compare two different requirements. Requirement 1 Customer ID = 1052 ↓ Retrieve Customer The system already knows exactly what it needs. The challenge is retrieving it efficiently. Requirement 2 Available Drivers ↓ Find Nearest Driver ↓ Assign Ride The system doesn't know the answer yet. It must compare multiple candidates before making a decision. These two behaviours may look similar. In reality, they solve completely different engineering problems. Every Software System Doesn't Search the Same Way Consider these questions. Find Order #50231 versus Find the highest priority order. Or: Retrieve Product ID = P1042 versus Recommend the most popular product. Or: Find Employee ID = 2107 versus Find the employee with the highest sales this month. The first question always

This article describes an anonymized enterprise implementation. Company names, internal domains, repository identifiers, ticket numbers, and proprietary control names have been intentionally removed or generalized. Multi-factor authentication is often described as a login problem: enter a password, receive a code, confirm identity. That model was not enough for the system described in this case study. The product ran in an in-store environment where the same tablet could be used by several people during a transaction: an employee initiating the process; a manager approving or supporting it; a customer reviewing and signing on their own device. The challenge was not simply to prove that a user knew a six-digit code. We needed to create a secure, short-lived handoff between a shared in-store session and the customer’s personal phone, without leaking the downstream signing session or allowing multiple devices to claim the same transaction. This post explains the architecture, the security model, the trade-offs, and the production practices behind that platform. The actual problem: secure device handoff The workflow started on a shared tablet. At a certain point, the customer needed to continue part of the process on their own phone. The platform therefore had to answer several questions: How does the phone prove that it belongs to the customer currently standing in front of the employee? How does the shared tablet know that the correct phone claimed the correct session? What happens if the QR code is scanned twice? How do we prevent session identifiers and tokens from appearing in URLs, browser history, logs, or referrer headers? How do we notify the phone immediately when verification succeeds? How do we ensure that a single-use signing URL is never exposed before verification? Those constraints turned a seemingly small MFA feature into a distributed-system problem involving identity, real-time communication, concurrency, edge delivery, infrastructure, and operational

Designing AI Systems That Outlive Today's Models If there's one lesson this series has taught me, it's this: Don't build your application around a model. Build it around a capability. That might sound like a small distinction. It isn't. Because models change. Constantly. A few months ago everyone was talking about GPT-4. Then Claude. Then Gemini. Then DeepSeek. Then Qwen. By the time you're reading this, there's probably another model making headlines. Imagine rewriting your application every time that happens. That's not innovation. That's technical debt. One mistake I see quite often is developers tightly coupling their applications to one provider. Your business logic knows it's talking to GPT-4. Your prompts are written specifically for GPT-4. Your output parsing assumes GPT-4. Your error handling assumes GPT-4. Now imagine your company decides to switch providers. What should have been a configuration change suddenly becomes weeks of refactoring. That's avoidable. Your application shouldn't know who answered the request. It should only know that the capability it asked for was delivered. Summarize this document. Generate this code. Classify this text. Translate this paragraph. Those are capabilities. The provider is simply an implementation detail. One thing I regret not doing earlier was versioning prompts. Most developers version everything else. Source code. Database migrations. Infrastructure. Configuration. Then prompts end up looking like this: typescript id = " a8fj21 " const prompt = " You are a helpful assistant... " ; Three months later someone tweaks a sentence. Responses change. Nobody knows why. Sound familiar? Prompts deserve the same engineering discipline as code. Version them. Review them. Document why changes were made. Roll them back when needed. Prompt engineering isn't magic. It's software development. Imagine you've found a brand-new reasoning model that performs better than your current one. Do you deploy it to every user immediately? Pro

"The real magic of a HashMap isn't that it stores data. It's that it knows where to start looking." In the previous article, we learned that a HashMap organises information around unique keys. Instead of searching every stored object one by one, it uses the key to retrieve information quickly. That naturally raises another question. "If millions of objects are stored inside a HashMap, how does it know where to begin?" Surely it isn't remembering the location of every object individually. The answer lies in one of the most important ideas in computer science: Hashing. Don't worry if the word sounds intimidating. Despite its name, the idea behind hashing is surprisingly simple. Imagine a Huge Apartment Building Suppose you're visiting a friend who lives in a building with 5,000 apartments. If nobody told you the apartment number, what would you do? Probably something like this. Apartment 1 ↓ Apartment 2 ↓ Apartment 3 ↓ ... ↓ Friend's Apartment That would take a long time. Now imagine your friend simply tells you: Apartment 1842 Suddenly, you don't search the building. You walk directly to Apartment 1842. The apartment number isn't your friend. It simply tells you where to begin. Hashing works in exactly the same way. Keys Need Locations Suppose our application stores customers. Customer ID → Customer 1001 → Alice 1002 → Bob 1003 → Charlie 1004 → David The system needs a way to answer one question. "Where should Customer 1002 be stored?" Searching every location first would defeat the purpose of using a HashMap. Instead, the system calculates where that key should go. Notice something important. It doesn't compare Customer 1002 against every other customer. It calculates a location directly. Think of a School Locker System Imagine a school with thousands of students. Every student receives a locker. Student ID ↓ Locker Number ↓ Locker Students don't spend every morning searching hundreds of lockers. Their Student ID determines where they should go. The locker number is

"The best software engineers don't begin by choosing data structures. They begin by understanding what the system needs to do." In the previous article, we learned that data structures never stopped being important after DSA. Their role simply changed. During coding interviews, we often ask ourselves: "Which data structure will solve this problem efficiently?" In Low-Level Design, experienced engineers ask a different question: "What behaviour should this system optimise?" At first glance, these questions sound similar. In reality, they lead to completely different ways of thinking. This article is about understanding why behaviour—not implementation—is where every good design begins. Why Beginners Often Think About Data Structures Too Early Imagine someone asks you to design an online food delivery platform. Many beginners immediately start thinking: Should I use a HashMap? Will I need a Queue? Should I store everything in a Tree? Would a Graph be useful? These aren't bad questions. They're simply being asked too early. Before choosing any data structure, we need to understand what the system is actually expected to do. Software engineering isn't about selecting tools first. It's about understanding problems first. Every Software System Is Really a Collection of Behaviours Let's consider a food delivery application. From a user's perspective, it looks like this. Customer Places Order │ Restaurant Accepts │ Assign Delivery Partner │ Track Delivery │ Order Delivered It looks like one workflow. But an engineer sees something very different. Each step represents a different behaviour. Let's break them apart. Behaviour 1 — Retrieve Existing Information A customer opens an order they placed yesterday. Customer ↓ Order ID ↓ Retrieve Order The system already knows exactly which order it needs. The challenge is retrieving it quickly. Behaviour 2 — Choose the Best Candidate A restaurant has multiple delivery partners nearby. Available Drivers ↓ Choose Best Driver ↓ Assign Ri

Your monitoring catches problems on everything except itself. Here is how an always-firing Watchdog alert plus an external heartbeat check turns silence into a signal, so you find out when your own alerting dies. TL;DR A monitoring system can't reliably monitor its own failure, so use a dead man's switch. Create an always-firing Prometheus Watchdog alert and route it to an independent external heartbeat service. As long as the monitoring pipeline is working, the Watchdog continuously refreshes the heartbeat. If Prometheus, Alertmanager, or the delivery path fails, the heartbeat stops and the external service alerts you through a separate channel. The key is independence: the system responsible for detecting that your monitoring is down must not depend on the monitoring stack itself. One of the traps of creating alerts on a monitoring stack is the hidden assumption that the mechanism evaluating the alert is running properly and has the ability to evaluate it. Prometheus watches your hosts and Alertmanager delivers the warnings. But what watches Prometheus? If something goes wrong and the monitoring stack fails in the middle of the night, no alerts are going out but there is definitely a problem. That is the failure mode that you should be most concerned about, because it is the one your monitoring cannot report on. The fix is an old idea with a grim name: a dead man's switch. A train's dead man's switch stops the train when the operator stops holding it down. The safe state requires continuous positive action, while the absence of that action is what triggers the response. Applied to monitoring, it means building one alert whose silence is itself the alarm. Step one: an alert that always fires This feels backwards the first time you see it, and it took me a little time to get it right. Basically, you create an alert with a condition that is always true, so it fires constantly, forever, on purpose. In the Prometheus world this is conventionally called Watchdog. - aler

Writing software that scales from a small monolith into a multi-team distributed system requires strict architectural discipline. The SOLID principles —coined by Robert C. Martin ("Uncle Bob")—serve as fundamental guidelines for object-oriented design and system architecture. When improperly understood, developers often fall into two extreme traps: creating monolithic "God objects" that break with every change, or over-engineering systems into hyper-fragmented, unmaintainable micro-services. In this deep-dive guide, we will break down each of the 5 SOLID principles from low-level class design up to high-level distributed systems design, complete with bad vs. refactored Java examples, system architecture diagrams, trade-off analyses, and a comprehensive cheat sheet. SOLID Principles Cheat Sheet Principle Core Concept Anti-Pattern / Code Smell Refactoring Solution Single Responsibility (SRP) A class or module should have one, and only one, reason to change (serving one business actor/domain). God Class / Micro-Fragmentation: Classes handling payment, DB, and notifications, OR over-fragmented single-function classes. Split by domain responsibility. Use orchestrator/coordinator components for workflows. Open/Closed (OCP) Software entities should be open for extension, but closed for modification . Conditional Bloat: Cascading if-else or switch statements checking object types or channels. Strategy Pattern, Dependency Injection, and Event-Driven Pub/Sub messaging (e.g., Kafka). Liskov Substitution (LSP) Subtypes must be completely substitutable for their base types without breaking client behavior. Runtime Exceptions: Subclasses throwing UnsupportedOperationException or silently breaking logic. Split fat inheritance hierarchies into granular, capability-specific interfaces. Interface Segregation (ISP) No client should be forced to depend on methods it does not use. Fat Interfaces: Monolithic interfaces forcing callers to mock or implement irrelevant methods. Role-focused

This series follows a fictional conversation between an experienced engineer and his nephew. Every episode explores one stage of how software moves from an idea to production. 👦 Nephew: Uncle, requirements are clear. I checked the codebase — there's already a FavoritesService I can extend for Wishlist. Now can I open VS Code? 👨‍🦳 Uncle: Almost. Tell me — what do you think HLD even is ? You've heard the term in every interview. What do you think it actually means? 👦 Nephew: Some kind of... diagram? Boxes connected with lines, before you start coding? 👨‍🦳 Uncle: That's what it looks like. That's not what it's for . Let me ask differently. Why do you think experienced engineers insist on drawing this before touching code, when they could just start building? 👦 Nephew: ...to plan the work? 👨‍🦳 Uncle: Closer, but still not it. Here's the real answer: HLD exists to decide, in advance, where the walls go — so that six months from now, when someone adds a new feature, they know exactly which room to build it in, without knocking down a wall that was holding up the ceiling. 👦 Nephew: That's a strange way to describe a diagram. 👨‍🦳 Uncle: Then let me show you, instead of describing it. That's the only way this actually lands. What Talks to What 👨‍🦳 Uncle: Suppose we're building this at Flipkart. Not a college project — a company with hundreds of live services, where breaking one thing can affect ten others you've never even heard of. Here's the simplest picture for Wishlist. Frontend ↓ Wishlist API ↓ Wishlist Service ↓ Database 👦 Nephew: That's it? Four boxes? 👨‍🦳 Uncle: That's it. HLD answers exactly one question, and nothing more — what talks to what. Not how the button looks. Not what fields the database table has. Just: which component calls which, and in what direction. 👦 Nephew: Then why does everyone treat it like it's such a big deal? This took ten seconds to draw. 👨‍🦳 Uncle: Because the value isn't in the ten seconds you spend drawing it today. The value is in what i

Disclosure: This post includes affiliate links; I may receive compensation if you purchase products or services from the different links provided in this article. Credit: codemia.io Hello Devs, if you're preparing for software engineering interviews, particularly in MAANG, you already know that Data Structures & Algorithms (DSA) and System Design are two key areas where you will be rigorously tested. While LeetCode is the go-to platform for DSA, system design has always been a challenge. While there are many websites and platforms to prepare for System Design Interviews like ByteByteGo , DesignGurus.io , Exponent , Educative , and Udemy , there is nothing like LeetCode. These are great resources to learn fundamentals, go through case studies, and understand the theory part of system design, but LeetCode-style practice is one thing that is missing - until now. I recently found Codemia.io , and I must say, it feels like the LeetCode for System Design. If you've struggled with structuring your system design answers, getting real feedback, or knowing whether your approach is correct, Codemia.io is a game-changer. They not only have the biggest collection of System Design and OOP Design problems for practice, but they also have a free System Design course called Tackling System Design Interview Problems , which is a great free resource to learn essential System Design concepts. It's a short course with 2 hours of content, but it's powerful and also has quizzes to test your skills. Here are all the key System Design topics you can learn on this free course: Now, let's check out how Codemia.io can help you to prepare better for your System design and OOP Design interview, and why I think it's like Leetcode for System design. Most system design resources today are long, text-heavy articles or expensive courses. The problem? No hands-on practice - Reading about system design isn't enough; you need to actively design solutions. No structured progression --- Unlike DSA, where

Most architecture decision records fail for the opposite reason people think. The issue usually isn't that teams forget to write them. It's that the ones they write are filled with the wrong content. The key information a reader needs—why this option instead of the others—often gets buried on page three under a list of API changes. An ADR has one job: capture a decision that is costly to reverse, along with the reasoning that led to it, while that reasoning is still fresh. That's all. It isn't a design document, a specification, or a collection of research. If you keep that focus, everything else about what to include or leave out will follow naturally. The format that still works Michael Nygard's original ADR template from 2011 (title, status, context, decision, consequences) has lasted for a reason. It directly addresses the key questions a future reader has: What was the situation? What did we decide? What did we give up? Teams that add ten extra sections, like owners, review dates, risk matrices, or approval lists, usually end up with a document no one finishes reading, which defeats the purpose. If your ADR template is longer than the time it takes to fill it out for a simple decision, cut sections until it’s more concise. A useful rule of thumb is that an ADR longer than a page and a half is often a design document masquerading as an ADR. This isn’t a strict rule, but I haven’t seen a truly good ADR exceed 600 words. The decisions worth documenting this way can be stated, justified, and owned in about a page. If that's not possible, the record isn’t the issue. The decision is probably still tied up with other unresolved matters. What belongs The decision, stated clearly. "We will use event-driven integration between the order and inventory services instead of synchronous REST calls" is a decision. "The order service integrates with inventory" simply describes the current state and belongs in a wiki, not an ADR. The challenges faced. Describe the two or three a

It's easy to reach for the popular tool before asking what your system actually needs. For job queueing, the usual advice is to use RabbitMQ or Kafka. The underlying burden of using these tools will be additional processes to deploy, monitor and reason about. But what if using your existing database is possible? I built a job queueing system with a PostgreSQL database that cleared the bar without adding infrastructure. The next question will be, does this solution meet the criteria of a job queue? A job needs a few things to execute properly within a system. It needs to be persistent, surviving unforeseen crashes. Jobs must not be processed more than once by workers; one job should be processed once by one worker. Also, a job's state has to be tracked through every step. A job state must show when it's pending, completed or failed. That's the bar any solution must clear. With the Postgres approach, persistence comes free. Jobs live in a table so when a worker dies mid-job, the job still exists in a row in the db. Whereas with in-memory queues, a crash loses everything still in memory. A broker like RabbitMQ has to be configured for persistence and if configured wrongly, jobs get lost. A database however is fundamentally built for durability. Now, let's say three workers poll the queue at the same instant and run the same query. They'll see the same pending job at the top and nothing stops them all from grabbing it. If that job is a payment, the customer gets charged three times for one service. All the workers successfully process the job with no indication of an error or alerts. This is the requirement that seems to demand a real message broker, and it's exactly where people assume a database can't compete. It can. Postgres has a specific tool for exactly this. The SQL clause FOR UPDATE is used to lock rows. This can be called on a job when a worker picks it up to process. By default other workers will get blocked during this process, they'll wait for the lock to r

Back-of-the-envelope estimation for system design interviews Most people don't fail capacity math because the arithmetic is hard. They fail because they do it silently, produce a number they can't defend, and then never use it again for the rest of the interview. The math itself is trivial. The method is what's worth learning. Why interviewers ask Capacity estimation isn't a numeracy test. It's checking two things: Can you tell whether a design is physically possible before you commit to it? Do you know which constraint actually binds — storage, read throughput, write throughput, or bandwidth? A candidate who estimates 30,000 reads/sec and 200 writes/sec has learned something that changes the design. A candidate who computes petabytes of storage and then never mentions it again has just performed arithmetic. Round aggressively Precision is a trap. You're not producing a capacity plan; you're finding the order of magnitude. The single most useful substitution: 1 day = 86,400 seconds ≈ 10^5 seconds That's a 16% error and it makes every subsequent division doable in your head. Nobody will challenge it. Everyone will notice if you spend forty seconds long-dividing by 86,400. A few more worth having ready: 1 million requests/day ≈ 12/sec — round to 10 1 KB × 1 million = 1 GB 1 KB × 1 billion = 1 TB Peak traffic ≈ 2–3× average Replicated storage ≈ 3× raw ## Work in one direction Users → requests → QPS → storage → bandwidth. Don't jump around. Say each assumption out loud and label it as an assumption, so the interviewer can correct you early rather than watch you build on sand. A worked example Say we're designing a social feed. Given: 100M daily active users. Assumptions (stated, not smuggled in): Each user posts 0.2 times/day Each user reads their feed 10 times/day A post averages 1 KB including metadata A feed page shows 20 posts Writes 100M × 0.2 = 20M posts/day 20M / 10^5 = 200 writes/sec Peak (3×) = 600 writes/sec Reads 100M × 10 = 1B feed loads/day 1B / 10^5 = 10,0

8 min read · telecom provisioning platform, 30M+ subscribers Most companies avoid migrating their monolith for one reason: they imagine it as a single, terrifying event — months of a feature freeze, a weekend cutover, and a rollback plan that's really just hope. That fear is reasonable. A big-bang rewrite of a live system serving 30M+ subscribers really would be terrifying. So we didn't do that. We used a pattern that lets you migrate a monolith one slice at a time, while the system keeps running and the product team keeps shipping features — the same pattern Martin Fowler named Strangler Fig , after the vine that grows around a host tree, gradually taking over, until eventually the original tree is no longer needed. Why the "just rewrite it" instinct is usually wrong The instinct to rewrite a legacy monolith from scratch is understandable — the old code is scary, undocumented, and nobody wants to touch it. But a full rewrite has a well-known failure pattern: it takes far longer than estimated, the business can't freeze feature development for that long, and by the time the rewrite is "done," the old system has changed underneath it and the rewrite is already out of date. The alternative isn't "don't migrate." It's: migrate in slices small enough that each one is boring , and never require the business to stop shipping while you do it. The pattern: a facade, and one slice at a time Strangler Fig works by putting a routing layer — a facade or API gateway — in front of the monolith. At first, 100% of traffic passes through to the old system untouched. Then, one capability at a time: Build the new version of that one capability as an independent service. Update the facade to route just that capability's traffic to the new service. Run both in parallel long enough to trust the new one (see "shadow traffic" below). Retire that piece of the old monolith. Repeat for the next capability. At every point in this process, the system is fully functional. There's no "half-migrat

Every engineer eventually hits this phase: “My design looks okay… but something feels off.” No compile errors. No obvious bugs. But still: responsibilities feel scattered services feel too big entities feel too thin logic feels duplicated boundaries feel unclear This is normal. Because domain modeling is not about getting it right in one attempt. It is about refining structure until the business behavior becomes clear. Step 1 — Start With the Symptom, Not the Code If your design feels wrong, don’t immediately rewrite everything. First identify the symptom: Common symptoms: too many “Manager” services logic repeated in multiple places unclear ownership of rules too many dependencies between modules frequent “if-else explosion” Each symptom points to a specific modeling issue. Step 2 — Check If Invariants Are Scattered Ask: “Where are my business rules living?” Bad sign: Rules inside services + controllers + helpers This leads to: inconsistent behavior duplicated validation broken business guarantees Good design: invariants live close to the entity or aggregate root Step 3 — Check Entity vs Service Confusion A very common issue: Entities become dumb: only fields no behavior Services become overloaded: all logic all rules all decisions This creates: Anemic Domain Model + Fat Services Fix mindset: Entity = owns behavior + protects state Service = coordinates workflows Step 4 — Check Your Aggregate Boundaries Ask: “What must stay consistent together?” If your answer is unclear, you likely have: wrong aggregates or missing aggregates Example problem: Cart and Order sharing logic This causes: inconsistent pricing unclear lifecycle ownership Fix: Cart = intent Order = truth Step 5 — Look for “Hidden Coupling” Hidden coupling happens when: one module depends on internal state of another multiple services modify same data business rules are duplicated across boundaries This leads to fragile systems. Strong design ensures: each domain owns its own truth. Step 6 — Validate Stat

In this article, we're going to explore Event Schema evolution with Event versioning 10. Event Schemas Will Eventually Change No event schema stays the same forever. As businesses grow, regulations shift, products gain new features, and processes become more complex, the data shared between services must evolve as well. This evolution is not optional—it is a natural consequence of a system adapting to changing requirements. Many teams initially assume they can simply update an event whenever needed. This assumption may hold when there is only one producer and one consumer, but real-world systems rarely remain that simple. Over time, multiple consumers emerge, each with its own responsibilities and release cycles. A typical system often looks like this: OrderConfirmed | +------------------+-------------------+ | | | v v v Inventory Billing Notification | v Analytics | v Customer Insights Each consumer evolves independently. Some services may deploy updates weekly, while others might release changes quarterly. In some cases, consumers may even belong to external teams with entirely different priorities and timelines. Because of this, producers cannot assume that all consumers will upgrade simultaneously. Schema evolution, therefore, is not just about modifying data structures. It is fundamentally about maintaining compatibility across independently evolving systems. Compatibility Is More Important Than Version Numbers When discussing schema evolution, teams often focus immediately on versioning. While versioning is useful, compatibility is far more critical. Without compatibility, versioning alone cannot prevent system breakage. Consider the following event: { "orderId" : "ORD-1001" , "customerId" : "CUS-501" , "totalAmount" : 249.99 } Now imagine a new requirement introduces currency. One approach might replace the existing field entirely: { "orderId" : "ORD-1001" , "customerId" : "CUS-501" , "amount" : { "value" : 249.99 , "currency" : "USD" } } Although the data mo

"We're using Kafka with exactly-once semantics, so we don't have to worry about duplicates." I've heard this in architecture reviews, design docs, and postmortem explanations. It represents a misunderstanding of what Kafka's exactly-once guarantee actually covers, and the gap between the promise and the reality has caused real production incidents. What Kafka's Exactly-Once Actually Covers Kafka's exactly-once semantics (EOS), introduced in 0.11.0, operates at two levels: Producer idempotence ( enable.idempotence=true ): The producer assigns each message a sequence number. The broker deduplicates messages with the same producer ID and sequence number. This prevents duplicates caused by producer retries — the message lands in the Kafka partition exactly once, regardless of retry count. Transactions ( transactional.id ): Allows a producer to write to multiple partitions atomically. Either all writes commit or none do. Combined with isolation.level=read_committed on consumers, readers only see committed transactions. Together, these give you exactly-once message delivery within the Kafka cluster. What Exactly-Once Does Not Cover Here's the boundary that engineers miss: Kafka's exactly-once guarantee is scoped to the Kafka cluster. The moment your consumer does anything outside Kafka — writes to a database, calls a REST API, publishes to a cloud queue — you're outside the transaction boundary. Consider a typical consumer: consumer . poll ( records ); for ( record : records ) { database . save ( process ( record )); // External write — outside Kafka transaction } consumer . commitSync (); If the application crashes after database.save() but before commitSync() , Kafka re-delivers the message. The consumer reprocesses it. The database now has two writes for the same event. Enabling producer idempotence on the consumer's Kafka writes does not fix this. The Patterns That Actually Give You End-to-End Safety Idempotent Consumers Design consumer processing logic to be idempote

The Quest Begins (The “Why”) I still remember the night our API started to sputter under a sudden traffic spike. Users were seeing 502 errors, the monitoring dashboard looked like a neon rainstorm, and I felt like I was stuck in a lobby waiting for the elevator that never arrives. We had a simple round‑robin load balancer sitting in front of three identical services. It worked fine when traffic was smooth, but as soon as a burst hit, one node would get overloaded while the others twiddled their thumbs. Honestly, I thought we just needed more servers. Throwing hardware at the problem felt like using a sledgehammer to crack a nut—expensive and messy. After a few frantic Slack threads and a lot of coffee, I realized the real issue wasn’t capacity; it was how we distributed the work. The balancer was oblivious to the actual load on each backend, treating every request like it was the same weight. That moment became my quest: design a load balancer that reacts to real‑time load, stays simple enough to operate, and doesn’t cause a reshuffling nightmare when we scale the cluster. The Revelation (The Insight) The breakthrough came when I read about least‑connections load balancing combined with a slow‑start period for new hosts. The core insight is deceptively simple: Send each new request to the backend that currently has the fewest active connections. Why does that work? Immediate fairness – If one node is handling long‑running requests, it will naturally have a higher connection count and receive fewer new ones until it catches up. Burst absorption – During a traffic spike, requests spill over to the less‑busy nodes instead of piling onto a single overloaded instance. Predictable scaling – When we add a new server, it starts with zero connections, so it gets a fair share of traffic right away—but we temper that with a slow‑start window to avoid overwhelming a cold host. Compare that to round‑robin, which blindly cycles through the list regardless of each node’s state. In

The dashboard says forty instances, up from twelve this morning. The autoscaler did its job: it saw latency climb and threw hardware at it. And latency got worse. Not flat. Worse. You're paying for three times the compute to serve a slower product. Somewhere under all forty of those boxes is a single thing they're all waiting in line for, and every instance you add makes the line longer. Horizontal scaling multiplies work that doesn't have to coordinate. The instant the work does have to coordinate, more instances make it slower. Amdahl wrote this down in 1967: the serial fraction of a job sets a hard ceiling on how much faster you can go, no matter how much hardware you throw at the parallel part. Neil Gunther's Universal Scalability Law goes further: past a certain point, the cost of nodes coordinating with each other bends the curve back down. Add capacity, get less throughput. That ceiling was not set by the autoscaler, and it will not be moved by the autoscaler. It was set a long time before this morning, in a room, by whoever decided where the state lives and who has to touch it at the same instant. Now hand the service to a fleet of agents. It writes you something that looks built to scale: stateless handlers, a tidy repo, green tests, a canary that bakes fine at 1% traffic. Every gate you trust says ship it. And the bottleneck is sitting right there in the design, invisible to all of it, because the mistake isn't in the lines, it's in the shape. You cannot catch a shape problem by reading a diff. Name the hot state before you pick a framework. Where does the contended state live, and which requests touch it at the same instant? Answer that out loud, before anyone opens an editor. The tool is downstream of that answer, every time. Originally published at https://imacto.com/writing/scale-is-a-design-not-a-dial . Written with Claude Opus 4.8.

Introduction So imagine you are focused on your cappuccino-frappuccino doing something very important on you win laptop and then have a cringe attack due to the unknown Phone Link notification : Complete linking devices Your PC and mobile device are almost linked. Click here to continue linking devices. via Phone Link Then you switch off bluetooth, wifi, laptop - and you are right. What to do next ? Basic checks Settings -> Bluetooth & devices -> Mobile devices Settings -> Accounts -> Email & accounts Inspect recent notifications in Event Viewer eventvwr.msc Applications and Services Logs └ Microsoft └ Windows └ Notifications Applications and Services Logs └ Microsoft └ Windows └ Shell-Core Digital forensics Windows stores toast notifications in a local database, hence you need to install sqlite Get-ChildItem " $ env : LOCALAPPDATA \Microsoft\Windows\Notifications" output : Directory: C:\Users\$ USERNAME \A ppData \L ocal \M icrosoft \W indows \N otifications Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 1/1/2026 0:00 AM wpnidm -a---- 1/1/2026 0:00 AM 1000000 wpndatabase.db -a---- 1/1/2026 0:00 AM 10000 wpndatabase.db-shm -a---- 1/1/2026 0:00 AM 1000000 wpndatabase.db-wal wpndatabase.db is a SQLite database. connect to the database : sqlite3 " $env :LOCALAPPDATA \M icrosoft \W indows \N otifications \w pndatabase.db" query the Notification table . headers on . mode column SELECT Notification . Id , Notification . HandlerId , Notification . Type , Notification . ArrivalTime , Notification . Payload FROM Notification LIMIT 20 ; Then you will have something like : Id: [REDACTED] HandlerId: [REDACTED] Type: toast ArrivalTime: [REDACTED] Payload: <?xml version="1.0"?> <toast activationType= "protocol" launch= "ms-phone:fre/?cid=[REDACTED]&ref=FreIncompleteToast&reason=IncompleteNotificationsToast" > <visual> <binding template= "ToastGeneric" > <text hint-maxLines= "1" > Complete linking devices </text> <text> Your PC and mobile device are almost li

At my workplace, I worked on an ERP platform used by fashion businesses to manage customers, body measurements, products, orders, invoices, inventory, staff, and other day-to-day operations. Each business also had a public storefront where customers could browse products and check out. The storefront started as a simple sharing feature. Businesses could publish products, copy a link, and send it to customers outside the main workspace. That worked well because the storefront was mostly a product catalogue, and most of the sales process still happened after the customer contacted the business. As the platform evolved, the storefront became much more than a catalogue. Customers were discovering businesses through shared links, browsing products, placing an order, and tracking orders directly from the storefront. That introduced new technical requirements around branded storefronts, SEO, server-rendered metadata, public checkout, pricing, and analytics. This article explores how I designed the storefront around wildcard subdomains, immutable shop identities, server-side shop resolution, and a scalable analytics pipeline. Table of Contents Giving the Storefront Its Own Identity Business Names, Reserved Names, and Subdomains Resolving a Storefront Active and Inactive Storefronts Location and Currency Product Pages and Share Previews Storefront Event Ingestion Processing Raw Events Counting Unique Visitors With HyperLogLog Domain Routing and Local DNS 1. Giving the Storefront Its Own Identity The original storefront was fairly simple. It was a React application that fetched a business and rendered its products. Beyond that, there wasn't much to it. There were no branded storefronts, analytics, subdomains, or even a separate identity beyond the business itself. Introducing those capabilities meant the storefront needed its own data model. I introduced a dedicated shop entity to represent the public storefront. The business remained the source of operational data such as cu