标签：#design

Payment data pipelines fail in ways that ruin a payments engineer’s week, and the failures rhyme. The dashboards froze. Fraud scores arrived after the transaction had already cleared. Settlement reports came in stale. Nobody slept. The frustrating part is that the same data architecture had run fine for years. So, what changed? The honest answer is that batch thinking does not survive contact with real-time payments. A lot of banks built their data foundations in an era when nightly jobs were good enough. Load the warehouse overnight, run the reports in the morning, move on. That rhythm worked when money moved slowly. It does not work when a customer expects an instant confirmation and a fraud engine has milliseconds to make a call. Here is where things crack. Real-time payment rails push a constant stream of events instead of a tidy nightly dump. Your pipeline now has to ingest, transform, and serve data while transactions are still happening. Add ISO 20022 into the mix and the pressure climbs. ISO 20022 messages are rich. They carry far more structured detail than the old formats, which is wonderful for analytics and miserable for a pipeline that was never designed to parse that much context at speed. This is not a fringe concern either. Swift reported that by the time its MT/ISO 20022 coexistence period closed in November 2025, around 80% of daily traffic was already running on the ISO 20022 format, with more than 3.1 million of these messages exchanged every day. The rich-data era is the default now, not the roadmap. Then there is the fraud-scoring window. Fraud models need fresh features. Account behaviour over the last few minutes, velocity checks, device signals. If your pipeline takes thirty seconds to surface that data, the fraud decision is already too late. You are essentially detecting fraud after the loss. That gap between when data is created and when it becomes usable is the silent killer in most payment systems. And the cost of getting it wrong runs

If your LLM costs are climbing, the instinct is almost always the same: swap to a cheaper model. GPT-4 to GPT-4-mini. Claude Opus to Claude Haiku. Sometimes that helps a little. It rarely fixes the actual problem. The actual problem, in most workflows I've looked at, is that every step gets routed through the LLM, even the steps that don't need language reasoning at all. This post breaks down a simple mental model for deciding what should and shouldn't touch an LLM, with a working example you can adapt. The four components of any AI workflow Every automated workflow — whether it's a support ticket router, a fraud check, or a content pipeline — is built from some combination of four building blocks. They get treated the same once a workflow diagram is drawn flat, but they have wildly different cost and latency profiles. Component What it does Think of it as Typical cost Trigger Starts the workflow The doorbell ~$0 Deterministic ML Structured predictions — classify, score, rank The calculator Cents per 1,000 calls LLM / Generative Reads, writes, reasons in language The writer Dollars per 1,000 calls Tool / API Fetches or writes real data The hands Cents per 1,000 calls The gap between row 2 and row 3 is the whole article. A classifier and an LLM call can solve the exact same problem, but one costs roughly 100-1000x more than the other, depending on model and provider. If you're not deliberately deciding which one handles which step, you're probably defaulting to the expensive one — because in frameworks like LangChain or a quick custom agent loop, it's just easier to shove everything into a prompt. Where this actually shows up Here's a workflow I see constantly: an automated support ticket triage system. flowchart LR A[New support ticket] --> B{Classify intent} B --> C[Route to team] B --> D[Auto-draft response] D --> E[Update CRM] A naive build sends the entire ticket text to an LLM and asks it to do everything at once: classify the intent, decide routing, draft a re

Good architecture is not only about how a system is built. It is also about how well the team can understand that system once it is running. That is where observability belongs in the architecture conversation. It is common for observability to be treated as something that comes after the main engineering work. The service gets built. The API works. The deployment succeeds. Then, somewhere near the end, the team starts thinking about logs, dashboards, alerts, and operational visibility. That approach creates a gap. The architecture may look clean on paper, but once the system is in production, the team has to understand how it behaves under real conditions. Real users do not follow the happy path perfectly. Dependencies slow down. Queues back up. Data arrives in unexpected shapes. Deployments change behavior in ways that are not always obvious. If the system does not give the team a way to see those things clearly, the architecture is incomplete. Observability is not decoration around the system. It is part of the system design. Architecture Describes the System. Observability Shows the Truth. Architecture is built on assumptions. During design, teams make reasonable guesses about usage patterns, service boundaries, dependency behavior, data flow, scale, latency, and failure modes. Some of those assumptions are based on experience. Some are based on current requirements. Some are simply the best call the team can make with the information available at the time. That is normal. The problem is not that architecture contains assumptions. Every architecture does. The problem is when those assumptions cannot be tested once the system is real. A design might assume that an external dependency will be reliable enough. Production may show that it is the slowest part of the request path. A queue might look like a clean decoupling point during design. Production may reveal retry behavior, duplication, or ordering concerns that were not obvious upfront. A serverless function m

My Testing Setup I used Midjourney V7 (midjourney.com, Standard plan at $30/mo for this project — volume was too high for Basic) over five weeks across six product photo projects: lifestyle context images, background replacement concepts, packaging mockups, and mood-board style reference images for briefing photographers. Some outputs went live in ads. Others were used internally. A few were scrapped entirely. Two specific examples: I generated 12 lifestyle context images showing a skincare product in a bathroom setting — no actual product in the image, just the environment and mood — and used them as ad backgrounds with the real product composited in afterward. Results were strong. I also tried to generate images of the actual product itself from reference photos. That failed in ways I will explain. Pricing: Standard plan at $30/mo. For product photo work at volume, Basic at $10/mo runs out fast. Budget for Standard if this is a regular workflow. 1. Lifestyle Context and Environment Images This is where Midjourney V7 earns its place in a product photo workflow. Generating the environment — a kitchen countertop, a gym bag, a coffee shop table — without needing to stage or shoot it is genuinely useful. I needed eight lifestyle backgrounds for a supplement brand's ad campaign. Real location shoots for eight setups would have cost $3,000 and taken two weeks. I generated the environments in Midjourney, exported them, and composited the real product in using Photoshop. Total cost: $30 for the month's Midjourney subscription and four hours of compositing work. The images ran in paid Meta ads for six weeks. CTR was in line with our studio-shot creative. Nobody asked if the backgrounds were AI-generated. The key: generate the environment only. Do not try to put your specific product into the Midjourney image. Composite it in post. That division of labor is where the workflow holds up. 2. Packaging Mockups for Concepts That Do Not Exist Yet Before you manufacture a product o

In this article, the author explores data poisoning as a threat to machine learning systems, covering techniques such as label flipping, backdoors, clean-label poisoning, and gradient manipulation. The article reviews real-world incidents, discusses the challenges of detecting poisoned data, and presents practical defenses, tools, and operational practices for securing ML training pipelines. By Igor Maljkovic

We asked Johan Ejdemo to list which Ikea items populate his home. He also tells us about his all-time favorite. (No, it's not a Billy bookcase.)

I bombed a system design interview last year — not because I didn't know the architecture, but because I spent the first 5 minutes fighting Excalidraw. So I built SystemDesignBoard — a free, keyboard-first whiteboard specifically for system design interviews. What it does You open it, press a key, and start drawing. No account, no onboarding, no drag-from-a-sidebar friction. R → place a Service node C → place a Database/Cache/Queue A → connect two nodes N → open the scratchpad for scale math The features I'm most proud of Animated connectors that show communication type Instead of just drawing arrows, connectors visually encode how services talk: ⇄ sync — paired dashes (request + ACK) ≋ stream — near-solid fast line with glow (continuous pipeline) This matters in interviews — your interviewer can glance at your diagram and immediately understand the communication pattern. Cloud provider badges Tag any node as AWS (EC2, Lambda, RDS, S3), GCP (GKE, Cloud Run, Firestore), or Azure. Each subtype has its own icon. Trade-off logging Right-click any node → Log Trade-offs → attach your CAP theorem stance, consistency level, and scaling strategy directly to the component. Diagram-as-Code Type: [Mobile App] -> [API Gateway] [API Gateway] -> [Auth Service] [Auth Service] -> [Users DB] [Feed Service] -> [Posts DB x3] [Feed Service] -> [Redis Cache] Hit Apply — it auto-lays out the whole architecture in seconds. Export to animated GIF Export your diagram as a GIF that shows live traffic flow animations. Great for sharing after an interview or in a design doc. Tech stack React + TypeScript + Vite @xyflow/react (ReactFlow v12) for the canvas Zustand + Immer for state with full undo/redo html-to-image + gifshot for PNG/GIF export It's free and open No signup required. Works entirely in the browser. Free during beta. 👉 systemdesignboard.com Would love feedback — especially from anyone who's done system design interviews recently. What's missing? What's annoying? Drop a comment below

As software engineers, we often architect solutions in a virtual ideal: fast networks, elastic resources, and servers that never physically degrade. But what happens when your carefully crafted systems need to interact with the messy, unpredictable physical world? Think factory floor monitors, real estate camera networks, or remote tracking devices. Suddenly, those cloud assumptions about infinite uptime and perfect connectivity crumble. My journey, particularly architecting and maintaining a continuous 24/7 camera livestream for a real estate group over six years, has been a masterclass in this reality. It's revealed that true reliability in the physical realm demands a hybrid approach – one that intelligently merges the power of edge computing with the scalability and data insights of the cloud. This isn't just about connecting devices; it's about building resilience into the very fabric of your architecture. In this article, I'll share the battle-tested strategies and design principles that enable systems to not just survive, but thrive, despite the harsh realities of physical deployment. 1. The Core Strategy: Smart Edge, Simple Cloud One of the most common pitfalls in hybrid architecture design is treating the edge device as a mere 'dumb' terminal, solely responsible for streaming raw data to a powerful cloud backend. This approach creates a critical single point of failure: if the network drops, the entire system grinds to a halt. Instead, I advocate for a Smart Edge, Simple Cloud architecture. This principle establishes a clear division of responsibility: The Edge : This is where the magic happens locally. The edge system should be robust enough to handle local processing , data filtering , buffering , and immediate hardware control . Critically, it must be capable of operating autonomously for extended periods without an active cloud connection. Think of it as a mini data center, designed for self-sufficiency. Benefits of a Smart Edge : Reduced bandwidth cost

The Counterintuitive Truth: Feature Flags Are Not Config Files Most engineers first encounter feature flags as a simple abstraction: a key-value lookup that returns true or false. That mental model works fine for a single service handling a few hundred requests per minute. It becomes actively dangerous at scale. A mature feature flag system isn't a config file with an API wrapper — it's a distributed control plane . The distinction matters architecturally. A control plane manages the real-time behavior of a running system across many nodes simultaneously, with its own consistency guarantees, failure semantics, and propagation latency. That's a fundamentally different design problem than reading a YAML file on startup. One constraint drives every downstream decision: user traffic must never block on a remote flag service call. If evaluation requires a synchronous RPC, you've coupled your request path to the availability and latency of an external system. Netflix's Archaius library enforces this by evaluating flags entirely in-process against a locally-cached configuration snapshot. A network round-trip per evaluation injects 10–50ms of tail latency at p99 — catastrophic when you're competing on streaming start times measured in hundreds of milliseconds. Google, Meta, and Netflix collectively evaluate flags against millions of requests per second with sub-millisecond overhead. That figure is only achievable through local evaluation backed by an async synchronization layer, not RPC. The other failure mode engineers underestimate is flag sprawl . Systems accumulate flags the way codebases accumulate dead functions — gradually, then all at once. I've seen services carrying thousands of flags where fewer than 10% were actively managed. The operational weight alone becomes a liability: which flags are safe to remove? Which ones are kill switches for production behavior that no one documented? Knight Capital's $440M loss in 45 minutes in 2012 remains the canonical cautionar

Enterprise Design Patterns in Python: Repository & Unit of Work 🐍🏗️ Series: Enterprise Application Architecture | Source: Fowler's EAA Catalog | Code: GitHub Repository 🧠 What Are Enterprise Design Patterns? Martin Fowler's Patterns of Enterprise Application Architecture (2002) is one of the most influential books in software engineering. It documents recurring architectural solutions — patterns — that solve common problems in enterprise systems: how to organize domain logic, how to talk to databases, how to handle transactions, and more. In this article, we'll explore two of the most powerful and widely-used patterns from that catalog: Pattern Category Core Purpose Repository Data Source Abstracts data access behind a collection-like interface Unit of Work Data Source Tracks object changes and commits them as a single transaction These two patterns work beautifully together — and you'll see exactly why with a real-world example. 🛒 The Problem: An E-Commerce Order System Imagine you're building a backend for an online store. When a customer places an order: A new Order is created Each Product 's stock is decremented A Payment record is registered If any of these steps fail midway, the entire operation should roll back — no partial state. This is exactly the problem the Unit of Work pattern solves, and the Repository pattern makes it all cleanly testable. 📁 Repository Pattern Definition "A Repository mediates between the domain and data mapping layers using a collection-like interface for accessing domain objects." — Martin Fowler, PoEAA The Repository acts as an in-memory collection of domain objects. Your business logic never knows if it's talking to PostgreSQL, SQLite, or even a mock list — it just calls .add() , .get() , .list() . Domain Model # models.py from dataclasses import dataclass , field from typing import List from uuid import uuid4 @dataclass class Product : id : str name : str price : float stock : int @dataclass class OrderItem : product_id : str qua

Failures are obvious. Retries are sneaky. When something fails, everyone notices. An alert goes off. A request errors out. Someone starts investigating. Retries are different. They look harmless. Most of the time, they save the system. But sometimes, retries create bigger problems than the original failure. Imagine an API call times out. No problem. The system retries. But what if the first request actually succeeded and only the response was lost? Now the retry creates: duplicate orders repeated emails inconsistent records workflows running twice The failure happened once. The retry multiplied it. Another thing I've seen: One slow dependency causes requests to pile up. Retries start firing. Those retries create even more traffic. Which slows things down further. Which triggers even more retries. Suddenly, the system is spending more effort retrying than doing useful work. Retries also hide problems. A temporary issue gets retried five times and eventually succeeds. Everything looks normal. Meanwhile: latency increases queues grow users experience delays Nothing technically failed. But the system is getting less healthy. What changed for me is that I stopped treating retries as free. Every retry has a cost. It consumes resources. It increases load. And if actions aren't designed carefully, retries can repeat side effects that should only happen once. Now when I build something, I don't ask: "What happens if this fails?" I ask: "What happens if this runs again?" Because in production, things almost always run again. And if the answer is "bad things happen," the retry mechanism isn't helping. It's making things worse. Failures are part of every system. Retries are too. The difference is that failures usually happen once. Retries can turn one problem into hundreds if you don't design for them. This is something we think about constantly at BrainPack when operating long-running workflows across multiple systems. AI and automation layers make retries even more common, wh

Single-model systems are simple. Multi-model systems are powerful. The challenge isn't choosing models — it's designing the architecture that orchestrates them. A multi-model system isn't about having more models. It's about having the right model for the right task at the right time. Architecture patterns Five patterns cover most use cases: Pattern Complexity When to use Tradeoff Single Model Lowest Prototyping, simple tasks Limited capability Sequential Low Multi-step workflows Higher latency Parallel Medium Independent tasks Higher cost Hierarchical High Complex reasoning Complex orchestration Ensemble Highest Critical decisions Highest cost Pick the simplest one that works. Complexity is real, and it compounds. Sequential architecture Process tasks through a chain of models, each specializing in a step. Pattern 1: Pipeline Pipeline pattern — each model's output feeds the next: class ModelPipeline : def __init__ ( self ): self . models = [ { " model " : " qwen2.5-1.5b " , " task " : " classify " }, { " model " : " qwen2.5-7b " , " task " : " extract " }, { " model " : " qwen2.5-32b " , " task " : " reason " }, ] def process ( self , input : str ) -> str : current = input for model_config in self . models : current = self . call_model ( model_config [ " model " ], self . create_prompt ( model_config [ " task " ], current ) ) return current Latency adds up. Three models in sequence means three times the latency. Only use this when each step actually needs a different model. Pattern 2: Router Router pattern — classify the task, route to the specialist: class ModelRouter : def __init__ ( self ): self . classifier = " qwen2.5-1.5b " self . specialists = { " code " : " qwen2.5-coder-7b " , " math " : " qwen2.5-32b " , " creative " : " claude-sonnet-4 " , " general " : " qwen2.5-7b " , } def route ( self , prompt : str ) -> str : task_type = self . classify ( prompt ) model = self . specialists . get ( task_type , self . specialists [ " general " ]) return self . call_m

Building a high-performance Polymarket trading bot requires mastering its unique hybrid architecture:...

Vinay Chella and Akshat Goel discuss the challenges of running traditional CDC across heterogeneous databases during peak order traffic. They explain how Debezium hit limits under high load and share how they built Write-Ahead Intent Log (WAIL) - a custom architecture that utilizes a dumb producer proxy and a smart consumer pattern to cleanly separate the intent from the state payload. By Vinay Chella, Akshat Goel

Adobe is introducing some new capabilities for its Firefly AI assistant, alongside a "reimagined" AI studio that lets you edit and generate new designs from a single interface. The new Firefly experience launching today in private beta is designed to give you "persistent context, reusable assets, and organized workflows" across your projects, according to Adobe, […]

Adobe's plan to stick AI assistants into all of its Creative Cloud suite is now fully underway, with new chatbots now rolling out to its biggest editing and design apps. As part of a public beta launching today, Photoshop, Premiere, Illustrator, InDesign, and Frame.io now each have a bespoke AI Assistant that can be used […]

VS Code 1.123 adds a two-hour delay before auto-updating extensions to newly published versions, creating a revocation window against supply chain attacks. The delay does not apply to trusted publishers like Microsoft, GitHub, and OpenAI. Similar cooldown mechanisms have now spread across pip, RubyGems, npm, pnpm, Yarn, and Bun. By Steef-Jan Wiggers

One of the most common system design interview questions is: "How would you scale a web application from 100 users to 100 million users?" The answer is rarely a single technology. Instead, systems evolve through multiple stages, with each stage solving a specific bottleneck. This article walks through the typical evolution of a scalable system and explains why , how , and when each component is introduced. 1. Single Server Why Start Here? Every application starts simple. In the beginning: Traffic is low Development speed matters more than scalability Infrastructure costs should be minimal What Is It? A single machine handles everything: Frontend Backend Database Users | v Single Server ├── Application └── Database How Does It Work? User sends request. Application processes request. Database stores and retrieves data. Response is returned. Everything happens on one machine. Problem As traffic grows: CPU becomes overloaded Memory becomes insufficient Database competes with application for resources A single server becomes a bottleneck. Interview One-Liner A single server architecture is simple and cost-effective but becomes a bottleneck as traffic and resource usage increase. 2. Application and Database Separation Why Do We Need It? The application and database have different workloads. Application Server: Uses CPU Handles business logic Database Server: Uses memory and storage Handles queries Keeping them together causes resource contention. How Does It Work? Move the database to a separate machine. Users | v Application Server | v Database Server Benefits Independent scaling Better resource utilization Improved performance Example Suppose an e-commerce website receives thousands of requests. The application handles: Authentication Order processing API responses The database handles: Product data Orders User information Separating them prevents one workload from affecting the other. Interview One-Liner Separating the application and database allows each layer to scal

When organizations scale, a simple monolithic architecture often becomes difficult to maintain, deploy, and scale. This is where microservices come into the picture. However, moving to microservices introduces new challenges: How do we migrate from a monolith safely? How do we handle transactions across multiple services? How do we scale read-heavy applications efficiently? Three popular patterns solve these problems: Strangler Pattern – Monolith to Microservices Migration Saga Pattern – Distributed Transaction Management CQRS (Command Query Responsibility Segregation) – Read/Write Scalability 1. Strangler Pattern Why Do We Need It? Most companies cannot shut down a production monolith and rewrite everything from scratch. A complete rewrite is risky because: Development takes a long time. Existing customers are affected. Bugs can impact business operations. Rollback becomes difficult. The Strangler Pattern allows teams to migrate gradually with minimal risk. What Is It? The Strangler Pattern is a migration strategy where new microservices slowly replace parts of a monolithic application until the monolith is no longer needed. The name comes from the strangler fig tree, which gradually grows around another tree and eventually replaces it. How Does It Work? [Insert diagram here showing Client → API Gateway → Monolith + Microservices] Step 1 All requests go to the monolith. Client | v Monolith Step 2 Introduce an API Gateway (or Controller). Client | v API Gateway | v Monolith Step 3 Extract one module into a microservice. Client | v API Gateway |------> Order Service | v Monolith Step 4 Gradually move more modules. Client | v API Gateway |------> Order Service |------> Payment Service |------> Inventory Service | v Monolith Step 5 Eventually remove the monolith completely. Example Consider an e-commerce application. Initially, everything exists inside one application: Monolith ├── Orders ├── Payments ├── Inventory └── Users Over time: Orders become Order Service Payme

Amazon Fulfillment: The Three Tiers of Optimization Amazon processes billions of orders annually through a network of over 175 fulfillment centers globally. To maintain their 1-2 day (or same-day) delivery guarantees, they built a 3-tier optimization architecture: ┌─────────────────────────────────────────────────────────────┐ │ TIER 1: ANTICIPATORY SHIPPING (Long-term — weeks/months) │ │ → ML predicts demand → Moves inventory close to customers │ │ BEFORE they place an order │ ├─────────────────────────────────────────────────────────────┤ │ TIER 2: REGIONALIZATION (Medium-term — days/weeks) │ │ → Partitions the fulfillment network into autonomous zones│ │ → Ensures 70-80% of orders are fulfilled intra-region │ ├─────────────────────────────────────────────────────────────┤ │ TIER 3: CONDOR (Short-term — hours) │ │ → Continuously re-optimizes the fulfillment plan within │ │ a 5-6 hour window before pick-and-pack begins. │ └─────────────────────────────────────────────────────────────┘ Anticipatory Shipping — Shipping Before You Buy A Crazy but Effective Idea Amazon holds a patent (US Patent 8,615,473) describing a system that begins shipping items BEFORE a customer places an order . It sounds like science fiction, but it's a reality. Traditional Model: Customer orders → Warehouse processes → Ships → Delivered (2-5 days) Anticipatory Shipping: ML predicts: "Customers in Region X will buy 200 iPhone 16s in the next 3 days" → Amazon ships 200 iPhones from a central hub to local delivery hubs in Region X → Customer places order → The item is already locally staged → Delivered same-day! ML Model Input Features Input Feature Significance Purchase history What do they buy, and how often? Browsing behavior What are they looking at? Cart abandonment? Wishlists Explicitly desired items Seasonal patterns Winter coats in November, sunscreen in June Regional demographics High-income areas? Young families? College towns? Trending products Items going viral on social media Weathe