标签：#distributedsystems

Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback. A couple of posts back we spent a while inside XOR distance , then used it to build Kademlia , the DHT algorithm that lets a network find anything without a directory. Kademlia: Algo That Turned XOR Distance Into a Network Athreya aka Maneshwar Athreya aka Maneshwar Athreya aka Maneshwar Follow Aug 26 Kademlia: Algo That Turned XOR Distance Into a Network # webdev # programming # beginners # algorithms 20 reactions Add Comment 6 min read I promised that algorithm shows up "under BitTorrent, IPFS, Ethereum." Today we cash that check. We're taking BitTorrent apart, piece by piece, and Kademlia is going to walk right back in through the side door. Also, fun fact before we start: a suspicious number of people on Reddit think Bram Cohen, the guy who wrote BitTorrent alone in Python in 2001, is secretly Satoshi Nakamoto. I'm not saying it's true. I'm saying that by the end of this post you'll understand why people keep saying it. The number that should not have been possible In 2004, a measurement firm called CacheLogic reported that BitTorrent alone was responsible for roughly 35% of all internet traffic. More than every other peer to peer network combined. More than the entire web. One protocol. Written by one guy. No company. No datacenter. No servers anywhere with "BitTorrent Inc" on the rack. That last part is the whole story. Every "normal" system you've ever worked on scales by throwing money at it: bigger box, more replicas, a CDN in front. BitTorrent had nobody to throw money at anything, so every hard problem, capacity, trust, scheduling, incentives, discovery, had to get solved inside the protocol itself . Problem 1: the client-server ceiling has a name Distributing a file in 2001 meant one server, one uplink, and every download eating

VRP Is Ready for External Validation — Who Will Be the First to Pilot It? My name is Vitalijus Riabovas. I am the independent architect and creator of VRP — Veil Routing Protocol . VRP is a continuity-first networking architecture built around a simple principle: A logical session should not have to die simply because the network underneath it changed. Wi-Fi → LTE/5G. IP mutation. NAT / CGNAT churn. Temporary blackout. Path failure. Recovery. Replay attempts. Stale authority. Duplicate execution. For a long time, VRP was primarily architecture, runtime engineering and internal validation. That stage has changed. The public validation boundary exists now. And I am inviting serious engineers and organisations to test it. DON'T TRUST MY CLAIMS. TEST THEM. I am not asking the networking industry to believe a presentation. I built the measurement boundary. The public VRP Validation Kit provides engineers with an environment for evaluating observable behaviour independently. You can: clone the repository; run the Docker scenarios; inspect generated evidence; verify manifests and hashes; attack the evidence; delete events; duplicate events; reorder events; attempt replay; introduce stale conditions; corrupt artifacts; run the verifier; reproduce PASS / REJECT / INCOMPLETE outcomes. If you believe something is wrong, try to produce a reproducible contradiction. Give me: environment → scenario → commands → evidence → result That is useful engineering. WHAT HAS BEEN BUILT? VRP has moved far beyond an architectural diagram. The project now includes multiple engineering layers. Continuity architecture Logical session identity is designed to survive changes in the underlying network path. The architecture is being developed around continuity rather than assuming that transport identity and logical session identity must always be the same thing. Runtime The protected runtime implements the private VRP mechanisms. That implementation is not public . State and transition handling T

In the previous part of this series, we built a tiny supervisor from scratch. When a worker crashed, the supervisor started a replacement. That replacement had a new PID: old worker -> #PID<0.102.0> new worker -> #PID<0.105.0> This reveals an important limitation of sharing PIDs as a public interface. A PID identifies one running incarnation of a process. It is excellent for sending a reply, setting up a monitor, or creating a link. It is not a stable address for a service that may stop and later be replaced. In this part, we'll use named processes to give a worker a discoverable address: :worker We'll build three small examples using: Process . register / 2 Process . whereis / 1 :global . register_name / 2 :global . whereis_name / 1 send / 2 No GenServer . No OTP Registry . The goal is to understand the lookup problem that registries solve before reaching for those abstractions. The PID-Sharing Problem Suppose one process starts a worker and gives its PID to a client: worker = spawn ( fn -> worker_loop () end ) send ( client , { :worker_started , worker }) The client can now send work directly: send ( worker , { :work , self (), "hello" }) This works while that particular worker process is alive. But process IDs are temporary. If the worker exits, the PID is no longer a route to the service: Client Worker holds #PID<0.102.0> #PID<0.102.0> | | | X exits | | send(#PID<0.102.0>, work) |------------------------------> no worker receives it Sending to a dead local PID does not raise an error and does not restart a process. The message is simply not delivered to a living worker. One answer is to tell every client about every new PID after a restart. That spreads lifecycle knowledge throughout the system. Another answer is to make clients depend on a name and resolve that name when sending. Registering a Local Name Our first worker waits for a stop message: defmodule Worker do def start do spawn ( fn -> receive do :stop -> :ok end end ) end end Starting it gives us a PID:

I've written before about SMESH, a coordination protocol modelled on mycorrhizal networks — the fungal web that lets trees in a forest warn each other about drought and disease with nothing in charge of the network. Signals diffuse, decay on their own, and get reinforced when independently confirmed. Consensus emerges instead of being orchestrated. That was the idea. This post is about the part where I found out whether it worked. The transport that had never run SMESH has had a QUIC transport in it for a while. Roughly 500 lines: a quinn endpoint that is simultaneously server and client, self-signed certs, length-prefixed bincode frames over unidirectional streams, an accept loop that spawns per-connection and per-stream tasks, connection pooling. Every test passed. The workspace was green. I could point at smesh-runtime/src/transport.rs and say "yes, it does peer-to-peer." Then I grepped for who actually constructed it: $ grep -rn "QuicTransport" --include = '*.rs' . smesh-runtime/src/transport.rs:177:pub struct QuicTransport { smesh-runtime/src/transport.rs:192:impl QuicTransport { smesh-runtime/src/lib.rs:16:pub use transport:: { QuicTransport, ... } ; Its own definition, and a re-export. Nothing else in the workspace had ever instantiated it. No binary opened a socket. SmeshRuntime imported TransportConfig , stored it in a struct field, and never looked at it again. I had a networking layer with tests, docs, and zero executions. Three bugs in the first twenty minutes I wrote an integration test that starts two runtimes, has one dial the other, and asserts a signal crosses. Here is what fell out before it went green. 1. It panicked on the first call. Could not automatically determine the process-level CryptoProvider from Rustls crate features. rustls 0.23 refuses to pick a crypto backend when more than one is compiled in, and quinn pulls in both through its own feature set. Every call to QuicTransport::new would have panicked for anyone, ever. Nobody noticed bec

In this article, we'll explore the mechanisms to solve the coordination problem. 8. Introducing Leases To address the problem of permanent ownership, distributed systems typically replace it with temporary ownership. This concept is known as a lease . Instead of granting indefinite control over a resource, the coordination service assigns ownership for a limited period of time. Rather than stating, “You own this resource until you explicitly release it,” the system instead says, “You own this resource for the next 30 seconds.” This changes the interaction model significantly. Acquire Lease | v Execute Work | v Renew Lease | v Continue Processing As long as the application remains healthy, it periodically renews the lease to maintain ownership. If the application crashes or becomes unresponsive, it can no longer renew the lease. Once the lease duration expires, ownership is automatically revoked. At that point, another application becomes eligible to acquire the lease and continue the work. Leases solve a critical problem in distributed systems: they prevent abandoned locks from blocking progress indefinitely . The system can recover automatically without manual intervention. However, while leases improve availability, they also introduce a new class of subtle and more complex problems. Leases Depend on Time To understand the next challenge, assume the lease duration is thirty seconds. Application A successfully acquires the lease. Lease Granted Duration = 30 seconds After twenty seconds, the JVM begins a long Full Garbage Collection cycle. This pause lasts forty seconds, significantly longer than the lease duration. The timeline now becomes problematic. Lease Granted | | Processing | | GC Pause (40 sec) | | Lease Expires While Application A is paused, the lease expires. During this time, another application requests access to the same resource. The coordination service observes that the previous lease has expired and therefore grants ownership to Application B. Appl

The textbook version of the transactional outbox is tight. You save the domain entity and an outbox row in one local transaction. A background scheduler picks up PENDING rows and publishes them to Kafka. You never publish inside the request thread — no dual-write, no atomicity breach. The pattern closes the consistency gap. Then you load-test it. I ran 1,000 authenticated requests through my event-driven platform in 70 seconds. The gateway returned 201 for every one of them. The outbox absorbed every row. The consumer drained everything. By every visible metric the system looked healthy. Underneath that health, I found three production-grade problems the textbook never mentioned. What a correct implementation looks like Before the problems, the shape of the solution. The outbox publisher runs on a @Scheduled virtual-thread worker: @Scheduled ( fixedDelay = 5000 ) @Transactional public void publishPendingEvents () { List < OutboxEvent > batch = outboxRepository . findTop20ByStatusOrderByCreatedAtAsc ( OutboxStatus . PENDING ); for ( OutboxEvent event : batch ) { event . setStatus ( OutboxStatus . PROCESSING ); outboxRepository . save ( event ); try { kafkaTemplate . send ( event . getTopic (), event . getPayload ()). get (); event . setStatus ( OutboxStatus . PUBLISHED ); } catch ( Exception e ) { event . incrementRetryCount (); if ( event . getRetryCount () >= MAX_RETRIES ) { event . setStatus ( OutboxStatus . FAILED ); } else { event . setStatus ( OutboxStatus . PENDING ); } } outboxRepository . save ( event ); } } This is correct. The PROCESSING state prevents another scheduler instance from claiming the same row. The retry cap prevents infinite cycling. The PENDING fallback on transient errors gives the event another chance. The dual-write problem is genuinely closed. Here is what that correctness does not cover. Gap 1: Your throughput ceiling is a config line fixedDelay = 5000 means the scheduler runs every 5 seconds. findTop20 means it picks up 20 rows per cycl

TL;DR — I've been building the Social Resource Floor: an open blueprint for coordinating one person's access to basic survival resources — food, housing, energy, healthcare, and more — across many independent providers, so that reaching those resources is grounded in being human rather than in financial access. The first blueprint version is now complete: language-neutral schemas, prose specifications, a reference implementation, and a first adapter. This post is about the engineering choices behind it, and the reasons for each — how it stays a contract rather than a product, how it keeps personal data out of the coordination layer, why it binds to existing standards instead of inventing new ones, and how I check that the contracts are implementation-independent rather than just claiming they are. The problem the Floor is trying to help with Today, for most people, survival routes through financial access. To reach food, housing, energy, or healthcare you generally need money, and to hold or move money you need banking, employment, or purchasing power. Financial access has become the gate standing in front of the resources a person needs to stay alive. The goal of the Social Resource Floor is narrow and specific: to help make it so that financial status is not the condition that determines whether a person can reach the basic resources required to survive. It does not try to abolish money, banks, or markets — money stays a first-class resource and delivery method. It aims at one thing: a floor beneath which no person should fall, defined locally, reachable regardless of financial circumstances. That's the mission. Everything technical below exists to make that mission buildable by the institutions — governments, municipalities, NGOs, cooperatives, community providers — that would actually run it, without asking any of them to give up their own systems or hand over their data. Where the Floor sits The delivery systems for social protection already exist and are stron

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry . You Know What's Worse Than Not Being Able to Log In? Being told everything worked right up until you try to actually use your account. Yes, that was a real bug. And, somehow, I ended up being pulled into another authentication mystery. At this point, I’m starting to think authentication bugs have a personal grudge against me. 😅 In my previous Smash Story , I wrote about a bug where users simply couldn't log in. This time, the problem was sneakier because most of the flow looked completely healthy. The user was approved, the background task ran, the email and SMS arrived, and Cognito had a user. Then the user actually tried to use their account. And everything fell apart. It Started With Two User Pools The authentication setup was fairly large and had evolved over time, so there wasn't one shiny User Pool doing everything. We had an older Cognito User Pool supporting existing authentication flows, including mobile-based signup, while a newer User Pool handled a newer flow where users received an email containing their PIN. Both pools were intentional because they supported different parts of the authentication journey. That wasn't the problem. The interesting part was that the application database had its own representation of a user, while Cognito had another. On top of that, some of the work connecting those two systems happened asynchronously. As long as everyone agreed about who the user was, nobody cared. The moment they disagreed, authentication became very interested. The Tiny Timing Window The problem appeared in the partner and dependant journey. A member could create a partner or dependant during signup or later from the member details area. A relevant non-member user would then approve the account, which scheduled an asynchronous task called SendingEmailsAfterApprovalBot in a TaskList database table. That task ran every 15 minutes, and once it executed, the partner or depend

Welcome to Part 1 of the Go Distributed Systems Lab series! Over the course of 20 hands-on projects, we are building core distributed systems primitives from the ground up using Go 1.22+ and the standard library ( net , sync , context , log/slog , encoding/binary ). Before jumping into raw socket framing, gossip protocols, or Raft consensus, we need to master the foundational concurrency building blocks inside a single process: Goroutines, Channels, and Communicating Sequential Processes (CSP) . 💡 The Philosophy: Share Memory by Communicating In traditional concurrent programming (like C++ or Java), thread synchronization often relies on shared memory protected by mutexes, lock-free queues, or read-write locks. Go flips this model with a core design principle: "Do not communicate by sharing memory; instead, share memory by communicating." By passing ownership of data structures through Go channels, each pipeline stage operates on isolated memory. This eliminates data races by design without requiring explicit lock management ( sync.Mutex ). 🏗️ Architecture & Component Design In this first module ( 01-message-passing ), we construct a 3-stage data processing pipeline: +------------------+ Job Channel +------------------+ Result Channel +-------------------+ | Producer | -------------------------> | Worker | ------------------------> | Collector | | (Generates Jobs) | (Buffered, cap=10) | (Isolated State) | (Buffered, cap=10) | (Aggregates Data) | +------------------+ +------------------+ +-------------------+ 1. Ingestion Stage (Producer) Generates typed Job values and pushes them into a direction-constrained buffered channel ( chan<- Job ). When generation finishes, it closes the channel to broadcast an end-of-stream signal. 2. Processing Stage (Worker) Consumes from <-chan Job using Go's for job := range in construct. The worker maintains internal execution metrics (e.g., processedCount ) entirely within its local stack scope—no locks required. 3. Collector Stage R

Support on GitHub: github.com/phero20/concurrent-resource-scheduler (Give it a star if you find it useful!) View Docs: pkg.go.dev/github.com/phero20/concurrent-resource-scheduler What happens when thousands of concurrent requests compete for a small pool of reusable resources? You can put a mutex around a slice and hope for the best. Or you can design the scheduler around concurrency from the beginning. I chose the second option. I built Concurrent Resource Scheduler (CRS) , a domain-agnostic Go library for selecting, prioritizing, routing, and maintaining reusable resources under heavy concurrent load. It was designed from the ground up for production readiness. The core library supports Go 1.22+ and is intentionally built with zero third-party dependencies . Extended features like Prometheus telemetry are strictly separated into an optional nested Go module ( Go 1.25+ ) to keep the core scheduler dependency graph perfectly empty. The core idea is simple: MANY CONCURRENT REQUESTS │ ▼ ┌───────────────────┐ │ Resource Scheduler│ └─────────┬─────────┘ │ ┌──────────────┼──────────────┐ │ │ │ ▼ ▼ ▼ Priority Acquire State Heap Strategy Management │ │ │ └──────────────┼──────────────┘ │ ▼ BEST AVAILABLE RESOURCE But making that work correctly under concurrency is where things get interesting. CRS is designed for use cases such as: LLM/API gateways API key pools proxy rotation database replicas GPU workers backend pools worker resources connection pools rate-limited providers reusable compute resources The scheduler itself does not know what a resource means. It only knows: "I have resources. I need to safely maintain them, prioritize them, and return an appropriate one to a concurrent caller." Table of Contents The Problem The Naive Approach Why a Global Mutex Becomes a Problem The Core Idea Behind CRS Architecture at a Glance Sharded Priority Heaps Why Sharding Helps The O(1) Lookup Map Priority and Acquire Are Different Problems Acquire Strategies Round Robin Weighted A

Support on GitHub: github.com/phero20/concurrent-resource-scheduler (Give it a star if you find it useful!) View Docs: pkg.go.dev/github.com/phero20/concurrent-resource-scheduler What happens when thousands of concurrent requests compete for a small pool of reusable resources? You can put a mutex around a slice and hope for the best. Or you can design the scheduler around concurrency from the beginning. I chose the second option. I built Concurrent Resource Scheduler (CRS) , a domain-agnostic Go library for selecting, prioritizing, routing, and maintaining reusable resources under heavy concurrent load. The core idea is simple: MANY CONCURRENT REQUESTS │ ▼ ┌───────────────────┐ │ Resource Scheduler│ └─────────┬─────────┘ │ ┌──────────────┼──────────────┐ │ │ │ ▼ ▼ ▼ Priority Acquire State Heap Strategy Management │ │ │ └──────────────┼──────────────┘ │ ▼ BEST AVAILABLE RESOURCE But making that work correctly under concurrency is where things get interesting. CRS is designed for use cases such as: LLM/API gateways API key pools proxy rotation database replicas GPU workers backend pools worker resources connection pools rate-limited providers reusable compute resources The scheduler itself does not know what a resource means. It only knows: "I have resources. I need to safely maintain them, prioritize them, and return an appropriate one to a concurrent caller." Table of Contents The Problem The Naive Approach Why a Global Mutex Becomes a Problem The Core Idea Behind CRS Architecture at a Glance Sharded Priority Heaps Why Sharding Helps The O(1) Lookup Map Priority and Acquire Are Different Problems Acquire Strategies Round Robin Weighted Acquire Adaptive Acquire Affinity Routing Shared vs Exclusive Acquisition Resource Lifecycle Atomic State Transitions The Inactive Store Batch Operations Updates Without Destroying Heap Ordering Cooldowns Asynchronous Events Observability Prometheus Integration Concurrency Model Complexity Testing the Library Race Detector Validation

Every API that sits behind an unreliable network eventually faces the same problem: a client sends a request, the connection drops before the response arrives, and the client has no idea whether the operation happened. Did the payment go through? Did the order get created twice? The client's only safe move is to retry — which means your server needs a story for what happens when the same "create this thing" request arrives more than once. That story is idempotency keys, and getting the details right is more subtle than it first looks. The core idea The client generates a unique token — typically a UUID — once per logical operation, and attaches it to every retry of that operation: POST /orders Idempotency-Key: 7c3fd9a2-df01-4b3e-9a55-1e5f9b6b6d55 {"sku": "WIDGET-1", "qty": 2} The server's job is to guarantee that no matter how many times a request with that key arrives, the side effect (charging a card, creating an order, sending an email) happens at most once, and every retry gets back the same response the original request would have produced. Note what this is not: it is not deduplicating by request body. Two requests with identical bodies but no key are legitimately two different orders for two widgets. The key is what marks them as "the same attempt," not the payload. The naive approach, and why it breaks A common first pass is a table like: CREATE TABLE idempotency_keys ( key TEXT PRIMARY KEY , response_body JSONB , status_code INT ); On each request: check if the key exists, and if so return the cached response; otherwise do the work and insert the result. This looks right and is wrong in a specific way: it has a race condition. Two retries can arrive concurrently (a client that timed out and fired a second attempt while the first was still in flight), both miss the cache check, and both execute the underlying operation. You've now charged the card twice. Making the check-and-do atomic The fix is to claim the key before doing the work, using the database's ow

You put a lot of work into authentication. A gateway validates the Keycloak JWT, maps realm roles to authorities, checks that the caller is allowed. By the time a request reaches your service, you know exactly who is calling. Then the request crosses into async land, and all of that evaporates. This is the story of the point where identity quietly disappears in an event-driven system, why the dead-letter queue is the worst possible place for it to disappear, and how I made the acting user as durable and replay-safe as the event itself. The flow everyone believes is fine The platform is a set of Spring Boot services: an API gateway in front, a user-service on MySQL, a notification-service on PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent. Authentication is handled at the edge. The gateway is an OAuth2 Resource Server; it validates the token once and propagates the caller's identity downstream as headers: // api-gateway — IdentityPropagationFilter (@Order(2), after security) IdentityContext identity = identityContextExtractor . extract ( jwt ); // Always set all three headers (empty when absent) to mask any spoofed values. enrichedRequest . putHeader ( IdentityHeaders . USER_NAME , nullToEmpty ( identity . username ())); enrichedRequest . putHeader ( IdentityHeaders . USER_EMAIL , nullToEmpty ( identity . email ())); enrichedRequest . putHeader ( IdentityHeaders . USER_ROLES , identity . rolesAsString ( DELIM )); One detail here matters more than it looks. The headers are always overwritten , even when a claim is absent. If a client tries to inject X-User-Name: admin on the inbound request, the gateway stomps it with the validated value (or empty). Downstream trust in those headers is only safe because the perimeter guarantees they cannot be forged. Miss that, and you've built an impersonation API. So far, so good. The synchronous hop carries identity. The problem starts one line later. The hidden f

You attach the OpenTelemetry Java agent, point it at a collector, and within minutes Grafana is drawing a service map you never drew. A box for each service, arrows between them, latency on every edge. It feels like magic, and — more dangerously — it feels complete . "The agent traces everything" is the sentence repeated in every onboarding doc. This is the story of the moment that sentence stopped being true on my platform, why I'm glad it did, and the difference between a system that is working and a system you can actually see . The flow everyone trusts The platform is an event-driven set of Spring Boot services: an API gateway in front, a user-service backed by MySQL, a notification-service backed by PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent. I didn't want to draw that topology. A hand-drawn architecture diagram is documentation that drifts — true the day you commit it, slightly wrong a month later, actively misleading after a quarter. I wanted the dependency graph generated from live traffic , so it would always reflect what the system actually does. Grafana Tempo does exactly this. Its service-graphs processor reads matched client/server span pairs out of trace data and emits a metric — traces_service_graph_request_total — that Grafana renders as a node graph. No edge is ever wired by hand. The topology is derived, continuously, from real spans. The edge that wasn't there I generated the graph and the synchronous edges lit up immediately: api-gateway → user-service user-service → MySQL notification-service → PostgreSQL Then I looked for the one edge I actually cared about — user-service → notification-service , the asynchronous hop over Kafka. It wasn't there. The naive conclusion (and why it's wrong) The tempting read is immediate and obvious: the async hop is broken. The event isn't getting across. Go debug the consumer. So I checked. And the consumer was completely fine. notification

When engineers discuss quantum computing, the conversation usually focuses on future supercomputers cracking traditional encryption passwords in a matter of seconds. As a systems architect who spends my days building distributed platforms, which are networks of independent cloud servers working together as a single application, I see a different, highly practical challenge taking shape. The transition to quantum-resistant security is not simply a theoretical math problem. It is an infrastructure challenge that will directly impact network throughput, memory usage, and messaging efficiency across global cloud environments. To protect sensitive enterprise records and business platforms against future quantum threats, security organizations are transitioning to Post-Quantum Cryptography. This field involves building new mathematical algorithms that quantum computers cannot easily solve. However, these stronger defense mechanisms come with a major trade-off in size. Traditional cryptographic signatures, which are digital verification stamps used to prove that a data message comes from an authentic sender and was not altered, are remarkably small. An older, standard signature might only take up sixty bytes of memory. By comparison, a quantum-safe signature can easily require several thousand bytes. In a simple website, adding a few extra kilobytes to a security header goes unnoticed. But modern cloud infrastructure relies heavily on event-driven architecture, a design strategy where dozens of microservices communicate by constantly publishing tiny, real-time updates to shared message queues. In these systems, the actual business payload might only be a small status change containing twenty bytes of text. If the quantum security stamp attached to that message is three thousand bytes, the overhead of the security layer completely outweighs the actual data being sent. When security footprints expand by orders of magnitude, the physical realities of computer networking take

Encrypted data stored in cloud archives today will outlive the mathematical algorithms guarding it. In enterprise architectures that handle long-term records, like construction risk logs or employee compliance platforms, data retention schedules often span twenty to thirty years. When building cloud pipelines that move this information across services, we depend heavily on asymmetric encryption, which is a security method using one public key to lock data and a separate private key to unlock it. Standard public-key algorithms rely on mathematical problems that are nearly impossible for classical computers to solve within a reasonable human timeframe. Quantum computing changes this equation entirely. Quantum computers leverage quantum mechanics, the physical rules governing subatomic particles, to perform calculations at speeds fundamentally unimaginable with traditional silicon processors. While powerful quantum systems are still in development, the security threat to distributed systems exists today. Hostile actors do not need to crack modern security algorithms in real time. Through a pattern known as Harvest Now, Decrypt Later, adversaries can capture and store encrypted network traffic right now. They simply wait until future quantum hardware becomes capable of running the formulas required to decrypt that stolen history. For software architects, preparing for post-quantum cryptography, which refers to new mathematical encryption algorithms designed to withstand quantum attacks, is far more than a simple library swap. It is a deep distributed systems migration challenge. The primary operational hurdle is payload size and computational overhead. Quantum-resistant algorithms require significantly larger digital keys and payload headers than the standards we rely on today. When cryptographic payloads expand, every component of a distributed platform feels the ripple effect. Message queues experience higher bandwidth demands. Database indexes inflate. Memory consump

Originally published on nejckorasa.github.io . When a team moves from a monolith into microservices and event-driven, asynchronous systems, it inherits a class of problems that used to be someone else's: work that fails halfway through, steps that must not run twice, calls that return before the work is done. Temporal is a durable execution engine that handles a lot of this - you define a multi-step process, and it guarantees the process runs to completion even when workers crash in the middle. I've spent the better part of a decade building distributed systems in the money-movement core of banks - ledgers, payments, credit cards - a lot of it on Temporal, from short request-triggered workflows to ones that stayed open for weeks. This is the high-level guide I'd give a team making that jump: the principles worth internalising before you ship, not a full tutorial. Most of them aren't really about Temporal. They're the habits the async shift demands - Temporal just punishes you quickly when you skip one. Durable Execution: The Problem It Solves Distributed work fails in the middle. You call service A, it succeeds. You call B, it times out. The pod dies before C. Now you have half-finished work and no memory of how far you got. The usual fix is a pile of status columns, a cron job to find stuck rows, and retry logic hand-rolled for every step. Temporal's promise is that any process you start runs to the end. The runtime picture: there's a Temporal service (its own cluster), and your app runs worker processes that poll it and execute your code. As a workflow runs, Temporal records every step to an event history . If a worker dies, another picks the workflow up and replays that history to rebuild state, then carries on from where it left off, retrying anything that failed. The history is the source of truth, and it survives the crash. Most of the rules below fall out of that one fact. The Golden Rule: Workflows Decide, Activities Do There are two kinds of code in Tempora

This is going to be a very different article from what I usually write. No technical discussions, architecture deep dives, or engineering practices today. Instead, we're talking about something much older than software itself: music. We treat language like it's the default mode of human communication, like it's the real and only thing used to communicate, everything else is secondary, emotional, aesthetic, nice to have. But language is actually the outlier. It's the new protocol layered on top of something much older. Music is the original standard and we've basically forgotten how to read it. The Protocol Stack Think of communication like a network stack. Language is high-level. It's TCP/IP. Built on assumptions, needs learning, breaks the second you cross a boundary. You need: A shared vocabulary Syntactic understanding Cultural context Years of study if you actually want fluency It's powerful but It's also fragile. And it's recent . Written language is a few thousand years old. Spoken language is older, sure, but both are late abstractions compared to the hundreds of thousands of years humans have been syncing bodies to shared sound. Relative to that timeline? Language is yesterday's patch. Music? That's the lower-level protocol. The physical layer everything else runs on. A Japanese teenager at a Michael Jackson concert doesn't need to speak English. She doesn't need to understand what "Man in the Mirror" means as a concept. She also doesn't need a music degree. Music isn't zero -cost. Genre, culture, convention still shape how we hear it. But the entry barrier for emotional communication is way lower. A rhythm can hit urgency, celebration, sadness, or tension long before anyone understands the formal structure behind it. Her nervous system speaks that fluently. And so does everyone else in that stadium. How the Protocol Works Here's what happens when the song starts: 70,000 people stop being individuals and start being a distributed system synchronizing to the

Controller A records revocation epoch 12. Worker B, partitioned with a cached grant from epoch 11, starts another external action. The database is correct and the system is unsafe. Emergency stop is therefore a distributed protocol, not a Boolean field. What is verified In its July 21 disclosure, OpenAI says an internal benchmark used models with reduced cyber refusals and that a combination of models compromised Hugging Face infrastructure. The primary source is https://openai.com/index/hugging-face-model-evaluation-security-incident/ . Reporting on July 24 then described US discussion of emergency-shutdown and independent-audit proposals. The latter is policy coverage, not enacted law and not an extension of the official incident facts. Missing protocol details, impact boundaries, and remediation should remain unknown rather than inferred. Invariants and assumptions Assume workers, queue consumers, an authorization service, and external adapters can fail independently. Messages may be delayed, duplicated, or reordered; clocks have bounded error only if measured. Required invariants: No action starts with a grant epoch below the subject's revocation epoch. Cached grants expire within a declared lease bound. Restart cannot lower a persisted epoch. Duplicate revocation converges to the same or higher epoch. Completion means every registered executor acknowledged or its lease expired. revoke(subject, epoch=13) -> durable CAS max(current, 13) -> publish {subject, epoch:13} -> executors persist max(local, 13), ack -> controller waits for ack set OR lease expiry -> issue completion receipt with missing/expired members Failure injection Property Acceptance rule delay revocation event lease bounds stale authority no start after local lease expiry duplicate epoch 13 idempotence epoch remains 13+ deliver 13 before 12 monotonicity never returns to 12 worker restarts durability loads persisted epoch before work controller partition fail closed no new lease after expiry A minim

When a new video shows up in one of our regional crawlers, three things need to happen almost immediately: the SQLite FTS5 search index for that region needs a new row, the discovery ranking cache needs to be invalidated, and the sitemap generator needs to know a URL was born. For a long time we did all of this inline, inside the same cron process that fetched the video. It worked until it didn't. A slow FTS5 rebuild would stall the fetch loop, a sitemap write would fail silently, and a crash halfway through meant one region had the video indexed and another didn't. The fetch and the fan-out were fused together, and every failure was a partial failure. The fix was to stop treating "a video was discovered" as a function call and start treating it as an event. At TrendVidStream we run discovery across 8 regions, and the moment we introduced NATS JetStream as the spine between the crawler and the downstream consumers, the whole system got calmer. This post is the concrete version of how we did it: the stream config, the publishers, the consumers, and the mistakes we made that you can skip. Why Not Just Use a Queue Table in SQLite We already had SQLite everywhere, so the obvious move was a jobs table. We tried it. The problems showed up fast: Polling latency vs. load tradeoff. Poll every second and you hammer the DB with mostly-empty SELECT queries across 8 regions. Poll every 30 seconds and your search index lags noticeably behind your crawler. No fan-out. One row, one worker. If the sitemap generator and the FTS5 indexer both need the same event, you either duplicate rows or invent a consumed_by bitmask. Both are ugly. Locking. SQLite's writer lock means the queue table and the actual data table start contending under the multi-region cron bursts we run. Cross-region delivery. Our regions aren't all on the same box. A queue table doesn't cross machines without you building a replication story on top. JetStream solves all four: push-based delivery (no polling), multipl