标签：#systems

Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback. A couple of posts back we spent a while inside XOR distance , then used it to build Kademlia , the DHT algorithm that lets a network find anything without a directory. Kademlia: Algo That Turned XOR Distance Into a Network Athreya aka Maneshwar Athreya aka Maneshwar Athreya aka Maneshwar Follow Aug 26 Kademlia: Algo That Turned XOR Distance Into a Network # webdev # programming # beginners # algorithms 20 reactions Add Comment 6 min read I promised that algorithm shows up "under BitTorrent, IPFS, Ethereum." Today we cash that check. We're taking BitTorrent apart, piece by piece, and Kademlia is going to walk right back in through the side door. Also, fun fact before we start: a suspicious number of people on Reddit think Bram Cohen, the guy who wrote BitTorrent alone in Python in 2001, is secretly Satoshi Nakamoto. I'm not saying it's true. I'm saying that by the end of this post you'll understand why people keep saying it. The number that should not have been possible In 2004, a measurement firm called CacheLogic reported that BitTorrent alone was responsible for roughly 35% of all internet traffic. More than every other peer to peer network combined. More than the entire web. One protocol. Written by one guy. No company. No datacenter. No servers anywhere with "BitTorrent Inc" on the rack. That last part is the whole story. Every "normal" system you've ever worked on scales by throwing money at it: bigger box, more replicas, a CDN in front. BitTorrent had nobody to throw money at anything, so every hard problem, capacity, trust, scheduling, incentives, discovery, had to get solved inside the protocol itself . Problem 1: the client-server ceiling has a name Distributing a file in 2001 meant one server, one uplink, and every download eating

Kasia Trapszo discusses how Netflix evolved its commerce platform from a U.S. DVD service into global infrastructure. She explains navigating international payment realities, adapting to strict regulatory mandates, decomposing monolithic architectures along domain boundaries, and re-architecting systems for massive live-event demand - proving great systems survive by continually evolving. By Kasia Trapszo

Cloudflare recently open-sourced Cloudflare OS. It allows enterprise teams to output work artifacts grounded in enterprise knowledge, know-how, and provisioned connectors, automate repetitive workflows with optimized token cost (with AI assistance only where needed), and build personal, shareable, customizable work software that caters to specific, complex use cases within a secure sandboxed model By Bruno Couriol

VRP Is Ready for External Validation — Who Will Be the First to Pilot It? My name is Vitalijus Riabovas. I am the independent architect and creator of VRP — Veil Routing Protocol . VRP is a continuity-first networking architecture built around a simple principle: A logical session should not have to die simply because the network underneath it changed. Wi-Fi → LTE/5G. IP mutation. NAT / CGNAT churn. Temporary blackout. Path failure. Recovery. Replay attempts. Stale authority. Duplicate execution. For a long time, VRP was primarily architecture, runtime engineering and internal validation. That stage has changed. The public validation boundary exists now. And I am inviting serious engineers and organisations to test it. DON'T TRUST MY CLAIMS. TEST THEM. I am not asking the networking industry to believe a presentation. I built the measurement boundary. The public VRP Validation Kit provides engineers with an environment for evaluating observable behaviour independently. You can: clone the repository; run the Docker scenarios; inspect generated evidence; verify manifests and hashes; attack the evidence; delete events; duplicate events; reorder events; attempt replay; introduce stale conditions; corrupt artifacts; run the verifier; reproduce PASS / REJECT / INCOMPLETE outcomes. If you believe something is wrong, try to produce a reproducible contradiction. Give me: environment → scenario → commands → evidence → result That is useful engineering. WHAT HAS BEEN BUILT? VRP has moved far beyond an architectural diagram. The project now includes multiple engineering layers. Continuity architecture Logical session identity is designed to survive changes in the underlying network path. The architecture is being developed around continuity rather than assuming that transport identity and logical session identity must always be the same thing. Runtime The protected runtime implements the private VRP mechanisms. That implementation is not public . State and transition handling T

The database platform Harper advocates for a single-runtime architecture that keeps application code and data together, with its benchmark against a Vercel-based stack reporting significantly better performance on live, personalized-data workloads. Harper recently released version 5.2, with a new record cache and more throughput per node. By Renato Losio

In the previous part of this series, we built a tiny supervisor from scratch. When a worker crashed, the supervisor started a replacement. That replacement had a new PID: old worker -> #PID<0.102.0> new worker -> #PID<0.105.0> This reveals an important limitation of sharing PIDs as a public interface. A PID identifies one running incarnation of a process. It is excellent for sending a reply, setting up a monitor, or creating a link. It is not a stable address for a service that may stop and later be replaced. In this part, we'll use named processes to give a worker a discoverable address: :worker We'll build three small examples using: Process . register / 2 Process . whereis / 1 :global . register_name / 2 :global . whereis_name / 1 send / 2 No GenServer . No OTP Registry . The goal is to understand the lookup problem that registries solve before reaching for those abstractions. The PID-Sharing Problem Suppose one process starts a worker and gives its PID to a client: worker = spawn ( fn -> worker_loop () end ) send ( client , { :worker_started , worker }) The client can now send work directly: send ( worker , { :work , self (), "hello" }) This works while that particular worker process is alive. But process IDs are temporary. If the worker exits, the PID is no longer a route to the service: Client Worker holds #PID<0.102.0> #PID<0.102.0> | | | X exits | | send(#PID<0.102.0>, work) |------------------------------> no worker receives it Sending to a dead local PID does not raise an error and does not restart a process. The message is simply not delivered to a living worker. One answer is to tell every client about every new PID after a restart. That spreads lifecycle knowledge throughout the system. Another answer is to make clients depend on a name and resolve that name when sending. Registering a Local Name Our first worker waits for a stop message: defmodule Worker do def start do spawn ( fn -> receive do :stop -> :ok end end ) end end Starting it gives us a PID:

Sam Newman discusses the concept of progressive collapse in civil engineering and how it applies to distributed systems. Using real-world examples - from the 1968 Ronan Point tower failure to AWS outages - he shares crucial resilience engineering strategies for software leaders. Learn how to strengthen components, isolate failures, and reduce interconnections to prevent catastrophic cascades. By Sam Newman

I've written before about SMESH, a coordination protocol modelled on mycorrhizal networks — the fungal web that lets trees in a forest warn each other about drought and disease with nothing in charge of the network. Signals diffuse, decay on their own, and get reinforced when independently confirmed. Consensus emerges instead of being orchestrated. That was the idea. This post is about the part where I found out whether it worked. The transport that had never run SMESH has had a QUIC transport in it for a while. Roughly 500 lines: a quinn endpoint that is simultaneously server and client, self-signed certs, length-prefixed bincode frames over unidirectional streams, an accept loop that spawns per-connection and per-stream tasks, connection pooling. Every test passed. The workspace was green. I could point at smesh-runtime/src/transport.rs and say "yes, it does peer-to-peer." Then I grepped for who actually constructed it: $ grep -rn "QuicTransport" --include = '*.rs' . smesh-runtime/src/transport.rs:177:pub struct QuicTransport { smesh-runtime/src/transport.rs:192:impl QuicTransport { smesh-runtime/src/lib.rs:16:pub use transport:: { QuicTransport, ... } ; Its own definition, and a re-export. Nothing else in the workspace had ever instantiated it. No binary opened a socket. SmeshRuntime imported TransportConfig , stored it in a struct field, and never looked at it again. I had a networking layer with tests, docs, and zero executions. Three bugs in the first twenty minutes I wrote an integration test that starts two runtimes, has one dial the other, and asserts a signal crosses. Here is what fell out before it went green. 1. It panicked on the first call. Could not automatically determine the process-level CryptoProvider from Rustls crate features. rustls 0.23 refuses to pick a crypto backend when more than one is compiled in, and quinn pulls in both through its own feature set. Every call to QuicTransport::new would have panicked for anyone, ever. Nobody noticed bec

In this article, we'll explore the mechanisms to solve the coordination problem. 8. Introducing Leases To address the problem of permanent ownership, distributed systems typically replace it with temporary ownership. This concept is known as a lease . Instead of granting indefinite control over a resource, the coordination service assigns ownership for a limited period of time. Rather than stating, “You own this resource until you explicitly release it,” the system instead says, “You own this resource for the next 30 seconds.” This changes the interaction model significantly. Acquire Lease | v Execute Work | v Renew Lease | v Continue Processing As long as the application remains healthy, it periodically renews the lease to maintain ownership. If the application crashes or becomes unresponsive, it can no longer renew the lease. Once the lease duration expires, ownership is automatically revoked. At that point, another application becomes eligible to acquire the lease and continue the work. Leases solve a critical problem in distributed systems: they prevent abandoned locks from blocking progress indefinitely . The system can recover automatically without manual intervention. However, while leases improve availability, they also introduce a new class of subtle and more complex problems. Leases Depend on Time To understand the next challenge, assume the lease duration is thirty seconds. Application A successfully acquires the lease. Lease Granted Duration = 30 seconds After twenty seconds, the JVM begins a long Full Garbage Collection cycle. This pause lasts forty seconds, significantly longer than the lease duration. The timeline now becomes problematic. Lease Granted | | Processing | | GC Pause (40 sec) | | Lease Expires While Application A is paused, the lease expires. During this time, another application requests access to the same resource. The coordination service observes that the previous lease has expired and therefore grants ownership to Application B. Appl

The textbook version of the transactional outbox is tight. You save the domain entity and an outbox row in one local transaction. A background scheduler picks up PENDING rows and publishes them to Kafka. You never publish inside the request thread — no dual-write, no atomicity breach. The pattern closes the consistency gap. Then you load-test it. I ran 1,000 authenticated requests through my event-driven platform in 70 seconds. The gateway returned 201 for every one of them. The outbox absorbed every row. The consumer drained everything. By every visible metric the system looked healthy. Underneath that health, I found three production-grade problems the textbook never mentioned. What a correct implementation looks like Before the problems, the shape of the solution. The outbox publisher runs on a @Scheduled virtual-thread worker: @Scheduled ( fixedDelay = 5000 ) @Transactional public void publishPendingEvents () { List < OutboxEvent > batch = outboxRepository . findTop20ByStatusOrderByCreatedAtAsc ( OutboxStatus . PENDING ); for ( OutboxEvent event : batch ) { event . setStatus ( OutboxStatus . PROCESSING ); outboxRepository . save ( event ); try { kafkaTemplate . send ( event . getTopic (), event . getPayload ()). get (); event . setStatus ( OutboxStatus . PUBLISHED ); } catch ( Exception e ) { event . incrementRetryCount (); if ( event . getRetryCount () >= MAX_RETRIES ) { event . setStatus ( OutboxStatus . FAILED ); } else { event . setStatus ( OutboxStatus . PENDING ); } } outboxRepository . save ( event ); } } This is correct. The PROCESSING state prevents another scheduler instance from claiming the same row. The retry cap prevents infinite cycling. The PENDING fallback on transient errors gives the event another chance. The dual-write problem is genuinely closed. Here is what that correctness does not cover. Gap 1: Your throughput ceiling is a config line fixedDelay = 5000 means the scheduler runs every 5 seconds. findTop20 means it picks up 20 rows per cycl

A stored fact can be wrong in a quiet way. The answer still reads clean. A preference from an old exchange gets reused, the message goes out with confidence, and later nobody can tell why that detail was allowed back into the result. That is the failure I built around. When a system returns remembered material, the caller needs the text plus the reason it passed the reuse check. A log line found after the action is weak evidence. The object that leaves the memory service has to carry the admission record with it. 1. Keep the outside surface small This is the pattern I used in Holographic, Law-Bound Memory (HLM), a stand-alone memory brain outside application code. The README describes public Application Programming Interface (API) routes under /api/brain/* , with internal /api/v1/* services behind that layer. The outside shape is intentionally thin: register an agent, write a fact, build a capsule. The Python Software Development Kit (SDK) in sdks/python/hlm_sdk/client.py shows the boundary without exposing table names or policy code: import httpx class HLMClient : def __init__ ( self , base_url : str , token : str | None = None ): self . base_url = base_url . rstrip ( " / " ) self . _client = httpx . AsyncClient ( headers = { " Authorization " : f " Bearer { token } " } if token else None ) async def register_agent ( self , name : str ): r = await self . _client . post ( f " { self . base_url } /api/brain/agents/register " , json = { " name " : name }) r . raise_for_status return r . json async def write_fact ( self , text : str , tags : list [ str ] | None = None , selectors : list [ str ] | None = None ): r = await self . _client . post ( f " { self . base_url } /api/brain/memory/facts " , json = { " text " : text , " tags " : tags or [], " selectors " : selectors or []}) r . raise_for_status return r . json async def build_capsule ( self , query : str , budget_tokens : int = 2048 ): r = await self . _client . post ( f " { self . base_url } /api/brain/context/cap

TL;DR — I've been building the Social Resource Floor: an open blueprint for coordinating one person's access to basic survival resources — food, housing, energy, healthcare, and more — across many independent providers, so that reaching those resources is grounded in being human rather than in financial access. The first blueprint version is now complete: language-neutral schemas, prose specifications, a reference implementation, and a first adapter. This post is about the engineering choices behind it, and the reasons for each — how it stays a contract rather than a product, how it keeps personal data out of the coordination layer, why it binds to existing standards instead of inventing new ones, and how I check that the contracts are implementation-independent rather than just claiming they are. The problem the Floor is trying to help with Today, for most people, survival routes through financial access. To reach food, housing, energy, or healthcare you generally need money, and to hold or move money you need banking, employment, or purchasing power. Financial access has become the gate standing in front of the resources a person needs to stay alive. The goal of the Social Resource Floor is narrow and specific: to help make it so that financial status is not the condition that determines whether a person can reach the basic resources required to survive. It does not try to abolish money, banks, or markets — money stays a first-class resource and delivery method. It aims at one thing: a floor beneath which no person should fall, defined locally, reachable regardless of financial circumstances. That's the mission. Everything technical below exists to make that mission buildable by the institutions — governments, municipalities, NGOs, cooperatives, community providers — that would actually run it, without asking any of them to give up their own systems or hand over their data. Where the Floor sits The delivery systems for social protection already exist and are stron

While mass-producing web tools with an AI, I've changed how I lock the design in three stages. The previous post I wrote about that got this comment: "I'd like to see the JSON approach and the design-system approach side by side." Taken at face value, I should just put the two side by side. But first, let me add a short preface. I don't want to frame this as "the JSON approach versus the design-system approach." When I called the JSON approach a "failure" in that post, I didn't mean the method is inferior; I meant it didn't suit my particular set of tools. A page made with the JSON approach does look thin. But where that thinness comes from is easily misread. Whether the design drifts and whether it looks rich are decided separately. What stops the drift is locking the design; whether it looks rich is how much you build out. What locking with JSON removes is drift in the items you specified in the schema. Whether the screen becomes rich, on the other hand, is determined by how much you've built out the machinery that turns that JSON into a screen. So it isn't that locking with JSON is what made it look like a spreadsheet. In the previous post, too, I wrote that fattening the schema and the renderer does increase the expression itself. But that came with a caveat: past a point, it heads toward rebuilding HTML and CSS by hand. What I really want to check is one step past that. If the template sets the ceiling on expression, then building out the JSON side's template as much as the current one should produce the same screen. So what does that build-out demand? I actually built it and measured. I'll share the result, along with the JSON-approach and design-system-approach screens placed side by side under matched test conditions. I'll admit up front: at the time, I chose the design system without running this comparison. So this is me building the road I didn't take, after the fact, and measuring what that cost consists of. Same order, same one-shot So that the comparis

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry . You Know What's Worse Than Not Being Able to Log In? Being told everything worked right up until you try to actually use your account. Yes, that was a real bug. And, somehow, I ended up being pulled into another authentication mystery. At this point, I’m starting to think authentication bugs have a personal grudge against me. 😅 In my previous Smash Story , I wrote about a bug where users simply couldn't log in. This time, the problem was sneakier because most of the flow looked completely healthy. The user was approved, the background task ran, the email and SMS arrived, and Cognito had a user. Then the user actually tried to use their account. And everything fell apart. It Started With Two User Pools The authentication setup was fairly large and had evolved over time, so there wasn't one shiny User Pool doing everything. We had an older Cognito User Pool supporting existing authentication flows, including mobile-based signup, while a newer User Pool handled a newer flow where users received an email containing their PIN. Both pools were intentional because they supported different parts of the authentication journey. That wasn't the problem. The interesting part was that the application database had its own representation of a user, while Cognito had another. On top of that, some of the work connecting those two systems happened asynchronously. As long as everyone agreed about who the user was, nobody cared. The moment they disagreed, authentication became very interested. The Tiny Timing Window The problem appeared in the partner and dependant journey. A member could create a partner or dependant during signup or later from the member details area. A relevant non-member user would then approve the account, which scheduled an asynchronous task called SendingEmailsAfterApprovalBot in a TaskList database table. That task ran every 15 minutes, and once it executed, the partner or depend

Welcome to Part 1 of the Go Distributed Systems Lab series! Over the course of 20 hands-on projects, we are building core distributed systems primitives from the ground up using Go 1.22+ and the standard library ( net , sync , context , log/slog , encoding/binary ). Before jumping into raw socket framing, gossip protocols, or Raft consensus, we need to master the foundational concurrency building blocks inside a single process: Goroutines, Channels, and Communicating Sequential Processes (CSP) . 💡 The Philosophy: Share Memory by Communicating In traditional concurrent programming (like C++ or Java), thread synchronization often relies on shared memory protected by mutexes, lock-free queues, or read-write locks. Go flips this model with a core design principle: "Do not communicate by sharing memory; instead, share memory by communicating." By passing ownership of data structures through Go channels, each pipeline stage operates on isolated memory. This eliminates data races by design without requiring explicit lock management ( sync.Mutex ). 🏗️ Architecture & Component Design In this first module ( 01-message-passing ), we construct a 3-stage data processing pipeline: +------------------+ Job Channel +------------------+ Result Channel +-------------------+ | Producer | -------------------------> | Worker | ------------------------> | Collector | | (Generates Jobs) | (Buffered, cap=10) | (Isolated State) | (Buffered, cap=10) | (Aggregates Data) | +------------------+ +------------------+ +-------------------+ 1. Ingestion Stage (Producer) Generates typed Job values and pushes them into a direction-constrained buffered channel ( chan<- Job ). When generation finishes, it closes the channel to broadcast an end-of-stream signal. 2. Processing Stage (Worker) Consumes from <-chan Job using Go's for job := range in construct. The worker maintains internal execution metrics (e.g., processedCount ) entirely within its local stack scope—no locks required. 3. Collector Stage R

Support on GitHub: github.com/phero20/concurrent-resource-scheduler (Give it a star if you find it useful!) View Docs: pkg.go.dev/github.com/phero20/concurrent-resource-scheduler What happens when thousands of concurrent requests compete for a small pool of reusable resources? You can put a mutex around a slice and hope for the best. Or you can design the scheduler around concurrency from the beginning. I chose the second option. I built Concurrent Resource Scheduler (CRS) , a domain-agnostic Go library for selecting, prioritizing, routing, and maintaining reusable resources under heavy concurrent load. It was designed from the ground up for production readiness. The core library supports Go 1.22+ and is intentionally built with zero third-party dependencies . Extended features like Prometheus telemetry are strictly separated into an optional nested Go module ( Go 1.25+ ) to keep the core scheduler dependency graph perfectly empty. The core idea is simple: MANY CONCURRENT REQUESTS │ ▼ ┌───────────────────┐ │ Resource Scheduler│ └─────────┬─────────┘ │ ┌──────────────┼──────────────┐ │ │ │ ▼ ▼ ▼ Priority Acquire State Heap Strategy Management │ │ │ └──────────────┼──────────────┘ │ ▼ BEST AVAILABLE RESOURCE But making that work correctly under concurrency is where things get interesting. CRS is designed for use cases such as: LLM/API gateways API key pools proxy rotation database replicas GPU workers backend pools worker resources connection pools rate-limited providers reusable compute resources The scheduler itself does not know what a resource means. It only knows: "I have resources. I need to safely maintain them, prioritize them, and return an appropriate one to a concurrent caller." Table of Contents The Problem The Naive Approach Why a Global Mutex Becomes a Problem The Core Idea Behind CRS Architecture at a Glance Sharded Priority Heaps Why Sharding Helps The O(1) Lookup Map Priority and Acquire Are Different Problems Acquire Strategies Round Robin Weighted A

Support on GitHub: github.com/phero20/concurrent-resource-scheduler (Give it a star if you find it useful!) View Docs: pkg.go.dev/github.com/phero20/concurrent-resource-scheduler What happens when thousands of concurrent requests compete for a small pool of reusable resources? You can put a mutex around a slice and hope for the best. Or you can design the scheduler around concurrency from the beginning. I chose the second option. I built Concurrent Resource Scheduler (CRS) , a domain-agnostic Go library for selecting, prioritizing, routing, and maintaining reusable resources under heavy concurrent load. The core idea is simple: MANY CONCURRENT REQUESTS │ ▼ ┌───────────────────┐ │ Resource Scheduler│ └─────────┬─────────┘ │ ┌──────────────┼──────────────┐ │ │ │ ▼ ▼ ▼ Priority Acquire State Heap Strategy Management │ │ │ └──────────────┼──────────────┘ │ ▼ BEST AVAILABLE RESOURCE But making that work correctly under concurrency is where things get interesting. CRS is designed for use cases such as: LLM/API gateways API key pools proxy rotation database replicas GPU workers backend pools worker resources connection pools rate-limited providers reusable compute resources The scheduler itself does not know what a resource means. It only knows: "I have resources. I need to safely maintain them, prioritize them, and return an appropriate one to a concurrent caller." Table of Contents The Problem The Naive Approach Why a Global Mutex Becomes a Problem The Core Idea Behind CRS Architecture at a Glance Sharded Priority Heaps Why Sharding Helps The O(1) Lookup Map Priority and Acquire Are Different Problems Acquire Strategies Round Robin Weighted Acquire Adaptive Acquire Affinity Routing Shared vs Exclusive Acquisition Resource Lifecycle Atomic State Transitions The Inactive Store Batch Operations Updates Without Destroying Heap Ordering Cooldowns Asynchronous Events Observability Prometheus Integration Concurrency Model Complexity Testing the Library Race Detector Validation

Netflix has described how it redesigned the streaming pipeline behind Service Topology, its real-time service dependencies map, to support production scale. The system uses three stages to separate intermediary resolution from enrichment and persistence, propagates backpressure to Kafka rather than dropping records, and uses server-sent events instead of gRPC for high-volume internal transfers. By Eran Stiller

Every API that sits behind an unreliable network eventually faces the same problem: a client sends a request, the connection drops before the response arrives, and the client has no idea whether the operation happened. Did the payment go through? Did the order get created twice? The client's only safe move is to retry — which means your server needs a story for what happens when the same "create this thing" request arrives more than once. That story is idempotency keys, and getting the details right is more subtle than it first looks. The core idea The client generates a unique token — typically a UUID — once per logical operation, and attaches it to every retry of that operation: POST /orders Idempotency-Key: 7c3fd9a2-df01-4b3e-9a55-1e5f9b6b6d55 {"sku": "WIDGET-1", "qty": 2} The server's job is to guarantee that no matter how many times a request with that key arrives, the side effect (charging a card, creating an order, sending an email) happens at most once, and every retry gets back the same response the original request would have produced. Note what this is not: it is not deduplicating by request body. Two requests with identical bodies but no key are legitimately two different orders for two widgets. The key is what marks them as "the same attempt," not the payload. The naive approach, and why it breaks A common first pass is a table like: CREATE TABLE idempotency_keys ( key TEXT PRIMARY KEY , response_body JSONB , status_code INT ); On each request: check if the key exists, and if so return the cached response; otherwise do the work and insert the result. This looks right and is wrong in a specific way: it has a race condition. Two retries can arrive concurrently (a client that timed out and fired a second attempt while the first was still in flight), both miss the cache check, and both execute the underlying operation. You've now charged the card twice. Making the check-and-do atomic The fix is to claim the key before doing the work, using the database's ow

You put a lot of work into authentication. A gateway validates the Keycloak JWT, maps realm roles to authorities, checks that the caller is allowed. By the time a request reaches your service, you know exactly who is calling. Then the request crosses into async land, and all of that evaporates. This is the story of the point where identity quietly disappears in an event-driven system, why the dead-letter queue is the worst possible place for it to disappear, and how I made the acting user as durable and replay-safe as the event itself. The flow everyone believes is fine The platform is a set of Spring Boot services: an API gateway in front, a user-service on MySQL, a notification-service on PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent. Authentication is handled at the edge. The gateway is an OAuth2 Resource Server; it validates the token once and propagates the caller's identity downstream as headers: // api-gateway — IdentityPropagationFilter (@Order(2), after security) IdentityContext identity = identityContextExtractor . extract ( jwt ); // Always set all three headers (empty when absent) to mask any spoofed values. enrichedRequest . putHeader ( IdentityHeaders . USER_NAME , nullToEmpty ( identity . username ())); enrichedRequest . putHeader ( IdentityHeaders . USER_EMAIL , nullToEmpty ( identity . email ())); enrichedRequest . putHeader ( IdentityHeaders . USER_ROLES , identity . rolesAsString ( DELIM )); One detail here matters more than it looks. The headers are always overwritten , even when a claim is absent. If a client tries to inject X-User-Name: admin on the inbound request, the gateway stomps it with the validated value (or empty). Downstream trust in those headers is only safe because the perimeter guarantees they cannot be forged. Miss that, and you've built an impersonation API. So far, so good. The synchronous hop carries identity. The problem starts one line later. The hidden f