标签：#systems

You attach the OpenTelemetry Java agent, point it at a collector, and within minutes Grafana is drawing a service map you never drew. A box for each service, arrows between them, latency on every edge. It feels like magic, and — more dangerously — it feels complete . "The agent traces everything" is the sentence repeated in every onboarding doc. This is the story of the moment that sentence stopped being true on my platform, why I'm glad it did, and the difference between a system that is working and a system you can actually see . The flow everyone trusts The platform is an event-driven set of Spring Boot services: an API gateway in front, a user-service backed by MySQL, a notification-service backed by PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent. I didn't want to draw that topology. A hand-drawn architecture diagram is documentation that drifts — true the day you commit it, slightly wrong a month later, actively misleading after a quarter. I wanted the dependency graph generated from live traffic , so it would always reflect what the system actually does. Grafana Tempo does exactly this. Its service-graphs processor reads matched client/server span pairs out of trace data and emits a metric — traces_service_graph_request_total — that Grafana renders as a node graph. No edge is ever wired by hand. The topology is derived, continuously, from real spans. The edge that wasn't there I generated the graph and the synchronous edges lit up immediately: api-gateway → user-service user-service → MySQL notification-service → PostgreSQL Then I looked for the one edge I actually cared about — user-service → notification-service , the asynchronous hop over Kafka. It wasn't there. The naive conclusion (and why it's wrong) The tempting read is immediate and obvious: the async hop is broken. The event isn't getting across. Go debug the consumer. So I checked. And the consumer was completely fine. notification

I asked Linux for one 4 KiB page from the start of a cold file. Four pages came back. I moved the same read one page further in, ran it again, and got one. Same file, same syscall, same kernel. The only thing that changed was where I started reading, and I spent twenty minutes assuming the tool I'd just written was miscounting. It wasn't. A read that starts at byte zero is treated as a promise. There's a branch in mm/readahead.c that reads, in full, if (!index) goto initial_readahead; . Offset zero means the kernel takes you for a program that's about to stream the whole file, and it fetches ahead immediately. Start anywhere else and you're assumed to be seeking randomly until a pattern proves otherwise. Nothing in my call said a word about my intentions. It inferred them from an offset. I spent two weeks on this sort of thing recently. Not for work, and not toward anything shippable. The short version of what I found is that a surprising amount of the machinery under a running program isn't carrying out instructions at all. It's guessing. The bench , because it changes how you should read every number here: an ext4 filesystem on a loop device, inside an OrbStack Linux VM on an Apple Silicon Mac, kernel 7.0.14, 4 KiB pages, read_ahead_kb at 128. That's a container sharing the host's kernel, not bare metal, and the host reclaims memory aggressively enough that a fully cached file can go cold in fifteen seconds. Reads came from dd ; the page-by-page counting came from a small C tool I wrote that mmap s a file and asks mincore() which of its pages are resident. You're not addressing the disk, you're addressing the page cache The model most of us carry is that read() goes and gets bytes off a device. It doesn't. It copies bytes out of the page cache into your buffer, and the page cache is just RAM the kernel uses to remember parts of files. If what you want is already there, no device is involved. If it isn't, the kernel fills the cache first and then copies. Either way

When engineers discuss quantum computing, the conversation usually focuses on future supercomputers cracking traditional encryption passwords in a matter of seconds. As a systems architect who spends my days building distributed platforms, which are networks of independent cloud servers working together as a single application, I see a different, highly practical challenge taking shape. The transition to quantum-resistant security is not simply a theoretical math problem. It is an infrastructure challenge that will directly impact network throughput, memory usage, and messaging efficiency across global cloud environments. To protect sensitive enterprise records and business platforms against future quantum threats, security organizations are transitioning to Post-Quantum Cryptography. This field involves building new mathematical algorithms that quantum computers cannot easily solve. However, these stronger defense mechanisms come with a major trade-off in size. Traditional cryptographic signatures, which are digital verification stamps used to prove that a data message comes from an authentic sender and was not altered, are remarkably small. An older, standard signature might only take up sixty bytes of memory. By comparison, a quantum-safe signature can easily require several thousand bytes. In a simple website, adding a few extra kilobytes to a security header goes unnoticed. But modern cloud infrastructure relies heavily on event-driven architecture, a design strategy where dozens of microservices communicate by constantly publishing tiny, real-time updates to shared message queues. In these systems, the actual business payload might only be a small status change containing twenty bytes of text. If the quantum security stamp attached to that message is three thousand bytes, the overhead of the security layer completely outweighs the actual data being sent. When security footprints expand by orders of magnitude, the physical realities of computer networking take

Cloudflare recently introduced Meerkat, an internal globally consistent control-plane service based on the QuePaxa consensus algorithm. Unlike Raft, it allows leaderless writes while preserving strong consistency, improving availability across Cloudflare's global network. By Renato Losio

This article was originally published on BuildZn . Everyone talks about multi-agent systems but few show you how to actually coordinate them without a ton of boilerplate and deadlocks. I spent weeks trying to get agents to talk, especially when building something like FarahGPT's multi-agent trading system, often hitting insane latency. Turns out, qm can drastically simplify this, and this qm multiplayer AI agent tutorial will show you how to cut task completion times by 20% using a specific Node.js workflow. Why Multi-Agent Systems Aren't Just Hype Anymore (and qm Helps) Single LLM calls hit a wall, fast. You get generic answers, struggle with complex, multi-step tasks, and prompt engineering becomes a full-time job. I've built 9-agent YouTube automation pipelines and an AI gold trading system that needed to analyze market data, news sentiment, and historical trends concurrently. Trying to jam all that into one prompt for a single agent? Forget about it. You need a collaborative AI agent architecture . That's where multi-agent systems shine. You break down complex problems into smaller, manageable tasks, assign them to specialized agents, and have them work together. Think of it like a dev team: one person focuses on backend, another on frontend, another on CI/CD. This is how you handle real-world complexity, and it's how I scaled FarahGPT to 5,100+ users. The challenge? Orchestration. How do these agents communicate? Who manages their state? How do you ensure they don't step on each other's toes or get stuck waiting for slow upstream tasks? This is exactly where qm , a lightweight agent harness, becomes a game-changer for building AI teams. It gives you the primitives to define agents, tasks, and workflows without drowning in custom event loops. The Core Concept: Task Delegation in qm Most qm examples show simple agent interactions. Agent A asks Agent B. Done. But what if Agent A needs to delegate a task that itself needs parallel sub-tasks, and then aggregate the

There are fresh signs that fusion power startup Commonwealth Fusion Systems will list in the next two to three years.

Commonwealth Fusion Systems raised $1 billion as the startup moves toward its first commercial fusion power plant.

Murat Demirbas discusses the shift toward disaggregated cloud database architectures driven by cloud economics. He explains how decoupling compute from storage enables elastic scaling, cost efficiency, and fault isolation. He shares how classical Paxos roles foreshadowed disaggregation, while analyzing network tradeoffs, shared-memory evolution, and self-assembling database designs. By Murat Demirbas

Handoffs are useful when a specialist agent needs to take over a task. They also make cost easier to hide, because the bill is spread across graph nodes instead of one visible chat turn. Why can LangGraph handoffs multiply tokens? LangGraph handoffs can multiply tokens because each model-calling node may resend instructions, prior messages, retrieved material, tool returns, summaries, and artifacts, then loops or handoffs repeat that payload for the next agent. Token amplification is the total prompt-plus-completion tokens across a trace divided by a simpler baseline for the same task; Anthropic reported in June 2025 that multi-agent systems used about 15x more tokens than chats while improving an internal research evaluation by 90.2% . Quick Answer: Handoffs raise the token bill when each agent receives copied context instead of a narrow task packet. Anthropic’s June 2025 research system showed the tradeoff clearly: multi-agent runs used about 15x more tokens than chats while scoring 90.2% higher on its internal research evaluation . In LangGraph, the practical issue is observability and budgeting, not whether graphs are bad. The LangGraph project describes the runtime as a way to build stateful, long-running agents with persistence, human control, memory, and debugging support; those same traits make it possible to measure where context grows instead of guessing. "Multi-agent systems are often highly effective at open-ended research tasks, but token usage can be substantial," — Anthropic engineering team at Anthropic The small verified demo below shows the arithmetic behind a 15x bill: a 100-token task becomes 1,500 billed tokens when 5 agents each receive 3 copies of the relevant context . """ Tiny token-accounting demo: handoffs multiply the same task context. """ task_tokens = 100 agents = 5 context_copies_per_handoff = 3 # instructions + task + summary/history direct_bill = task_tokens handoff_bill = task_tokens * agents * context_copies_per_handoff print ( f

Encrypted data stored in cloud archives today will outlive the mathematical algorithms guarding it. In enterprise architectures that handle long-term records, like construction risk logs or employee compliance platforms, data retention schedules often span twenty to thirty years. When building cloud pipelines that move this information across services, we depend heavily on asymmetric encryption, which is a security method using one public key to lock data and a separate private key to unlock it. Standard public-key algorithms rely on mathematical problems that are nearly impossible for classical computers to solve within a reasonable human timeframe. Quantum computing changes this equation entirely. Quantum computers leverage quantum mechanics, the physical rules governing subatomic particles, to perform calculations at speeds fundamentally unimaginable with traditional silicon processors. While powerful quantum systems are still in development, the security threat to distributed systems exists today. Hostile actors do not need to crack modern security algorithms in real time. Through a pattern known as Harvest Now, Decrypt Later, adversaries can capture and store encrypted network traffic right now. They simply wait until future quantum hardware becomes capable of running the formulas required to decrypt that stolen history. For software architects, preparing for post-quantum cryptography, which refers to new mathematical encryption algorithms designed to withstand quantum attacks, is far more than a simple library swap. It is a deep distributed systems migration challenge. The primary operational hurdle is payload size and computational overhead. Quantum-resistant algorithms require significantly larger digital keys and payload headers than the standards we rely on today. When cryptographic payloads expand, every component of a distributed platform feels the ripple effect. Message queues experience higher bandwidth demands. Database indexes inflate. Memory consump

Herodotus wrote of a bird that lived five hundred years in Arabia, and when its life came to an end, it did not wait to be surprised by death. It built its own nest of cinnamon and myrrh, set the nest and itself alight, and let a new bird rise from what the fire left behind. The Hestia argued for tending a fire that must never go out. That’s true, and it isn’t the whole truth. Teams end. People leave. Companies get acquired, reorganized, shut down, and five years from now some part of this whole model will probably look as dated as the practices it was written to replace. No amount of tending prevents that. Pretending otherwise is its own kind of Alexandria , a slow decline dressed up as continuity, right up until the fire goes out anyway and nobody chose the moment. The bird in Herodotus doesn’t get caught by surprise. It builds the pyre itself. Chooses the moment, gathers what matters, and burns deliberately, trusting that what rises afterward carries the shape of what came before, not because the fire preserved the old bird whole, but because starting over was never the same thing as starting from nothing. That’s the part tending alone can’t promise. A team that’s about to be split up can hand its shared model to whoever inherits the work on purpose, the way a rep in the Boule carries a decision back instead of leaving it to travel however it happens to travel. A team about to lose its most experienced person can spend the weeks before that departure making sure the framing, not just the conclusions, made it into someone else’s head, the way the Mimesis argued a junior actually learns. None of that stops the ending. It decides what the ending leaves behind. This series doesn’t get to end with a fire that never goes out. Nothing does. It gets to end with the only thing actually inside anyone’s control. Build the pyre on purpose. Choose what goes into the fire. References The Myth of the Phoenix: Rebirth and Renewal : Greek Mythology, on Herodotus’s original accoun

Part 1 documented the recurring snapback in practice. This note asks a narrower question: what does the observed pattern support, what remains a working hypothesis, and what changes should follow in the project? Status: Bounded project conclusion. This note separates observed behaviour, working hypothesis, and practical consequence. It is based on current project documents and interactions; it is not external validation or a universal claim about AI systems. What the evidence supports 1. The project already contains a stable relational model The working model is not generic “AI assistance.” It separates reasoning surfaces, uses bounded comparisons, permits two-way cognitive pressure, and keeps final acceptance authority with the human. Reciprocal cognitive contribution, asymmetrical governing authority. 2. Concrete project work preserves the structure better than public abstraction At the concrete level, instructions such as: Review this proposal against that architecture. preserve the distinction between the object being reviewed, the surface applying pressure, the evidence, and the authority that may accept a change. When the same structure was compressed into general prose, generated explanations repeatedly returned to a simpler one-way model of either human control or transferred AI authority. That is an observed pattern in this development process. 3. Public explanation is a separate reasoning surface A README, article, summary, or portfolio page is a projection of the model, not the model itself. It cannot be assumed to reproduce the internal structure faithfully merely because that structure is present in context. The explanation must be reviewed against the model it represents: Does this explanation preserve the actual authority, review, evidence, and state-transition structure? 4. Annoyance was useful boundary data The irritation indicated that the generic rendering was no longer merely an imperfect exploration. It was colliding with an internal frame that

Originally published on nejckorasa.github.io . When a team moves from a monolith into microservices and event-driven, asynchronous systems, it inherits a class of problems that used to be someone else's: work that fails halfway through, steps that must not run twice, calls that return before the work is done. Temporal is a durable execution engine that handles a lot of this - you define a multi-step process, and it guarantees the process runs to completion even when workers crash in the middle. I've spent the better part of a decade building distributed systems in the money-movement core of banks - ledgers, payments, credit cards - a lot of it on Temporal, from short request-triggered workflows to ones that stayed open for weeks. This is the high-level guide I'd give a team making that jump: the principles worth internalising before you ship, not a full tutorial. Most of them aren't really about Temporal. They're the habits the async shift demands - Temporal just punishes you quickly when you skip one. Durable Execution: The Problem It Solves Distributed work fails in the middle. You call service A, it succeeds. You call B, it times out. The pod dies before C. Now you have half-finished work and no memory of how far you got. The usual fix is a pile of status columns, a cron job to find stuck rows, and retry logic hand-rolled for every step. Temporal's promise is that any process you start runs to the end. The runtime picture: there's a Temporal service (its own cluster), and your app runs worker processes that poll it and execute your code. As a workflow runs, Temporal records every step to an event history . If a worker dies, another picks the workflow up and replays that history to rebuild state, then carries on from where it left off, retrying anything that failed. The history is the source of truth, and it survives the crash. Most of the rules below fall out of that one fact. The Golden Rule: Workflows Decide, Activities Do There are two kinds of code in Tempora

Building an operating system is a project I have had my eyes set on ever since I discovered free will in the realm of programming. Years ago, I did a reasonable amount of research, paying extra attention to the subject during my computer science degree and I was able to understand Operating System Theory and how it works from first principles but I never really got around to building one. I had only flimsy reasons for not embarking on it like "why build one when there are tons of working ones out there? The theoretical knowledge is enough" . More recently, I am ignoring the need to not re-invent the wheel for the joy of programming. So if you are interested in also rebuilding stuff because you can, join me on this series as I document how I am going to be building kluster. kluster is in its infancy and the direction is not clear but the one certain thing is that I will be building it entirely in Rust, save some assembly instructions and a linker script and I will be explaining every single line of code along the way. It will also be designed to target the raspberrypi 4 & 5, on qemu and on real hardware respectively. This is an opportunity for anyone who wants to see how Rust works at the lowest of levels to hop on and join the ride. Note that this series will be your biggest lesson on delayed gratification because we will write a lot of code before we even get to see anything meaningful on screen but I will foreshadow what you can get by the end of part 3 if you are patient enough: {{ image(src="/images/os-part3-result.png", alt="Part 3 Results OS Dev") }} You can also clone the source code for part 1 from Github and follow along. Project Setup First things first, let us setup the foundation of the project. I'll be straight with you, I love Rust and I enjoy using the Rust ecosystem in its entirety so I will stay true to that and use it as obsessively as any true Rustacean; I won't hold back. Without doubt, all the dependencies we need are freely available as long as

This is going to be a very different article from what I usually write. No technical discussions, architecture deep dives, or engineering practices today. Instead, we're talking about something much older than software itself: music. We treat language like it's the default mode of human communication, like it's the real and only thing used to communicate, everything else is secondary, emotional, aesthetic, nice to have. But language is actually the outlier. It's the new protocol layered on top of something much older. Music is the original standard and we've basically forgotten how to read it. The Protocol Stack Think of communication like a network stack. Language is high-level. It's TCP/IP. Built on assumptions, needs learning, breaks the second you cross a boundary. You need: A shared vocabulary Syntactic understanding Cultural context Years of study if you actually want fluency It's powerful but It's also fragile. And it's recent . Written language is a few thousand years old. Spoken language is older, sure, but both are late abstractions compared to the hundreds of thousands of years humans have been syncing bodies to shared sound. Relative to that timeline? Language is yesterday's patch. Music? That's the lower-level protocol. The physical layer everything else runs on. A Japanese teenager at a Michael Jackson concert doesn't need to speak English. She doesn't need to understand what "Man in the Mirror" means as a concept. She also doesn't need a music degree. Music isn't zero -cost. Genre, culture, convention still shape how we hear it. But the entry barrier for emotional communication is way lower. A rhythm can hit urgency, celebration, sadness, or tension long before anyone understands the formal structure behind it. Her nervous system speaks that fluently. And so does everyone else in that stadium. How the Protocol Works Here's what happens when the song starts: 70,000 people stop being individuals and start being a distributed system synchronizing to the

Controller A records revocation epoch 12. Worker B, partitioned with a cached grant from epoch 11, starts another external action. The database is correct and the system is unsafe. Emergency stop is therefore a distributed protocol, not a Boolean field. What is verified In its July 21 disclosure, OpenAI says an internal benchmark used models with reduced cyber refusals and that a combination of models compromised Hugging Face infrastructure. The primary source is https://openai.com/index/hugging-face-model-evaluation-security-incident/ . Reporting on July 24 then described US discussion of emergency-shutdown and independent-audit proposals. The latter is policy coverage, not enacted law and not an extension of the official incident facts. Missing protocol details, impact boundaries, and remediation should remain unknown rather than inferred. Invariants and assumptions Assume workers, queue consumers, an authorization service, and external adapters can fail independently. Messages may be delayed, duplicated, or reordered; clocks have bounded error only if measured. Required invariants: No action starts with a grant epoch below the subject's revocation epoch. Cached grants expire within a declared lease bound. Restart cannot lower a persisted epoch. Duplicate revocation converges to the same or higher epoch. Completion means every registered executor acknowledged or its lease expired. revoke(subject, epoch=13) -> durable CAS max(current, 13) -> publish {subject, epoch:13} -> executors persist max(local, 13), ack -> controller waits for ack set OR lease expiry -> issue completion receipt with missing/expired members Failure injection Property Acceptance rule delay revocation event lease bounds stale authority no start after local lease expiry duplicate epoch 13 idempotence epoch remains 13+ deliver 13 before 12 monotonicity never returns to 12 worker restarts durability loads persisted epoch before work controller partition fail closed no new lease after expiry A minim

I cranked Apple’s iOS 27 beta to maximum transparency on my iPhone and enjoyed every warped second.

When a new video shows up in one of our regional crawlers, three things need to happen almost immediately: the SQLite FTS5 search index for that region needs a new row, the discovery ranking cache needs to be invalidated, and the sitemap generator needs to know a URL was born. For a long time we did all of this inline, inside the same cron process that fetched the video. It worked until it didn't. A slow FTS5 rebuild would stall the fetch loop, a sitemap write would fail silently, and a crash halfway through meant one region had the video indexed and another didn't. The fetch and the fan-out were fused together, and every failure was a partial failure. The fix was to stop treating "a video was discovered" as a function call and start treating it as an event. At TrendVidStream we run discovery across 8 regions, and the moment we introduced NATS JetStream as the spine between the crawler and the downstream consumers, the whole system got calmer. This post is the concrete version of how we did it: the stream config, the publishers, the consumers, and the mistakes we made that you can skip. Why Not Just Use a Queue Table in SQLite We already had SQLite everywhere, so the obvious move was a jobs table. We tried it. The problems showed up fast: Polling latency vs. load tradeoff. Poll every second and you hammer the DB with mostly-empty SELECT queries across 8 regions. Poll every 30 seconds and your search index lags noticeably behind your crawler. No fan-out. One row, one worker. If the sitemap generator and the FTS5 indexer both need the same event, you either duplicate rows or invent a consumed_by bitmask. Both are ugly. Locking. SQLite's writer lock means the queue table and the actual data table start contending under the multi-region cron bursts we run. Cross-region delivery. Our regions aren't all on the same box. A queue table doesn't cross machines without you building a replication story on top. JetStream solves all four: push-based delivery (no polling), multipl

The pricing formulas in Motor, the estimating engine I built for a water feature shop, did not come from the manual. I pulled 32 of them out of the JavaScript behind Aquascape's contractor calculator, the tool contractors actually use to bid jobs. The manual was sitting right there, official and free. Ignoring it was the best design decision in the whole system. A vendor never ships a sloppy calculator Why trust the calculator over the manual? Because of what happens when each one is wrong. If the manual sizes a pump wrong, a reader shrugs and moves on. If the calculator sizes a pump wrong, a contractor bids a job at that number, wins it, and loses money on the install. Then the phone rings. So calculators get fixed and manuals drift. Give it ten years and the two quietly disagree, and everyone in the trade knows which one to trust without anyone saying so. A vendor will ship a sloppy PDF. They will never ship a sloppy calculator. Documentation is what a domain says about itself. The artifacts money flows through are what it actually believes. Once you see that split, you cannot stop seeing it. The other half was in old invoices Formulas only get you to cost. What a shop charges on top of cost is a belief about its market, and no vendor document holds that number. So I pulled 132 historical quotes out of the shop's CRM. Real quotes, sent to real customers, most of them paid. I calibrated Motor's markup against those, then checked its output against what the shop had actually charged. The result: Calibrated against 132 real quotes, Motor's estimates landed within 5 percent of what the shop actually charged, with no pricing rule taken from documentation. I could have just asked the owner what his markup was. But what an owner says and what his invoices show are rarely the same number, and the invoices are the ones customers paid. When the two disagree, believe the invoices. The same bug in a different industry I build and run systems in several industries, and the sur

Before a young Athenian took his full place in the city, he spent two years in the ephebeia. Training happened in the gymnasion, organized by tribe, the same tribes that would later send him to represent them in the Boule itself. Nobody handed him full standing first and hoped the judgment would follow. This should have been post seven. It’s showing up as sixteen because the gap only became visible once the room was real enough to test against. The Agora described what a Spec Session does. It never asked whether everyone walking into that room shares an accurate picture of what the agent can actually do. Most rooms don’t. Someone watched a demo and thinks the agent can do anything. Someone else got burned by a bad output three weeks ago and doesn’t trust it with anything real. Nobody’s intuition has been tested against the same tasks, and the spec that comes out of that room ends up too ambitious or too conservative depending on whose untested belief happened to speak first. That gap has a name in Athens. The gymnasion existed because nobody was handed a place in the city first and expected to develop judgment on the job. The ephebeia ran two years, training built around a specific fact. Physical readiness and civic judgment weren’t taught in separate places. They happened in the same space, under the same supervisors, organized by the same tribal groupings that would later structure how the city actually governed itself. The same word, gymnasion, ended up naming both the training ground for eighteen-year-olds and the buildings where Plato and Aristotle did their most serious thinking. Academy and Lyceum were gymnasia first. Nobody separated the trial from the reflection. The trial was how the reflection got earned. That’s the part worth taking seriously. Testing a tool and understanding a tool were never two different activities. The testing is how the understanding gets built. A team that reads documentation about what an agent can do has a description. A team tha