标签：#os

Making something run automatically and having it actually get handled to completion turned out to be two different sentences This is the English version of a post originally written in Korean for my algorithmic trading system devlog (new tab). I designed this system to judge and trade on its own, without needing human approval for each decision. The order execution and safety-guard layer (new tab) was built around that same goal. Over the past month, though, there were three separate moments where I had to step in and act directly on the live account. Each one failed for a structurally different reason. 1. An allocation rule structurally starved one order I noticed a particular ticker had a sell plan queued for several days running, yet it never actually went out. The system kept nominating it as a sell candidate every day, but the order never reached execution. The cause was the rule that splits the daily trading budget across multiple rounds. This position's order size was consistently larger than whatever sell budget remained that day. No matter how the rounds were reordered, it could never clear. The budget cap itself was working exactly as designed. The problem was that the underlying assumption — "if it doesn't clear this time, it'll clear next time" — never held for this position. New opportunities kept arriving, but none of them was ever big enough. I executed the one blocked order by hand. It was an irreversible sell on the live account. Afterward, I reworked the allocation rule itself so the same starvation pattern couldn't recur. 2. A safety guard confused a snapshot with a cumulative reading On a different day, the opposite kind of failure happened. A guard meant to detect cumulative drawdown wrongly blocked two legitimate buy orders. The index was nearly flat that day, but the loss figure this guard was tracking read much larger. Looking closer, the guard was designed to measure "decline from peak," but the current value it compared against was an intra

로그조차 안 쌓이니, 돌고 있는지 죽어 있는지 구분할 방법이 없었습니다 이 블로그의 개발일지는 매일 밤 자동으로 마무리됩니다. 그날 대화로 초안을 썼으면 변환해서 로그에 남기고, 없으면 스킵했다는 한 줄만 남깁니다. 최근 이 파이프라인을 들여다볼 일이 있었는데, 7월 24일 이후로 로그가 통째로 비어 있었습니다. 무슨 일이 있었나 자동화 로그( .automation.log ) 마지막 줄이 2026-07-24였습니다. 그 뒤로 8월 22일까지, 거의 한 달 가까이 스킵 기록조차 한 줄도 없었습니다. 자동화가 아예 안 돌았나 싶어서 실행 로그( cron_output.log )를 열어봤습니다. 그런데 거기엔 매일 밤 실행된 흔적이 빼곡했습니다. 날짜 확인하고, 초안 있으면 내용 정리하고, 크로스링크까지 챙긴 요약이 매일 밤 남아 있었습니다. 일은 하고 있었는데, 결과물만 하나도 남지 않고 있었던 겁니다. 왜 아무도 몰랐나 실행 로그를 읽어보니 원인은 매일 같았습니다. 파일 쓰기 권한 승인을 기다리다가 그대로 끝난 겁니다. "workspace has not been trusted"라는 경고가 매 실행마다 찍혀 있었습니다. 초안을 잘 정리해놓고도, 마지막 파일 쓰기 한 줄이 승인 대기에 막혀서 아무것도 저장되지 않은 채 세션이 끝나는 패턴이 한 달 가까이 반복됐습니다. 문제는 이게 하필 로그를 남기는 단계 자체가 막힌 상황 이었다는 겁니다. 스킵한 날엔 스킵했다는 한 줄도 못 남겼습니다. 그러니 로그만 보면 "자동화가 멈췄다"와 "쓸 게 없어서 조용했다"를 구분할 수가 없었습니다. 무인 자동화이니 매일 밤 누가 화면을 지켜보는 것도 아닙니다. 결과적으로 이 공백은 사람이 우연히 로그 파일을 열어보기 전까지는 발견될 방법이 없는 구조였습니다. 진짜 원인 원인은 신뢰(trust) 설정이었습니다. 이 헤드리스 자동화 세션은 프로젝트 폴더 단위로 파일 쓰기 권한을 신뢰받아야 동작하는데, 그 신뢰 설정 키가 이 블로그 폴더가 아니라 상위 디렉터리(프로젝트들이 모여 있는 루트) 단위로 걸려 있었습니다. 즉 이 블로그 폴더만 놓고 보면 "아직 한 번도 대화형으로 신뢰 승인을 받은 적 없는 새 작업공간" 취급을 받고 있었던 셈입니다. 설정 파일 안에 이미 허용 규칙 10개가 들어 있었는데도, 그 규칙들이 걸려 있는 범위 자체가 무시되고 있었습니다. 헤드리스로 도는 야간 자동화는 대화형 승인 프롬프트에 응답할 사람이 없습니다. 범위가 어긋난 신뢰 설정 하나가, 매일 밤 정확히 같은 지점에서 조용히 실행을 무력화하고 있었던 겁니다. 흥미로운 디테일 하나 — 유령 완료 기록 이 공백을 되짚어보다가 특이한 줄 하나를 발견했습니다. 8월 22일 낮 12시 41분에 "weekly: 완료"라는 로그 한 줄이 남아 있었는데, 그 시각에 대응하는 실제 산출물 파일은 없었습니다. 같은 날 오후 4시에 다시 수동으로 실행된 기록이 있었고, 이번엔 실제 파일까지 정상적으로 만들어졌습니다. 앞선 12시 41분 기록이 왜 실물 없이 "완료"라고만 남았는지는 원인을 특정하지 못했습니다. 권한 문제가 한창이던 구간이라 어떤 형태로든 쓰기 절차 일부만 성공하고 일부는 실패한 걸로 추정만 할 뿐입니다. 다만 이 한 줄은 별도로 눈에 띄는 교훈을 남겼습니다. "완료"라고 적힌 로그도 그 자체로 완전히 믿을 수는 없다 는 겁니다. 로그와 실제 산출물을 따로 대조하지 않았다면 이 유령 기록을 그냥 지나쳤을 겁니다. 어떻게 고쳤나 신뢰 설정을 이 블로그 폴더 기준으로 다시 걸어주니, 그날 밤부터 바로 정상화됐습니다. 별도의 복잡한 조치는 필요 없었습니다. 문제는 고치는 방법이 아니라, 한 달 가까이 그 문제를 놓치고 있었다는 사실 쪽이었습니다. 일반화된 교훈 이번 일로 다시 확인한 건, 무인 자동화에서 "로그가 없다"는 상태 자체가 하나의 신호라는 겁니다. 그런데 그 신호를 신호로 취급하려면, 애초에 "침묵"과 "성공적인 무동작"을 구분할 수 있게 설계돼 있어야 합니다. 이번 파이프라인은 스킵한 날에도 로그 한 줄을 남기게 되어 있었습니다. 그 설계 덕분에, "로그가 아예 안

Bruna Pereira explains how DoorDash built a content-agnostic AI moderation platform. She covers replacing costly LLM-only pipelines with a hybrid pattern: using fast internal models to filter obvious cases, LLM multi-axis scoring for nuanced decisions, and no-code workflows with backtesting. Discover how this architectural pattern cut safety incidents while scaling to millions of daily messages. By Bruna Pereira

The Nvidia Cloverleaf data center partnership announced on Friday tells you where the real constraint in AI has moved, and it is not the chip. TechCrunch reported that Nvidia has taken a minority stake in Cloverleaf , a company founded in 2024 that raised $300 million that year and sits between utility companies and data centers, arranging power and site infrastructure. Nvidia did not buy a chip designer. It bought a piece of the electricity supply chain. That reframing is worth thinking about if you build software from anywhere outside a well-supplied grid. 🔌 The bottleneck moved from silicon to substations Read the deal literally. Cloverleaf's product is not compute. It is power sourcing and site infrastructure — the interconnect agreements, the substations, the land next to a utility that can actually deliver load. Nvidia buying into that layer is an admission that shipping more GPUs does not help if nobody can plug them in. This was not a one-off either. The same week, per the reporting: Deal Announced Reported size What Nvidia bought into SB Energy (OpenAI-linked, Ohio) 17 Aug 2026 $1.5 billion Data center project with an energy parent Cloverleaf 21 Aug 2026 Several hundred million (WSJ), minority stake (Reuters) Utility-to-data-center power intermediary Terms were not disclosed by either company, so treat the dollar figures as press reporting rather than filings. Key takeaway: When the company that sells the shovels starts buying the mines, the scarce input is no longer shovels. For AI in 2026, the scarce input is grid capacity. 💰 The circular money problem you should price in Here is the part I would not skip over. Nvidia sells GPUs to data centers. Nvidia is now also investing in the companies that build and power those data centers . Some of that capital flows back as GPU orders. That is not illegal or even unusual in capital-intensive industries. Telecom vendors financed carriers for decades. But it has a specific consequence for you as a buyer of compute:

Thunderstorms make seismic waves that can be used to find sub-surface features.

The private Android-based OS will expand beyond Pixels next year.

A few years ago, I discovered the Nostr protocol 1 thanks to Jack (Jack Dorsey) 2 . I've been using it actively ever since. I already had an interest in distributed social networks for years. I tried ActivityPub 3 for a while, but something just didn't click. Around that time, Jack started ATProto 4 within Twitter, but something felt off there too. I hadn't even bothered trying the methods that came before those anyway. It always bothered me—and many others, of course—that the internet became so centralized when the web is distributed by nature. The main reason for this was social media platforms. The trend that started with guestbooks evolved into forums, then to Friendfeed-like walls, MySpace-like platforms, and eventually Facebook and Twitter. The internet, which had a much more distributed structure with blogs prior to Facebook and Twitter, suddenly started to centralize. Due to the maintenance costs of distributed structures and unsettled systems, humanity took the easy way out and chose centralization, as always. Facebook and Twitter grew steadily, and many other platforms alongside them transformed into centralized hubs using the exact same playbook. One of the things people realized too late was that centralized power always possesses the ability to manipulate, a fact that eventually came to light through the Facebook and Twitter lawsuits and similar cases over time. Moreover, there was no such thing as account ownership in any of these centralized systems; we were essentially no different from non-paying tenants (or tenants paying with our data). As time passed, we saw what happened when the landlords started evicting people. Nowadays, even just posting anti-government content can get you kicked out of these centralized platforms in an instant, and you have absolutely no ownership rights over your data or anything else. We've actually been living in a crappy system for a long time, and this system is still going on. From the era I caught onto, ActivityPub w

In this article, we review useEditorContext in n8n codebase. You will learn: Composables in Vue useEditorContext as composable in n8n. Composables in Vue In the context of Vue applications, a "composable" is a function that leverages Vue's Composition API to encapsulate and reuse stateful logic. When building frontend applications, we often need to reuse logic for common tasks. For example, we may need to format dates in many places, so we extract a reusable function for that. This formatter function encapsulates stateless logic: it takes some input and immediately returns expected output. There are many libraries out there for reusing stateless logic - for example lodash and date-fns , which you may have heard of. By contrast, stateful logic involves managing state that changes over time. A simple example would be tracking the current position of the mouse on a page. In real-world scenarios, it could also be more complex logic such as touch gestures or connection status to a database. This is just like React Hooks. Learn more about Composables . useEditorContext as composable in n8n. Now that we understand what a composable is in Vue.js, btw, n8n editor-ui is written in Vue, let's understand how useEditorContext is used. Below is a comment I picked from useEditorContext.ts file. /** * Per - editor host overrides for the current editor context . * * Editor hosts ( e . g . the Instance AI artifact preview ) scope their embedded * editor by providing ` EditorEnabledFeaturesKey ` - the capabilities the host * supersedes . AI features can only be restricted: an explicit ` false ` turns one * off , while omitted ( or ` true `) features fall back to their store values . * ` readOnly ` is a direct flag - ` true ` forces the canvas read - only . When no host * provides the key , AI features fall back to their store values and the canvas * is editable (` readOnly ` is ` false `) . * ` executionSuccessToasts ` / ` executionErrorToasts ` are direct flags too - each * ` true `

A manual psql check answers exactly one question: does this policy work right now, against today's schema, with today's roles. It says nothing about tomorrow. Three ordinary changes are enough to quietly break tenant isolation without anyone noticing at review time. A migration that drops and recreates a table loses RLS entirely, since it's a per-table flag, not something that travels with column definitions. A new service role for a background job can skip the policy if nobody remembers to apply it. And the most common one: someone grants BYPASSRLS during an incident and never revokes it. Most guides point you at pgTAP here and stop. pgTAP is fine, but it's a separate SQL-based framework with its own runner. If your backend is already on Jest, you don't need a second test framework, you need a Jest test that actually proves a leak can't happen. The core pattern: seed a row as tenant A, query as tenant B, assert the result is empty. Run it through a dedicated low-privilege role, since table owners and superusers bypass RLS by default even with FORCE enabled for the owner. I break down the full pattern, the queryAsTenant helper, testing WITH CHECK on INSERT/UPDATE, catching accidental BYPASSRLS grants, and wiring it into GitHub Actions here: https://devencyclopedia.com/blog/postgres-rls-testing-jest If you're doing this across more than one or two tables, I also built RLSBuilder, a browser tool that generates the CREATE POLICY SQL and a matching Jest test from the same three inputs so they can't drift apart: https://devencyclopedia.com/tools/rls-builder

Mark (aka Winsaucerer) here to show you how you can test your PostgreSQL database like a sorcerer. We are going to be using Spawn, a SQL build system supporting migrations and testing. You do not need to be using Spawn for migrations in order to use it for testing. Spawn does not require any extension installed. All you need is the spawn CLI and a psql connection to the database for Spawn to connect through. Spawn was built to solve some migration pains I've experienced, but I happily discovered that when used for testing, it is very powerful. To show you some of that power, we're going to use a contrived database example. It uses golden file testing to determine success. When the test runs, we capture the stdout and stderr output from psql, and compare that to expected output. Testing with Spawn involves these steps: Create a new test with spawn test new <name> and fill out the test steps Check test outputs with spawn test run <name> (or view the SQL that will be sent to psql via spawn test build <name> ) When outputs are as expected, create the golden file with spawn test expect <name> Run the test and compare to expected output with spawn test compare <name> For now, Spawn only supports connecting via psql, which means that you have access to all the features that psql provides. To get started, follow the Spawn install instructions: Install Spawn And then create a new folder on your system, and initialise a new project with a docker compose config ready for us to play with: # inside your new folder: spawn init --docker docker compose up -d You now have a running docker based PostgreSQL database and a spawn.toml file configured to connect to it. We are not assuming that you are using Spawn or any other tool for migrations, so you can manually create and update the database by connecting directly using psql: docker exec -ti postgres-db psql -U postgres Create the database ⚠️ Caution This post is not intended as an example of how to build an orders database. The des

Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is...

An iOS app can feel slow even when its interface looks well-designed and responsive. The problem may not always be the UI or the code running on the device. Often, the real issues are hidden in network requests, API responses, WebSocket connections, and background activity. For developers, finding these problems requires visibility into what is happening behind the screen. This is where Owlse , a network inspection and debugging tool for iOS and macOS developers, can help. 1. What Actually Makes an iOS App Feel Slow? Several hidden network issues can affect an app's performance: Slow API responses Too many network requests Large data payloads Connection delays Failed or repeated requests Background network activity A user may simply see a loading screen or delayed response, while several network operations are happening in the background. Understanding these operations is the first step toward finding the actual cause of the problem. 2. Why Traditional Debugging Can Make These Issues Hard to Find Network-related problems are not always easy to identify through standard debugging. Developers may need to switch between different tools to inspect requests, analyze timing, investigate WebSockets, and understand application issues. When an app generates hundreds of requests, finding one problematic request can also take considerable time. Without a clear view of network activity, developers often have to rely on assumptions. A dedicated network debugging workflow can make this process much easier. 3. Meet Owlse: Network Debugging for iOS & macOS Owlse is built to give iOS and macOS developers greater visibility into their application's network activity. Instead of treating network behavior as something happening in the background, Owlse helps developers inspect and understand it. With features including live request streaming, request inspection, timing analysis, WebSocket inspection, mocking, crash reporting, search, HAR export, and timeline debugging, Owlse brings impo

Disclosure: I build Schemity , a desktop ERD tool - this post is from our blog and uses it for the examples. TL;DR: The database has a built-in place to document a column - COMMENT ON COLUMN in PostgreSQL, the COMMENT attribute in MySQL - and almost nobody fills it in, because a sentence of prose has to travel the same path as a schema change: a migration file, a review, a deploy. Schemity keeps field descriptions in the diagram instead, where editing one generates no SQL, reads existing database comments in on import, and exports the result as a data dictionary. You can document a database column without touching the database: write the description in the model rather than in the schema. That sounds like a dodge until you price the alternative. The database's own mechanism for column documentation, COMMENT ON COLUMN in PostgreSQL and the COMMENT attribute in MySQL, sends a sentence of prose down exactly the same path as a change to how data is stored - a migration file, a code review, an approval, a deploy window - and on MySQL it does something worse than that. Schemity keeps field descriptions in the diagram, where editing one produces no SQL at all. This is why so many production schemas have thousands of columns and almost no comments. Not because nobody wanted to write them. Because writing one costs a deploy. How do I document a database column without running a migration? Keep the description in the model rather than in the storage engine. A field description is a fact about what the column means to your team; it changes no type, no constraint, no index, and nothing about what the database will accept. When it lives in the diagram, editing it is like editing a comment in a code file: you change it, review it in the same pull request as everything else, and nothing has to run against production for it to take effect. The moment that description is a column comment, it stops being prose and becomes DDL. Now it needs a migration file, and the migration needs a

Every one of my publishing lanes went dark for four days, and every script involved exited with status 0. Nothing had crashed. The files themselves had quietly stopped existing on disk — macOS had uploaded them to iCloud and deleted the local copies to "optimize storage." Why This Matters What it means for automation to depend on its environment When you run 160+ launchd jobs around the clock, the execution environment itself becomes a failure source before your script logic does. Ports get exhausted, processes orphan and pile up, memory never frees — I wrote about that class of resource leak last time. This is a completely different kind of total failure that happened the very next day. The files had become fatal to read . Not a bug in my code. Not a filesystem bug. An unintended side effect of a mechanism macOS runs under the name "optimization." What optimize-storage actually does macOS's "Optimize Storage" (System Settings → General → Storage → Optimize Storage), on a machine with iCloud Drive enabled, uploads files under Desktop and Documents to iCloud and deletes the local copies when free disk space gets tight . In Finder they still look like normal icons, but there is no local data — they are in a "dataless" state. Click one and it downloads automatically. For a human user, that's an acceptable tradeoff. The problem is automation scripts. python3 's open() , pathlib.Path.read_text() , cat , jq , cp — all of them die instantly on a dataless file with Errno 11: EDEADLK: Resource deadlock avoided . The name "Resource deadlock" makes you suspect a deadlock, but this is a POSIX errno code that macOS repurposes to mean "waiting for a file download." No lock is contended. No thread is stuck. The mere fact that "the data isn't local" surfaces to the process as a fatal error code. You can also get EAGAIN (resource temporarily unavailable). That one shows up as a race right after a download starts. The actual damage: four days of zero posts On August 6, 2026, note's a

The API makes it look trivial. const number = await carrier . numbers . buy ({ phone_number : " +1... " }); await db . insert ( " rented_numbers " , { user_id , e164 : number . phone_number }); await stripe . subscriptions . create ({ customer , price }); Three lines, one number, done. Ship it. What you actually wrote is a distributed transaction across three systems. They share no transaction log, they have no two-phase commit, and none of them can roll back the others. The carrier will keep charging you for a number your database has never heard of. Stripe will stop charging for a number your database still thinks is paid up. Neither one is going to mention it. I run a virtual phone number product. Below are the failure modes that actually cost us money, roughly in order of how much. The orphan taxonomy Write down the states first, because the interesting ones are the states nobody designs for. Three systems, each holding an opinion about a single number: Your DB Carrier Stripe What is actually happening active owns it active The happy path. Rare in the tail. no row owns it nothing You pay monthly rent on a number nobody can see or use. active released active You bill a customer for a number you no longer own. pending_cancellation owns it canceled Customer stopped paying. You are still paying the carrier. active owns it canceled You provide service for free, indefinitely. cancelled owns it canceled Release failed at teardown. Silent monthly bleed. Every row under the first one is reachable from a plain network timeout at a bad moment. The first orphan class is the worst, because you cannot see it from inside your own product. No row, no user, no support ticket. The number sits in the carrier's inventory producing an invoice line every month until somebody actually reads the invoice. The second class is the one that generates a complaint. The rest leak money in one direction or the other, quietly. Reconcile, don't prevent The instinct is to armour the write path. S

Short answer: the best way to reduce LLM cost for a product catalog is to measure cost per accepted record, then route each job by difficulty. Count prompt tokens before the call, use a small model for the easy summarize/classify/extract-JSON cases, reserve a stronger model for exceptions, and batch work that can wait. The winning option is the one that preserves the required fields at the lowest cost per accepted result, not the one with the lowest token rate. I build RAG and agent features in Python, so I want this decision to survive contact with production. A catalog description such as “blue jacket, recycled nylon, sizes S–XL” looks easy until one tenant sends HTML, another sends translated attributes, and a third puts three products in one paragraph. The useful unit is therefore a tenant-scoped record with an outcome, a token count, a retry count, and a review decision. Start with the record, not the model For this workflow, the data path is short: normalize a description, count the tokens in the planned prompt, choose a route, request a typed result, validate it, and write the result and its cost to an append-only ledger. A rejected result must remain visible. Otherwise, a cheap model can appear successful simply because invalid JSON and missing attributes disappeared from the report. The ledger needs enough identity to answer a boring but important question: which tenant paid for this retry? I use a job ID, tenant ID, source revision, model route, input token count, output token count, latency, validation status, and review status. Keep the raw description out of ordinary cost dashboards when it contains customer data; the ledger can hold a reference to the source revision instead. Prompt-token counting is a gate, not a decorative metric. If an HTML description expands the prompt beyond its budget, the worker can strip markup, split the record, send it to review, or select a different route. Those choices have different quality consequences, so record the re

There's a moment every developer hits the first time they connect an AI assistant to a real database: it works beautifully, the model writes a clean SELECT , you get your answer in seconds — and then a small, cold thought arrives. What if it had written DELETE instead? That worry is healthy. An AI agent that can query your production database is also, by default, an AI agent that can UPDATE , DROP , and TRUNCATE it. Large language models are probabilistic. They hallucinate. They misread a vague prompt like "clean up the test users" as an instruction to actually delete rows. You don't want the only thing standing between a confused model and your orders table to be good intentions. The fix isn't to keep AI away from your data. It's to make write operations structurally impossible — read-only by design, enforced at layers the model can't talk its way past. This post walks through how to do that properly, from the database grant all the way up to query-level guardrails. Why "just prompt it to be careful" fails The tempting shortcut is to add "only run SELECT queries, never modify data" to your system prompt and call it a day. Don't rely on this. Prompt instructions are suggestions, not enforcement. A cleverly worded user request, an injected instruction hidden in some data the model reads, or a plain misunderstanding can all lead the model to generate a destructive statement anyway. Real read-only access is enforced below the model — in places where no amount of clever text can override it. Think of it as defense in depth, with at least three independent layers: Layer What it stops Enforced by Database permissions Any write reaching the engine SQL GRANT / REVOKE Connection / replica Writes even being routed to a writable node Read replica, read-only transaction Query parser / broker Non-SELECT statements before they run SQL parsing, allowlists Any one of these is decent. All three together mean a write has to defeat your database engine, your routing, and your parser s

I run self-hosted Chatwoot as the WhatsApp inbox for a dozen or so small Israeli businesses. Two servers, a few thousand conversations a week, a drip-sequence engine bolted on the side. Chatwoot is good software. The self-hosting docs will get you to a running container. What they will not tell you is which failures actually happen at month six, when you have real customers and real volume. These five all bit me in production, and none of them looked like what they were. 1. Your disk fills from somewhere Postgres never sees I got a disk alert at 86 percent and immediately went looking at the database. That was the wrong place. DB (postgres): 680 MB chatwoot_storage_data: 17 GB Attachments live in ActiveStorage, on a Docker volume, not in Postgres. Every image, voice note, and PDF a customer sends is a file on disk, and none of it shows up when you check database size. If your monitoring watches the DB, it will report everything is fine right up until the container cannot write. The growth curve is a function of how many accounts you host, not how busy any one of them is. Mine sat at roughly 0.05 GB a month until I onboarded seven new businesses over two months, and then it hit 16 GB a month. Check the right volume: docker system df -v | grep chatwoot_storage_data 2. Forty-four percent of my outbound storage was duplicate files This is the part that surprised me. When I actually measured what was on that volume, almost half the outbound media was byte-identical copies of the same file. One 14.5 MB video was stored 48 separate times. One image was stored 325 times. Chatwoot creates a new blob and a new file on disk on every send, even when the bytes are identical. That is correct behavior for a chat app where every message owns its attachment. It becomes expensive the moment you have anything that fans one file out to many conversations. In my case it was not campaigns at all, it was the drip engine sending the same media to 48 separate conversations as ordinary outbo

This is a writeup of building a required CI gate for degraded-network behavior. The system under test is a robotics fleet substrate, but the finding applies to anyone shaping networks in CI. Ganglion exists to reach robots on networks nobody controls. Warehouse Wi-Fi, carrier CGNAT, a hospital VLAN, a customer firewall that was configured once in 2019 and has not been touched since. Until this week that claim was a sentence on a website. CI ran on clean loopback, everything was green, and the failure modes that actually matter in the field were the exact ones the test suite could never produce. That is now a required gate. Every push to main runs the full deploy, invoke and verify round trip over the relay against five shaped network profiles, and all five have to pass before anything merges. I build Ganglion, so treat the enthusiasm accordingly. The part worth your time is not that it went green. It is what I got wrong on the way there. The five profiles clean : baseline, no shaping. If this one fails, something else is broken. lossy : packet loss with light reordering. high-latency : 250ms round trip. asymmetric : plentiful downlink, starved uplink. This is the one nobody tests and the one teleop actually dies on, because control acknowledgements go the starved direction. nat-relay : endpoints with no route to each other at all, forcing hole punching to fail and relay fallback to carry the session. The last two are the ones I care about. Loss and latency are what people imagine a bad network is. Asymmetry and no-direct-route are what a bad network usually is. What I got wrong The original design assumed you can pin netem's seed and get a repeatable lossy run. Two profiles: a pinned-seed one that gates the build, and a nastier randomized one that runs nightly and is allowed to fail. You cannot pin netem's seed. Its loss and jitter draw from the kernel RNG and there is no seed parameter to set. A "deterministic lossy netem profile" is not a thing that exists. This m

"I will tear apart this forest and bring back what it took from me."