Testing the claim: a degraded-link matrix as a required CI gate

This is a writeup of building a required CI gate for degraded-network behavior. The system under test is a robotics fleet substrate, but the finding applies to anyone shaping networks in CI. Ganglion exists to reach robots on networks nobody controls. Warehouse Wi-Fi, carrier CGNAT, a hospital VLAN, a customer firewall that was configured once in 2019 and has not been touched since. Until this week that claim was a sentence on a website. CI ran on clean loopback, everything was green, and the failure modes that actually matter in the field were the exact ones the test suite could never produce. That is now a required gate. Every push to main runs the full deploy, invoke and verify round trip over the relay against five shaped network profiles, and all five have to pass before anything merges. I build Ganglion, so treat the enthusiasm accordingly. The part worth your time is not that it went green. It is what I got wrong on the way there. The five profiles clean : baseline, no shaping. If this one fails, something else is broken. lossy : packet loss with light reordering. high-latency : 250ms round trip. asymmetric : plentiful downlink, starved uplink. This is the one nobody tests and the one teleop actually dies on, because control acknowledgements go the starved direction. nat-relay : endpoints with no route to each other at all, forcing hole punching to fail and relay fallback to carry the session. The last two are the ones I care about. Loss and latency are what people imagine a bad network is. Asymmetry and no-direct-route are what a bad network usually is. What I got wrong The original design assumed you can pin netem's seed and get a repeatable lossy run. Two profiles: a pinned-seed one that gates the build, and a nastier randomized one that runs nightly and is allowed to fail. You cannot pin netem's seed. Its loss and jitter draw from the kernel RNG and there is no seed parameter to set. A "deterministic lossy netem profile" is not a thing that exists. This m