Rebuilding the Hull at Sea
The box that ran everything started dying in April. Not dramatically. Machines almost never die dramatically. It started with instability... the kind you explain away once, side-eye twice, and start losing sleep over the third time production goes down while you're in the middle of something else. The host under my entire stack... public site, analytics, security tooling, the AI crew's memory layer... was getting flaky. And flaky hardware only trends one direction. Here's the thing about a homelab that lives in a 40ft fifth wheel: there is no second team. No vendor escalation. No change advisory board. No maintenance window negotiated three weeks out. There's me, a crew of governed AI instances, and a reclaimed Dell T3600 about to get the biggest promotion of its second life. So we didn't try to heal the sick box. We built a new hull alongside it and started moving the ship... plank by plank... while it was still sailing. One ground rule, set day one: the old host stays untouched and keeps serving production until the new hull is proven. Not "mostly proven." Proven. Hold that thought, it matters at the end. Moving containers is the easy part. Docker made that boring years ago, and boring is a compliment in infrastructure. What's never boring is the inventory of everything you assumed and never wrote down. A migration doesn't test your stack. It tests your assumptions. Here's what mine were hiding. 01: The umbilical nobody documented Security stack went first. Suricata, Zeek, Wazuh, CrowdSec, Falco, the whole alphabet, up clean on the new hull. Then the MCP server, the piece that gives the AI crew its hands, refused to come up right. It was hard-wired over HTTP to the crew's memory backend. A live dependency, in production for months, documented exactly nowhere. The crew that documents every f*cking thing had never documented its own umbilical cord. Fix was trivial once we could see it: deploy the brain before the hands. Reorder, redeploy, done. But the lesson isn't