标签：#chaosengineering

I Deliberately Destroyed My Kubernetes Cluster at 2 AM. Here's What Died First. Chaos engineering is not about breaking things. It's about discovering that your "production-grade" homelab is held together by hope and a single etcd snapshot before someone else finds out for you. The Setup I was lying in bed at 1:47 AM, staring at the ceiling, unable to sleep. Not because of caffeine. Because of a thought that had been gnawing at me for weeks: If one of my nodes died right now, would my cluster actually survive? I run a 4-node bare-metal Kubernetes cluster on Talos Linux. Dell OptiPlex control plane. Three Raspberry Pi workers. Cilium eBPF. ArgoCD. Longhorn distributed storage. Prometheus. Grafana. The whole cloud-native stack, shoehorned into $220 of scrap hardware and stubbornness. From the outside, it looks solid. ArgoCD syncs green. Cilium status shows healthy. Longhorn volumes are replicated across three nodes. I have etcd snapshots every 6 hours to S3. On paper, I'm resilient. But I had never actually tested it. Not a controlled test. Not a graceful node drain. I mean chaos . Sudden death. The kind of failure that happens at 3 AM when a power supply dies, or a kernel panics, or a neighbor's construction crew hits the wrong breaker. So I got out of bed, walked to my desk, and installed Chaos Mesh. Why Chaos Engineering on a Homelab? Professionally, I design AWS infrastructure with multi-AZ failover, auto-scaling groups, and managed services that abstract failure away. At Siemens, if an EKS node dies, the managed node group replaces it before I finish reading the alert. But my homelab has no managed control plane. No AWS SLA. No auto-repair. If a Pi's USB boot drive corrupts, that node is gone until I physically fix it. I needed to know: What dies first when a worker vanishes? Not "what should die" — what actually dies. Does Longhorn really failover? Three replicas sound great until you realize two of them were on the same node. Does Cilium handle network partitio