I recorded my Kubernetes AI agent failing, on purpose
Most demos of AI-for-infrastructure tools show a clean run. I think that is exactly backwards, and I want to explain why by walking through the thirty seconds of my own demo that a normal product video would have cut. The scene A deployment is crash-looping. The agent has already diagnosed it. I ask it to restart the deployment. It stops and asks for approval. I approve. The restart runs, and the command succeeds. Then I ask the follow-up question: did the restart change anything? No. The pods fail with the same error. A restart was never going to supply a missing environment variable. The agent re-reads the pods, re-reads the events, and names the same root cause it found at the very beginning: DATABASE_URL is not set, so the container exits 1. The restart was a reasonable thing to try and the wrong thing to fix it, and the system is the one saying so. Why that scene is the product The worst failure for an incident-response tool is not being unhelpful. It is emitting something that looks like a real diagnosis and is not. At 3am a confident wrong answer costs more than no answer, because it sends a tired human down a wrong path with false authority. So the design follows from that. The approval gate is at the tool boundary, not in the prompt. A guardrail written into a system prompt is a suggestion, and models can be argued out of suggestions. Instead every mutating operation passes through one chokepoint that returns exactly three answers: do it, ask a human, or refuse. The model does not get a vote, so there is nothing to talk around. Detectors are compiled predicates, not model calls. They are always on, they run on every observation, and they cost zero tokens. The LLM is only invoked once a detector actually fires. Watching your cluster is free; thinking about it is the expensive part, so it happens last. Every decision is appended to a hash-chained log. A run can be replayed afterwards instead of remembered. If you are going to let software touch production, "w