The Fallacies of GenAI Development

In 1994, Peter Deutsch published the Fallacies of Distributed Computing — eight assumptions that every developer building distributed systems makes, discovers are wrong, and pays for in production. The network is reliable. Latency is zero. Bandwidth is infinite. Each assumption sounds true. Each leads to system failures that could have been avoided. Thirty years later, we're making the same category of mistakes with generative AI. The trough of disillusionment for AI-assisted development has begun. Byron Cook, VP and Distinguished Scientist at Amazon, founder of AWS's Automated Reasoning Group (300+ scientists, 15+ teams), says it plainly: "Generative AI is sliding into the trough of disillusionment." The headlines are shifting. The "summer of vibe coding" is over. The disillusionment isn't caused by AI being useless. AI-assisted coding delivers real productivity gains. The disillusionment is caused by false assumptions about WHERE the gains come from and WHAT changes when generation gets fast. Teams expected 10x engineering. They got 10x code generation and 1x everything else. The gap between expectation and reality is the trough. This series names the eight assumptions, explains why each one fails, and presents the resolution — not from theory, but from domains that hit the same wall and climbed out. The Eight Fallacies 1. Faster code generation means faster engineering. You made one sub-system 10x faster. Seven others didn't change. The system doesn't get faster — it breaks at the interfaces. The CPU-memory wall tells you exactly what happens and what fixes it. 2. If the output looks correct, it is correct. AI-generated code is optimized for plausibility, not correctness. It compiles, passes tests, and reads well — while violating properties nobody tested. Plausible is not correct. The gap is where production failures live. 3. You can verify AI output with another AI. Guardrails, LLM-as-judge, AI code review — the verifier has the same failure modes as the thing