The Test Framework Is Not the Product

A few years ago, the hardest part of building a browser test framework was getting started. You had to choose a runner, configure browsers, create page objects, wire up reporting, add retries, manage secrets, connect it to CI, and convince someone else on the team to learn how the whole thing worked. Today, you can open an AI assistant and ask it to generate most of that before lunch. That sounds like a dramatic improvement. In some ways, it is. But it also moves the bottleneck. The question is no longer, “Can we create a framework?” The question is, “Can we operate what was created?” That distinction matters more than it appears. Generation cost is not ownership cost A generated framework feels cheap because the first version arrives quickly. The code compiles, a few tests pass, and the pull request looks more complete than anything you could have written in an afternoon. Then reality starts applying pressure. The application changes. Authentication behaves differently in staging. A shared helper starts hiding failures. Parallel workers collide over test data. Someone upgrades a dependency and three reporters stop agreeing with one another. The initial generation was fast. The ownership cost was merely deferred. This is the central problem described in what actually breaks when Claude generates a large Playwright framework . Large generated systems often fail in the seams: fixtures, abstractions, environment assumptions, test data, and conventions that were never explicitly agreed upon. The code may be readable line by line while the system remains difficult to reason about as a whole. That is a dangerous form of complexity because it looks productive. More code can hide less understanding Teams sometimes evaluate AI-generated automation by counting output: number of test files; number of scenarios; number of passing checks; number of prompts completed; number of lines added. Those numbers are easy to produce and easy to report. They are also weak proxies for confi