Why API Breaking Changes Still Reach Production Even With CI/CD

Why API Breaking Changes Still Reach Production Even With CI/CD A few years ago I watched a "tiny" API change take down checkout for about forty minutes. The change was a one-liner. The pull request had two approvals. CI was green across the board. And it still broke production, because the thing that actually mattered was never tested. If you run microservices at any real scale, you have lived some version of this. Let's talk about why it keeps happening even with a mature pipeline, and what the teams who don't keep getting paged do differently. The Problem Here's the change that caused the outage. A payments service had a response that looked like this: { "status" : "ok" , "transaction_id" : "txn_8842" , "amount_cents" : 4200 } Someone renamed amount_cents to amount and switched it to a decimal, because "cents is confusing." Cleaner field, better docs. The producing service's tests were updated to match, everything passed, it shipped. The problem: three downstream services still read amount_cents . One of them was the order service, which now received undefined , multiplied it by a quantity, and wrote NaN into the database. The failures didn't even surface in the payments service. They surfaced two hops away, in a service the original author had never opened. This is the core issue. A breaking change is not defined by the service that makes it. It's defined by the consumers who depend on it. And the producer's CI pipeline has no idea those consumers exist. Why Existing Approaches Fail The natural reaction is "we need more tests." But look at what each layer actually checks. Unit tests verify the code does what the author intended. The author intended to rename the field. The unit tests were updated to expect amount . They passed because they were testing the new, broken behavior. Green unit tests told us nothing. Integration tests verify the service works with its own dependencies — its database, its cache, the APIs it calls. They almost never spin up the services