DevOps Questions After We Broke The Release Handshake
Answers from the incident where every dashboard looked politely wrong. The release-api deployment had already been marked complete when the invoice page began returning 503s. The new container was serving traffic, the PostgreSQL migration had committed, and the feature flag was on. A NetworkPolicy added in another repository prevented the new pod from reaching tax-rate-cache . The application team saw errors, the database team saw a clean migration, and Platform saw green nodes. By the time we put all three facts in one incident channel, 63 deployment messages had buried the one that mattered. “Is This Actually A DevOps Failure Or Just One Bad Deploy?” It was a DevOps failure because four teams completed valid local work and nobody owned the release handoff between them. Calling it “just a bad deploy” would have been convenient. We could have fixed the policy, replayed the release, written a short incident note, and carried on pretending that a green Argo CD application means a service is ready for users. The pod was healthy according to Kubernetes. It was also unable to call a dependency required to render an invoice. Both things can be true, which is why a deployment status alone is a fairly poor witness. Our old release process had hidden contracts in too many places: The service repository declared its image and Helm values. The infrastructure repository held network rules. Database migrations ran from a separate GitHub Actions workflow. Feature flags lived in LaunchDarkly, owned by whoever had last touched the feature. The runbook lived in Confluence, where it had last been edited in February. We’ve started putting the release dependencies in the service repository, close to the code that needs them. It is not a clever system. It is a file that a human can read during an incident and a pipeline can check before promotion. release : service : release-api requires : - dependency : tax-rate-cache namespace : finance port : 8080 network_policy : allow-release-api-t