标签：#sre

Table of Contents The Moment That Changed Everything What It Actually Is The Setup Nobody Believes Is This Simple Focused Sessions — One Agent, One Mission Sub-Agents With Specific Skill Sets Building MCPs for Your Own Stack What We've Actually Achieved How It Became a Force Multiplier for Incident Response From OpenCode to FRIDAY — The Agent That Investigates Incidents Autonomously From FRIDAY to JARVIS — Thinking About Write-Path Autonomy Deleting 130,000 Accounts Without Writing Code Learning the Entire Product in Conversations Why This Is Different From ChatGPT The Uncomfortable Truth What's Next The Moment That Changed Everything It was 11pm on a Tuesday. A cache migration in our production environment had just caused thousands of authentication failures for two of our largest enterprise customers. Our VP of Product wanted answers. Our support team was fielding escalations. And our engineers were alt-tabbing between AWS console, Datadog, GitHub, Azure DevOps, and PagerDuty trying to piece together what happened. Three weeks later, when we needed to attempt the same change again, an engineer typed this into a terminal: "Review the ADO change ticket, compare the MOP against the actual ElastiCache configuration in prod region, check the K8s config repo for how Redis env vars are wired on the Green cluster, and tell me if this approach avoids the token validation failure that caused the previous customer impact." Fourteen seconds later , the system had pulled the work item, queried AWS ElastiCache across four regions, read the Kubernetes configuration from GitHub, cross-referenced the deployment patches, and delivered a precise technical assessment including a risk it identified that the team hadn't documented: in-flight tokens during the 30–60 second Global Accelerator propagation window. That system is OpenCode — an AI-powered CLI assistant connected to our entire operational stack through the MC(Model Context Protocol). And it has fundamentally changed how a 20-

I run a paid infrastructure service. Alone. No co-founder, no on-call rotation, no senior engineer to escalate to. My only collaborator is Claude Code, and after about a year, my persistent memory has grown to 60+ entries. Those entries have become more valuable than any runbook I've written. They've also taught me — painfully — what makes memory architecture work and what makes it quietly fail. If you're running anything solo with an AI agent, here are five lessons I wish I'd burned into my brain on day one. 1. Write the why , not the what The first instinct when you start using persistent memory is to log what you did. "Migrated service X from tool A to tool B." "Switched protocol from X to Y." Six months later, when something breaks, that information is worthless . You don't need to know what you did — git log and git blame already tell you that. You need to know why you made that choice. What constraint forced it. What you ruled out. Real example. The bad version of an entry I once wrote: Switched the worker pool from Docker containers to systemd units on host. Tells me nothing my git history doesn't. The rewritten version: systemd units on the host instead of Docker containers on this VPS provider. Why: the provider runs aggressive kernel-wide OOM scoring across tenants; containers were getting reaped by oom-killer triggered by other customers' workloads. systemd processes survive because they're scored as system processes. How to apply: any VPS where dmesg | grep -i oom shows kills from PIDs you don't recognize — don't run containers there, run systemd. That one entry has saved me three rebuilds. Because the next time I'm tempted to "just dockerize it, it'll be cleaner," the memory entry says: no, you already learned this, you'll be back here in a week. The pattern: always include Why: and How to apply: lines. If a memory entry can't answer those two questions, delete it. 2. Memory rots — prune or pay About six months in, I did a memory audit. Of 60 entries, 1

Chaos engineering sounds expensive. Netflix built Chaos Monkey to randomly kill production servers. Google runs DiRT (Disaster Recovery Testing) across their entire infrastructure. Amazon does game days where they intentionally take down services. You're building a Node.js API. You don't have a platform team. You don't have a chaos infrastructure. But you still need to know: what happens when your dependencies get slow? The good news is that 80% of the value of chaos engineering comes from one question, and you can answer it locally in five minutes. The one question that matters What does my application do when a dependency responds slowly or not at all? Not "what if the server catches fire" — that's infrastructure chaos. What about application chaos: the database is slow, the payment API is timing out, Redis is having a bad day. These happen constantly in production and they're almost never tested. The failure modes look like this: Your DB gets slow under load → your API response times climb → your timeout fires → you retry → now you're sending twice the load to an already-slow DB Your Redis cache goes down → every request hits Postgres directly → Postgres gets slow → same cascade Stripe's API takes 3 seconds instead of 200ms → your checkout route times out → users get errors → you're losing revenue Every one of these is a latency failure , not a crash. The service is still up. It's just slow. And slow is the hardest failure mode to test because your local environment is fast. Why local testing misses this When you test locally, your "database" is either: A real local Postgres running on the same machine (sub-millisecond latency, not production-like) A mock that returns instantly ( jest.fn().mockResolvedValue(data) ) A fake with a flat delay ( await sleep(200) ) None of these produce realistic latency. A real production database has: Fast responses most of the time (p50 ~5ms) Occasional slowdowns (p95 ~50ms) Rare but real spikes (p99 ~200ms, p99.9 ~2000ms) The spik

A model fallback that only works in a diagram is not resilience. It is a TODO with better branding. If your product depends on AI agents, one slow provider, rate-limit spike, regional restriction, malformed response, or model behavior change can turn a useful workflow into a confusing user experience. The dangerous part is not always a clean outage. The dangerous part is a half-working fallback that silently changes schemas, drops tool state, skips citations, or gives users lower-confidence output without saying so. This guide shows how to run practical AI model failover drills before production traffic teaches you the lesson the hard way. The goal is not to make every model interchangeable. The goal is to keep the user workflow safe, honest, and recoverable when the primary model cannot do the job. Why model failover needs drills, not just retries Most teams start with a simple fallback chain: try the primary model, then a backup model, then show an error. That is better than nothing, but it misses the real problems in AI applications. Traditional APIs usually fail in obvious ways: timeout, 500, bad credentials, quota exceeded. AI systems can fail more subtly: The backup model returns valid JSON with different field meanings. A cheaper model ignores part of the tool policy. A provider accepts the request but streams tokens too slowly. A fallback model does not support the same function-calling format. A regional policy or access rule changes availability. The model completes the answer but loses citation discipline. The agent retries and burns the tenant budget. The final response looks polished but skipped the expensive verification step. Recent AI infrastructure conversations are pointing in the same direction: the system around the model now matters as much as the model. Agent benchmarks, provider reliability, AI cost pressure, and model routing are all active developer concerns. Search results also show many broad posts about LLM fallback strategy, but fewer pr

The Runbook That Lied to Me at 3am The pager went off at 3:14am for a wedged OpenStack Neutron agent. I did what any tired engineer does: I opened the runbook. It told me to restart a service that had been renamed eighteen months earlier, pointed at a Grafana dashboard that 404'd, and assumed a network topology we'd migrated off of two quarters back. The runbook wasn't just unhelpful. It was actively lying to me, and I burned twenty minutes trusting it before I gave up and went to read the source. That's the real problem with documentation. It isn't that we don't write it. It's that the moment we finish writing it, it starts rotting, and the cost of keeping it fresh is high enough that nobody pays it until the document has already betrayed someone at 3am. A runbook your team doesn't trust is worse than no runbook, because no runbook at least forces you to think. This is where AI actually earns its keep in a platform org, and not in the way the marketing decks suggest. AI is not going to own your documentation. It's going to do the tedious first-draft labor — turning a resolved incident, a chunk of shell history, or a deploy diff into a structured skeleton — so a human engineer can spend their scarce attention on the part that matters: verifying the commands, marking what's unproven, and editing the robotic tone out so the team actually reads it. AI drafts. You verify and sign off. That distinction is the whole game. Why "Humanizing" AI Is the Job, Not a Slogan Let me be precise about what I mean by "humanizing AI," because the phrase gets abused. I don't mean making AI sound human to fool a reader. I mean keeping a human in the loop as the editor and owner of record, and doing the unglamorous work of turning a competent-but-soulless machine draft into something a colleague trusts. Two things break trust in AI-drafted docs, and both are fixable by a human pass: Unverified claims stated with confidence. An LLM will happily tell you to run systemctl restart neutron-l3-

Your runbooks are out of date. I don't know your team, but I'd bet money on it. Most teams write runbooks once, in a panic after an outage, and then never touch them again until the next outage proves them wrong. How runbooks rot Steps reference deprecated tools. The grafana dashboard moved, the CLI command was renamed, the bastion host got retired. Nobody updated the runbook because nobody re-ran it during the calm months. The team that owned the system left. Three of the five engineers who wrote it are gone. The remaining two haven't actually run the runbook in 18 months because the outage type it covers stopped happening. Half-truths from the start. The original author skipped the obvious steps (because they were obvious to them) and the new on-call engineer can't reproduce the recovery. The result: at 2 AM during the actual incident, the runbook sends you down a dead end. Now you're improvising under stress. What working runbooks have in common I've seen exactly two patterns work: 1. Runbooks live with the code. Put them in the repo of the system they document. When the code changes, the runbook PR is part of the same review. Out-of-repo wikis die first because the cognitive distance is too great. 2. Runbooks are reviewed by people who weren't there. Have a junior engineer run through the runbook quarterly on a non-incident day. Every place they get stuck is a real bug in the document. Fix it then. The author will be too close to see the gaps. The three sections that matter A useful runbook has exactly three sections: Symptom : how do I know this is the right runbook? Concrete signals, with example screenshots if visual. First 5 minutes : what to do RIGHT NOW to stop the bleed. Not the root cause investigation, just the triage actions. Investigation : where to look, what queries to run, what to escalate. That's it. Anything else (architecture diagrams, history, philosophy) goes in a separate doc that the runbook links to. The cultural part The hardest part isn't

Failures are obvious. Retries are sneaky. When something fails, everyone notices. An alert goes off. A request errors out. Someone starts investigating. Retries are different. They look harmless. Most of the time, they save the system. But sometimes, retries create bigger problems than the original failure. Imagine an API call times out. No problem. The system retries. But what if the first request actually succeeded and only the response was lost? Now the retry creates: duplicate orders repeated emails inconsistent records workflows running twice The failure happened once. The retry multiplied it. Another thing I've seen: One slow dependency causes requests to pile up. Retries start firing. Those retries create even more traffic. Which slows things down further. Which triggers even more retries. Suddenly, the system is spending more effort retrying than doing useful work. Retries also hide problems. A temporary issue gets retried five times and eventually succeeds. Everything looks normal. Meanwhile: latency increases queues grow users experience delays Nothing technically failed. But the system is getting less healthy. What changed for me is that I stopped treating retries as free. Every retry has a cost. It consumes resources. It increases load. And if actions aren't designed carefully, retries can repeat side effects that should only happen once. Now when I build something, I don't ask: "What happens if this fails?" I ask: "What happens if this runs again?" Because in production, things almost always run again. And if the answer is "bad things happen," the retry mechanism isn't helping. It's making things worse. Failures are part of every system. Retries are too. The difference is that failures usually happen once. Retries can turn one problem into hundreds if you don't design for them. This is something we think about constantly at BrainPack when operating long-running workflows across multiple systems. AI and automation layers make retries even more common, wh

Most AI coding tools assume you're sitting in front of a repo. There's a working directory, some source files, tests, maybe a CI pipeline. The AI reads your code, suggests changes, ru****ns tests. Great model — if you're a developer writing code on a Tuesday afternoon. Now picture the other scenario. It's 2am. PagerDuty fires. You SSH into a box you haven't touched in three months. Something is broken, you're not sure what, and the runbook was last updated by someone who left the company in 2022. You're not thinking about repos. You're thinking: what OS is this thing running? What just failed? Is it safe to restart that service or will I make it worse? These are two fundamentally different workflows. But almost every AI terminal tool I've seen is built for the first one. The 30 minutes nobody builds for There's a ton of tooling for the world before you log into the box: Prometheus, Grafana, PagerDuty, incident.io, runbooks, dashboards. All useful. No complaints. But there's this practical 30-minute window after you SSH in where you're basically doing archaeology with journalctl and grep. You check systemd. You look at disk. You read logs that were clearly written by someone who hated future-you. You copy-paste terminal output into Slack so your teammate can squint at it from a different timezone. This is where I think AI could actually help. Not by replacing Grafana. Not by building another dashboard. Just by being present in the shell while you're debugging. Please, for the love of uptime, don't replace my shell I've seen a few AI terminal projects that basically build an entire new terminal experience from scratch. New keybindings, new UI, new everything. Here's the thing: ops people have muscle memory. We have aliases we wrote in 2019 and forgot about. We have tmux configs we'd defend with our lives. We SSH through jump hosts with key forwarding chains that barely work but have worked for years so nobody touches them. If your AI tool requires me to abandon all of

Why This Matters: The 2 AM Problem It's 2 AM. Your phone rings. Your production database is down. Customers can't log in. Revenue is dropping by the second. You call the Ops team. They restart the server. Downtime: 45 minutes. Cost: $100K in lost sales. Root cause? Unknown. This happens thousands of times a week at companies worldwide. The question isn't "Will your system break?" It's " When it breaks, are you ready? " That's where SRE comes in. What is SRE? (The Real Definition) SRE (Site Reliability Engineering) = Applying software engineering principles to build reliable, scalable infrastructure and systems. It's not just about keeping servers running. It's about: Reliability : Systems that don't break unexpectedly Scalability : Systems that handle growth without collapsing Infrastructure : Automating how systems are built, deployed, and monitored Measurability : Knowing exactly how your system is performing at any moment Traditional operations manages infrastructure reactively — when something breaks, you fix it. SRE manages infrastructure proactively — you engineer it so it rarely breaks, and when it does, it heals itself. The Key Insight: Reliability is Engineered, Not Hoped For Here's the critical shift in thinking: Old mindset : "Let's build this system and hope it doesn't break." SRE mindset : "Let's measure what 'reliable' means, design the system to achieve that, and automate the monitoring and recovery." But reliability isn't just uptime. It includes: Uptime : Is the system available? Latency : How fast does it respond? (A slow system is effectively broken) Error rate : What percentage of requests fail? Throughput : Can it handle the traffic? User experience : Does the system meet user expectations? All of these are engineered and measured. A Simple Analogy: The Bridge Imagine you're managing a bridge. Traditional approach : Engineers patrol daily, react to problems, work around the clock fixing issues SRE approach : Engineers design monitoring that aler

The thesis Quorum is built on is uncomfortable and true: the tools a team uses to coordinate an incident often live in the same region as the thing that is failing. When the region goes, the incident response goes with it. You are now coordinating a region outage over a status page that the region outage took down. Quorum is an incident command plane designed to survive a region loss. This post is about how the failover works, what the live demo does and does not prove, and where the survival story currently ends, because a database audience will ask all three and they deserve a straight answer. What DSQL gives you A multi-region DSQL cluster in the US set is three regions: two full regions, which for Quorum are us-east-1 and us-east-2, and a log-only witness in us-west-2 that has no cluster endpoint of its own. Both full-region endpoints present a single logical database with strong consistency, and the architecture is designed for 99.999% multi-region availability with no single point of failure and automated failure recovery . The behavior that matters for an incident tool is stated plainly in the GA announcement : applications can keep reading and writing with strong consistency even when they are unable to connect to a region's cluster endpoint, and the third region acts as a log-only witness with no cluster resource or endpoint. The survivor keeps serving; the witness holds the log so the surviving region keeps commit quorum. Quorum is, in effect, a live demonstration of that reference behavior with an incident-command product wrapped around it. Quorum's failover layer AWS's guidance for multi-region DSQL is to put routing in front of the endpoints: either DNS-based routing with Route 53, or application-level routing logic, so traffic redirects automatically when an endpoint becomes unreachable. This is laid out in Implement multi-Region endpoint routing for Amazon Aurora DSQL . Quorum, a Next.js app on Vercel, does the application-level version: it detects an

If you're a DevOps, SRE, or Cloud engineer in India — or hiring one — the market in 2026 looks very different from a few years ago. Instead of guessing, we analyzed 800+ live DevOps/SRE/Cloud/Platform Engineering roles currently on PuneOps to see what's actually being hired for right now. Here's what the data shows. 1. Bangalore dominates, but the market is genuinely national DevOps hiring in India is no longer a one-city story. Of the live roles: Bangalore — the clear leader, ~25% of all listings Pune — a strong #2 (and a serious DevOps hub, not just an IT-services town) Hyderabad, Mumbai, Delhi NCR, Chennai — all with steady, healthy demand Remote / Pan-India — roughly a third of all roles don't tie you to a city at all Takeaway for candidates: you're no longer limited to wherever you live. Remote and pan-India DevOps roles are a huge and growing slice of the market. 2. This is a senior-heavy market The single most striking pattern: DevOps hiring in India skews experienced. The largest band by far is 5–10 years of experience A meaningful chunk wants 10+ years (architects, principals, platform leads) Entry-level (0–2 years) roles are comparatively rare Takeaway: DevOps remains a hard field to break into directly. Most roles assume you've already done software, sysadmin, or cloud work. If you're junior, the path in is usually via a software/ops role, then specializing. 3. The skills employers actually ask for Across the listings, the same technologies show up again and again: Kubernetes — effectively table stakes now Terraform / IaC — infrastructure-as-code is expected, not bonus AWS / Azure / GCP — cloud fluency, often multi-cloud CI/CD pipelines, observability, and Python for automation Takeaway: if you're leveling up, Kubernetes + Terraform + one major cloud is the core combination Indian employers are screening for in 2026. 4. Salary ranges (market benchmarks, 2026) Compensation varies widely by company type (product vs. services), city, and exactly how senior t

Error budgets are useless without a policy. 'We're out of error budget' should trigger consequences. If it doesn't, you don't have an error budget — you have a vanity metric. Here's a policy that actually works. The four states Healthy (< 70% of budget used). Business as usual. Feature development proceeds at full speed. Watch (70-90% used). Feature velocity continues but new risky changes require explicit sign-off from an SRE. No gate, just attention. Constrained (90-100% used). Feature freezes. Only reliability work and critical bug fixes until we're back below 90%. Breached (> 100% used). Incident-level response. Leadership informed. Post-mortem for why we blew through. Feature work stays frozen until we recover and identify systemic causes. The part most policies miss The feature freeze in 'constrained' state is the part that actually changes behavior. Everything else is documentation. Without consequences, teams ignore the budget. The freeze has to be real . Leadership can't override it for a 'really important feature' — that's exactly the time the freeze matters. The only exception is a legitimate emergency fix, and those should be rare. Selling this to leadership Executives hate feature freezes. They see it as slowing the business. Counter-argument: feature freezes during budget exhaustion protect the business. Shipping features onto broken infrastructure creates more breakage, which burns more budget, which is a doom loop. Frame it as: 'the feature freeze is a safety valve. When it triggers, it's because something's wrong and we need to fix it before making it worse.' Also: a good policy lets you spend the budget aggressively when you have it. Feature teams should be encouraged to experiment, deploy fast, and take risks when you're at 30% budget used. The freeze is only for when the safety margin is gone. The review cadence Weekly error budget review, 15 minutes max. Who attended: SRE lead, engineering manager, maybe a PM. Decisions: are we in healthy/watch/

In the summer of 2016, Pokémon GO launched to a user base roughly fifty times larger than its capacity planning had anticipated. The engineering team had done load testing. They had throughput thresholds. They had autoscaling configured. Within hours of launch, the service was degraded globally — not because the infrastructure could not scale, but because it scaled too slowly against an arrival rate that exceeded every modelled scenario, and because the metric that was driving scaling decisions (CPU utilisation) lagged behind the actual saturation signal by several minutes. By the time CPU registered critical, the request queue had already grown to the point where p99 latency had crossed into the range where users were abandoning sessions faster than new sessions were being created. The engineering post-mortem identified the same root cause that appears in the post-mortems of most capacity-related incidents: the organisation's operational metrics were measuring how hard the infrastructure was working, not how much work the service could safely accept. CPU percentage is a resource utilisation metric. Memory percentage is a resource utilisation metric. IOPS is a resource utilisation metric. None of them is a service throughput metric. None of them tells you, with precision, at what arrival rate your SLO begins to degrade. Safe Operating Throughput is that metric. It is not a new concept in queueing theory or systems engineering — the idea of a safe operating ceiling predates modern distributed systems. What is new is its treatment as a first-class SRE metric: formally derived from load test data and SLO targets, continuously monitored for drift, and operationally enforced as a constraint in autoscaling configuration, capacity planning decisions, and deployment pipeline gates. Why Existing Capacity Metrics Are Insufficient The canonical capacity management approach in most organisations works like this: observe CPU or memory utilisation, set an autoscaling threshold (t

Three weeks ago, one of the teams we work with had a checkout outage. The root cause a malformed...

Three years ago we had our first real outage. Six hours of downtime. Thousands of angry users. Multiple executives on the call. Here's what we did right, what we did wrong, and what we'd do differently. What we did right 1. Communicated immediately. The moment we knew we had a problem, we updated the status page and emailed our biggest customers personally. Not when we had answers. When we had a question. 2. Had a single incident commander. One person making calls. Not a committee. When the CEO tried to direct technical work, the IC politely rerouted and told her where her help was actually needed (talking to customers). 3. Took care of our people. During hour 4, I ordered food. During hour 5, I forced the primary engineer off the call for 20 minutes to walk outside. Long incidents destroy people. You have to feed them and force them to rest. 4. Wrote it down as we went. We had a shared doc with a live timeline. When the post-mortem came, we had every decision captured. What we did wrong 1. Tried to fix the root cause during the incident. For the first 2 hours, we were digging into why the database was struggling. We should have been mitigating (rolling back) first. 2. Let too many people 'help.' By hour 3, we had 12 engineers in the call. Half of them were useless. The IC should have kicked people out sooner. 3. Gave optimistic estimates. 'We'll be back in 30 minutes.' We were not back in 30 minutes. That miscommunication was worse than saying 'unknown.' 4. Didn't prepare the executive communication. The CEO had to answer customer questions in real time with no script. We should have drafted talking points for her after hour 1. What we'd do differently Mitigate first, investigate second. Always. Cap the number of active engineers at 4 during an incident. Others go on standby. Default to 'unknown' for estimates. Only give a number when we're sure. Assign someone explicitly to 'executive liaison.' Their job is to keep the C-suite informed without interrupting the tec

Production is down. Slack is on fire. Your phone is ringing. You've seen this exact error before — ConnectionResetError: [Errno 104] cascading through your FastAPI worker pool — but you can't remember exactly which Redis configuration tweak fixed it last time, who applied it, or how long the incident lasted. You're starting from zero again. Twenty minutes of context-building before you even touch a fix. I got tired of that feeling. So I built an AI agent that never forgets. The Problem With Generic AI in Production When production breaks, most engineers reach for their LLM of choice and paste in the stack trace. And the response is almost always the same: a competent, thoughtful, completely useless answer. The model has no idea that your team already tried increasing max_connections six weeks ago and it made things worse. It doesn't know that your infrastructure runs on a specific internal Kubernetes setup that changes how standard fixes apply. It gives you textbook advice for textbook problems, and your problems are never textbook. This is what I started calling the Round 1 problem. Round 1 — generic response: Error: ConnectionResetError: [Errno 104] Connection reset by peer Stack: redis.exceptions.ConnectionError in worker pool The agent responds with something like: "This typically indicates your Redis connection pool is exhausted. Try increasing max_connections in your Redis client config, add retry logic with exponential backoff, and check network stability between your app and Redis instance." Technically correct. Practically useless if you've already tried all three. The agent is reasoning from general knowledge, not from your specific production history. It has no memory of your past incidents. Every error feels like the first error. What I Built: Code Memory's Incident Agent Code Memory is a developer workspace I built in Next.js with a three-pane interface — a file explorer, a code viewer with syntax highlighting, and a real-time AI fix panel. But the core

Using threat modelling to make system dependability observable, testable, and actionable Executive summary Security and Reliability address system degradation. Security addresses degradation from intentional actions, such as denial-of-service attacks, while reliability addresses degradation from failure, load, dependency behaviour, operational change, or complexity. The underlying engineering question is the same: which critical system property can degrade, how would users experience that degradation, how would we detect it, and what controls would prevent, contain, or recover from it? This document proposes a practical way to bridge the two disciplines: anchor analysis on Critical User Journeys, express expected behaviour through SLOs and SLIs, use RAMSS to ensure coverage across dependability dimensions, and adapt PASTA-style threat modelling to reliability scenarios. The goal is not to merge Security and Reliability into one generic practice. The goal is to reuse the strongest habits of each discipline: security's adversarial modelling and reliability's production-oriented measurement, validation, and recovery loops. The most useful outcome is a shared model of degradation scenarios. A degradation scenario links a critical user journey to a concrete reliability or security threat, the system weakness that makes it possible, the signal that would detect it, the objective it would violate, and the mitigation or experiment that would validate the control. This makes risk easier to discuss with engineering teams because it connects abstract concerns to user impact, SLO burn, business loss, and testable remediation. 1. The problem: two disciplines, one degradation model After working in both Reliability and Security, I found that the two domains share much in common: both focus on objectives, weaknesses, control effectiveness, incident response, prioritisation, and residual risk, but often use different rituals, terminology, metrics, and boundaries. This separation ca

The test passed. The runbook completed. Infrastructure came back online inside the RTO window. None of that means the organization can recover from an actual disaster. Disaster recovery testing is designed to succeed. Clean environments, pre-staged dependencies, known failure modes, available staff — each design decision is operationally reasonable. Collectively they remove the conditions that make real recovery hard. What the test validates is test completion, not recovery capability. The Test Is Designed to Pass Every design decision in a standard DR test tilts toward a successful outcome. The test window is pre-announced, so the right engineers are available. The scope is pre-defined, so unexpected systems don't surface mid-exercise. The environment is either isolated or pre-staged, so competing failures don't complicate the recovery sequence. The data state is known and clean, so integrity issues don't slow the restore. The declaration point is assumed, so nobody has to make an ambiguous call under pressure. A test designed to remove the variables that make recovery hard cannot produce evidence about what happens when those variables are present. What Disaster Recovery Testing Actually Excludes Declaration threshold. In a DR test, recovery starts at a pre-agreed time. In a real incident, recovery starts when someone decides the situation has crossed the threshold for declaration — a decision that is rarely clean and routinely delayed 45 minutes to several hours. That delay is inside the real outage window and outside the test clock. Dependency assumptions. DR tests run against known, pre-cleared dependencies. Real incidents surface undocumented dependencies that were never in scope — a configuration service that hasn't been touched in two years, an authentication endpoint that wasn't in the architecture diagram. Data state. Test environments use clean or pre-staged data. Real recovery requires handling whatever state the data was in at the moment of failure — pa

The DORA research programme is the most rigorous empirical study of software delivery performance ever conducted. Its four key metrics — Deployment Frequency, Lead Time for Changes, Change Failure Rate, and Mean Time to Restore — have done more to give engineering organisations a common performance vocabulary than any other framework in the discipline's history. If you work in software and you have not read the State of DevOps Report, stop and read it before finishing this paragraph. Now: the DORA Four were derived primarily from organisations with cloud-native architectures, on-demand deployment infrastructure, and relatively unconstrained ability to release software when it is ready. The research cohort skews toward technology companies that have already made the cultural and architectural investments that make high-frequency, low-risk deployment possible. This is not a criticism of the research. It is an observation about its generalisability — and it has a specific consequence for practitioners who work in regulated enterprises: banks, healthcare systems, utilities, insurance carriers, government agencies. In these environments, the DORA Four are necessary but structurally insufficient. They measure the delivery pipeline accurately. They do not measure the operational sustainability of the team running that pipeline — and in regulated enterprises, operational sustainability is where SRE programmes go to die quietly, years before anyone realises the damage is permanent. This post proposes a fifth metric. Not to replace the DORA Four, but to complete them — to close the measurement gap that leaves regulated enterprise SRE teams flying blind on the dimension that most reliably predicts long-term programme failure. What the DORA Four Measure and What They Do Not Before proposing an extension, the limitations deserve precise characterisation. Imprecise criticism of a well-validated framework is noise. The limitations described here are structural — arising from the d

Production incidents almost never break in one place. The alert fires in one tool. The broken deploy is in Netlify. The suspicious change is in GitHub. The stack trace is in Sentry. The human context is in Slack. The runbook is in Notion. The "is this actually paging someone?" answer is in PagerDuty. A normal chatbot can sound helpful in that situation. It can say things like "you should check your recent deployments" and "look for related errors in Sentry." But that is not triage. That is a polished to-do list. I wanted something more useful: an agent that could go get the evidence, connect the dots across sources, show its work, and give an operator-grade answer grounded in real system data. The design constraint from the start was simple: no evidence, no answer. That became ReefWatch , a Coral-powered production triage agent built to investigate instead of improvise. It discovers the tools connected to a workspace at runtime, queries them as evidence, correlates records across systems, and produces a compact answer only when the facts support one. Coral became the backbone because it turns the messiest part of agent tooling into something the model can actually reason about: SQL . What This Guide Builds By the end of this route, you will have a blueprint for an agent that can: discover connected Coral sources at runtime query production systems through read-only SQL correlate evidence across code, deploys, errors, alerts, chats, and runbooks stream every query and row count into an inspectable UI run the same investigation workflow from a CLI when you want a scriptable path generate an incident report only when the evidence supports one stay focused with policy layers instead of a giant prompt blob In one sentence: ReefWatch is a Coral-powered investigation workspace that lets an agent discover connected tools at runtime, query them with read-only SQL, stream the evidence trail, and generate an incident report only when the facts actually support one. Why Coral B