标签：#sre

Using threat modelling to make system dependability observable, testable, and actionable Executive summary Security and Reliability address system degradation. Security addresses degradation from intentional actions, such as denial-of-service attacks, while reliability addresses degradation from failure, load, dependency behaviour, operational change, or complexity. The underlying engineering question is the same: which critical system property can degrade, how would users experience that degradation, how would we detect it, and what controls would prevent, contain, or recover from it? This document proposes a practical way to bridge the two disciplines: anchor analysis on Critical User Journeys, express expected behaviour through SLOs and SLIs, use RAMSS to ensure coverage across dependability dimensions, and adapt PASTA-style threat modelling to reliability scenarios. The goal is not to merge Security and Reliability into one generic practice. The goal is to reuse the strongest habits of each discipline: security's adversarial modelling and reliability's production-oriented measurement, validation, and recovery loops. The most useful outcome is a shared model of degradation scenarios. A degradation scenario links a critical user journey to a concrete reliability or security threat, the system weakness that makes it possible, the signal that would detect it, the objective it would violate, and the mitigation or experiment that would validate the control. This makes risk easier to discuss with engineering teams because it connects abstract concerns to user impact, SLO burn, business loss, and testable remediation. 1. The problem: two disciplines, one degradation model After working in both Reliability and Security, I found that the two domains share much in common: both focus on objectives, weaknesses, control effectiveness, incident response, prioritisation, and residual risk, but often use different rituals, terminology, metrics, and boundaries. This separation ca

The test passed. The runbook completed. Infrastructure came back online inside the RTO window. None of that means the organization can recover from an actual disaster. Disaster recovery testing is designed to succeed. Clean environments, pre-staged dependencies, known failure modes, available staff — each design decision is operationally reasonable. Collectively they remove the conditions that make real recovery hard. What the test validates is test completion, not recovery capability. The Test Is Designed to Pass Every design decision in a standard DR test tilts toward a successful outcome. The test window is pre-announced, so the right engineers are available. The scope is pre-defined, so unexpected systems don't surface mid-exercise. The environment is either isolated or pre-staged, so competing failures don't complicate the recovery sequence. The data state is known and clean, so integrity issues don't slow the restore. The declaration point is assumed, so nobody has to make an ambiguous call under pressure. A test designed to remove the variables that make recovery hard cannot produce evidence about what happens when those variables are present. What Disaster Recovery Testing Actually Excludes Declaration threshold. In a DR test, recovery starts at a pre-agreed time. In a real incident, recovery starts when someone decides the situation has crossed the threshold for declaration — a decision that is rarely clean and routinely delayed 45 minutes to several hours. That delay is inside the real outage window and outside the test clock. Dependency assumptions. DR tests run against known, pre-cleared dependencies. Real incidents surface undocumented dependencies that were never in scope — a configuration service that hasn't been touched in two years, an authentication endpoint that wasn't in the architecture diagram. Data state. Test environments use clean or pre-staged data. Real recovery requires handling whatever state the data was in at the moment of failure — pa

The DORA research programme is the most rigorous empirical study of software delivery performance ever conducted. Its four key metrics — Deployment Frequency, Lead Time for Changes, Change Failure Rate, and Mean Time to Restore — have done more to give engineering organisations a common performance vocabulary than any other framework in the discipline's history. If you work in software and you have not read the State of DevOps Report, stop and read it before finishing this paragraph. Now: the DORA Four were derived primarily from organisations with cloud-native architectures, on-demand deployment infrastructure, and relatively unconstrained ability to release software when it is ready. The research cohort skews toward technology companies that have already made the cultural and architectural investments that make high-frequency, low-risk deployment possible. This is not a criticism of the research. It is an observation about its generalisability — and it has a specific consequence for practitioners who work in regulated enterprises: banks, healthcare systems, utilities, insurance carriers, government agencies. In these environments, the DORA Four are necessary but structurally insufficient. They measure the delivery pipeline accurately. They do not measure the operational sustainability of the team running that pipeline — and in regulated enterprises, operational sustainability is where SRE programmes go to die quietly, years before anyone realises the damage is permanent. This post proposes a fifth metric. Not to replace the DORA Four, but to complete them — to close the measurement gap that leaves regulated enterprise SRE teams flying blind on the dimension that most reliably predicts long-term programme failure. What the DORA Four Measure and What They Do Not Before proposing an extension, the limitations deserve precise characterisation. Imprecise criticism of a well-validated framework is noise. The limitations described here are structural — arising from the d

Production incidents almost never break in one place. The alert fires in one tool. The broken deploy is in Netlify. The suspicious change is in GitHub. The stack trace is in Sentry. The human context is in Slack. The runbook is in Notion. The "is this actually paging someone?" answer is in PagerDuty. A normal chatbot can sound helpful in that situation. It can say things like "you should check your recent deployments" and "look for related errors in Sentry." But that is not triage. That is a polished to-do list. I wanted something more useful: an agent that could go get the evidence, connect the dots across sources, show its work, and give an operator-grade answer grounded in real system data. The design constraint from the start was simple: no evidence, no answer. That became ReefWatch , a Coral-powered production triage agent built to investigate instead of improvise. It discovers the tools connected to a workspace at runtime, queries them as evidence, correlates records across systems, and produces a compact answer only when the facts support one. Coral became the backbone because it turns the messiest part of agent tooling into something the model can actually reason about: SQL . What This Guide Builds By the end of this route, you will have a blueprint for an agent that can: discover connected Coral sources at runtime query production systems through read-only SQL correlate evidence across code, deploys, errors, alerts, chats, and runbooks stream every query and row count into an inspectable UI run the same investigation workflow from a CLI when you want a scriptable path generate an incident report only when the evidence supports one stay focused with policy layers instead of a giant prompt blob In one sentence: ReefWatch is a Coral-powered investigation workspace that lets an agent discover connected tools at runtime, query them with read-only SQL, stream the evidence trail, and generate an incident report only when the facts actually support one. Why Coral B