标签：#incident

The Post-Mortem Nobody Learns From I've sat through hundreds of post-mortems. Most follow the same pattern: something breaks, someone writes a Google Doc, we have a meeting, we list action items, nobody follows up, the same thing happens again in 3 months. Here's how to break the cycle. The Blameless Culture Trap "Blameless" doesn't mean "actionless." The biggest failure mode I see is teams that use blameless culture as an excuse to avoid accountability. Blameless means: we don't punish the person who pushed the bad deploy. Blameless does NOT mean: nobody is responsible for fixing the systemic issue. My Post-Mortem Template # Incident: [SERVICE] [SYMPTOM] on [DATE] ## Impact - Duration: X minutes - Users affected: N - Revenue impact: $X - SLO budget consumed: X% ## Timeline (UTC) - HH:MM - First alert fired - HH:MM - On-call acknowledged - HH:MM - Root cause identified - HH:MM - Fix deployed - HH:MM - Service recovered - HH:MM - All-clear declared ## Root Cause [2-3 sentences. Technical but readable.] ## Contributing Factors 1. [Factor that made the incident possible] 2. [Factor that made detection slow] 3. [Factor that made resolution slow] ## What Went Well - [Something that worked] - [Something that helped] ## What Went Wrong - [Process failure] - [Technical gap] ## Action Items | Action | Owner | Priority | Due Date | Status | |--------|-------|----------|----------|--------| | ... | ... | P1/P2/P3 | ... | Open | ## Lessons Learned [1-2 paragraphs of genuine insight] The Action Item Problem Action items from post-mortems have a 30% completion rate industry-wide. That's terrible. Here's why: Too many items (I've seen post-mortems with 15 action items) No clear ownership No deadline No follow-up mechanism Competing with feature work The Fix: Three Rules Rule 1: Maximum 3 action items per post-mortem. If you can't narrow it to 3, you haven't identified the real problems. Rule 2: Every action item gets a JIRA ticket linked to the next sprint. Not "someday." Not "bac

You forwarded the phishing email to the security channel about ninety seconds too late. The laptop is already cooperating with someone else. Your personal access token, the one you minted "just for that one script", is on its way to whatever Discord pays for stolen tokens this week. Now what? For users on GitHub Enterprise, what was previously a clickthrough checklist you complete while your hands shake is now one button. On June 24 the GitHub Changelog announced a self-service credential revocation flow under Settings, Credentials. From that view a user can see counts of every credential they have generated or authorized through SSO, then revoke or delete all of them in a single action. Personal access tokens, SSH keys, OAuth tokens, SSO authorizations: gone together. What actually shipped Containment used to be a manual scavenger hunt. PATs sat under Developer Settings. SSH keys lived one tab over. OAuth apps you forgot you authorized two years ago hid behind a different submenu. SSO was its own world. In practice that meant during an incident you forgot something, and the something you forgot was the credential the attacker actually wanted. The new view collapses that surface onto one screen. Counts on one side, a revoke-or-delete-everything action on the other. Whoever wrote it had clearly pictured the 3am screenshot: a user who has just been told to "rotate everything" and has no idea where "everything" lives. GitHub frames this as a complement to an earlier enterprise-owner capability that lets admins with the "Manage enterprise credentials" permission bulk-revoke across one user or many. So there are now two pairs of hands on the kill switch: the user, and the org. (Whichever one notices first.) Why a pipeline owner should care Because users are the trust boundary you keep pretending is somebody else's problem. A leaked PAT in a CI pipeline is rarely a CI bug. It is a human who pasted the token into a script, then a laptop, then a sync folder, then a backup,

Sean Klein discusses why "human error" is a dangerous myth in complex systems. Sharing the inside story of Azure’s 2023 global WAN outage, he explains how modern incident analysis looks past the "Five Whys" to uncover systemic issues. Learn how engineering leaders can move away from blame, improve Standard Operating Procedures, and design resilient systems that actively protect their engineers. By Sean Klein

Three years ago we had our first real outage. Six hours of downtime. Thousands of angry users. Multiple executives on the call. Here's what we did right, what we did wrong, and what we'd do differently. What we did right 1. Communicated immediately. The moment we knew we had a problem, we updated the status page and emailed our biggest customers personally. Not when we had answers. When we had a question. 2. Had a single incident commander. One person making calls. Not a committee. When the CEO tried to direct technical work, the IC politely rerouted and told her where her help was actually needed (talking to customers). 3. Took care of our people. During hour 4, I ordered food. During hour 5, I forced the primary engineer off the call for 20 minutes to walk outside. Long incidents destroy people. You have to feed them and force them to rest. 4. Wrote it down as we went. We had a shared doc with a live timeline. When the post-mortem came, we had every decision captured. What we did wrong 1. Tried to fix the root cause during the incident. For the first 2 hours, we were digging into why the database was struggling. We should have been mitigating (rolling back) first. 2. Let too many people 'help.' By hour 3, we had 12 engineers in the call. Half of them were useless. The IC should have kicked people out sooner. 3. Gave optimistic estimates. 'We'll be back in 30 minutes.' We were not back in 30 minutes. That miscommunication was worse than saying 'unknown.' 4. Didn't prepare the executive communication. The CEO had to answer customer questions in real time with no script. We should have drafted talking points for her after hour 1. What we'd do differently Mitigate first, investigate second. Always. Cap the number of active engineers at 4 during an incident. Others go on standby. Default to 'unknown' for estimates. Only give a number when we're sure. Assign someone explicitly to 'executive liaison.' Their job is to keep the C-suite informed without interrupting the tec

Kyle Lexmond explains how to handle the high-pressure environment of severe production outages. He discusses the critical distinction between mitigation and root-cause resolution, sharing personal experiences from harrowing incident rooms. He shares valuable operational strategies on overcoming cognitive overload, establishing blameless cultures, and optimizing systems for faster recovery. By Kyle Lexmond

A lot of Linux incident response starts with a login question, not a malware sample. Someone sees a spike of failed SSH attempts. A root login appears in the wrong time window. A service account logs in from an address nobody recognizes. A helpdesk ticket says "the server looks weird" and the only concrete clue is a username or IP address. At that point, the useful question is not "is this host compromised?" It is more boring and more important: Did anyone actually authenticate? Which account was involved? Was it password, key, sudo, su, or a scheduled task? Was the same IP seen in web logs, current sockets, process context, or command history? Did persistence, services, packages, or recent files change near the same time? Can another responder review exactly what evidence was collected? That last point matters. If you let an AI assistant freely run shell commands during the first pass, you can get speed, but you also create a new risk: the model may over-collect, mutate the host, or produce a confident answer that nobody can audit later. For a login anomaly, I prefer a read-only evidence loop. A practical first pass Start with the narrow clue if you have one. If the alert names a user: oi login --user root -s 7d If the alert names an IP address: oi login --ip 203.0.113.44 -s 7d If the alert is vague, start wider: oi login -s 7d oi scan -s 7d The goal of the first pass is not to prove every detail. The goal is to build a timeline that a human responder can challenge. For a suspicious SSH login, I want the initial report to answer five things. 1. Authentication pattern Look for the difference between noise and access. A server can receive thousands of failed SSH attempts from the internet. That is useful background, but it is not the same as a successful session. The first split should be: failed attempts only successful login after many failures accepted key from an unusual source login by an account that normally should not be interactive root login where root SSH