标签：#sre

Three years ago we had our first real outage. Six hours of downtime. Thousands of angry users. Multiple executives on the call. Here's what we did right, what we did wrong, and what we'd do differently. What we did right 1. Communicated immediately. The moment we knew we had a problem, we updated the status page and emailed our biggest customers personally. Not when we had answers. When we had a question. 2. Had a single incident commander. One person making calls. Not a committee. When the CEO tried to direct technical work, the IC politely rerouted and told her where her help was actually needed (talking to customers). 3. Took care of our people. During hour 4, I ordered food. During hour 5, I forced the primary engineer off the call for 20 minutes to walk outside. Long incidents destroy people. You have to feed them and force them to rest. 4. Wrote it down as we went. We had a shared doc with a live timeline. When the post-mortem came, we had every decision captured. What we did wrong 1. Tried to fix the root cause during the incident. For the first 2 hours, we were digging into why the database was struggling. We should have been mitigating (rolling back) first. 2. Let too many people 'help.' By hour 3, we had 12 engineers in the call. Half of them were useless. The IC should have kicked people out sooner. 3. Gave optimistic estimates. 'We'll be back in 30 minutes.' We were not back in 30 minutes. That miscommunication was worse than saying 'unknown.' 4. Didn't prepare the executive communication. The CEO had to answer customer questions in real time with no script. We should have drafted talking points for her after hour 1. What we'd do differently Mitigate first, investigate second. Always. Cap the number of active engineers at 4 during an incident. Others go on standby. Default to 'unknown' for estimates. Only give a number when we're sure. Assign someone explicitly to 'executive liaison.' Their job is to keep the C-suite informed without interrupting the tec

Production is down. Slack is on fire. Your phone is ringing. You've seen this exact error before — ConnectionResetError: [Errno 104] cascading through your FastAPI worker pool — but you can't remember exactly which Redis configuration tweak fixed it last time, who applied it, or how long the incident lasted. You're starting from zero again. Twenty minutes of context-building before you even touch a fix. I got tired of that feeling. So I built an AI agent that never forgets. The Problem With Generic AI in Production When production breaks, most engineers reach for their LLM of choice and paste in the stack trace. And the response is almost always the same: a competent, thoughtful, completely useless answer. The model has no idea that your team already tried increasing max_connections six weeks ago and it made things worse. It doesn't know that your infrastructure runs on a specific internal Kubernetes setup that changes how standard fixes apply. It gives you textbook advice for textbook problems, and your problems are never textbook. This is what I started calling the Round 1 problem. Round 1 — generic response: Error: ConnectionResetError: [Errno 104] Connection reset by peer Stack: redis.exceptions.ConnectionError in worker pool The agent responds with something like: "This typically indicates your Redis connection pool is exhausted. Try increasing max_connections in your Redis client config, add retry logic with exponential backoff, and check network stability between your app and Redis instance." Technically correct. Practically useless if you've already tried all three. The agent is reasoning from general knowledge, not from your specific production history. It has no memory of your past incidents. Every error feels like the first error. What I Built: Code Memory's Incident Agent Code Memory is a developer workspace I built in Next.js with a three-pane interface — a file explorer, a code viewer with syntax highlighting, and a real-time AI fix panel. But the core

Using threat modelling to make system dependability observable, testable, and actionable Executive summary Security and Reliability address system degradation. Security addresses degradation from intentional actions, such as denial-of-service attacks, while reliability addresses degradation from failure, load, dependency behaviour, operational change, or complexity. The underlying engineering question is the same: which critical system property can degrade, how would users experience that degradation, how would we detect it, and what controls would prevent, contain, or recover from it? This document proposes a practical way to bridge the two disciplines: anchor analysis on Critical User Journeys, express expected behaviour through SLOs and SLIs, use RAMSS to ensure coverage across dependability dimensions, and adapt PASTA-style threat modelling to reliability scenarios. The goal is not to merge Security and Reliability into one generic practice. The goal is to reuse the strongest habits of each discipline: security's adversarial modelling and reliability's production-oriented measurement, validation, and recovery loops. The most useful outcome is a shared model of degradation scenarios. A degradation scenario links a critical user journey to a concrete reliability or security threat, the system weakness that makes it possible, the signal that would detect it, the objective it would violate, and the mitigation or experiment that would validate the control. This makes risk easier to discuss with engineering teams because it connects abstract concerns to user impact, SLO burn, business loss, and testable remediation. 1. The problem: two disciplines, one degradation model After working in both Reliability and Security, I found that the two domains share much in common: both focus on objectives, weaknesses, control effectiveness, incident response, prioritisation, and residual risk, but often use different rituals, terminology, metrics, and boundaries. This separation ca

The test passed. The runbook completed. Infrastructure came back online inside the RTO window. None of that means the organization can recover from an actual disaster. Disaster recovery testing is designed to succeed. Clean environments, pre-staged dependencies, known failure modes, available staff — each design decision is operationally reasonable. Collectively they remove the conditions that make real recovery hard. What the test validates is test completion, not recovery capability. The Test Is Designed to Pass Every design decision in a standard DR test tilts toward a successful outcome. The test window is pre-announced, so the right engineers are available. The scope is pre-defined, so unexpected systems don't surface mid-exercise. The environment is either isolated or pre-staged, so competing failures don't complicate the recovery sequence. The data state is known and clean, so integrity issues don't slow the restore. The declaration point is assumed, so nobody has to make an ambiguous call under pressure. A test designed to remove the variables that make recovery hard cannot produce evidence about what happens when those variables are present. What Disaster Recovery Testing Actually Excludes Declaration threshold. In a DR test, recovery starts at a pre-agreed time. In a real incident, recovery starts when someone decides the situation has crossed the threshold for declaration — a decision that is rarely clean and routinely delayed 45 minutes to several hours. That delay is inside the real outage window and outside the test clock. Dependency assumptions. DR tests run against known, pre-cleared dependencies. Real incidents surface undocumented dependencies that were never in scope — a configuration service that hasn't been touched in two years, an authentication endpoint that wasn't in the architecture diagram. Data state. Test environments use clean or pre-staged data. Real recovery requires handling whatever state the data was in at the moment of failure — pa

The DORA research programme is the most rigorous empirical study of software delivery performance ever conducted. Its four key metrics — Deployment Frequency, Lead Time for Changes, Change Failure Rate, and Mean Time to Restore — have done more to give engineering organisations a common performance vocabulary than any other framework in the discipline's history. If you work in software and you have not read the State of DevOps Report, stop and read it before finishing this paragraph. Now: the DORA Four were derived primarily from organisations with cloud-native architectures, on-demand deployment infrastructure, and relatively unconstrained ability to release software when it is ready. The research cohort skews toward technology companies that have already made the cultural and architectural investments that make high-frequency, low-risk deployment possible. This is not a criticism of the research. It is an observation about its generalisability — and it has a specific consequence for practitioners who work in regulated enterprises: banks, healthcare systems, utilities, insurance carriers, government agencies. In these environments, the DORA Four are necessary but structurally insufficient. They measure the delivery pipeline accurately. They do not measure the operational sustainability of the team running that pipeline — and in regulated enterprises, operational sustainability is where SRE programmes go to die quietly, years before anyone realises the damage is permanent. This post proposes a fifth metric. Not to replace the DORA Four, but to complete them — to close the measurement gap that leaves regulated enterprise SRE teams flying blind on the dimension that most reliably predicts long-term programme failure. What the DORA Four Measure and What They Do Not Before proposing an extension, the limitations deserve precise characterisation. Imprecise criticism of a well-validated framework is noise. The limitations described here are structural — arising from the d

Production incidents almost never break in one place. The alert fires in one tool. The broken deploy is in Netlify. The suspicious change is in GitHub. The stack trace is in Sentry. The human context is in Slack. The runbook is in Notion. The "is this actually paging someone?" answer is in PagerDuty. A normal chatbot can sound helpful in that situation. It can say things like "you should check your recent deployments" and "look for related errors in Sentry." But that is not triage. That is a polished to-do list. I wanted something more useful: an agent that could go get the evidence, connect the dots across sources, show its work, and give an operator-grade answer grounded in real system data. The design constraint from the start was simple: no evidence, no answer. That became ReefWatch , a Coral-powered production triage agent built to investigate instead of improvise. It discovers the tools connected to a workspace at runtime, queries them as evidence, correlates records across systems, and produces a compact answer only when the facts support one. Coral became the backbone because it turns the messiest part of agent tooling into something the model can actually reason about: SQL . What This Guide Builds By the end of this route, you will have a blueprint for an agent that can: discover connected Coral sources at runtime query production systems through read-only SQL correlate evidence across code, deploys, errors, alerts, chats, and runbooks stream every query and row count into an inspectable UI run the same investigation workflow from a CLI when you want a scriptable path generate an incident report only when the evidence supports one stay focused with policy layers instead of a giant prompt blob In one sentence: ReefWatch is a Coral-powered investigation workspace that lets an agent discover connected tools at runtime, query them with read-only SQL, stream the evidence trail, and generate an incident report only when the facts actually support one. Why Coral B