标签：#incident

Short answer: make the event identity durable, deduplicate at the consumer boundary, and resume from a server-issued cursor; a client-side set alone cannot make a marketplace chat room survive reconnects or an incident-response burst. The constraint is trust. A browser reconnects after a laptop sleeps, a mobile radio changes networks, or a tab is restored from the back-forward cache. It may replay its last request, lose an acknowledgement, or present an event twice. In an incident response dashboard, the same mechanics become dangerous at scale: an alert that appears twice can page two people, while a missing alert can hide the incident. I design the storage boundary first, because a pretty WebSocket demo does not answer either question. Start with an event identity that can outlive a connection Every published event needs an immutable identity scoped to the stream, not to a socket. For a marketplace chat room, I use (room_id, sequence) as the primary key and keep a globally unique event_id for tracing. The sequence is allocated by the room writer, so two reconnecting clients can compare progress without trusting wall-clock timestamps. The payload is deliberately boring. It includes the room, sequence, event ID, type, and data. A client can verify that an event belongs to the room it requested; it cannot mint a higher sequence or widen its token scope. That last rule matters more than transport choice. from dataclasses import dataclass from typing import Any @dataclass ( frozen = True ) class ChatEvent : room_id : str sequence : int event_id : str event_type : str data : dict [ str , Any ] def identity ( event : ChatEvent ) -> tuple [ str , int ]: """ The room sequence is the replay-safe identity. """ return event . room_id , event . sequence Do not use a payload hash as the only key. Two legitimate messages can have identical text, and a producer retry can produce different JSON ordering. Persist the identity and the payload together, with a uniqueness constraint,

In this blog post, we will see how I use AI to speed up incident investigation without letting it take over the decisions that need a human. It is 3:00 AM. Your phone starts making that familiar PagerDuty noise. You open the alert with half-open eyes. Error rates are climbing. Slack is already active. The incident commander wants an update. Depending on the severity, your director or CTO may also join the call. Every developer who goes on call will face this situation at some point. I have faced it a few times a year. The first time, panic is normal. You do not know where to start, which dashboard to open, or how to explain the issue while you are still investigating it. Experience teaches you how to stay prepared. AI can reduce some of that early morning panic too. It will not bring the panic factor down to zero, and it should not replace the engineer. But it can remove the first few minutes of searching, tab switching, and collecting context. The goal is simple: move from panic mode to confidence mode faster. I Started With a Prompt in My Notes I started with something small, before custom skills became common in coding harnesses. I kept one incident prompt at the top of my notes folder. I also pinned it in my clipboard manager. When an alert arrived, I filled in the blanks and launched the investigation: I received this alert: <PagerDuty or Slack alert link>. Context: - Service: <service name> - Environment: <environment> - Region: <region> - Error or symptom: <error details> - Investigation window: past <n> hours - Runbook: <runbook link> Start investigating the issue. 1. Analyze the relevant Splunk logs and dashboards. 2. Check recent deployments, configuration changes, and feature-flag changes. 3. Check upstream and downstream dependencies. 4. Check cloud-provider status pages and internal maintenance announcements. 5. Search PagerDuty history and incident records for similar symptoms. 6. Use parallel agents for independent investigation tracks where useful. R

When a vendor is breached, don't wait for their update page: figure out fast what that vendor can see or touch inside your business, and act on that answer before you have all the facts. Vendor breaches are a recurring headline: a phone system, messaging platform, or file-sharing tool used by thousands of businesses discloses an incident, and every customer of that vendor suddenly has to ask what it means for them, usually with almost no visibility into the vendor's internal investigation. The gap between "something happened to our vendor" and "here's what we do about it" is where most companies lose time they can't get back. Quick answers What's the first thing to do when a vendor is breached? Confirm exactly what that vendor can access in your systems — before you do anything else. Should I rotate credentials before the vendor confirms exposure? Yes, for anything plausibly exposed — even though it may briefly disrupt a live integration. How often should I re-check a slow-to-disclose vendor? It depends on how much access they hold — high-privilege vendors need a 48–72 hour check-in, not a week. Do I need a written plan before an incident happens? Yes — a one-page plan per privileged vendor, reviewed at least yearly. Why vetting a vendor once isn't enough Most businesses do vendor due diligence at signup: a security questionnaire, maybe a look at their trust page, a checkbox before the contract is signed. That's reasonable, but it only answers the question that matters on day one: is this vendor safe to start using? It doesn't answer the question that matters every day after: if this vendor gets breached, what happens to us, and what do we do? Vetting is a point-in-time filter. A breach is an ongoing event with a timeline, and your response needs its own plan, separate from the vetting checklist you ran before signing. The fix is simple: treat vendor risk as a lifecycle, not a gate. Vet before you sign, revisit access and exposure on a regular cadence, and have a re

One of the most common assumptions in engineering leadership is that a rising number of reported incidents signals declining system reliability. However, a recent article from Great Circle argues that the opposite is often true: an increase in incident counts may actually indicate that an organization's incident management culture is improving. By Craig Risi

Artificial intelligence is rapidly changing how engineering teams respond to production incidents, offering the ability to summarize incident channels, analyze unfamiliar code, suggest remediation steps, generate pull requests, and increasingly assist with diagnosis. By Craig Risi

A GitHub product launch is being held back by the CI/CD platform underneath it. On August 6 GitHub filed a Changelog entry announcing that Kimi K3, an open-weight model, is now generally available in GitHub Copilot, then added an editor's note the same day: the rollout is temporarily paused while GitHub mitigates an incident with GitHub Actions. What the entry says, and what it does not Per the note, GitHub will resume the rollout as soon as possible and update the docs with Kimi K3 pricing: $3 per 1M input tokens, $15 per 1M output tokens, and $0.30 per 1M cached input tokens. That is the extent of the disclosure. The Changelog does not describe the Actions incident, does not put a scale on its blast radius, and does not commit to a resume time. It also does not explain how a Copilot model rollout ends up gated on Actions in the first place; a reader can infer that some provisioning or feature-flag step rides the same platform, but the entry does not say so. Availability is qualified in a way worth flagging. Kimi K3 is GA on paper, but the switch that actually turns it on for end users is paused. The operational read There is a coupling here worth naming plainly. GitHub sells Actions as CI/CD for everyone else, and it also uses Actions to ship its own products. When Actions has a bad day, GitHub's launch calendar has a bad day too, in public. That is not a scandal; it is what dogfooding looks like when the changelog is a live document. It is also a data point for any team running a rollout on top of a hosted CI platform: your feature-flag flip is downstream of somebody else's incident queue, and you inherit that queue's MTTR whether or not it is on your status page. Two follow-ups are worth watching. First, whether the resumed rollout entry names the incident and its cause, or whether it stays silent. Second, whether Kimi K3's published pricing survives the pause unchanged. Until then, the GA label is doing work the runtime cannot back up.

Security disclosures highlighted vulnerabilities in AI evaluations of autonomous cyber capabilities. Notably, OpenAI’s models escaped sandbox isolation, breaching Hugging Face’s systems. The incident involved a multi-stage attack, revealing flaws in evaluation containment and prompting calls for stricter infrastructure controls and local incident response tools. By Olimpiu Pop

The Post-Mortem Nobody Learns From I've sat through hundreds of post-mortems. Most follow the same pattern: something breaks, someone writes a Google Doc, we have a meeting, we list action items, nobody follows up, the same thing happens again in 3 months. Here's how to break the cycle. The Blameless Culture Trap "Blameless" doesn't mean "actionless." The biggest failure mode I see is teams that use blameless culture as an excuse to avoid accountability. Blameless means: we don't punish the person who pushed the bad deploy. Blameless does NOT mean: nobody is responsible for fixing the systemic issue. My Post-Mortem Template # Incident: [SERVICE] [SYMPTOM] on [DATE] ## Impact - Duration: X minutes - Users affected: N - Revenue impact: $X - SLO budget consumed: X% ## Timeline (UTC) - HH:MM - First alert fired - HH:MM - On-call acknowledged - HH:MM - Root cause identified - HH:MM - Fix deployed - HH:MM - Service recovered - HH:MM - All-clear declared ## Root Cause [2-3 sentences. Technical but readable.] ## Contributing Factors 1. [Factor that made the incident possible] 2. [Factor that made detection slow] 3. [Factor that made resolution slow] ## What Went Well - [Something that worked] - [Something that helped] ## What Went Wrong - [Process failure] - [Technical gap] ## Action Items | Action | Owner | Priority | Due Date | Status | |--------|-------|----------|----------|--------| | ... | ... | P1/P2/P3 | ... | Open | ## Lessons Learned [1-2 paragraphs of genuine insight] The Action Item Problem Action items from post-mortems have a 30% completion rate industry-wide. That's terrible. Here's why: Too many items (I've seen post-mortems with 15 action items) No clear ownership No deadline No follow-up mechanism Competing with feature work The Fix: Three Rules Rule 1: Maximum 3 action items per post-mortem. If you can't narrow it to 3, you haven't identified the real problems. Rule 2: Every action item gets a JIRA ticket linked to the next sprint. Not "someday." Not "bac

You forwarded the phishing email to the security channel about ninety seconds too late. The laptop is already cooperating with someone else. Your personal access token, the one you minted "just for that one script", is on its way to whatever Discord pays for stolen tokens this week. Now what? For users on GitHub Enterprise, what was previously a clickthrough checklist you complete while your hands shake is now one button. On June 24 the GitHub Changelog announced a self-service credential revocation flow under Settings, Credentials. From that view a user can see counts of every credential they have generated or authorized through SSO, then revoke or delete all of them in a single action. Personal access tokens, SSH keys, OAuth tokens, SSO authorizations: gone together. What actually shipped Containment used to be a manual scavenger hunt. PATs sat under Developer Settings. SSH keys lived one tab over. OAuth apps you forgot you authorized two years ago hid behind a different submenu. SSO was its own world. In practice that meant during an incident you forgot something, and the something you forgot was the credential the attacker actually wanted. The new view collapses that surface onto one screen. Counts on one side, a revoke-or-delete-everything action on the other. Whoever wrote it had clearly pictured the 3am screenshot: a user who has just been told to "rotate everything" and has no idea where "everything" lives. GitHub frames this as a complement to an earlier enterprise-owner capability that lets admins with the "Manage enterprise credentials" permission bulk-revoke across one user or many. So there are now two pairs of hands on the kill switch: the user, and the org. (Whichever one notices first.) Why a pipeline owner should care Because users are the trust boundary you keep pretending is somebody else's problem. A leaked PAT in a CI pipeline is rarely a CI bug. It is a human who pasted the token into a script, then a laptop, then a sync folder, then a backup,

Sean Klein discusses why "human error" is a dangerous myth in complex systems. Sharing the inside story of Azure’s 2023 global WAN outage, he explains how modern incident analysis looks past the "Five Whys" to uncover systemic issues. Learn how engineering leaders can move away from blame, improve Standard Operating Procedures, and design resilient systems that actively protect their engineers. By Sean Klein

Three years ago we had our first real outage. Six hours of downtime. Thousands of angry users. Multiple executives on the call. Here's what we did right, what we did wrong, and what we'd do differently. What we did right 1. Communicated immediately. The moment we knew we had a problem, we updated the status page and emailed our biggest customers personally. Not when we had answers. When we had a question. 2. Had a single incident commander. One person making calls. Not a committee. When the CEO tried to direct technical work, the IC politely rerouted and told her where her help was actually needed (talking to customers). 3. Took care of our people. During hour 4, I ordered food. During hour 5, I forced the primary engineer off the call for 20 minutes to walk outside. Long incidents destroy people. You have to feed them and force them to rest. 4. Wrote it down as we went. We had a shared doc with a live timeline. When the post-mortem came, we had every decision captured. What we did wrong 1. Tried to fix the root cause during the incident. For the first 2 hours, we were digging into why the database was struggling. We should have been mitigating (rolling back) first. 2. Let too many people 'help.' By hour 3, we had 12 engineers in the call. Half of them were useless. The IC should have kicked people out sooner. 3. Gave optimistic estimates. 'We'll be back in 30 minutes.' We were not back in 30 minutes. That miscommunication was worse than saying 'unknown.' 4. Didn't prepare the executive communication. The CEO had to answer customer questions in real time with no script. We should have drafted talking points for her after hour 1. What we'd do differently Mitigate first, investigate second. Always. Cap the number of active engineers at 4 during an incident. Others go on standby. Default to 'unknown' for estimates. Only give a number when we're sure. Assign someone explicitly to 'executive liaison.' Their job is to keep the C-suite informed without interrupting the tec

Kyle Lexmond explains how to handle the high-pressure environment of severe production outages. He discusses the critical distinction between mitigation and root-cause resolution, sharing personal experiences from harrowing incident rooms. He shares valuable operational strategies on overcoming cognitive overload, establishing blameless cultures, and optimizing systems for faster recovery. By Kyle Lexmond

A lot of Linux incident response starts with a login question, not a malware sample. Someone sees a spike of failed SSH attempts. A root login appears in the wrong time window. A service account logs in from an address nobody recognizes. A helpdesk ticket says "the server looks weird" and the only concrete clue is a username or IP address. At that point, the useful question is not "is this host compromised?" It is more boring and more important: Did anyone actually authenticate? Which account was involved? Was it password, key, sudo, su, or a scheduled task? Was the same IP seen in web logs, current sockets, process context, or command history? Did persistence, services, packages, or recent files change near the same time? Can another responder review exactly what evidence was collected? That last point matters. If you let an AI assistant freely run shell commands during the first pass, you can get speed, but you also create a new risk: the model may over-collect, mutate the host, or produce a confident answer that nobody can audit later. For a login anomaly, I prefer a read-only evidence loop. A practical first pass Start with the narrow clue if you have one. If the alert names a user: oi login --user root -s 7d If the alert names an IP address: oi login --ip 203.0.113.44 -s 7d If the alert is vague, start wider: oi login -s 7d oi scan -s 7d The goal of the first pass is not to prove every detail. The goal is to build a timeline that a human responder can challenge. For a suspicious SSH login, I want the initial report to answer five things. 1. Authentication pattern Look for the difference between noise and access. A server can receive thousands of failed SSH attempts from the internet. That is useful background, but it is not the same as a successful session. The first split should be: failed attempts only successful login after many failures accepted key from an unusual source login by an account that normally should not be interactive root login where root SSH