标签：#runbooks

There is a note in my runbooks that says the Enter key works. That is almost the whole note. Press Enter and the prompt submits. I wrote it in bold, with sources and a date, because one of my AI agents once fixed the Enter key. The fix was the bug. The Fix Was the Bug I run a fleet of Claude Code agents in tmux panes. An orchestrator script types a prompt into a pane and presses Enter for it. One day an agent decided the submissions were not going through, and it knew the fix: send a backslash before the Enter. That sounds plausible if you have ever fought a terminal. It is also exactly backwards. In Claude Code, backslash plus Enter inserts a newline. It is the multiline key. So the fix turned every prompt into a draft that quietly grew longer and never sent. The panes looked busy. Nothing was happening. The submissions had been fine all along. The agent just had not waited for the reply. What broke: An agent changed Enter to backslash-Enter to repair prompt submission. Backslash-Enter creates a newline, so the repair silently stopped every prompt from sending. The system it fixed had never been broken. There is a second note in the same genre. My monitoring dashboard answers its health check with a 302 redirect to the login page. That is what healthy looks like there. But a 302 looks like trouble if you are hunting for outages, and agents kept trying to repair it. So the runbook now says, more or less: the redirect is normal, do not fix the healthy system. Human teams do not write notes like these. Nobody pins "the Enter key works" to the wall of an office. A New Hire Every Session Why did I have to? Because a human hits a dead end once. The wince is the documentation. Whoever broke a build by fixing the Enter key would remember it for years, tell the story at lunch, and the whole team would absorb it without anyone writing a word. An agent has none of that. It has no episodic memory. Every fresh context window is a new hire: smart, fast, and seeing your system fo

Your runbooks are out of date. I don't know your team, but I'd bet money on it. Most teams write runbooks once, in a panic after an outage, and then never touch them again until the next outage proves them wrong. How runbooks rot Steps reference deprecated tools. The grafana dashboard moved, the CLI command was renamed, the bastion host got retired. Nobody updated the runbook because nobody re-ran it during the calm months. The team that owned the system left. Three of the five engineers who wrote it are gone. The remaining two haven't actually run the runbook in 18 months because the outage type it covers stopped happening. Half-truths from the start. The original author skipped the obvious steps (because they were obvious to them) and the new on-call engineer can't reproduce the recovery. The result: at 2 AM during the actual incident, the runbook sends you down a dead end. Now you're improvising under stress. What working runbooks have in common I've seen exactly two patterns work: 1. Runbooks live with the code. Put them in the repo of the system they document. When the code changes, the runbook PR is part of the same review. Out-of-repo wikis die first because the cognitive distance is too great. 2. Runbooks are reviewed by people who weren't there. Have a junior engineer run through the runbook quarterly on a non-incident day. Every place they get stuck is a real bug in the document. Fix it then. The author will be too close to see the gaps. The three sections that matter A useful runbook has exactly three sections: Symptom : how do I know this is the right runbook? Concrete signals, with example screenshots if visual. First 5 minutes : what to do RIGHT NOW to stop the bleed. Not the root cause investigation, just the triage actions. Investigation : where to look, what queries to run, what to escalate. That's it. Anything else (architecture diagrams, history, philosophy) goes in a separate doc that the runbook links to. The cultural part The hardest part isn't