标签：#aiagentoperations

The evals for my agent skills scored 0% for as long as I had records. Not low. Not noisy. Exactly zero, every skill, every run. And I believed it. For months I thought my skills were bad, because the number said so and the number never wavered. Then one night I actually read the harness. It had fallen back to the wrong auth token. Every call it made came back 401, and it quietly graded each one a failure. The skills never got a chance to fail on their own. I was not measuring them at all. I was reading a broken thermometer. Real weakness is jagged Here is what took me too long to see. When a system is genuinely bad, it scores 40% one week and 60% the next. It passes the easy cases and trips over the hard ones. It has good days. Incompetence has texture, because an incompetent system is still in contact with the world, and the world varies. A flat number has no texture. A flat number means the measurement stopped touching the thing being measured somewhere upstream, and what you are reading is the instrument's resting state. Doctors know this. A heart monitor drawing a perfectly straight line does not mean the patient is calm. Only a broken thermometer writes the same number every time. Key insight: A performance number with no variance is a reading of the instrument, not of the thing being measured. The same bug in three industries I run systems in advertising, in healthcare billing, and in agent operations, and the same shape shows up in all of them. In advertising I found a dashboard figure that had been hardcoded for two years. Nobody questioned it, because it looked right, and it looked right because it never moved. In agent operations, an account-rotation bug in one of my pipelines overwrote every real error with the same generic message, "no active accounts," so for a while every distinct failure in that system looked identical. And in the denial-assessment engine I run for a medical-billing operation, an agreement metric came back at 44.7%, alarmingly low, un

There is a note in my runbooks that says the Enter key works. That is almost the whole note. Press Enter and the prompt submits. I wrote it in bold, with sources and a date, because one of my AI agents once fixed the Enter key. The fix was the bug. The Fix Was the Bug I run a fleet of Claude Code agents in tmux panes. An orchestrator script types a prompt into a pane and presses Enter for it. One day an agent decided the submissions were not going through, and it knew the fix: send a backslash before the Enter. That sounds plausible if you have ever fought a terminal. It is also exactly backwards. In Claude Code, backslash plus Enter inserts a newline. It is the multiline key. So the fix turned every prompt into a draft that quietly grew longer and never sent. The panes looked busy. Nothing was happening. The submissions had been fine all along. The agent just had not waited for the reply. What broke: An agent changed Enter to backslash-Enter to repair prompt submission. Backslash-Enter creates a newline, so the repair silently stopped every prompt from sending. The system it fixed had never been broken. There is a second note in the same genre. My monitoring dashboard answers its health check with a 302 redirect to the login page. That is what healthy looks like there. But a 302 looks like trouble if you are hunting for outages, and agents kept trying to repair it. So the runbook now says, more or less: the redirect is normal, do not fix the healthy system. Human teams do not write notes like these. Nobody pins "the Enter key works" to the wall of an office. A New Hire Every Session Why did I have to? Because a human hits a dead end once. The wince is the documentation. Whoever broke a build by fixing the Enter key would remember it for years, tell the story at lunch, and the whole team would absorb it without anyone writing a word. An agent has none of that. It has no episodic memory. Every fresh context window is a new hire: smart, fast, and seeing your system fo