I've Spent Months Grading AI Agents' Code for a Living. Here's the Pattern Nobody's Talking About

Everyone's talking about agentic AI shipping production code. Nobody's talking about what happens when you actually sit down and grade thousands of lines of it against a rubric, line by line, for months. I have. And the failure pattern that shows up over and over isn't the one Twitter/X is arguing about. The job title that didn't exist two years ago "AI evaluator." "AI trainer." "Expert contributor to frontier model training data." None of these existed as job titles when I started my career. Now they're where a chunk of the most interesting engineering signal in the industry is actually happening — quietly, behind NDAs, far from the demo videos. Here's what the job actually is: agentic coding outputs land on your desk, and you grade them against a structured rubric — correctness, instruction adherence, quality, edge-case handling. You design adversarial prompts to find where the model's reasoning breaks. You decide which checks can be programmatic and deterministic, and which genuinely need a human who's shipped production systems to make the call. This is RL environment design and LLMOps in its rawest form, and it's a completely different skill from "prompt engineer" or "ML researcher." It's closer to being a QA lead for a junior engineer who never sleeps, never gets embarrassed, and will confidently ship the wrong answer with perfect syntax. The pattern: agents are great at code, bad at consequences Here's the uncomfortable part. The failure mode people are loudest about — hallucinated APIs, made-up library functions — is the easy failure mode. It's loud, it's obvious, and any decent test suite catches it in seconds. The failure mode that actually matters, the one that slips past a surface read and even past a naive test suite, looks like this: The code is syntactically perfect and semantically wrong about failure. It handles the happy path beautifully and quietly assumes the retry, the timeout, the partial write, the duplicate message never happens. It optimises