My Agent Found Real Improvements. The Statistics Still Killed the Promotion.

Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own Prompts · The Edit That Fixed 4 Tasks and Broke 1 · The Gate Is the Product · The Doctor Who Diagnosed Every Patient · 4 Models, 0 Promotable Edits In v0.1.0, an edit fixed 4 tasks and broke 1. Net +3 on 26 tasks. p=0.23. Gate rejected. The ceiling was clear: if you do not move enough tasks, the gate should say no. In v0.2.0, we expanded the A/B corpus to 40 tasks. We fixed the pipeline bugs. We added rejection context. We tested stronger models. The math got cleaner, not kinder. The ceiling shifted. It did not disappear. The v0.1.0 Result: 26 Tasks, 5 Movable, p=0.23 The edit was real. It fixed 4 tasks and broke 1: Task Prompt A Prompt B Expected Change classify-015 technical urgent urgent FIXED classify-023 security urgent, security urgent, security FIXED classify-024 feature feature, billing feature, billing FIXED classify-029 feature other other FIXED classify-014 technical feature technical BROKEN Net: +3. Mean delta = 0.115. p=0.23. The permutation test computes this by shuffling task labels 1,000 times and counting how often random chance produces a delta ≥ 0.115. 23% of the time — above the 5% threshold. The sign-test floor with 5 discordant pairs out of 26 is ~0.031 one-sided. Even a flawless edit that fixed all 5 would barely clear p<0.05 two-sided. The v0.2.0 Result: 40 Tasks, Still Nothing We expanded to 40 tasks. We should have more power. Here's what Mistral 24B, our strongest analyzer, produced: Iter p-value Mean delta Accuracy 1 0.55 +0.025 64% 2 0.52 +0.025 64% 3 1.0 0.0 64% 4 0.52 +0.025 64% 5 0.77 -0.025 64% Mistral produced positive deltas in 3 of 5 iterations. That is real signal. But the delta is +0.025 — 2.5% improvement on 40 tasks. At p~0.5, there is roughly a coin-flip chance this is noise. The ceiling did not disappear. It moved: with 40 tasks, the sign-test floor for a flawless edit that moves 5 tasks is ~0.016 one-sided — clearable. But Mistral