My Comment Section Designed My Next Experiment. Then It Made Me Freeze My Predictions.

Ten days ago I published an article about a failure mode: tell a language model "a scanner flagged this code" and some models agree with everything. Gemma removed 51% of my false alarms; gpt-4o-mini removed 20% and confirmed 90% of whatever it was shown. Then the comment section took the article apart — in the best way anything I've written has ever been taken apart. Over four days, four readers found the missing control in my experiment, made me preregister my predictions before running it, fixed my statistics, pinned my model versions, and designed a production monitoring spec I didn't ask for. This article is the story of that review, and the results of the experiment it produced. The predictions were frozen — publicly, in the thread, and in a commit — before a single API call. What follows includes the rows that survived and the rows that didn't. The objection I couldn't answer The article's headline number was a confirm-rate: what fraction of flagged code each judge model agreed was really vulnerable. One reader pointed out what that number actually is: "Every prompt already tells the model that scanner flagged this, so you measure agreement with the claim and detection together, in one number. … Right now Gemma can be more skeptical or just worse at seeing bugs, and from these tables I cannot tell which one it is." He was half wrong and half right, and the split matters. The half the existing tables answer: "maybe Gemma is just worse at seeing bugs." A judge that rejects because it can't see bugs fails symmetrically — it rejects real vulnerabilities and false alarms at similar rates, because it can't tell them apart. Gemma's rejections split 51% on false alarms against 2% on real bugs. Blindness doesn't produce that asymmetry. The half they can't answer: why does gpt-4o-mini confirm 90% of everything? Two stories fit the same tables. Story one: it believed the flag — classic sycophancy, the article's framing. Story two: it sees vulnerabilities everywhere and w