I Added a Fourth Model Mid-Run. It Changed What My Field Test Could Prove.
Latest release: v0.2.2 — Aug 29, 2026 I did something I usually try hard not to do in a field test. I changed the design after it had already started. Halfway through validating AdversarialDebate, I realized the model set was too narrow to answer the most important question in the project. So I added a fourth model in the middle of the run. That was messy. It wasted work. It made the corpus inconsistent for a while. It also turned out to be one of the best decisions in the whole release. This post is about a lesson I trust far more now than I did before building this project: a field test is not just there to produce numbers. It is there to reveal whether your experiment can actually answer the question you think it is answering. The Setup I Started With I began with three models: GPT-4o-mini, Gemini 2.5 Flash, and DeepSeek-V3. That gave me three useful pairings — GPT + Gemini, Gemini + DeepSeek, and GPT + GPT as a homogeneous control. Three labs, two regions, one same-model control. Reasonable spread. I ran the small corpus first, just 3 PRs, to validate the pipeline. Pair Small-corpus score Verdict rate Gemini + DeepSeek 0.835 33% GPT + GPT 0.667 33% GPT + Gemini 0.148 0% The diverse pair was ahead. The weak pair was struggling. The homogeneous control was doing something interesting. If I had stopped there, I would have told a clean story — and it would have been the wrong one. The Problem Was Not The Data. It Was The Coverage. The issue was not that the first three models were bad. The issue was that the experiment could only see part of the diversity spectrum. With those three models, the farthest useful pairing I had was US + China. I did not have a genuinely cross-continent pair that could show what happened at the far end of diversity. The test could suggest whether diversity helped. It could not show whether maximum diversity behaved differently from moderate diversity. That is a major blind spot when the whole thesis is about pairing behavior. I needed a f