Your reasoning model isn't dumb. Your parser is throwing away its best answers.

I benchmarked a vision-language model and scored it at 0.31. The real number was 0.70. Same model, same weights, same hardware, same 100 questions. The only thing that changed was how I read its output. I had already written up the 0.31 as a capability finding and concluded the model was unsuitable. That conclusion was wrong, and the failure was entirely in my harness. Here is the mistake, because I doubt I am the only one making it. The setup I was evaluating a batch of open-weight and frontier models on a multiple-choice benchmark: multi-view driving scenes, four options per question, one correct answer. Standard stuff. The prompt asked for reasoning followed by a final line, Answer: X . My scoring code did the obvious thing: m = re . search ( r " Answer:\s*([A-D]) " , output ) pred = m . group ( 1 ) if m else None # None scores as wrong That last comment is the bug. What actually happened The model I was testing is a "thinking" model. It emits a long internal reasoning trace before it commits to an answer. I had a generation budget of 1024 tokens. On easy questions it reasoned briefly, emitted Answer: B , and scored fine. On hard questions it reasoned at length, hit the token cap mid-thought, and never emitted the answer line at all. So the harness scored every one of those as wrong. 64 of 100 questions returned no parseable answer. Zero of those were image-loading errors or crashes. They were all truncation. And the truncation was not random: Uncertainty 0/8 answered Counterfactual 0/3 answered Safety-critical Planning 1/11 answered Safety-critical Prediction 3/12 answered Look at that distribution. The questions the model failed to answer were precisely the questions that required the most reasoning. My harness was systematically discarding the model's performance on exactly the hard subset I was trying to measure, and reporting the result as a capability ceiling. Of the 36 it did answer, it got 86% right. The model was fine. My measurement was garbage. The fix