You're Not Comparing Models. You're Comparing Contracts.

You're Not Comparing Models. You're Comparing Contracts. Two teams publish scores on the same agent benchmark. One lands in the low sixties. The other clears seventy. A procurement team reads the spread and makes a call. What they do not see: both teams may be running the same model. They did not need to change the weights for the gap to appear. The spread can come from scaffold alone. One team wrapped the model in a harness with better retries. Different tool defaults. A planner step the other team had skipped. None of that appears on the leaderboard. The comparison that drove the decision was not between two agents. It was between two contracts. There Is No Benchmark The mistake hiding behind this story is a category error. People talk about agent benchmarks as if they measure a thing called “the model.” They do not. They measure a coupled system. The model is one component. The rest is a stack of protocol decisions that are almost never disclosed and almost always matter. The score is the output of that stack. Change any layer and you change what the number means. Recent research on agent evaluation has named those layers explicitly. There are at least seven. Deployment regime. Observation channel. Harness and scaffold. Metric and action. Configured evaluator. Grader protocol. Audit bundle. Each is a contract. Each is negotiable. And each can silently change the verdict while the headline looks the same. That is what a benchmark actually is. Not a measurement of a model. A measurement of an entire testing contract, of which the model is one slot. There is structural reason the seven layers are the seven layers. They cluster into three corners that show up in almost every published agent-evaluation failure. What the model is rewarded for. How that reward is optimised. And how the test contract differs from production. Once you hold those three corners in view, the seven-layer stack stops feeling like a checklist and starts behaving like the actual shape of what is