My eval said a perfect MCP server was broken. It was the eval that was lying.

Originally published at tengli.dev When I added an LLM-powered eval to mcpgrade , the first real run produced a result that looked like a scoop: context7 — a server with a perfect static score — failed tool selection 62% of the time. A model shown its two-tool catalog picked the "wrong" tool on 5 of 8 tasks. If I had shipped that number, it would have been wrong. Not slightly wrong — systematically, unfairly wrong. This post is about how I caught it, because the failure mode generalizes to most agent benchmarks people are building right now. The setup mcpgrade's --eval mode works like this: it reads a server's tool catalog, synthesizes realistic single-step tasks ("find the Slack channel where the incident was discussed"), shows a model the full catalog, and measures three things — does it pick the right tool, does it fill valid arguments, and does it correctly refuse tasks that no tool can handle. Round 1, on three real servers, cost about twelve cents and produced this: Server Static score Tool selection Args Refusal context7 (2 tools) 100 38% 100% 100% server-memory (9 tools) 81 93% 100% 100% server-slack (8 tools) 97 54% 100% 100% Two servers with excellent static scores, apparently failing live. Either static analysis was worthless, or the eval was broken. The eval was broken Every "miss" traced to one cause. Slack's post_message needs a thread_ts — a value you can only get from a previous call to get_channel_history . context7's get-library-docs needs a library ID that comes from resolve-library-id . These are pipelined tools : their required arguments are produced by other tools. My task synthesizer didn't know that. It generated tasks like "reply to the thread about the outage" — without a thread timestamp. The model, quite sensibly, picked get_channel_history first (to find the thread), or declined. My grader marked both choices wrong. The model wasn't confused. The model was right . The benchmark was grading correct multi-step reasoning as failure — and me