I Tried to Verify an AI Agent Benchmark. Here's the Bundle I Wish Everyone Shipped
Nearly every AI agent benchmark you read is unfalsifiable. Not wrong, necessarily - unfalsifiable. There's a blog post with a bar chart, a claim that framework A beat framework B, and no way for you to check it. No run count. No model version. No raw output. Often no cost. You are asked to trust a summary statistic produced by people with an interest in the result. We publish agent benchmarks, so this is our problem too. This post is about the evidence bundle we settled on, and how you can pull one down and take it apart in about two minutes. Every command below is one I actually ran while writing this, with its real output pasted in. The claim we're going to try to break From one of our pilot runs: LangGraph 1.2.9 and Pydantic AI 2.13.0 both completed 20 of 20 tasks under gpt-4o , at a total spend of $0.094275. That's the sort of sentence you'd normally have to take on faith. Let's not. Two minutes to verify it yourself The bundle is a directory in a public repo. Pull it: BASE = "https://raw.githubusercontent.com/benchclawio/harness/main/results/gpt-4o-vs-gpt-4o-mini-tool-calling-2026-07-24" for f in SHA256SUMS README.md gpt4o-pilot-manifest-v0.4.0.json \ scored-pilot-gpt4o-raw-2026-07-24.jsonl \ scored-pilot-raw-2026-07-24.jsonl \ scored-pilot-analysis-2026-07-24.json \ scored-pilot-gpt4o-analysis-2026-07-24.json \ real-pilot-status-manifest-v0.3.0.json ; do curl -sfO " $BASE / $f " done First question: is this the same data we published, or has something drifted? sha256sum -c SHA256SUMS README.md: OK gpt4o-pilot-manifest-v0.4.0.json: OK real-pilot-status-manifest-v0.3.0.json: OK scored-pilot-analysis-2026-07-24.json: OK scored-pilot-gpt4o-analysis-2026-07-24.json: OK scored-pilot-gpt4o-raw-2026-07-24.jsonl: OK scored-pilot-raw-2026-07-24.jsonl: OK That's the cheapest integrity control there is and almost nobody ships it. It costs one line in your run script and it means a reader can tell the difference between the file you published and a file someone edited afte