标签：#benchmarking

OpenAI published an audit of SWE-Bench Pro on July 8, 2026 and estimated that roughly 30% of its tasks are broken. The reported issues make a familiar leaderboard assumption unsafe: every task in the denominator is a valid, equally interpretable trial. Primary source: OpenAI, “Separating signal from noise in coding evaluations” . The operational response should not be “ignore all benchmarks.” It should be: version task validity, preserve disputed cases, and publish how conclusions change across plausible denominators. Model task state separately from model result task validity: unreviewed | valid | broken | disputed model result: pass | fail | infrastructure_error | missing Never convert infrastructure_error to model failure without reporting that policy. Never delete broken tasks while retaining an old score label. A row needs provenance: { "task_id" : "repo-issue-17" , "dataset_revision" : "sha256:..." , "harness_revision" : "git:..." , "model_config" : "immutable-config-id" , "validity" : "disputed" , "result" : "pass" , "review_revision" : 3 , "evidence" : [ "fixture.log" , "review.json" ] } Publish three denominators Let: P_v , N_v : passes and total among reviewed-valid tasks; P_a , N_a : passes and total across all attempted tasks; D : disputed tasks. Report: valid-only score = P_v / N_v all-attempted score = P_a / N_a uncertainty interval = score if every disputed task hurts conclusion .. score if every disputed task helps conclusion This interval is not a statistical confidence interval. It is a sensitivity bound for unresolved task validity. A tiny sensitivity calculator #!/usr/bin/env python3 import json , sys rows = [ json . loads ( line ) for line in open ( sys . argv [ 1 ]) if line . strip ()] valid = [ r for r in rows if r [ " validity " ] == " valid " ] disputed = [ r for r in rows if r [ " validity " ] in ( " unreviewed " , " disputed " )] attempted = [ r for r in rows if r [ " result " ] in ( " pass " , " fail " )] rate = lambda passed , total : pa

"It works on my test input" is the most dangerous sentence in on-device AI development. I typed that sentence - or some version of it - a dozen times while building Redacto, our on-device PII redaction app running Gemma 4 E2B on a Samsung Galaxy S25 Ultra. The model would redact a patient name from a clinical note, I would nod, and I would move on. Then I would hand the phone to a teammate, they would type a police report, and the model would redact the suspect description instead of the victim name. The problem is not the model. The problem is that manual spot-checking is not validation. You are testing a single input against your own expectations, with all the confirmation bias that entails. When you have five domain modes (HIPAA, Financial, Tactical, Journalism, Field Service), three difficulty levels, and two candidate models, you need something systematic. You need a benchmark suite. This post covers how I built one - from dataset curation to scoring methodology to on-device infrastructure - for a hackathon app running entirely on a phone. No cloud. No API calls. No data leaving the device. Why Not Use an Existing Framework? The LLM evaluation space has mature tools. EleutherAI's lm-eval-harness is the community standard for evaluating language models against academic benchmarks like MMLU, HellaSwag, and ARC. Stanford's HELM (Holistic Evaluation of Language Models) provides a multi-metric evaluation framework with standardized scenarios. Google's BIG-bench offers hundreds of tasks for probing specific capabilities. These frameworks are excellent for what they do. They are also completely wrong for this problem, for three reasons. First, they assume server-side inference. lm-eval-harness expects to call a model through an API or load it in PyTorch on a GPU server. Redacto's model runs on a Qualcomm Hexagon NPU inside a phone. There is no Python runtime, no HuggingFace tokenizer at evaluation time, no way to hook into the framework's inference loop. Second, their

Explore how the GitHub Copilot agentic harness delivers strong results across multiple benchmarks and leading token efficiency, while maintaining flexibility to choose among more than 20 models. The post Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks appeared first on The GitHub Blog .