Stop Guessing: A Reproducible Harness for Evaluating Free AI Coding Models on Your Own Repo

Most "which AI coding model is best?" debates I see devolve into vibes. Someone pastes a cherry-picked diff, someone else counters with a different cherry-picked diff, and nobody learns anything transferable. The problem isn't the models — it's that we almost never evaluate them on our code, with our constraints, using a method we could rerun tomorrow. This article is the harness I wish more teams built before arguing. It's a small, language-agnostic evaluation loop you can point at any model you have access to — including free tiers — and get a defensible answer to a narrow question: does this model help with the tasks I actually do? The evaluation trap Public benchmarks (HumanEval-style tasks, leaderboard scores) measure performance on curated problems with clean specifications. Your work is rarely that. Real tasks look like: "Add retry logic to this half-migrated HTTP client without breaking the old call sites." "Write tests for a function whose behavior depends on a config file three directories up." "Refactor this 200-line function, but the ORM calls must stay in the same transaction." These tasks share a trait: correctness is checkable, but only by you . Your test suite, your type checker, your lint rules. That's actually good news — it means evaluation can be automated against artifacts you already have. The artifact: a task-runner harness The core idea is dumb on purpose. Define a set of tasks as directories. Each task has a prompt, a snapshot of the relevant code, and a verification command. The harness applies a model's patch and runs the verifier. No scoring model, no LLM-as-judge — just your own build. eval/ ├── tasks/ │ ├── 001-retry-http-client/ │ │ ├── prompt.md │ │ ├── repo/ # snapshot of the relevant files │ │ └── verify.sh # exit 0 = pass │ ├── 002-test-config-loader/ │ └── 003-split-billing-fn/ └── run_eval.py Here's a minimal runner (Python 3.10+, stdlib only): #!/usr/bin/env python3 """ run_eval.py — apply a model-produced patch to each task and