Stop Trusting Vibes: A Reproducible Harness for Comparing AI Coding Models on Your Own Codebase

Most comparisons of AI coding models are useless to you. Not because the authors are dishonest, but because they test on their problems: greenfield LeetCode-style prompts, demo TODO apps, or a framework you don't use. Your codebase has different failure modes — a weird build system, a legacy module nobody wants to touch, tests that take 40 minutes. This article is a small, reproducible harness you can run in an afternoon to compare coding models against your own repository, with scoring based on your own test suite instead of vibes. The artifact is ~120 lines of shell and Python, plus a scoring rubric you can adapt. The core idea Instead of asking "which model is best?", ask: on a fixed set of real tasks from my repo, which model produces patches that pass my tests, fastest, with the least hand-holding? That gives you three measurable axes: Correctness — does the resulting diff pass the relevant tests? Edit locality — did the model touch only the files it should have? Iteration cost — how many prompt rounds did it take to get there? Step 1: Build a task set from your own git history The cheapest source of realistic tasks is your own commit log. Find commits that fixed a bug or added a small feature, then check out the parent commit and ask the model to reproduce the fix (without showing it the actual fix). #!/usr/bin/env bash # extract_tasks.sh — mine candidate tasks from git history # Usage: ./extract_tasks.sh <repo_path> <count> set -euo pipefail REPO = " $1 " ; COUNT = " ${ 2 :- 8 } " cd " $REPO " # Small, self-contained commits: <= 3 files, <= 80 changed lines, has a test file touched git log --oneline --no-merges -n 300 | while read -r sha msg ; do files = $( git diff-tree --no-commit-id --name-only -r " $sha " | wc -l ) lines = $( git diff --shortstat " $sha ^" " $sha " | grep -oE '[0-9]+ insertion|[0-9]+ deletion' | grep -oE '[0-9]+' | paste -sd + | bc ) if [ " $files " -le 3 ] && [ " ${ lines :- 999 } " -le 80 ] ; then echo " $sha | $files | $lines | $msg "