Testing AI Coding Agents Beyond Code Generation: A Real-World Benchmark

1. Introduction Most public demonstrations of AI coding agents begin with an empty directory and end when a visible feature works. That is useful, but it measures only the first and easiest part of software engineering: generation. Real projects are stateful. They contain existing behavior, data that must survive, security boundaries, partially completed work, and verification steps that cannot be replaced by a convincing demo. This benchmark was designed as a small, practical record of that harder problem. It documents two Codex runs on the same full-stack task-manager project. Phase 1 was a greenfield build. Phase 2 returned to the existing application for an authentication and database-migration refactor. The second run also included a long, user-initiated interruption, making recovery part of the observed workflow. The narrow result is encouraging: the initial application was recorded as complete in approximately 34 minutes, and the later refactor took approximately 42 minutes of active execution, excluding the user-initiated pause. The Phase 2 run reported 16 backend tests, 1 legacy-migration test, and 25 frontend tests passing, along with lint, type, build, dependency, and browser-acceptance checks. Those numbers need boundaries. This is a single-environment case study, not an official benchmark, not a controlled comparison between providers, and not a production guarantee. The current documentation set does not retain the complete source tree, exact prompts, raw terminal output, migration artifact, database fixture, or browser trace. Results are therefore presented as supplied run records rather than independently reproduced evidence. 2. Why Simple Code Generation Tests Are Insufficient A prompt such as "build a task app" primarily tests synthesis. The agent chooses a stack, creates files, connects components, and produces a working path. It may reveal speed and basic tool use, but it says little about how the agent behaves when constraints collide. Maintenan