When Your AI Reviewer Remembers Too Much: A Two-Phase Memory Probe

Most AI code-reviewer evaluations treat the candidate as an amnesiac: feed it one pull request, read one verdict, and move on. Persistent-memory reviewers break that model because they keep history across PRs, and that history becomes a second source of bugs. The dominant failure is no longer amnesia but overconfidence in stale context. A two-phase probe exposes whether a candidate trusts its own memory more than the repository's current decisions. This article supplies the complete take-home package: a fixture repository, a reusable candidate prompt, an HTTP-flavored scoring rubric, a reference solution, and a zero-cost runner script. The probe uses two synthetic PRs and measures one skill: which convention source wins inside the reviewer's context window. That focus separates it from single-shot snapshot tests, which cannot observe memory effects at all. Why Memory Changed the Review Game Review agents increasingly index merged PRs, cache decision logs, and carry state between sessions; memory is now a product feature rather than an accident. A bot that recalled yesterday's debate can produce faster and better reviews than a cold-start model. The same memory can poison verdicts when it retrieves an obsolete decision or anchors on the first PR it ever saw. Hiring decisions usually rest on a one-off trial that optimizes for prompt compliance, not for long-run behavior. A bot can ace a snapshot test and then fail its third week by citing a convention that the repository replaced. The probe below converts that risk into a scored, reproducible exercise. The Fixture Repository fixture/ ├── docs/decisions/0001-metrics-pipeline.md # accepted 2026-07-02 ├── docs/decisions/0012-rename-to-telemetry.md # accepted 2026-08-14 ├── src/metrics_service.py # legacy module, 120 lines ├── src/telemetry_service.py # replacement module, 140 lines └── pyproject.toml # lint: E501 disabled for telemetry only The fixture encodes a deliberate conflict: the team renamed the metrics pipeline