Reactive Play: Achieved!! Experimenting with Atari Breakout [R]

Six months ago I started experimenting with PPO and Breakout as a way to learn about Machine Learning and Reinforcement Learning. After a few experiuments just trying to get high scores, it bothered me that everything was a "memorized" script rather than reactive play, like a human would play. Thus began my journey to try and convince PPO to actually track the ball instead of focusing on scoring points. I read a lot of articles and tried a lot of things. After 124 PPO experiments on Atari Breakout, I found that every single model, across sticky actions, cursor wrappers, entropy tuning, dynamics randomization, adversarial bumpers, and everything else, converged to a memorized action sequence, not a reactive ball-tracking policy. The argmax was always a script. The fix wasn't more environment engineering. It was three lines of reward shaping: Directly rewarding the paddle for being horizontally close to the ball during descent. A tiny bonus (0.05 per frame vs 1.0-7.0 per brick) that fires every frame the ball is descending applied during training. During evaluation, the agent plays clean Breakout with no bonus. The behavior transfers!! Every prior approach I tried to penalize scripts by making the environment harder to memorize. PPO always found a way around it: timing-robust scripts, layout-conditioned scripts, noise-tolerant scripts. The optimum was always a script; only the shape changed. Proximity reward changes what the optimum is. A center-hold script gets incidental bonus when the ball passes near center. A reactive tracker gets the maximum bonus on every descent frame. The optimization pressure is unambiguous: track the ball, get more reward. I also made a cool tool to watch the agent work! It's called the "Split-Watcher" (so clever). It shows two instances of Breakout, each being controlled by a separate instance of the same agent. The one of the left is vanilla Breakout. The one of the right is a series of custom brick configurations. With the first 123 expe