How LLMs Learned to Reason: SFT --> RLHF --> RLVR
1. The Starting Line: The Last Non-Reasoning Flagships GPT-4.5, DeepSeek-V3, and Claude 3.5 Sonnet share something that has nothing to do with benchmark scores: they were the last major models built entirely on the "pretrain, then instruct-tune" recipe. All internal computation was done in one forward pass per token, with no backtracking, verification or revision mechanisms in place. By late 2024, these labs had run into the same big wall: scaling pretraining data and compute had reached a point of saturation. The naive recipe of more tokens, bigger model, more GPU-compute no longer bought equivalent capability gains, particularly on multi-step reasoning tasks. Good output quality became the bottleneck before parameter count did. What came after wasn't a bigger version of the same thing. It was a different training paradigm applied on top of the existing ones. The field pivoted to training models to think before they answer. 2. RL Terminologies Recap Think of a Roomba cleaning a house. The environment is the house; the state is what the Roomba currently senses (position, dirt map, obstacles); an action is a movement decision (turn, advance, suck); a trajectory is one full cleaning run; and the reward is some measure of how much dirt got picked up, dispensed along the way or tallied at the end. Map this onto an LLM generating text: the state is the prompt plus every token generated so far; an action is the next token (or, at a coarser grain, the next reasoning step); a trajectory is the full generated sequence; and the reward is a scalar signal applied to that sequence, either at the very end or, in some setups, at intermediate points. 3. The Pipeline, Oversimplified Pretraining → SFT → RLHF / RLVR Pretraining gives the model raw capability and world knowledge from next-token prediction over a huge corpus. SFT and the RL stages that follow are about shaping that capability toward useful, correct, well-formed behaviour. The rest of this post is a section-by-section zo