Every LLM Request Has Two Halves. Only One Uses Your GPU Cores

Paste a long document into ChatGPT and hit enter. Nothing happens for a second or two. Then the answer starts appearing, word by word, at a steady pace until it finishes. You have seen this hundreds of times. Most people never think about it. But those are two completely different things happening inside one request, running on two different parts of the same GPU, limited by two different bottlenecks. Once you know what they are, a lot of confusing things about serving AI models stop being confusing. Including why a faster GPU sometimes makes no difference at all. This is part two of a series. Part one covered the split between VRAM and cores. This post covers what each of them does when a request arrives. The example we will follow I am going to trace one question the whole way through: what is the capital of france Simple, short, and it produces a short answer. Perfect for watching the machinery. Before anything happens, the model server has already loaded the model's weights into VRAM. That happened at startup and they stay there. Nothing about your request loads a model. Your words get chopped into tokens, which are just pieces of words turned into numbers, and placed in VRAM alongside the weights. Now the work begins. Phase one: prefill, or "the pause" Before the model can write anything, it has to read everything you gave it. Here is the important part, and it is the opposite of what most people assume: it reads your entire prompt at once. Not word by word. All six words go to the cores together, in a single pass. That is possible because your whole prompt is already there. Nothing has to wait for anything else. So thousands of cores fire simultaneously and chew through all of it in one burst. flowchart LR A["what is the capital of france<br/>all 6 tokens together"] --> C W["16 GB of model weights"] --> C C["GPU CORES<br/>every core busy"] --> O1["First word: 'The'"] C --> O2["Notes saved to VRAM<br/>the KV cache"] Two things come out of that single pass: The