LLMs Don't Have to Generate One Token at a Time: How Medusa and Multi-Token Prediction Cheat Autoregression

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. A modern LLM can contain hundreds of billions of parameters, run on extremely expensive accelerators, and still spend most of its inference time doing something that looks embarrassingly sequential: token 1 -> token 2 -> token 3 -> token 4 -> token 5 -> ... That is the awkward part of autoregressive generation. The model may process a whole prompt in parallel during the initial prefill, but once generation starts, the next token depends on the previous token. So generating 100 tokens looks conceptually like running the model 100 times. And for many serving workloads, that is exactly where the money goes. A family of techniques tries to break this bottleneck by asking a deceptively simple question: What if the model could predict several future tokens at once, then verify them in parallel? That idea leads to speculative decoding, Medusa-style multiple decoding heads, and the broader multi-token prediction approach used during training. The interesting part is that these are not merely "optimization tricks." They change the computational structure of decoding. This article develops that idea from first principles and then gets into the engineering details. 1. The problem: your GPU is doing an expensive sequential loop Consider ordinary autoregressive decoding. Given a prompt: The capital of France is the model predicts: Paris Then it feeds the new sequence back through the model: The capital of France is Paris and predicts the next token. Then again: The capital of France is Paris . and so on. Formally, the model factorizes the probability of a sequence as: P(x1, x2, ..., xT) = product over t of P(xt | x1, ..., x(t-1)) That conditional dependence is what makes language modeling so useful. It is also what makes decoding