It Fit in Memory and Was Still Unusable — Do the Bandwidth Arithmetic First

Originally published on hexisteme notes . "Will it fit on our hardware?" is the wrong first question. It's the one everyone asks, because it's free to answer — the thing either loads or it doesn't. Throughput costs you a measurement. So the capacity gate passes, and it feels like the decision is made. The measurement Mac Mini M4, 24GB unified memory, ~120GB/s memory bandwidth. A 27B model, IQ4_XS quantized, 15GB on disk. Capacity gate: pass. Metal's recommendedMaxWorkingSet is 17.76GB, the model is 15GB, ollama ps reports 100% GPU resident. No swap, no spillover. By every "does it fit" criterion this is a clean win. Generation: 5.6 tokens/second. That's not a usable interactive worker. It's barely a usable batch worker. And nothing about the capacity check hinted at it. The arithmetic that would have told me in advance Autoregressive generation reads the entire model's weights once per token. So: ceiling ≈ memory bandwidth ÷ bytes touched per operation = 120 GB/s ÷ 15 GB = 8 tokens/second Measured 5.6 against a ceiling of 8. Ratio 0.70. That ratio is the whole verdict. When measured throughput is a large fraction of the arithmetic ceiling, you are bandwidth-bound , and you now know something concrete: the bottleneck is not your configuration, not memory pressure, not thermal throttling. It's how fast bytes move. Rule of thumb I now use: ratio ≥ 0.5 → bandwidth-bound, and size-reduction fixes are dead. Why "just quantize harder" doesn't work The natural move when capacity is tight is to shrink. Lower quantization, smaller batch, heavier compression. It's the reflex, and in a bandwidth-bound regime it's close to useless. I was considering Q3_K_M at 13.8GB. Run the same division: 120 ÷ 13.8 = 8.7 tokens/second (up from 8) Under 9% more throughput. For a real drop in output quality, because quantization error doesn't scale linearly with size the way bandwidth does — you give up more than you get, every time, in this regime. I killed that plan without downloading anythin