标签：#localllama

Part 1 established the hardware, the runner, and the primary model. This entry covers what governs inference on that hardware — the two phases of inference, the cost of long context, and the cost of loading a model from disk — and compares the local machines against a free-tier cloud model. The two phases of inference Inference has two phases. Prefill processes the input prompt before any output appears; it is compute-bound and wants a GPU. Generation produces output tokens one at a time and is bound by memory bandwidth. Casual use is almost all generation and hides the difference; the cost of prefill surfaces only when prompts grow large. The machines, and how they were measured Machine CPU / RAM GPU (VRAM) Storage (read) Prefill (tok/s) Gen (tok/s) Load (18 GB) Primary desktop 5950X / ~80 GB DDR4 RX 6900XT (16 GB) NVMe (~2.1 GB/s) 360 18.3 8.4s Secondary box 5600G / 32 GB DDR4 GTX 1060 (6 GB) SATA SSD (~0.35 GB/s) 253 17.1 50.6s Laptop 8840U / 32 GB DDR5 Radeon 780M (none) NVMe (~2.4 GB/s) 20 10.0 7.5s All inference figures come from a controlled run: the same model (Gemma 4 26B, 18 GB) on each machine, a unique random prefix per prompt to defeat caching, a fixed 8,192-token context, warm, on an identical ~6,855-token prompt (generation timed over a 200-token output). Two things stand out. Prefill varies about eighteen-fold across the machines (360 to 20 tok/s) while generation varies less than twofold (18.3 to 10.0), and prefill is what dominates large-prompt workloads — so a machine can look fine on generation yet be useless in practice. Model-load time, separately, is set by storage rather than compute: the secondary box's budget SATA SSDs load the 18 GB model in 50 seconds against eight on NVMe, which turns a cold request into a minute-long stall. Secondary box Request time Warm (model resident) ~4s Cold (model reload) ~54s If the model is allowed to unload between calls, every call silently pays that reload — a real source of intermittent timeouts. The fix is

This is the first in a series of build-log posts documenting a local LLM project, in which models are run on owned consumer hardware rather than through a cloud API. The present entry covers the hardware, the software stack, and the benchmarks by which a primary model was selected. The hardware Two machines are used, both consumer-grade. All benchmarks reported below were obtained on the primary desktop. Machine CPU RAM GPU Primary desktop Ryzen 5950X ~80 GB DDR4 AMD RX 6900XT (16 GB) Secondary box Ryzen 5600G 32 GB NVIDIA GTX 1060 (6 GB) The software stack Ollama serves as the model runner across two GPU vendors: ROCm 5.7 for the AMD card on the primary desktop, and CUDA for the NVIDIA card on the secondary box. The primary model is Gemma 4 26B, a mixture-of-experts model with roughly 3.8B active parameters, quantized to Q4_K_M and occupying approximately 18 GB on disk. On the RX 6900XT it is run with an automatic GPU/CPU layer split, as the Q4 weights together with the KV cache exceed the 16 GB of available VRAM. Several Ollama settings were enabled to recover headroom: flash attention, and an 8-bit ( q8_0 ) KV cache, the latter approximately halving the cache footprint. A free cloud tier is retained for occasional heavier tasks, though the objective is to run as much as possible locally. Selecting a model: benchmarks Before a primary model was chosen, the installed models were benchmarked. Two properties were of interest: throughput and output quality. Throughput was measured on the primary desktop with a 500-word essay prompt ( ollama run <model> --verbose ): Model Tokens/sec Duration Tokens out gemma4:26b 18.86 50.11s 945 gemma4-26b (64K ctx) 17.96 51.99s 934 mistral:7b-instruct 34.81 10.17s 354 llama3.2 57.11 3.99s 228 The smaller models are substantially faster; their token counts, however, are lower, and in practice their responses were correspondingly shallower. Quality was assessed with a five-task suite spanning logic, coding, summarization, creative writ