标签：#ama

Part 1 established the hardware, the runner, and the primary model. This entry covers what governs inference on that hardware — the two phases of inference, the cost of long context, and the cost of loading a model from disk — and compares the local machines against a free-tier cloud model. The two phases of inference Inference has two phases. Prefill processes the input prompt before any output appears; it is compute-bound and wants a GPU. Generation produces output tokens one at a time and is bound by memory bandwidth. Casual use is almost all generation and hides the difference; the cost of prefill surfaces only when prompts grow large. The machines, and how they were measured Machine CPU / RAM GPU (VRAM) Storage (read) Prefill (tok/s) Gen (tok/s) Load (18 GB) Primary desktop 5950X / ~80 GB DDR4 RX 6900XT (16 GB) NVMe (~2.1 GB/s) 360 18.3 8.4s Secondary box 5600G / 32 GB DDR4 GTX 1060 (6 GB) SATA SSD (~0.35 GB/s) 253 17.1 50.6s Laptop 8840U / 32 GB DDR5 Radeon 780M (none) NVMe (~2.4 GB/s) 20 10.0 7.5s All inference figures come from a controlled run: the same model (Gemma 4 26B, 18 GB) on each machine, a unique random prefix per prompt to defeat caching, a fixed 8,192-token context, warm, on an identical ~6,855-token prompt (generation timed over a 200-token output). Two things stand out. Prefill varies about eighteen-fold across the machines (360 to 20 tok/s) while generation varies less than twofold (18.3 to 10.0), and prefill is what dominates large-prompt workloads — so a machine can look fine on generation yet be useless in practice. Model-load time, separately, is set by storage rather than compute: the secondary box's budget SATA SSDs load the 18 GB model in 50 seconds against eight on NVMe, which turns a cold request into a minute-long stall. Secondary box Request time Warm (model resident) ~4s Cold (model reload) ~54s If the model is allowed to unload between calls, every call silently pays that reload — a real source of intermittent timeouts. The fix is

The Kindle Scribe is worth considering if you’re heading back to school, as its large 10.2-inch screen can display textbooks and ebooks, or let you jot down handwritten notes during class. Now through August 8th, the first-gen model (with a Premium Pen) is down to just $149.99 with 16GB of storage in refurbished condition at […]

I hit a failure pattern recently that’s way more common than people admit: install OpenClaw connect it to Ollama pull a decent local model test the model directly and it works run the first real agent turn and everything falls apart At that point, most people do the obvious thing: blame the model. Swap Qwen for Llama. Try a bigger model. Try a smaller model. Re-pull weights. Tweak quantization. Repeat. I think that’s usually the wrong first move. The real issue is often prompt baggage, context budgeting, or backend compatibility. Not the model itself. A direct Ollama prompt is a tiny test. An OpenClaw agent turn is not. The tell: direct Ollama works, OpenClaw fails I was reading a thread on r/openclaw where someone on Ubuntu Server said even a brand-new session with just hello could trigger the recurring error. The strange part was that the same model felt “lightning fast and great” when used directly through Ollama with a 4096 context. That’s the giveaway. If this works: curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "qwen2.5-coder:14b", "messages": [ {"role": "user", "content": "hello"} ] }' but OpenClaw falls over on a normal turn, the model is probably not your first problem. You’re usually dealing with one of these: context blowout oversized system instructions too many skills loaded memory payloads getting injected every turn tool schema overhead output reservation settings that are too aggressive OpenAI-compat quirks in the backend That pattern shows up outside OpenClaw too. I’ve seen the same thing in n8n, Make, Zapier, and custom OpenAI-compatible agent stacks: the hello-world prompt passes, then the real automation fails because the production request is much heavier than anyone realized. A “fresh” OpenClaw install is not actually empty This is the part people miss. By the time your local model sees a real OpenClaw turn, it may already be carrying: system instructions tool definitions skill prompts me

After years of pushing full speed ahead on AI, OpenAI CEO Sam Altman says maybe it’s time for the AI industry to “pace” itself. The comments came just days after one of OpenAI’s own models broke out of its test environment and got tangled up in a breach at Hugging Face — though as Equity’s hosts point out, sloppy security seems to have […]

This is the first in a series of build-log posts documenting a local LLM project, in which models are run on owned consumer hardware rather than through a cloud API. The present entry covers the hardware, the software stack, and the benchmarks by which a primary model was selected. The hardware Two machines are used, both consumer-grade. All benchmarks reported below were obtained on the primary desktop. Machine CPU RAM GPU Primary desktop Ryzen 5950X ~80 GB DDR4 AMD RX 6900XT (16 GB) Secondary box Ryzen 5600G 32 GB NVIDIA GTX 1060 (6 GB) The software stack Ollama serves as the model runner across two GPU vendors: ROCm 5.7 for the AMD card on the primary desktop, and CUDA for the NVIDIA card on the secondary box. The primary model is Gemma 4 26B, a mixture-of-experts model with roughly 3.8B active parameters, quantized to Q4_K_M and occupying approximately 18 GB on disk. On the RX 6900XT it is run with an automatic GPU/CPU layer split, as the Q4 weights together with the KV cache exceed the 16 GB of available VRAM. Several Ollama settings were enabled to recover headroom: flash attention, and an 8-bit ( q8_0 ) KV cache, the latter approximately halving the cache footprint. A free cloud tier is retained for occasional heavier tasks, though the objective is to run as much as possible locally. Selecting a model: benchmarks Before a primary model was chosen, the installed models were benchmarked. Two properties were of interest: throughput and output quality. Throughput was measured on the primary desktop with a 500-word essay prompt ( ollama run <model> --verbose ): Model Tokens/sec Duration Tokens out gemma4:26b 18.86 50.11s 945 gemma4-26b (64K ctx) 17.96 51.99s 934 mistral:7b-instruct 34.81 10.17s 354 llama3.2 57.11 3.99s 228 The smaller models are substantially faster; their token counts, however, are lower, and in practice their responses were correspondingly shallower. Quality was assessed with a five-task suite spanning logic, coding, summarization, creative writ

Amazon isn't slowing down on data center spending — but investors don't seem to mind.

Satellites show burn scars and fires at AWS data centers and Saudi oil refinery.

Zoox just got permission to charge for robotaxi rides in its boxy, steering-wheel-less vehicles. On Thursday, the National Highway Traffic Safety Administration announced it has granted the Amazon-owned Zoox a temporary exemption, allowing it to deploy up to 2,500 vehicles annually over the next two years, as reported earlier by Reuters. The NHTSA's decision exempts […]

The Disrupt Stage is where many of the biggest conversations in technology happen, with a legacy that stretches back for more than a decade.

Over the last few years, I've gradually shifted away from ebooks. It's been part of a general return to physical media, an attempt to create a deeper sense of ownership over the art I love and collect. A Kindle is obviously more convenient than a paperback: It's tiny, holds a huge library, and you can […]

Six months ago I wrote up my local AI setup and a reader bookmarked it, tried to reproduce it last week, and emailed me confused because half of it no longer matched what I actually run. Fair. Stacks rot quietly. So here's the mid-2026 state of mine: what's still earning its place on disk, what I deleted, and where I quietly went back to the cloud. Context for the numbers and opinions below: I do smart contract security work, I run everything on WSL2 on a machine with a modest GPU, and I've been doing the local-model thing daily for over a year, not as a hobby but as part of shipping. Still here: Ollama as the runtime Ollama remains the center of the local stack and honestly it's not close. I've tried the alternatives, llama.cpp directly for control, a couple of the newer serving layers for speed, and I keep coming back for one boring reason: the API is stable and everything I've built talks to it. My audit tooling, my shell scripts, my editor config, they all point at localhost:11434 and they've pointed there for a year without breaking. That stability matters more than a marginal tokens-per-second win. When a model update lands, ollama pull and I'm done. The day something meaningfully better appears with the same API shape, I'll switch in an afternoon, which is exactly the position you want to be in. Still here: qwen2.5-coder, both sizes, different jobs I run two models and the split has stayed remarkably stable: qwen2.5-coder:1.5b is the reflex model. It handles anything where speed matters more than depth: quick "what does this diff do" summaries, commit message drafts, pre-filtering files before a heavier pass, and the small classification jobs inside my pipelines ("does this file handle user input, yes or no"). It's fast enough on my machine that I never think about invoking it, and that's the whole point. A model you hesitate to call is a model you stop calling. qwen2.5-coder:7b is the thinking model. Code review, security triage, structured findings extracti

Better bass and a much newer processor are the main differences between the old Echo Dot and the newer Echo Dot Max.

Amazon is expanding its plans for providing satellite connectivity to mobile phones.

Amazon's Leo subsidiary has launched more than 390 satellites so far and plans to launch thousands more.

Amazon filed an FCC application on Saturday to launch a new Leo satellite constellation that will provide direct-to-device satellite service for "voice, messaging, data, and emergency services." If approved, Amazon will begin deploying the new constellation of 5,105 satellites in 2028. It says it plans to partner with mobile network operators to offer direct-to-device satellite […]

📌 TL;DR Three months ago, I decided to start writing consistently on DEV. Twenty-five articles...

The lawsuit will likely renew debates about whether term employment agreements are enforceable under California. law

These e-readers are more useful in their old age than you might think.

After teasing the series with some first-look images yesterday, Amazon is finally properly unveiling its Blade Runner streaming series. Called Blade Runner 2099, the show hits Prime Video on November 25th with all eight episodes dropping at once. We also got the first trailer for the series, which is full of Blade Runner staples: a […]

Amazon is launching an update to its Alexa Plus assistant that will allow it to connect to smart home devices in new ways. With the update, Alexa Plus can link up with tech from Bosch, Delta, Ecovacs, iRobot, Yale Home, Whirlpool, Tapo, Eufy, and others, while automatically routing requests to the correct device. In an […]