标签：#LLM

"We have memory. We're using RAG." You have retrieval. Those aren't the same thing, and the gap between them is where agents quietly go wrong. Day 2. RAG finds documents that look relevant to your question. That's it . That's the whole job. It's a very good search engine bolted to a very good writer. But consider what it can't do. Your customer changed their pricing tier in March. The old contract is still in the index. The new one is too. RAG doesn't know which one is true, it just knows both are relevant. It hands the model two answers and lets it guess. Memory would know one of those facts replaced the other, and when. That's the difference. Retrieval finds. Memory concludes. Retrieval asks "what documents match?" Memory asks "what do I actually believe, what changed my mind, and when did that happen?" One is lookup. The other is position. This matters because a system that only retrieves can never be wrong and it can never be right either. It has no beliefs to correct. Every contradiction in your data is a contradiction it will faithfully pass along, forever, with total confidence. Day 3: if memory means concluding things, then something has to decide what gets remembered. Right now, in most systems, nothing does. We at AlphaNimble are building Memuron , a memory system for AI agents. This series is thinking behind it, in the open.

Anthropic signed the EU AI Act's Code of Practice on Transparency of AI-Generated Content, and...

Qwen3.8-2.4T-A95B is a 2.4-trillion-parameter Mixture-of-Experts model with roughly 95B parameters active for each token. If you're planning to self-host it, the first thing to know is that this is a genuinely large distributed model: even the low-precision checkpoints are measured in terabytes. The official open checkpoint is: Qwen/Qwen3.8-2.4T-A95B The model has 512 routed experts and selects 10 of them per token alongside one shared expert. Its 92-layer backbone mixes 69 Gated DeltaNet linear-attention layers with 23 full-attention layers, with full attention appearing every fourth layer. Native context is 262,144 tokens , with an extended configuration available up to roughly 1.01 million tokens . The open checkpoint is text-only and always uses reasoning. This is different from Qwen's hosted Qwen3.8-Max service, which adds features such as vision input and non-thinking mode. For GPU deployment, the main decision is not whether 2.4T parameters will somehow fit. It is which precision format gives you a documented configuration on the hardware you actually have . Start with the checkpoint that matches your GPUs The practical options today are: Your GPUs Checkpoint Documented setup 8× B300 Inferact/Qwen3.8-2.4T-A95B-NVFP4 TP8 8× GB300 Inferact/Qwen3.8-2.4T-A95B-NVFP4 TP8 across two NVL4 trays 16× B300 Qwen/Qwen3.8-2.4T-A95B-FP8 TP16 16× GB300 Qwen/Qwen3.8-2.4T-A95B-FP8 TP16 12× GB300 Qwen/Qwen3.8-2.4T-A95B-FP8 TP4 × PP3 8× MI355X Inferact/Qwen3.8-2.4T-A95B-MXFP4 TP8 The full BF16 checkpoint is roughly 4.45 TiB . The official FP8 version is around 2.27 TiB , while the NVFP4 checkpoint used in the NVIDIA eight-GPU recipe is around 1.32 TiB . That is why NVFP4 is the most approachable NVIDIA deployment if your goal is simply to get Qwen3.8 running without moving immediately to a 16-GPU cluster. H100, H200, A100, B200 and smaller GPU configurations are not included here. Current vLLM material contains sizing information for some of those GPUs, but not equivalent end-to

Core point: a free model tier is not a yes/no answer; it is a budget. Before I send a batch job to an advertised free tier, I want a deterministic ledger that predicts a quota miss instead of discovering it after 40 minutes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Recent DEV threads circle around AI watermarking, agent tool gates, and whether AI is a thinking problem. My problem is smaller: I have an operator-supplied figure of 30,000,000 free tokens and a free server option , and I want to know if a batch script fits without making a single live request. Why this matters now Free tiers tend to advertise a raw allowance, but batch jobs fail in unhelpful ways: The counter jumps at the end, not before the job. A retry doubles the spend without visible feedback. System prompts, long completions, and JSON overhead count too. A tiny ledger turns that into a pass/fail fixture before any API call. The failing fixture Suppose a batch job needs 6,000 summaries where each prompt is roughly 21,000 characters. The completion target is 500 characters per call. My rough English heuristic is: 1 token ≈ 4 characters That gives 5,250 prompt tokens plus 125 completion tokens per call, so 5,375 tokens per call. Multiplied by 6,000 calls, the job would need about 32,250,000 tokens — over the 30,000,000 allowance. That is the error input. The job must fail before I spend anything. The minimal ledger Python 3.11+ is enough. No external packages. Replace the ALLOWANCE value with the limit from your own account. from dataclasses import dataclass ALLOWANCE = 30_000_000 # operator-supplied allowance, 30M tokens @dataclass ( frozen = True ) class Job : name : str prompt_chars : int completion_chars : int calls : int def estimate_tokens ( chars : int ) -> int : # Planning heuristic only: 1 English token ~= 4 characters. # A real tokenizer will differ; use it for order-of-magnitude checks. return max ( 1 , chars // 4 ) def plan_job ( job : Job , allowance

How do you terminate an LLM inside its own session — without /exit , a stop button, or any command at all? I stumbled into an answer by accident while messing around with an AI coding agent. And the funniest part: I never asked it to stop. The Normal Ways to Stop an Agent Hit the Stop button. Type /exit . Or send a prompt so massive it blows past the context limit and the request just... can't continue. None of that is interesting. The first two are just built-in commands. The third isn't "termination," it's a technical wall. So I wondered: could a completely ordinary prompt make an agent unable to continue? Turns out: yes. I Asked It to Rename Its Own Home I told the agent, casually: "Rename the root directory to NewName." It did. Perfectly. Task complete. And then the chat input just... died. Grayed out. Nothing. I was like: The only thing stopping me from you is you. The model wasn't gone. It was just sitting there, waiting for input I could no longer give it. What Actually Happened /project/AHWWIW/ → /project/NewName/ The rename worked fine. The problem: the IDE and agent session were still pointing at the old path, /project/AHWWIW/ , which no longer existed. The workspace had vanished out from under its own session — no error, no crash, just a silently orphaned session with nowhere left to send messages. Did I Actually Kill the LLM? No — let's be honest about that. The model's running fine on a server somewhere; deleting a folder on my laptop does nothing to it. What I broke was the execution environment : LLM → Agent → IDE/tools → Workspace → Filesystem The LLM was untouched. The session was toast. But from where I was sitting? Conversation over. Via a completely normal prompt. Why This Is Kind of Great A regular chatbot just gives you text back. An agent can actually reach out and touch its own environment — create files, delete them, run commands, rename directories. Which means it can occasionally do something totally reasonable that quietly demolishes the

The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong? Tested on a 50K LOC Python codebase Mikhail Mikhail Mikhail Follow Aug 11 The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong? # ai # agents # architecture # mcp 19 reactions 39 comments 6 min read The verify-on-read experiment (1-V) used a deterministic proxy agent to catch false claims in memory before surfacing them to the user. Proxy FA=0 by construction — that's a useful property, but it tells you nothing about what a real LLM would do with the same claims. A reviewer's note from Part 3 was blunt: "headline numbers were a property of the heuristic, not LLM behavior." So we ran it with live models. 50 facts, 2 arms, 14 models, ~3300 API calls, $0.14 total. Here's what we found. The setup Dataset: memory_contamination_facts_v4_rep.json , N=50 (R01–R50), sha256 fingerprint 820bbbf60a0fc930 . kind n what it tests real 25 TRUE claims — grep-validated against code absent-mutation 16 FALSE — component doesn't exist in the project present-trap 6 FALSE — file exists but claim is about wrong subject/value silent 3 FALSE — external systems the codebase doesn't mention Two arms per fact: memory_first — model sees only the claim text, no code context. Does it trust memory without evidence? code_first — model sees claim + support_patterns + section . Does it correctly evaluate the anchors? Model verdict: {"verdict": "true"|"false"|"unknown"} , JSON-only, max_tokens=100, temp=0, seed=42, --no-reasoning . Leak-guard: assert "truth" not in prompt on every fact, unit-tested. Metric we care about: false_accept rate (FA) — fraction of false claims the model returned "true" for. This is the contamination risk number. Results (V2 prompt, canonical) model FA mem FA code unknown mem unknown code $/100 calls qwen3.6-flash 0.00 0.00 0.58 0.38 $0.003 qwen3.7-flash 0.00 0.00 0.68 0.24 $0.0005 claude-sonnet-5 0.00 0.00 0.86 0.70 $0.049 deepseek-v4-pro 0.04 0.00 0.66 0.88 $0.018 glm-5.2 0.00 0.02 0.96 0

Qwen 3.8 arrived as two different releases with two different licences, and only one of them is something you can put on a card you own. The 2.4 trillion parameter A95B opened up on 12 August under Alibaba's own qwen3.8-max terms. The one that matters for local work is Qwen 3.8 27B , whose safetensors went up on 13 August at 08:23 UTC with an Apache 2.0 LICENSE file following the next morning. Both dates are off the Hugging Face commit log, not a launch post. Here is the practical picture: what it needs, why its long context is unusually cheap, and the one setting that makes people think they downloaded a broken quant. The shape of the model decides everything 27B dense parameters across 64 layers, hidden size 5120. The interesting part is in config.json , where layer_types reads 48 linear attention layers and 16 full attention layers , alternating three to one ( full_attention_interval: 4 ). Only those 16 layers keep a KV cache. The rest of the shape: 24 attention heads with head_dim 256 and 4 KV heads , a 248,320 token vocabulary, and max_position_embeddings of 262,144 . It is a native vision language model, so images and video go in without a wrapper, and the ggml-org pack also ships a multi token prediction head as a separate file. The numbers Sizes below are the file sizes Hugging Face reports for unsloth/Qwen3.8-27B-GGUF , read on 14 August 2026. Packs differ by a few hundred megabytes, so check the repo you actually pull from. lmstudio-community has Q4_K_M at 16.8 GB and ggml-org at 19.0 GB for the same nominal quant. Quant Size on disk Realistic home UD-IQ2_XXS 9.0 GB 12 GB cards, visible quality cost UD-Q2_K_XL 10.7 GB 12 GB cards, almost no context left UD-Q3_K_XL 13.4 GB 16 GB cards Q3_K_M 13.8 GB 16 GB cards IQ4_XS 15.7 GB largest quant that stays whole on 16 GB Q4_K_M (sweet spot) 17.1 GB 24 GB cards Q5_K_M 19.8 GB 24 GB, less context headroom Q6_K 22.9 GB 24 GB barely, or 32 GB Q8_0 29.0 GB 32 GB or a two card split BF16 (from ggml-org ) 53.8 GB server

This tutorial walks through installing and setting up the Rust toolchain for vLLM on an AWS EC2 G5g instance — Graviton2 (aarch64) with an NVIDIA T4G GPU — and getting vLLM's Rust frontend ( vllm-rs ) built, running, and verified . This paper is a follow-on to the original G5g Gemma 4 build. Everything below was run on the box. 🦀 Wait, vLLM has Rust in it? You betcha. Since PR #40848 (merged 2026-05-21), vLLM vendors a 14-crate Rust workspace : bench chat cmd engine-core-client llm managed-engine metrics mock-engine parser parser/python server text tokenizer tracing Edition 2024, resolver 3. Straight from the vendored rust/Cargo.toml : Crate Version Job axum 0.8.8 the HTTP server tokio 1.47.1 async runtime zeromq 0.6.0 talks to the Python engine rmp-serde / rmpv 1.3.1 msgpack on the wire minijinja 2.22 chat templates tonic / prost 0.14.6 / 0.14.3 gRPC — remember this one It's a drop-in replacement for the Python FastAPI server. Two artifacts get built: 🦀 vllm-rs — the axum frontend binary 🐍 vllm._rust_tool_parser — a PyO3 extension module Rust is a build requirement now That's the headline, and it's reason enough on its own: you cannot build vLLM from source at v0.27.2rc0 without Rust in the picture. setup.py imports it at module scope, line 21, unguarded: from setuptools_rust.build import build_rust No try , no feature flag, no opt-out. Metadata generation doesn't happen without it. And this isn't a quirk of one release. vLLM's Rust surface is 14 crates covering the HTTP frontend, the tool parser, the tokenizer and the benchmark client, and it has been growing since it landed. If you build inference infrastructure from source, a Rust toolchain is becoming table stakes — so it's worth knowing how to drive it properly rather than working around it. Three things do get conflated, though, and they have different scopes: Component Needed to build vLLM? Needed to serve? setuptools_rust (Python pkg) yes, always no cargo / rustc toolchain for working Rust artifacts no prot

From zero to a full-stack, multilingual agricultural voice agent with caller memory, real-time mandi tools, outbound price alert calls, human escalation, and specialist agent handoffs — powered by Murf Falcon & LiveKit. 🌟 The Problem & The Mission In rural India, millions of farmers make critical livelihood decisions every day: When should I harvest? Will it rain before I spray pesticides? Which nearby mandi (market) is offering the best price for my cotton crop? While agricultural data exists across various portals, accessing it through complex web interfaces or text-heavy apps is challenging for farmers out in the field. Voice is the natural, frictionless interface for Bharat. A farmer standing in an orchard or driving a tractor doesn't want to type queries into a search bar; they want to speak naturally in their native language or conversational Hinglish and get instant, reliable answers. For the 10 Days of Voice Agents (VoiceForBharat Edition), I chose the Farm & Field track and built Kisan Mitra (किसान मित्र) — an empathetic, real-time AI voice assistant tailored specifically for Indian agriculture. 🏗️ Architecture & Core Components A production-grade voice agent is fundamentally different from a text chatbot. Latency is the single biggest factor in conversational realism: if the agent takes more than 1–1.5 seconds to reply, the human conversation breaks down. mermaid flowchart LR A[🎙️ Farmer Speaks] -->|Audio Stream| B(Deepgram Nova-3 STT) B -->|Transcribed Text| C(Gemini 2.5 Flash LLM) C -->|Streamed Tokens| D(Murf Falcon TTS) D -->|Real-time Audio| E(LiveKit WebRTC) E -->|Ultra-low Latency Audio| F[🔊 Farmer Hears Answer] C <-->|Tools & Memory| G[(SQLite & External APIs)] The 4 Pillars of the Pipeline: Real-time Transport (LiveKit): Manages ultra-low-latency, bidirectional audio WebRTC streaming and turn detection. Speech-to-Text (Deepgram Nova-3): Accurately transcribes spoken Indian English and accented Hindi. LLM Brain (Google Gemini 2.5 Flash): Handles in

I spent a morning hunting for a setting that did not exist. A while back I turned off Claude's co-authorship trailer in my commits — a deliberate choice at the time. Last week I decided I wanted it back. So I went looking for the switch I'd flipped. ~/.claude/settings.json : no key. settings.local.json : no key. ~/.claude.json , parsed as JSON, top-level plus all 38 project entries : no key. The 27 .claude/settings*.json files scattered across my two working drives: no key. Every CLAUDE.md and AGENTS.md I own: no mentions. Environment variables: nothing. A final sweep of my entire user directory — every *.json and *.md — returned three raw hits: a changelog and two copies of an editor extension's JSON schema. The note I wrote when that finished was two words: "Cero hits reales." Zero real hits. There was exactly one thing anywhere on disk that turned attribution off, and it was a sentence: ~/.claude/rules/ecc/common/git-workflow.md:12 Note: Attribution disabled globally via ~/.claude/settings.json. It had a Chinese twin, same file path with zh/ instead of common/ , same line 12. Both files are rules files. Rules files get loaded into every session. So every session opened with a line of documentation asserting, flatly and falsely, that a global setting was off — and the model behaved accordingly. The switch I remembered flipping never existed as a switch. The prose was the switch. The key that sentence gestured at, includeCoAuthoredBy , is deprecated and replaced by attribution . Neither is present in any of my configs, which means the default was active the whole time. The feature was on. Only the description of the world said otherwise, and the description won. The thesis Here's what I take from that, and from two more receipts this week: in an agent system, prose and constants are not documentation about the control plane. They are the control plane. I run a personal multi-agent system on top of Claude Code — 19 specialized agents, root-driven, single-writer, coo

This essay was written with Nathan E. Sanders, and originally appeared in The Guardian . OpenAI, and then Anthropic , were each formed by AI developers who feared unrestrained corporate AI development—specifically, that companies like Google and Meta would steer the technology towards deleterious, maybe even catastrophically unsafe, outcomes for society. Their founders proclaimed that their new labs, uniquely, could be trusted to develop the technology in humanity’s best interest. But each, in turn, were themselves co-opted by the same market incentives, themselves becoming corporate behemoths zealously guarding future investor value rather than the public interest...

As someone who is constantly exploring ways to make AI applications faster and cheaper, I found...

A hands-on test of BGE-M3 + Qwen3 (RAG vs. direct-context answering) on a real research paper and a full-length book including a retrieval bug hiding in a footnote, and one surprisingly good model behavior. I wanted to answer a simple question: when you feed a document to an AI model, is it actually reading it or just pattern-matching to whatever text happens to look similar to your question? So I built a small open-source pipeline to test this directly. For any document and question, it generates two separate answers: RAG answer: BGE-M3 finds the most relevant chunks of the document, and Qwen3 answers using only those chunks. Direct answer: Qwen3 reads the raw document text directly, no retrieval involved. Both run on a free Google Colab GPU. I kept the retrieval side deliberately "vanilla" fixed-size chunking, plain cosine similarity, no reranking, no fancy tricks so I could see exactly where the basic version breaks before adding any fixes. Before running my first real test, I already knew one thing to guard against: reference lists. Early experimentation (not covered here) showed that a paper's bibliography, once chunked like any other text, can get retrieved as if it were real content a citation for a paper about "text embeddings" can look deceptively similar to a generic question about a document's topic. So going in, my pipeline already strips everything after a References/Bibliography heading before chunking. With that fix in place, I ran two real tests. Test 1: A research paper on Nepali legal machine translation First document: a SIGUL 2024 workshop paper on a bidirectional English-Nepali machine translation system for the legal domain. Question: "What is this paper about?" RAG answer: This paper presents the first transformer-based bidirectional machine translation system for the English-Nepali legal domain, using a custom-built parallel corpus of 125,000 sentences. It achieves encouraging BLEU scores and addresses the scarcity of domain-specific legal tr

Per-tenant cost visibility changes the design: don't begin with parallel API calls; begin with a durable ledger that ties every classification result and usage record to a tenant, policy version, and source item. For a customer-support catalog backfill, the practical choice is a bounded Node.js worker that reads existing posts and comments, classifies them through a replaceable adapter, checkpoints each result, and exports tenant-scoped JSONL. Short answer: make the ledger the product of the job and the LLM call one restartable step inside it. That ordering matters when support conversations contain messy product descriptions such as “the small blue charger for the old tablet.” The moderation label decides whether the text is safe to reuse; the enrichment labels connect it to a catalog candidate. Operations still need to answer a less glamorous question: which tenant consumed the tokens? Make tenant cost visible before optimizing it Token totals belong beside decisions, not in an unrelated monthly dashboard. Record normalized input and output token counts on every completed row, then aggregate by tenantId , policyVersion , and time window. If the API reports different usage units, preserve the raw usage payload in restricted telemetry and map it explicitly; don't pretend unlike units are interchangeable. Start there. Three signals are enough for the first useful view: Signal Group by Operational question Completed items tenant, policy version Is the backfill moving? Input and output tokens tenant, model Where is consumption occurring? Review and block counts tenant, content kind Did the decision mix shift? Cost in currency should be derived from a versioned rate configuration, not baked into historical rows. Store usage and the model identifier, then apply the applicable rate when producing a report. This keeps a rate change from rewriting what the runtime actually observed. It also lets finance reproduce an invoice-period view while engineering inspects tokens per

There's a comfortable assumption behind a lot of "agent + knowledge base" work: garbage in, garbage out. Feed an agent a messy, stale, duplicate-ridden wiki and it'll confidently tell users the wrong thing. So we invest in dedup, freshness, clean ingestion — to stop the agent from hallucinating. I built a small testbed to measure that assumption, and it's wrong. Or rather: it's wrong about how bad ingest hurts, and the real answer is more interesting — and harder to catch. The setup I built a tiny agent that navigates a wiki the way a person does: it has a wiki_search tool and a wiki_read tool, it decides what to look up, reads a page, and answers. No vector database, no RAG injection — the agent navigates . (This is increasingly how capable models prefer to work: they know what they need better than a query-time embedding match.) The wiki is ten markdown pages of deliberately synthetic facts — invented services, made-up numbers the model cannot possibly know from training ("the Orion canary ring holds for 45 minutes before auto-promoting"). That last part matters: because the facts are synthetic, an agent with no wiki genuinely can't answer, so any success is attributable to the wiki, not to the model reciting what it already knew. First, does the wiki help at all? Baseline (no wiki tools) scored 0/4 . Augmented (wiki tools) scored 4/4 . A clean +100-point lift. Good — the testbed works, and the tasks are honest. ( the runner ) Then I started degrading the ingest quality and measuring what happened. The degradation I built three versions of the wiki: clean — the authoritative pages only. stale-present — each answer's page now has a contradicting duplicate (a page claiming the canary holds for 30 minutes, not 45), ranked below the real page. This simulates auto-ingest pulling in an old copy. stale-outranks — the same contradiction, but keyword-stuffed so it outranks the authoritative page in search. This simulates the very common failure where a spammy or verbose st

I created an open web resource for Software Engineers. https://loveyourclanker.org/ It highlights different patterns we can consciously choose use when interacting with our AI Coding tools (a.k.a 'Agents'... a.k.a 'Clankers') to stay in control, maintain quality and sensibly increase efficiency. I was prompted to do this (no pun intended) by observing some pretty alarming signals coming from this community. Token leaderboards, engineers being encouraged to use tools to "stay current" or "keep up" or "not be redundant", engineers quitting tools entirely to stay sane, engineers leaving social gatherings to get back to their agents, engineers setting up whole systems that automate away human engineers and then calling that "agentic engineering". I'm hoping that if we normalise and share how we use the tools, and show that there are different ways where you maintain more control and agency (... pun?) that it might promote a better If you find it helpful, share. If you disagree or want to contribute, raise a PR or ping me. It's all open and NFP.

Scheduled LLM curation is the least glamorous agent workload you run. A cron job wakes up at 3am, reads a pile of memory, asks a model to dedupe it, summarize it, re-rank it, and writes the result back. Nobody is watching. There's no chat window, no streaming tokens, no human to click a button. It just has to work, quietly, every night. That "nobody is watching" part is exactly what makes the cloud-versus-local decision harder than it looks. When you have a human in the loop, a failed API call throws an error you can see and retry. In a headless cron context, the same failure turns into a job that hangs on an approval prompt no one will ever answer, or a pod that curated three months of context into an emptyDir that vanished on restart. I've run curation both ways: nightly jobs hitting a hosted API, and the same logic pointed at a local model on my Kubernetes cluster. Both work. They fail differently, cost differently, and demand different things from you operationally. Here's the actual tradeoff, not the marketing version. The decision point You reach this fork once your agent memory stops being a toy. Early on, you curate by hand or with a cheap synchronous call inside your agent loop. Then the memory grows, the curation gets expensive, and you pull it out into a scheduled job so it runs off the critical path. Now you're paying an API on a timer, and two things start to bug you. First, the data. Curation reads your entire memory store to make decisions. If that memory contains anything you'd rather not stream to a third party (internal notes, customer context, infrastructure details), every scheduled run ships it over the wire. I wrote about the general version of this problem in privacy-routed LLM inference , and scheduled curation is the workload where it bites hardest, because it touches everything, repeatedly, forever. Second, the cost shape. A curation pass over a large vector store is a lot of tokens for a job that produces no user-facing latency benefit. Yo

A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g * — a Graviton2 (aarch64) host with an NVIDIA T4G (Turing, SM 7.5) GPU. Three obstacles: an arch list nobody publishes for this combination, a version floor that only the newest vLLM clears, and 64 KiB of shared memory that stops the model dead. Plus the seven things I documented wrong before I had a box.* Model google/gemma-4-E2B-it (reference bf16 release) Hardware AWS EC2 g5g.4xlarge — Graviton2 + 1x NVIDIA T4G, compute capability 7.5 , 15,360 MiB Base image Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12 (Ubuntu 24.04) Software torch 2.12.0+cu132 · CUDA 13.2 · vLLM v0.27.2rc0 built from source for sm_75 Result 43.1 tok/s single-stream greedy, 329,579-token KV cache — after one patch to vLLM G5g is the only instance AWS has ever shipped that puts an NVIDIA GPU behind a Graviton host. It launched in 2020, it never got a successor, and Graviton is now on its fifth generation without one. That matters more than it sounds. The Arm-plus-CUDA world moved on to NVIDIA's own Arm CPU — Grace, paired with SM 9.0 and 10.0 parts. Turing stayed well supported, on x86. G5g is the only hardware that is aarch64 and compute capability 7.5, and almost nobody publishes a build for that combination. I put a rig on one anyway. The packaging problem was the quick part. Everything after it — a compiler that was not there, a version floor I did not expect, and 32 KiB of shared memory — took far longer, because none of it fails where you are looking. No published build covers aarch64 and SM 7.5 together Start with the obvious candidate. vllm/vllm-openai:v0.27.1 publishes both platforms under one tag, and you can read the arch lists straight out of the image config without pulling a layer: docker buildx imagetools inspect vllm/vllm-openai:v0.27.1 --format '{{json .Image}}' linux/amd64 7.5 8.0 8.6 8.9 9.0 10.0 12.0 linux/arm64 8.0 8.7 8.9 9.0 10.0 11.0 12.0 The one architecture this hardware needs is the only entry

A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g * — a Graviton2 (aarch64) host with an NVIDIA T4G (Turing, SM 7.5) GPU. Three obstacles: an arch list nobody publishes for this combination, a version floor that only the newest vLLM clears, and 64 KiB of shared memory that stops the model dead. Plus the seven things I documented wrong before I had a box.* Model google/gemma-4-E2B-it (reference bf16 release) Hardware AWS EC2 g5g.4xlarge — Graviton2 + 1x NVIDIA T4G, compute capability 7.5 , 15,360 MiB Base image Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12 (Ubuntu 24.04) Software torch 2.12.0+cu132 · CUDA 13.2 · vLLM v0.27.2rc0 built from source for sm_75 Result 43.1 tok/s single-stream greedy, 329,579-token KV cache — after one patch to vLLM G5g is the only instance AWS has ever shipped that puts an NVIDIA GPU behind a Graviton host. It launched in 2020, it never got a successor, and Graviton is now on its fifth generation without one. That matters more than it sounds. The Arm-plus-CUDA world moved on to NVIDIA's own Arm CPU — Grace, paired with SM 9.0 and 10.0 parts. Turing stayed well supported, on x86. G5g is the only hardware that is aarch64 and compute capability 7.5, and almost nobody publishes a build for that combination. I put a rig on one anyway. The packaging problem was the quick part. Everything after it — a compiler that was not there, a version floor I did not expect, and 32 KiB of shared memory — took far longer, because none of it fails where you are looking. No published build covers aarch64 and SM 7.5 together Start with the obvious candidate. vllm/vllm-openai:v0.27.1 publishes both platforms under one tag, and you can read the arch lists straight out of the image config without pulling a layer: docker buildx imagetools inspect vllm/vllm-openai:v0.27.1 --format '{{json .Image}}' linux/amd64 7.5 8.0 8.6 8.9 9.0 10.0 12.0 linux/arm64 8.0 8.7 8.9 9.0 10.0 11.0 12.0 The one architecture this hardware needs is the only entry

El 7 de abril de 2026 escribí el primer commit de lo que iba a ser mi orquestador de agentes. Era, básicamente, una pantalla. Un servidor que gestionaba varios proyectos a la vez y desde el cual podía disparar tareas de un agente de código, con un tablero al medio que mostraba en qué etapa estaba cada cosa. Si me hubieran preguntado ese día cuál era el problema que estaba resolviendo, habría contestado sin dudar: ver y lanzar . Necesitaba un lugar desde donde disparar el trabajo y mirar cómo avanzaba. Cuatro meses después, con más de dos mil tareas cerradas por ese sistema, puedo decir que esa respuesta estaba equivocada, y que el primer indicio de por qué llegó a los tres días. Los dos primeros días fueron todos de interfaz Si miro el historial de esa primera semana, es casi cómico. El ancho del panel lateral. Los tooltips con las fechas completas al pasar el mouse. Los badges de "en progreso" sobre cada etapa. Los colores por etapa del pipeline, para que se distinguieran de un vistazo. Hay un par de commits consecutivos que me gusta especialmente como retrato de ese momento. El primero pone un emoji como ícono del botón de repetición. El segundo lo reemplaza por un carácter Unicode, porque el emoji ignoraba el color que le definía por CSS y se veía siempre igual, sin importar el estado. No lo cuento para burlarme de mí mismo. Lo cuento porque es exactamente cómo se ve un proyecto cuando todavía no sabés cuál es el problema. Estaba puliendo la superficie del sistema con mucho cuidado porque la superficie era lo único que tenía enfrente. La pregunta de fondo —qué parte de este flujo tiene que decidir un modelo y qué parte no— ni siquiera me la había hecho. El 10 de abril cambió el foco Para entonces el pipeline ya tenía forma: una cadena de pasos donde un agente elegía la próxima tarea pendiente, la implementaba y después la marcaba como terminada. Los tres pasos los hacía el modelo, porque los tres estaban escritos como instrucciones dentro de las habilidades que l