标签：#cuda

This tutorial walks through installing and setting up the Rust toolchain for vLLM on an AWS EC2 G5g instance — Graviton2 (aarch64) with an NVIDIA T4G GPU — and getting vLLM's Rust frontend ( vllm-rs ) built, running, and verified . This paper is a follow-on to the original G5g Gemma 4 build. Everything below was run on the box. 🦀 Wait, vLLM has Rust in it? You betcha. Since PR #40848 (merged 2026-05-21), vLLM vendors a 14-crate Rust workspace : bench chat cmd engine-core-client llm managed-engine metrics mock-engine parser parser/python server text tokenizer tracing Edition 2024, resolver 3. Straight from the vendored rust/Cargo.toml : Crate Version Job axum 0.8.8 the HTTP server tokio 1.47.1 async runtime zeromq 0.6.0 talks to the Python engine rmp-serde / rmpv 1.3.1 msgpack on the wire minijinja 2.22 chat templates tonic / prost 0.14.6 / 0.14.3 gRPC — remember this one It's a drop-in replacement for the Python FastAPI server. Two artifacts get built: 🦀 vllm-rs — the axum frontend binary 🐍 vllm._rust_tool_parser — a PyO3 extension module Rust is a build requirement now That's the headline, and it's reason enough on its own: you cannot build vLLM from source at v0.27.2rc0 without Rust in the picture. setup.py imports it at module scope, line 21, unguarded: from setuptools_rust.build import build_rust No try , no feature flag, no opt-out. Metadata generation doesn't happen without it. And this isn't a quirk of one release. vLLM's Rust surface is 14 crates covering the HTTP frontend, the tool parser, the tokenizer and the benchmark client, and it has been growing since it landed. If you build inference infrastructure from source, a Rust toolchain is becoming table stakes — so it's worth knowing how to drive it properly rather than working around it. Three things do get conflated, though, and they have different scopes: Component Needed to build vLLM? Needed to serve? setuptools_rust (Python pkg) yes, always no cargo / rustc toolchain for working Rust artifacts no prot

A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g * — a Graviton2 (aarch64) host with an NVIDIA T4G (Turing, SM 7.5) GPU. Three obstacles: an arch list nobody publishes for this combination, a version floor that only the newest vLLM clears, and 64 KiB of shared memory that stops the model dead. Plus the seven things I documented wrong before I had a box.* Model google/gemma-4-E2B-it (reference bf16 release) Hardware AWS EC2 g5g.4xlarge — Graviton2 + 1x NVIDIA T4G, compute capability 7.5 , 15,360 MiB Base image Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12 (Ubuntu 24.04) Software torch 2.12.0+cu132 · CUDA 13.2 · vLLM v0.27.2rc0 built from source for sm_75 Result 43.1 tok/s single-stream greedy, 329,579-token KV cache — after one patch to vLLM G5g is the only instance AWS has ever shipped that puts an NVIDIA GPU behind a Graviton host. It launched in 2020, it never got a successor, and Graviton is now on its fifth generation without one. That matters more than it sounds. The Arm-plus-CUDA world moved on to NVIDIA's own Arm CPU — Grace, paired with SM 9.0 and 10.0 parts. Turing stayed well supported, on x86. G5g is the only hardware that is aarch64 and compute capability 7.5, and almost nobody publishes a build for that combination. I put a rig on one anyway. The packaging problem was the quick part. Everything after it — a compiler that was not there, a version floor I did not expect, and 32 KiB of shared memory — took far longer, because none of it fails where you are looking. No published build covers aarch64 and SM 7.5 together Start with the obvious candidate. vllm/vllm-openai:v0.27.1 publishes both platforms under one tag, and you can read the arch lists straight out of the image config without pulling a layer: docker buildx imagetools inspect vllm/vllm-openai:v0.27.1 --format '{{json .Image}}' linux/amd64 7.5 8.0 8.6 8.9 9.0 10.0 12.0 linux/arm64 8.0 8.7 8.9 9.0 10.0 11.0 12.0 The one architecture this hardware needs is the only entry

A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g * — a Graviton2 (aarch64) host with an NVIDIA T4G (Turing, SM 7.5) GPU. Three obstacles: an arch list nobody publishes for this combination, a version floor that only the newest vLLM clears, and 64 KiB of shared memory that stops the model dead. Plus the seven things I documented wrong before I had a box.* Model google/gemma-4-E2B-it (reference bf16 release) Hardware AWS EC2 g5g.4xlarge — Graviton2 + 1x NVIDIA T4G, compute capability 7.5 , 15,360 MiB Base image Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12 (Ubuntu 24.04) Software torch 2.12.0+cu132 · CUDA 13.2 · vLLM v0.27.2rc0 built from source for sm_75 Result 43.1 tok/s single-stream greedy, 329,579-token KV cache — after one patch to vLLM G5g is the only instance AWS has ever shipped that puts an NVIDIA GPU behind a Graviton host. It launched in 2020, it never got a successor, and Graviton is now on its fifth generation without one. That matters more than it sounds. The Arm-plus-CUDA world moved on to NVIDIA's own Arm CPU — Grace, paired with SM 9.0 and 10.0 parts. Turing stayed well supported, on x86. G5g is the only hardware that is aarch64 and compute capability 7.5, and almost nobody publishes a build for that combination. I put a rig on one anyway. The packaging problem was the quick part. Everything after it — a compiler that was not there, a version floor I did not expect, and 32 KiB of shared memory — took far longer, because none of it fails where you are looking. No published build covers aarch64 and SM 7.5 together Start with the obvious candidate. vllm/vllm-openai:v0.27.1 publishes both platforms under one tag, and you can read the arch lists straight out of the image config without pulling a layer: docker buildx imagetools inspect vllm/vllm-openai:v0.27.1 --format '{{json .Image}}' linux/amd64 7.5 8.0 8.6 8.9 9.0 10.0 12.0 linux/arm64 8.0 8.7 8.9 9.0 10.0 11.0 12.0 The one architecture this hardware needs is the only entry

You don't need to know anything about Go to read this. The game is just the fixed yardstick. The story is a hardware benchmark: the same program, the same problem, the same settings — only the machine changed, from a 2017 GPU cluster to a single 2026 graphics card. That makes it a rare clean measurement of one decade of progress. What "solving" means here There are two very different things a computer can do with a board game. It can play it well — that's what AlphaGo did. Or it can solve it: mathematically prove the outcome under perfect play from both sides, leaving no doubt. Solving is the hard one. You explore an enormous tree of "if I play here, they play there…" move sequences until you have an airtight proof. Each node in that tree is one position examined. The target here is a single 7x7 opening called JA . In 2023, a NeurIPS paper ( Game Solving with Online Fine-Tuning , Wu et al.) proved its verdict — the attacker cannot win — using a cluster of twelve GTX 1080Ti GPUs running 384 parallel workers. The solver is guided by a neural network that estimates how hard each branch is, and crucially that network is fine-tuned online — it keeps learning during the solve. I rebuilt that exact solver (same code, same problem, same initial model, same search settings) and ran it on one RTX 5090 . It reached the identical proof . Everything but the hardware was held fixed, so the two runs line up as a generation-vs-generation benchmark — and it doubled as a full shakedown of the new Blackwell workstation. The numbers 1x RTX 5090 (2026) 12x GTX 1080Ti (2017) ratio Worker slots 24 384 1/16 the parallelism Per-slot throughput 284 nodes/s 141 nodes/s 2.01x faster Search work to proof 1.01B nodes 1.73B nodes 0.59x (41% less work) Avg work per sub-job 4,189 nodes 6,136 nodes shallower proofs Live model updates 4,007 208 19.3x more Wall-clock time 41.4 h 8.9 h 4.64x slower Verdict loss (proven) loss identical The single card finished slower in wall-clock time (41 h vs 9 h) — b