Running Gemma 4 on EC2 G5g: Graviton2 AMD with NVIDIA GPU
A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g * — a Graviton2 (aarch64) host with an NVIDIA T4G (Turing, SM 7.5) GPU. Three obstacles: an arch list nobody publishes for this combination, a version floor that only the newest vLLM clears, and 64 KiB of shared memory that stops the model dead. Plus the seven things I documented wrong before I had a box.* Model google/gemma-4-E2B-it (reference bf16 release) Hardware AWS EC2 g5g.4xlarge — Graviton2 + 1x NVIDIA T4G, compute capability 7.5 , 15,360 MiB Base image Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12 (Ubuntu 24.04) Software torch 2.12.0+cu132 · CUDA 13.2 · vLLM v0.27.2rc0 built from source for sm_75 Result 43.1 tok/s single-stream greedy, 329,579-token KV cache — after one patch to vLLM G5g is the only instance AWS has ever shipped that puts an NVIDIA GPU behind a Graviton host. It launched in 2020, it never got a successor, and Graviton is now on its fifth generation without one. That matters more than it sounds. The Arm-plus-CUDA world moved on to NVIDIA's own Arm CPU — Grace, paired with SM 9.0 and 10.0 parts. Turing stayed well supported, on x86. G5g is the only hardware that is aarch64 and compute capability 7.5, and almost nobody publishes a build for that combination. I put a rig on one anyway. The packaging problem was the quick part. Everything after it — a compiler that was not there, a version floor I did not expect, and 32 KiB of shared memory — took far longer, because none of it fails where you are looking. No published build covers aarch64 and SM 7.5 together Start with the obvious candidate. vllm/vllm-openai:v0.27.1 publishes both platforms under one tag, and you can read the arch lists straight out of the image config without pulling a layer: docker buildx imagetools inspect vllm/vllm-openai:v0.27.1 --format '{{json .Image}}' linux/amd64 7.5 8.0 8.6 8.9 9.0 10.0 12.0 linux/arm64 8.0 8.7 8.9 9.0 10.0 11.0 12.0 The one architecture this hardware needs is the only entry