标签：#tpu

Serving Gemma 4 E2B on a TPU v6e-1 A Cloud TPU v6e-1 (Trillium) costs 2.25× a v5e-1 and returns 1.62–1.68× the throughput on workloads that fit in a v5e, and 2.32–2.77× on workloads that do not. Per output token that makes v6e 34–39% dearer in the first regime and 3–19% cheaper in the second — so the case for the bigger chip is narrower than the memory ratio suggests, and break-even sits at roughly 270,000 KV tokens. v6e is not a general upgrade over v5e. It is a memory upgrade sold at a compute price: 32 GB against 16, a KV pool of 1,151,744 tokens against 321,376 (3.6×) , for 1.907× the bandwidth. Where the extra memory does nothing, the workload pays 2.25× for 1.6×. Two findings drive the rest: There is no capacity knee at any occupancy tested. TTFT = −8542 + 265 × concurrency , R² = 0.999996 , across 56% to 157% of the KV pool, with num_preemptions_total = 0 in every cell. A line fitted entirely below 100% occupancy predicts 157% to within 0.13%. Spot is more expensive than flex-start on this chip — $1.4033 against $1.35/chip-hr in us-east5, reversing the v5e ordering. The cheaper option is also the preemption-free one that stops billing by itself. Configuration: v6e-1 ( ct6e-standard-1t , one Trillium chip), vllm/vllm-tpu:nightly , vLLM 0.26.1rc1.dev256+gf5bb701fa , tpu-inference JAX backend, google/gemma-4-E2B-it at bf16, TP=1, max_model_len 32768, max_num_batched_tokens 4096, kv_cache_dtype=auto , prefix caching on. OUTPUT_LEN 128 throughout. v5e-1 comparison figures are from the same model and engine family on v5litepod-1 and are not a controlled A/B — read them as shape, not delta. Part 1 — Getting a v6e-1 at all 1.1 The gcloud spelling table On v5e, v5e is spelled v5litepod to gcloud. On v6e the marketing name and the CLI value coincide — which teaches a habit that breaks on the next chip. Context v5e single chip v6e single chip Prose, directory names v5e-1 / v5e1 v6e-1 / v6e1 --accelerator-type v5litepod-1 v6e-1 Flex-start runtime version v2-alpha-tpuv5-l

DeepSeek V4 Flash costs $0.14 per million input tokens and $0.28 per million output, with cache hits at $0.0028, and the retrained 0731 build that now ships under that name has a defect you need to route around: with thinking on (the default) and a strict json_schema , integer fields came back corrupted in 8 of 13 default-thinking runs across two independent request paths. Turning thinking off fixed every run and cut the extraction to a seventh of the tokens. We measured deepseek-v4-flash-0731 on day one: the corruption, the sharper off-switch cliff the retrain introduced, the budget floor that rescues it, the 1,024-token cache pages, and what still separates the preview build and V4 Pro . TL;DR With default thinking plus strict json_schema , deepseek-v4-flash-0731 corrupted integer fields in 8 of 13 runs on two request paths; V4 Pro corrupted 2 of 4, and only the preview stayed clean. The 0731 retrain sharpened the off-switch cliff: 2-hop math fell 6/6 to 0/6. The cache serves 1,024-token pages from roughly a 1.1K-token floor, hits 0.3 seconds after priming, and entries outlive 45 minutes. enable_thinking: false fixed every structured run at a seventh of the tokens; for 2-hop math the safe thinking budget is 256. How do the three V4 builds compare on paper and on the meter? Same tokenizer, same cache, same thinking machinery; different prices, different failure modes. Everything measured below comes from identical probes run against all three (a dash means we did not probe that cell); the day-one analysis threads focus on benchmarks, so this is the operational half of the comparison: Flash 0731 Flash preview V4 Pro List price, in / out per 1M $0.14 / $0.28 $0.14 / $0.28 $0.435 / $0.87 Cache-hit input per 1M $0.0028 $0.0028 $0.003625 Thinking default on on on Strict JSON with thinking on 5/5 corrupted (our path) 4/4 clean 2/4 corrupted 2-hop math with thinking off 0/6 2/6 4/4 thinking_budget token-exact token-exact honored (4/4 at 16) Cache pages 1,024 tokens, hit a

Measured 2026-07-21 on vllm/vllm-tpu:nightly (vLLM 0.23.1rc1.dev1076), a GCE flex-start ct6e-standard-1t (one TPU v6e chip, 32 GB HBM) in europe-west4-a. TL;DR The plain google/gemma-4-E2B-it serves well on one v6e chip; none of its QAT siblings load at all. The 2-billion-parameter "efficient" Gemma 4 sustains 213 tok/s for a single user with a 16 ms first token, scales to ~2,200 output tok/s across concurrent streams, handles OpenAI-style function calling — including parallel calls and refusal to hallucinate calls — without a miss, and answers simple vision questions accurately in ~200 ms. The QAT variants are a different story: the int4 compressed-tensors export hits an unimplemented quantization path, and the bf16 QAT export trips a loader bug — the Gemma 4 implementation demands per-layer norms that E2B's KV-sharing architecture legitimately doesn't have. Filed upstream as tpu-inference #3225 . One capability coupling to know about: with a reasoning parser configured, schema enforcement only engages when thinking is enabled — thinking-off requests sail through unconstrained with a 200 status. Config interaction, not TPU limitation; details below. 1. Getting to a serving endpoint The host is a GCE flex-start VM — capacity granted on request, billed until deleted, hard-stopped at a 4-hour max run, $1.35/chip-hour. A startup script installs Docker, pulls vllm/vllm-tpu:nightly , fetches the Hugging Face token from Secret Manager via the metadata server, and launches vLLM. Boot timeline: VM RUNNING at t+0 (200 GB boot disk — the 10 GB default cannot hold the vLLM image) → Docker installed ~t+1:00 → image pulled ~t+6:00 → weights downloaded, XLA compiled, health green ~t+8:30. Two environment quirks worth knowing: Direct SSH silently times out on some networks even when the VPC allows tcp:22 — the block is upstream of the VPC. IAP tunneling ( gcloud compute ssh --tunnel-through-iap ) rides over HTTPS and works; so does tunneling the API port with gcloud compute start-

In production we have ~1.5 TB of full-text court decisions and their vector embeddings, plus another ~550 GB of other legal data: registries, legislation, business entities, a Spanish case law corpus, EU-Lex. If we take this corpus and train an MoE model the size of DeepSeek V3, scaled to 860B parameters, on GCP — what comes out? We break down the dataset, architecture, compute cost, and the properties such a model would have on Ukrainian law. What's in the Dataset The entire corpus is what's already running in SecondLayer's production. No extra scrapes, no Common Crawl, no noise. EDRSR — the dataset core, ~1.5 TB. The Unified State Register of Court Decisions of Ukraine. 96.2 million full-text decisions (1,079 GB in PostgreSQL TOAST), 471 GB of vectors in Qdrant (voyage-3.5, 1024-dim), 28 GB of metadata (court, judge, date, case category, proceeding type, statute code). Breakdown by jurisdiction: civil 33.7M, administrative 14M+, criminal 12M+, commercial 6M+, misdemeanors 6M+. Largest annual cohort — 2024 (115 GB of TOAST text). OpenReyestr — 43 GB. Ukrainian public registries: 16.7M legal entities (EDR), ownership structures (beneficiaries, shareholders), debtors (State Enforcement Service), NAIS registries. This is the foundation for SneakyPiper — our due-diligence platform — but here it serves as raw corpus for the model. Legislation — ~40 GB. The Constitution, major codes (Civil, Criminal, Criminal Procedure, Civil Procedure, Commercial Procedure, Administrative Procedure, Labor, Tax, Customs), laws, and secondary legislation. All structurally annotated: articles, parts, clauses, revision dates with effective-date tracking. This isn't flat text: we know that Article 124 of the Constitution took effect on a specific date, carries particular references, and is cited in a precise number of decisions. Supreme Court review practices + lu_court_decisions — ~25 GB. SC plenary decisions, practice overviews, Grand Chamber rulings. This is the most valuable slice — the