标签：#tpu

Measured 2026-07-21 on vllm/vllm-tpu:nightly (vLLM 0.23.1rc1.dev1076), a GCE flex-start ct6e-standard-1t (one TPU v6e chip, 32 GB HBM) in europe-west4-a. TL;DR The plain google/gemma-4-E2B-it serves well on one v6e chip; none of its QAT siblings load at all. The 2-billion-parameter "efficient" Gemma 4 sustains 213 tok/s for a single user with a 16 ms first token, scales to ~2,200 output tok/s across concurrent streams, handles OpenAI-style function calling — including parallel calls and refusal to hallucinate calls — without a miss, and answers simple vision questions accurately in ~200 ms. The QAT variants are a different story: the int4 compressed-tensors export hits an unimplemented quantization path, and the bf16 QAT export trips a loader bug — the Gemma 4 implementation demands per-layer norms that E2B's KV-sharing architecture legitimately doesn't have. Filed upstream as tpu-inference #3225 . One capability coupling to know about: with a reasoning parser configured, schema enforcement only engages when thinking is enabled — thinking-off requests sail through unconstrained with a 200 status. Config interaction, not TPU limitation; details below. 1. Getting to a serving endpoint The host is a GCE flex-start VM — capacity granted on request, billed until deleted, hard-stopped at a 4-hour max run, $1.35/chip-hour. A startup script installs Docker, pulls vllm/vllm-tpu:nightly , fetches the Hugging Face token from Secret Manager via the metadata server, and launches vLLM. Boot timeline: VM RUNNING at t+0 (200 GB boot disk — the 10 GB default cannot hold the vLLM image) → Docker installed ~t+1:00 → image pulled ~t+6:00 → weights downloaded, XLA compiled, health green ~t+8:30. Two environment quirks worth knowing: Direct SSH silently times out on some networks even when the VPC allows tcp:22 — the block is upstream of the VPC. IAP tunneling ( gcloud compute ssh --tunnel-through-iap ) rides over HTTPS and works; so does tunneling the API port with gcloud compute start-

In production we have ~1.5 TB of full-text court decisions and their vector embeddings, plus another ~550 GB of other legal data: registries, legislation, business entities, a Spanish case law corpus, EU-Lex. If we take this corpus and train an MoE model the size of DeepSeek V3, scaled to 860B parameters, on GCP — what comes out? We break down the dataset, architecture, compute cost, and the properties such a model would have on Ukrainian law. What's in the Dataset The entire corpus is what's already running in SecondLayer's production. No extra scrapes, no Common Crawl, no noise. EDRSR — the dataset core, ~1.5 TB. The Unified State Register of Court Decisions of Ukraine. 96.2 million full-text decisions (1,079 GB in PostgreSQL TOAST), 471 GB of vectors in Qdrant (voyage-3.5, 1024-dim), 28 GB of metadata (court, judge, date, case category, proceeding type, statute code). Breakdown by jurisdiction: civil 33.7M, administrative 14M+, criminal 12M+, commercial 6M+, misdemeanors 6M+. Largest annual cohort — 2024 (115 GB of TOAST text). OpenReyestr — 43 GB. Ukrainian public registries: 16.7M legal entities (EDR), ownership structures (beneficiaries, shareholders), debtors (State Enforcement Service), NAIS registries. This is the foundation for SneakyPiper — our due-diligence platform — but here it serves as raw corpus for the model. Legislation — ~40 GB. The Constitution, major codes (Civil, Criminal, Criminal Procedure, Civil Procedure, Commercial Procedure, Administrative Procedure, Labor, Tax, Customs), laws, and secondary legislation. All structurally annotated: articles, parts, clauses, revision dates with effective-date tracking. This isn't flat text: we know that Article 124 of the Constitution took effect on a specific date, carries particular references, and is cited in a precise number of decisions. Supreme Court review practices + lu_court_decisions — ~25 GB. SC plenary decisions, practice overviews, Grand Chamber rulings. This is the most valuable slice — the