标签：#gcp

Serving Gemma 4 E2B on a TPU v6e-1 A Cloud TPU v6e-1 (Trillium) costs 2.25× a v5e-1 and returns 1.62–1.68× the throughput on workloads that fit in a v5e, and 2.32–2.77× on workloads that do not. Per output token that makes v6e 34–39% dearer in the first regime and 3–19% cheaper in the second — so the case for the bigger chip is narrower than the memory ratio suggests, and break-even sits at roughly 270,000 KV tokens. v6e is not a general upgrade over v5e. It is a memory upgrade sold at a compute price: 32 GB against 16, a KV pool of 1,151,744 tokens against 321,376 (3.6×) , for 1.907× the bandwidth. Where the extra memory does nothing, the workload pays 2.25× for 1.6×. Two findings drive the rest: There is no capacity knee at any occupancy tested. TTFT = −8542 + 265 × concurrency , R² = 0.999996 , across 56% to 157% of the KV pool, with num_preemptions_total = 0 in every cell. A line fitted entirely below 100% occupancy predicts 157% to within 0.13%. Spot is more expensive than flex-start on this chip — $1.4033 against $1.35/chip-hr in us-east5, reversing the v5e ordering. The cheaper option is also the preemption-free one that stops billing by itself. Configuration: v6e-1 ( ct6e-standard-1t , one Trillium chip), vllm/vllm-tpu:nightly , vLLM 0.26.1rc1.dev256+gf5bb701fa , tpu-inference JAX backend, google/gemma-4-E2B-it at bf16, TP=1, max_model_len 32768, max_num_batched_tokens 4096, kv_cache_dtype=auto , prefix caching on. OUTPUT_LEN 128 throughout. v5e-1 comparison figures are from the same model and engine family on v5litepod-1 and are not a controlled A/B — read them as shape, not delta. Part 1 — Getting a v6e-1 at all 1.1 The gcloud spelling table On v5e, v5e is spelled v5litepod to gcloud. On v6e the marketing name and the CLI value coincide — which teaches a habit that breaks on the next chip. Context v5e single chip v6e single chip Prose, directory names v5e-1 / v5e1 v6e-1 / v6e1 --accelerator-type v5litepod-1 v6e-1 Flex-start runtime version v2-alpha-tpuv5-l

The Problem with Fully Autonomous Remediation Every platform team eventually asks the same question: can we let something automatically fix production when it breaks? The instinct to say yes is understandable incidents at 3 a.m. are expensive, and a lot of Kubernetes failures follow recognizable patterns. But fully autonomous remediation has a bad failure mode: when the agent is wrong, it's wrong fast, and it's wrong at scale. AIOps agents for Kubernetes solve this by splitting the problem in two: let the agent do the work of detection, correlation, and proposal the parts humans are slow and inconsistent at and keep a human as the final decision-maker for anything with real consequences. This is the human-in-the-loop (HITL) model, and on Google Cloud it maps cleanly onto existing primitives: GKE for the runtime, Cloud Monitoring/Logging for signal, IAM and Kubernetes RBAC for guardrails, and Vertex AI or a self-hosted model for the reasoning layer. What the Agent Actually Does Strip away the buzzwords and an AIOps agent for Kubernetes does four things on a loop: Watch — consume events, metrics, and logs from the cluster and surrounding GCP services Correlate — connect a symptom (say, elevated 5xx rate) to a likely cause (a bad rollout, a starved node, an expired credential) Propose — generate one or more candidate remediations, each with a confidence score and an estimate of blast radius Act or Ask — execute directly if the action is pre-approved as low-risk, otherwise route to a human for a decision The engineering effort is disproportionately in steps 2 and 4. Step 2 (correlation) requires the agent to reason over multiple, often noisy signal sources rather than pattern-match a single metric. Step 4 (the human gate) requires a review surface good enough that a tired on-call engineer can make a correct decision in seconds, not minutes. Core Signals on GKE The Approval Gate, Concretely The human-in-the-loop gate is usually a chat-based approval flow, since on-call e

Google just made multi-region Cloud Run significantly easier. Here is the full picture; what changed, what it means in practice, and how to build it right. Most teams discover they need multi-region architecture the hard way and sadly, during an outage. Whether you're running a global e-commerce platform, a real-time gaming API, or a financial services application, users expect your service to be available whenever they need it. There is a conversation that happens in almost every engineering team at some point. It usually starts with a post-mortem. A regional Google Cloud outage or a Cloud Run service that hit a cold start spike, or a single-region deployment that could not handle the latency demands of users spread across Lagos, Nairobi, and London simultaneously, caused enough pain that someone finally asked: why are we only deployed in one region? The answer is usually one of three things: it felt complex, it felt expensive, or no one had prioritised it yet. In July 2026, Google moved Cloud Run Service Health to General Availability and the timing was hard to miss. Six days earlier, a power cut at Google's Netherlands data centre had knocked three services offline. The GA release brings automatic cross-region failover to Cloud Run with what Google describes as a two-step setup: add a readiness probe, set minimum instances to at least 1. The load balancer does the rest. This article covers the full architecture, what Service Health is, how readiness probes underpin it, how to set up the Global Load Balancer correctly, and how to test that failover actually works. It also covers the production details. What changed: Service Health and readiness probes Before Service Health, multi-region Cloud Run required you to implement a /health endpoint in your application and configure a separate HTTPS health check at the load balancer level. This worked, but it had a significant gap. The load balancer's health check only knew whether the Cloud Run service endpoint was respon

Using Firestore as "the app's primary database" is easy at first. Writes and reads complete in one place, and onSnapshot gives you realtime push for free. But as a service grows, keeping Firestore as the SoT (source of truth) exposes some fatal constraints. Where SoT-Firestore hurts Weak transaction boundaries, missing complex queries, the cost structure, the difficulty of migrating away. The harshest one: you cannot narrow related updates with a WHERE . For the use case "update a set of documents matching a condition, consistently, in one go," Firestore is structurally weak. Redefining it as a public cache In one project, I made Cloud SQL the SoT and treated Firestore as a "read-optimized projection." Writes go through the backend (Next.js / Cloud Run), and the SoT (Cloud SQL via Data Connect) and Firestore are both updated within the same write path . Separately, a Cloud Run reconciliation job runs and checks and repairs consistency against the SoT as authoritative. This reconciliation job is enqueued at the start of the request, before the DB updates. Because it's queued first, even if the write dies halfway, the consistency check always runs afterward and repairs the state. From the client's perspective, Firestore is a rebuildable cache — in the worst case it can be rebuilt from the SoT. // Writes go through the backend. The backend updates both the SoT // (Data Connect → Cloud SQL) and Firestore. Clients never write Firestore directly. await backend . updateEntity ({ id , title }); // Changes are pushed back in realtime via Firestore's onSnapshot unsubscribe = onSnapshot ( entityRef , ( snap ) => render ( snap . data ())); Benefits and costs The benefits are clear. The SoT side (SQL) brings transactional consistency and complex queries; the read side (Firestore) brings freedom in data modeling and realtime push; and consistent syncing with external systems (OpenSearch / BigQuery, etc.) coexists naturally. Since the backend updates Firestore at write time, the f

In production we have ~1.5 TB of full-text court decisions and their vector embeddings, plus another ~550 GB of other legal data: registries, legislation, business entities, a Spanish case law corpus, EU-Lex. If we take this corpus and train an MoE model the size of DeepSeek V3, scaled to 860B parameters, on GCP — what comes out? We break down the dataset, architecture, compute cost, and the properties such a model would have on Ukrainian law. What's in the Dataset The entire corpus is what's already running in SecondLayer's production. No extra scrapes, no Common Crawl, no noise. EDRSR — the dataset core, ~1.5 TB. The Unified State Register of Court Decisions of Ukraine. 96.2 million full-text decisions (1,079 GB in PostgreSQL TOAST), 471 GB of vectors in Qdrant (voyage-3.5, 1024-dim), 28 GB of metadata (court, judge, date, case category, proceeding type, statute code). Breakdown by jurisdiction: civil 33.7M, administrative 14M+, criminal 12M+, commercial 6M+, misdemeanors 6M+. Largest annual cohort — 2024 (115 GB of TOAST text). OpenReyestr — 43 GB. Ukrainian public registries: 16.7M legal entities (EDR), ownership structures (beneficiaries, shareholders), debtors (State Enforcement Service), NAIS registries. This is the foundation for SneakyPiper — our due-diligence platform — but here it serves as raw corpus for the model. Legislation — ~40 GB. The Constitution, major codes (Civil, Criminal, Criminal Procedure, Civil Procedure, Commercial Procedure, Administrative Procedure, Labor, Tax, Customs), laws, and secondary legislation. All structurally annotated: articles, parts, clauses, revision dates with effective-date tracking. This isn't flat text: we know that Article 124 of the Constitution took effect on a specific date, carries particular references, and is cited in a precise number of decisions. Supreme Court review practices + lu_court_decisions — ~25 GB. SC plenary decisions, practice overviews, Grand Chamber rulings. This is the most valuable slice — the

The difficulty If you’re using Google Cloud, you’re probably paying for stuff you don’t need anymore—a server someone forgot to turn off, a storage disk someone forgot to delete, or an IP address someone forgot to release. "That waste is costing us money." It consumes electricity. It generates carbon emissions. Almost nobody tracks those alongside the cost. I wanted a tool that could find both problems simultaneously, without me having to think. So I made one. Live demo: https://greenops-dashboard-845589445410.us-central1.run.app/ Code: https://github.com/raghu-putta/greenops-agent How it works: 4 AI agents in a row Imagine an assembly line in which each worker has only one job: Carbon Scout scans your Google Cloud project and reports on everything that looks like it might be unused or forgotten—idle servers, unattached storage disks, and unused reserved IP addresses. GreenOps Analyzer takes that list and adds two numbers to each item: how much it’s costing you in dollars, and how much it’s costing the planet in carbon emissions. Optimization Executor takes each finding and makes it an action: stop this server, delete this disk, release this address. It shows you the plan first; it only changes if you say go. Report Generator rolls up the entire run into a downloadable report so you (or your boss or your sustainability team) have something nice to look at. Powered by: FastAPI (the web framework), Google’s Agent Development Kit (a library for building AI agents), Gemini 2.5 Pro (the AI model doing the reasoning), and Google Cloud Run (where it’s all running). The “bug” that was not a bug: The dashboard has a terminal-style live window that shows you what each agent is doing in real time, using a technique called Server-Sent Events (basically, the server streams updates to your browser one at a time instead of making you refresh). And then at some point that window just stopped showing anything. Nothing. My backend logs told me that the updates were still being sent,

It's important to monitor Dataform - jobs executed by Dataform can be the primary source of BigQuery costs in a modern data platform. Forgetting to incrementalise a table, using a table instead of a view in the wrong place or performing complex window functions on a large table can all incur large costs and long run times. Using the WorkflowInvocationAction for each job we can extract its BigQuery Job ID, then extract key metadata for each BigQuery job by querying INFORMATION_SCHEMA.JOBS_BY_PROJECT , before writing the output back to BigQuery so that it can be analysed (maybe even by transforming it in Dataform). from google.cloud import dataform_v1 from google.cloud import bigquery from datetime import datetime # ------------------------------------------------------------ # CONFIG # ------------------------------------------------------------ PROJECT_ID = " my-project " REGION = " europe-west2 " REPOSITORY_ID = " analytics " WORKFLOW_INVOCATION_ID = " 123456789 " BQ_REGION = " region-europe-west2 " OUTPUT_TABLE = " my-project.raw_dataform_monitoring.raw_dataform_bigquery_metrics " # ------------------------------------------------------------ # CLIENTS # ------------------------------------------------------------ dataform = dataform_v1 . DataformClient () bq = bigquery . Client ( project = PROJECT_ID ) repository = dataform . repository_path ( PROJECT_ID , REGION , REPOSITORY_ID ) invocation_name = f " { repository } /workflowInvocations/ { WORKFLOW_INVOCATION_ID } " # ------------------------------------------------------------ # 1. GET WORKFLOW INVOCATION ACTIONS → EXTRACT JOB IDS # ------------------------------------------------------------ job_ids = set () actions = dataform . list_workflow_invocation_actions ( parent = invocation_name ) for action in actions : # only BigQuery actions contain job metadata if hasattr ( action , " bigquery_action " ) and action . bigquery_action : if action . bigquery_action . job_id : job_ids . add ( action . bigquery_action

Building a Benchmarking Agent with A2A and MCP This tutorial aims to build and test benchmarking Agents using the A2A protocol across several mainstream Cloud providers. A Master Orchestrator Agent is exposed via MCP to allow Antigravity CLI to be used as a MCP client to co-ordinate the benchmarks. Deja Vu — What is Old is New! This paper is a re-visiting of the original benchmark series with Gemini CLI over Node, GO, and Python: Cross Language A2A Agent Benchmarking with Gemini 3 and Gemini CLI In this updated version, the Antigravity CLI is used to push Rust Agents cross-cloud and co-ordinate Mersenne Prime Calculations. Why would I need Multi-Cloud Support? And Rust? Can’t I just use Python? Most mature Agent development tools and libraries are Python based. Python allows for rapid prototyping and evaluation of approaches. Python is also an interpreted language- which has trade-offs in memory safety, and performance. Other languages like GO and Rust offer high performance and memory safe operations. With a language neutral communication protocol — the actual Agent implementation of each Agent can be coded in the most appropriate language. What is this Approach actually Benchmarking? The high level goal was to measure the actual time spent running an algorithm in the native language code inside the A2A agent. Each language had a slightly different implementation due to the language syntax. After running the algorithm- each Agent was instructed to calculate and return the elapsed time for cross cloud comparison. What is the A2A protocol? The Agent2Agent (A2A) protocol, an open communication standard for AI agents, was initially introduced by Google in April 2025. It is specifically engineered to facilitate seamless interoperability within multi-agent systems, enabling AI agents developed by diverse providers or built upon disparate AI agent frameworks to communicate and collaborate effectively. A good overview of the A2A protocol can be found here: A2A Protocol Lan