标签：#gke

Upgrading Kubernetes clusters across a large enterprise fleet is often a balancing act between staying current with security patches and avoiding outages. By default, Google Kubernetes Engine (GKE) rolls out automatic upgrades progressively according to Google Cloud regional timelines. While regional rollout works well for standalone clusters, it does not understand your organization's business topology. If you run staging clusters in us-central1 and critical production clusters in us-east1 , a standard regional rollout could upgrade your production environment before your pre-production validation completes. The General Availability (GA) release of GKE rollout sequencing with custom stages solves this challenge. It provides platform teams with declarative control to sequence cluster upgrades across fleets, environments, and even distinct Google Cloud organizations according to business criticality rather than cloud geography. How rollout sequencing works Rollout sequencing builds on GKE fleet management. Fleets serve as logical boundaries for environments such as development, staging, and production. With rollout sequencing, you define an ordered pipeline of upgrade stages managed by a central resource called RolloutSequence . When GKE publishes a new automatic upgrade target for a release channel, or when you explicitly trigger a target version, the system creates a Rollout object. This rollout progresses through your defined stages sequentially: Control plane upgrades start in the first stage. Once all control planes in that stage reach the target version, a stage soak timer begins. Node upgrades run in parallel with control plane upgrades, respecting node pool upgrade strategies such as surge or blue-green. When both control planes and nodes complete their upgrade and satisfy the configured soak duration, the rollout advances to the next stage in the sequence. If an individual stage contains clusters that take longer than 30 days to finish upgrading—due to restr

Learn strategies for building interrupt-resilient AI workloads on Google Kubernetes Engine (GKE).

You’ve built your model, your training code is containerized, and you’re ready to scale up on Google Kubernetes Engine (GKE). You go to provision your nvidia-h100-80gb node pool and... QUOTA_EXCEEDED. It’s one of the most common (and frustrating) roadblocks in modern AI development. High-end accelerators like H100s, A100s, and TPUs are in massive demand, and securing permanent, on-demand quota for them can be difficult. But a lack of on-demand quota doesn't mean you're out of options. GKE provides two powerful, cost-effective strategies for acquiring these scarce resources when you can't get standard, on-demand instances: Spot VMs and the Dynamic Workload Scheduler (DWS) . Let's break down what they are, when to use each, and how to implement them. Strategy 1: Spot VMs Spot VMs are Google Cloud's excess compute capacity sold at a massive discount, up to 90% off the price of standard on-demand VMs. They are perfect for workloads that can be interrupted. The catch is that Spot VMs have no availability guarantee. Google Cloud can "preempt" (i.e., terminate) them at any time if that capacity is needed for on-demand customers. GKE gets a 30-second warning before the node is terminated. Kubernetes uses this window to gracefully shut down your application (giving non-system pods up to 15 seconds to wrap up) before the node vanishes. When to use Spot VMs for accelerators Spot VMs are ideal for workloads that are: Fault-tolerant and stateless: Your application can handle a node vanishing and having its pods rescheduled elsewhere. Batch processing: Jobs that can be easily restarted or have checkpointing built-in. CI/CD pipelines: Running tests or builds that don't need 100% uptime. How to use Spot VMs in GKE You can easily add a Spot VM node pool to your GKE Standard cluster. The key is to use Spot VMs for your workers, not your critical system pods. Create a dedicated Spot VM node pool: When creating a node pool, simply add the --spot flag and apply a taint so standard pods