标签：#Uber

Even in a future of robot taxis, someone still has to return the things passengers leave behind.

Uber's cutback has occurred after the company had reportedly encouraged staff to use AI as much as possible.

Learn strategies for building interrupt-resilient AI workloads on Google Kubernetes Engine (GKE).

A unicorn Beanie Baby. A 15-pound green bowling ball. A pair of dentures. These are just some of the items left behind in robotaxis in the past year, according to Uber's annual Lost and Found Index. For the first time, the company is expanding its annual of accounting of things forgotten in Uber vehicles to […]

Introduction Managed Kubernetes platforms such as Amazon EKS, Google Kubernetes Engine (GKE), and Azure Kubernetes Service (AKS) abstract away much of the operational complexity involved in running Kubernetes clusters. While this significantly improves developer productivity, it also hides many of the internal systems responsible for cluster orchestration, networking, node registration, and workload scheduling. As a result, many engineers interact with Kubernetes daily without fully understanding the components that keep a cluster operational behind the scenes. To better understand Kubernetes from an operational perspective, I set out to build and operate a self-managed Kubernetes platform on AWS using kubeadm. Unlike lightweight local environments such as Minikube or kind, kubeadm bootstraps Kubernetes in a way that closely resembles how real-world self-managed clusters are provisioned and operated. The objective of this project was not simply to install Kubernetes, but to explore: How the control plane components interact. How worker nodes register with the cluster. How Kubernetes networking behaves. How cloud integrations work. How traffic reaches workloads running inside the cluster. How operational failures surface during deployment and runtime. How production-style systems behave beneath managed abstractions. This article documents the architecture, implementation process, engineering decisions, operational lessons, and troubleshooting insights encountered during the effort to bring the platform to a healthy operational state. Project Objectives The primary objectives of this project were to: Provision infrastructure on AWS using Terraform. Bootstrap a self-managed Kubernetes cluster using kubeadm. Configure Kubernetes networking using Calico. Integrate Gateway API with AWS Load Balancer Controller. Expose workloads externally using AWS Application Load Balancers. Validate cluster functionality through application deployment. Understand the operational mechani

You’ve built your model, your training code is containerized, and you’re ready to scale up on Google Kubernetes Engine (GKE). You go to provision your nvidia-h100-80gb node pool and... QUOTA_EXCEEDED. It’s one of the most common (and frustrating) roadblocks in modern AI development. High-end accelerators like H100s, A100s, and TPUs are in massive demand, and securing permanent, on-demand quota for them can be difficult. But a lack of on-demand quota doesn't mean you're out of options. GKE provides two powerful, cost-effective strategies for acquiring these scarce resources when you can't get standard, on-demand instances: Spot VMs and the Dynamic Workload Scheduler (DWS) . Let's break down what they are, when to use each, and how to implement them. Strategy 1: Spot VMs Spot VMs are Google Cloud's excess compute capacity sold at a massive discount, up to 90% off the price of standard on-demand VMs. They are perfect for workloads that can be interrupted. The catch is that Spot VMs have no availability guarantee. Google Cloud can "preempt" (i.e., terminate) them at any time if that capacity is needed for on-demand customers. GKE gets a 30-second warning before the node is terminated. Kubernetes uses this window to gracefully shut down your application (giving non-system pods up to 15 seconds to wrap up) before the node vanishes. When to use Spot VMs for accelerators Spot VMs are ideal for workloads that are: Fault-tolerant and stateless: Your application can handle a node vanishing and having its pods rescheduled elsewhere. Batch processing: Jobs that can be easily restarted or have checkpointing built-in. CI/CD pipelines: Running tests or builds that don't need 100% uptime. How to use Spot VMs in GKE You can easily add a Spot VM node pool to your GKE Standard cluster. The key is to use Spot VMs for your workers, not your critical system pods. Create a dedicated Spot VM node pool: When creating a node pool, simply add the --spot flag and apply a taint so standard pods

There's a question that keeps surfacing across engineering teams — sometimes in architecture reviews,...

Appendix: Live System Output — Real Pipeline in Production All output below was captured live from the running pipeline on 2026-03-08. These are not mock outputs — they come from actual AWS infrastructure and Kubernetes clusters. ArgoCD — All 50 Applications Across 6 Clusters The following is the live output of argocd app list from the hub cluster ( myapp-production-use1 ). Every component of the pipeline is represented — security, logging, monitoring, backups, and the application itself. $ argocd app list --output wide NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH argocd/argo-rollouts-myapp-production-use1 myapp-production-use1 argo-rollouts production Synced Healthy argocd/argo-rollouts-myapp-production-usw2 myapp-production-usw2 argo-rollouts production Synced Healthy argocd/aws-lbc-myapp-production-use1 myapp-production-use1 kube-system production Synced Healthy argocd/eso-myapp-production-use1 myapp-production-use1 external-secrets production OutOfSync Healthy ← known false positive argocd/eso-myapp-production-usw2 myapp-production-usw2 external-secrets production OutOfSync Healthy ← known false positive argocd/falco-myapp-dev-use1 myapp-dev-use1 falco production Synced Healthy argocd/falco-myapp-dev-usw2 myapp-dev-usw2 falco production Synced Healthy argocd/falco-myapp-production-use1 myapp-production-use1 falco production Synced Healthy argocd/falco-myapp-production-usw2 myapp-production-usw2 falco production Synced Healthy argocd/falco-myapp-staging-use1 myapp-staging-use1 falco production Synced Healthy argocd/falco-myapp-staging-usw2 myapp-staging-usw2 falco production Synced Healthy argocd/fluent-bit-myapp-dev-use1 myapp-dev-use1 logging production Synced Healthy argocd/fluent-bit-myapp-dev-usw2 myapp-dev-usw2 logging production Synced Healthy argocd/fluent-bit-myapp-production-use1 myapp-production-use1 logging production Synced Healthy argocd/fluent-bit-myapp-production-usw2 myapp-production-usw2 logging production Synced Healthy argocd/fluent-bit-myapp-

DevSecOps Pipeline — Completion Runbook All code is written and pushed to GitHub. This runbook covers the remaining operational steps: Terraform applies, GitOps ARN updates, and ArgoCD deployment. Prerequisites Install these tools if not already present: # AWS CLI v2 winget install Amazon.AWSCLI # Terraform 1.6+ winget install HashiCorp.Terraform # Terragrunt # Download from https://github.com/gruntwork-io/terragrunt/releases # Place in C:\Windows\System32\ or add to PATH # kubectl winget install Kubernetes.kubectl # ArgoCD CLI winget install argoproj.argocd AWS Profile Setup The root terragrunt.hcl uses profiles named myapp-{env}-{region_alias} . Configure them in ~/.aws/config : [profile myapp-production-use1] region = us-east-1 role_arn = arn:aws:iam::591120834781:role/AdministratorAccess source_profile = default [profile myapp-production-usw2] region = us-west-2 role_arn = arn:aws:iam::591120834781:role/AdministratorAccess source_profile = default [profile myapp-staging-use1] region = us-east-1 role_arn = arn:aws:iam::690687753178:role/AdministratorAccess source_profile = default [profile myapp-staging-usw2] region = us-west-2 role_arn = arn:aws:iam::690687753178:role/AdministratorAccess source_profile = default [profile myapp-dev-use1] region = us-east-1 role_arn = arn:aws:iam::557702566877:role/AdministratorAccess source_profile = default [profile myapp-dev-usw2] region = us-west-2 role_arn = arn:aws:iam::557702566877:role/AdministratorAccess source_profile = default PHASE 1 — Terraform Applies Work from the myapp-infra/ directory. Run in the order shown — capture outputs for updating GitOps files in Phase 2. 1.1 WAF (production + staging) # Production us-east-1 terragrunt apply --terragrunt-working-dir live/production/us-east-1/waf # Output → webacl_arn (copy this value) # Production us-west-2 terragrunt apply --terragrunt-working-dir live/production/us-west-2/waf # Output → webacl_arn (copy this value) # Staging (no GitOps ARN needed, but good to have) terra

Threat-intel reports from recent years document campaigns in which attackers obtain AWS IAM credentials from developer workstations, use them to enumerate cloud accounts and access Kubernetes clusters. From there, attackers deploy poisoned container images to move laterally and harvest secrets. The MITRE ATT&CK chain maps to: T1552.001 ( Credentials in Files ) → T1078.004 ( Valid Accounts: Cloud Accounts ) → T1610 ( Deploy Container ) → T1496 ( Resource Hijacking ). This is not an isolated case. The Shai-Hulud supply chain attack harvested Kubernetes credentials from CI and developer workstations, feeding exactly this kind of attack chain. This research started with a short list of questions: What are Kubernetes secrets, exactly? What can an attacker do with them? How can defenders harden their clusters? So before we look at what we found in the wild, and how to harden clusters to mitigate impacts, let's define what Kubernetes secrets are. Three Surfaces, Three Secret Formats A simplified view of the cluster has three sides that matter for this post: A developer, or automation pipeline, talks to the Kubernetes API server with credentials. That is the canonical front door. On every node, the kubelet exposes its own HTTPS API. The same credentials can authenticate to it directly when it is reachable on the network. The cluster's nodes pull images from container registries (Docker Hub, GitHub Container Registry, ECR, Quay, GitLab, ACR…) using a second set of credentials. Kubernetes Architecture These are the attack surfaces where leaked secrets have the most impact, and three secret formats unlock them: TLS client certificates are used by humans through a kubeconfig file with the kubectl command to connect to a Kubernetes cluster. JSON Web Tokens, or Service Accounts, are non-human identities (NHI) used to automate cluster operations from CI/CD jobs, controllers, and integrations. By default, JWTs have no expiration date — which is why a JWT leaked years ago can still

Platform Engineering Labs has announced a major update to its open-source Infrastructure-as-Code platform, formae, introducing full Kubernetes support, native Helm integration, direct .tfvars compatibility, and a new public plugin hub aimed at simplifying cloud-native infrastructure management By Craig Risi