标签：#uber

I'm fairly new to SRE/DevOps, and one of the topics I recently spent time studying properly was EKS cluster upgrades . My first instinct, like most people starting out, was: "it's just a version bump, click upgrade in the console, done." That's basically what most beginner blog posts say too. But the more I read and the more I dug into real-world postmortems and discussions, the more I realized — the actual Kubernetes control plane upgrade is the easy part. Almost everything that can go wrong seems to happen around it, not because of it. Sharing what I learned here, mainly for my own notes, but hoping it's useful for anyone else early in their journey too. Learning #1: There's No "Undo" Button This was the first thing that surprised me. I assumed upgrades work like most software — if something breaks, you roll back. But with EKS, you cannot downgrade the control plane version once you upgrade it. So the plan can't be "upgrade, and if it breaks, revert." It has to be "test enough beforehand that breaking isn't really an option," and if something does go wrong, the fix is always moving forward, not backward. That single fact changes how you're supposed to approach the whole thing — testing has to happen before the button is clicked, not after. Learning #2: APIs Get Deprecated, and It's Usually Not Your Own Code That Breaks Kubernetes removes old API versions on a schedule. I already knew this conceptually, but what I didn't realize is that the risk usually isn't your own YAML files — it's the Helm charts and third-party tools you installed a while back and forgot about , which might still be using an older API version internally. There are tools built exactly for catching this before it becomes a problem: pluto detect-helm -owide pluto detect-files -d ./manifests kubent (kube-no-trouble) does something similar. I hadn't heard of either tool before researching this, and it made me realize how much of "being good at Kubernetes" is really just knowing which small tools e

Microsoft is placing greater emphasis on controlling disruption in Azure Kubernetes Service (AKS) Node Auto-Provisioning (NAP), publishing new guidance to help platform teams balance the efficiency benefits of automated node consolidation with application availability. By Craig Risi

Scalable Guardrail Service ASP.NET Core Kubernetes: Architecture, Code, and Ops Quick Answer Scalable Guardrail Service ASP.NET Core Kubernetes: A dedicated ASP.NET Core guardrail microservice on Kubernetes validates LLM requests, enables instant policy updates via Redis, and scales with custom HPA for high‑throughput. Scalable Guardrail Service ASP.NET Core Kubernetes: Why a Dedicated Guardrail Microservice Matters When you expose an LLM‑powered API to the world, every request is a potential compliance risk. A single malformed prompt can surface PII, trigger a policy violation, or even cause a brand‑damaging output. In my experience, the first version of such a system is a set of ad‑hoc filters sprinkled across controllers. Under load, those filters become latency bottlenecks, policy updates race, and audit trails vanish. The root cause is a missing architectural layer that treats guardrails as a first‑class microservice that can scale horizontally, be updated live, and be observed independently. Guardrail Layer Requirements We need a guardrail layer that: Validates every request before it hits the LLM engine. Can be updated without redeploying the entire API surface. Provides per‑tenant isolation and versioning. Logs every decision for compliance and red‑team analysis. Runs at the same scale as the LLM inference service. When This Fails in Production Policy updates are applied via a shared ConfigMap and the pods do not reload, so new rules are never enforced. The guardrail service is single‑instance; a spike in requests triggers a queue that exceeds the LLM engine’s rate limit, causing a cascading failure. Audit logs are written to local disk; a pod crash loses events. Latency spikes because each request performs a synchronous Redis lookup for every policy. Common Mistakes Engineers Make Embedding guardrail logic inside the API controller rather than a dedicated middleware. Using in‑memory policy caches without a TTL, leading to stale rules. Ignoring the fact that

Upgrading Kubernetes clusters across a large enterprise fleet is often a balancing act between staying current with security patches and avoiding outages. By default, Google Kubernetes Engine (GKE) rolls out automatic upgrades progressively according to Google Cloud regional timelines. While regional rollout works well for standalone clusters, it does not understand your organization's business topology. If you run staging clusters in us-central1 and critical production clusters in us-east1 , a standard regional rollout could upgrade your production environment before your pre-production validation completes. The General Availability (GA) release of GKE rollout sequencing with custom stages solves this challenge. It provides platform teams with declarative control to sequence cluster upgrades across fleets, environments, and even distinct Google Cloud organizations according to business criticality rather than cloud geography. How rollout sequencing works Rollout sequencing builds on GKE fleet management. Fleets serve as logical boundaries for environments such as development, staging, and production. With rollout sequencing, you define an ordered pipeline of upgrade stages managed by a central resource called RolloutSequence . When GKE publishes a new automatic upgrade target for a release channel, or when you explicitly trigger a target version, the system creates a Rollout object. This rollout progresses through your defined stages sequentially: Control plane upgrades start in the first stage. Once all control planes in that stage reach the target version, a stage soak timer begins. Node upgrades run in parallel with control plane upgrades, respecting node pool upgrade strategies such as surge or blue-green. When both control planes and nodes complete their upgrade and satisfy the configured soak duration, the rollout advances to the next stage in the sequence. If an individual stage contains clusters that take longer than 30 days to finish upgrading—due to restr

Uber said no one will be able to access the video once the ride ends

Uber is updating its teen account feature to allow parents to use the selfie cameras on Uber drivers' phones to check in on their teenage children during ridehail trips. Since first launching teen accounts in 2023, Uber has touted the myriad ways parents can keep tabs on their children, from PIN verification to live GPS […]

Short answer: for a Node.js app in Docker or Kubernetes, give startup, readiness, and liveness probes separate meanings, keep routine health traffic out of application logging, and measure state transitions instead of counting every successful check. For a property-management API rolling out a new pricing rule, this preserves useful metrics: whether an instance can calculate rent correctly and accept traffic, without turning each kubelet poll into noise. Which health signal should control each container decision? Start with the decision, not the endpoint name. Signal Question it answers Include Exclude Action Startup Has initialization completed? Configuration parsing, pricing-rule compilation, required local warm-up Long-term dependency health Allow the process more time before other probes apply Readiness Can this instance safely receive a new pricing request now? Ability to serve the active rule version and any required dependency state Optional analytics and background exports Remove the pod from Service endpoints Liveness Is the process stuck beyond local recovery? Event-loop progress or another narrow process invariant Database, cache, and third-party availability Restart the container This split is the main noise filter. A downstream dependency becoming unavailable can make a pod unready, but restarting the same healthy process usually doesn't repair that dependency. If the dependency is placed in liveness anyway, every pod can restart together. The health response has then amplified one problem into two: lost capacity plus a restart storm. The pricing rollout makes readiness more demanding than “the port is open.” Imagine rule version rent-2026-08 is enabled for one building cohort. A newly started instance has loaded configuration but hasn't compiled that version yet. It is alive. It isn't ready. Its startup check should hold back liveness and readiness until initialization finishes; afterward, readiness should stay false until the active rule can be evalua

By combining ProgressivePerGroup with Placement decision groups, you can roll out add-on configuration changes in the order dev → stg → prod. In this article, I use cluster-proxy as an example to explain the required configuration and how the rollout actually works. Overview flowchart TB Upgrade["helm upgrade<br/>change tag to vX.Y.Z"] subgraph Hub["Hub cluster"] direction TB Manager["cluster-proxy-addon-manager<br/>update Deployment"] Config["ManagedProxyConfiguration<br/>update spec"] ProxyServer["proxy-server<br/>update Deployment"] Hash["proxyAgent config<br/>update spec hash"] Rollout["OCM add-on manager<br/>ProgressivePerGroup"] Groups["progress through decision groups<br/>dev → stg → prod<br/>success + minSuccessTime before next group"] AddOn["ManagedClusterAddOn in current group<br/>Configured=True"] Render["cluster-proxy manager<br/>render agent chart"] Work["update ManifestWork"] end subgraph Spoke["spoke clusters in the current group"] direction TB WorkAgent["work-agent"] ProxyAgent["proxy-agent<br/>update Deployment"] end Upgrade -->|Helm updates directly| Manager Upgrade -->|Helm updates directly| Config Config -->|proxyServer.image<br/>not part of rollout| ProxyServer Config -->|proxyAgent.image<br/>part of rollout| Hash Hash --> Rollout Rollout --> Groups Groups -->|current group only| AddOn AddOn --> Render Render --> Work Work --> WorkAgent WorkAgent --> ProxyAgent WorkAgent -.->|Applied / Available| Work Work -.->|hash matches + Ready| Rollout classDef immediate fill:#fff3cd,stroke:#a66b00,color:#332200; classDef staged fill:#e8f3ff,stroke:#2563a6,color:#102a43; classDef spoke fill:#eaf7ed,stroke:#2f855a,color:#173d2a; class Manager,Config,ProxyServer immediate; class Hash,Rollout,Groups,AddOn,Render,Work staged; class WorkAgent,ProxyAgent spoke; Yellow indicates updates that happen immediately on the Hub. Blue indicates updates controlled by ProgressivePerGroup , and green indicates processing on the spoke clusters. Dashed lines represent status r

We have a web server container running the nginx image. The access and error logs generated by the web server are not critical enough to be placed on a persistent volume. However, Nautilus developers need access to the last 24 hours of logs so that they can trace issues and bugs. Therefore, we need to ship the access and error logs for the web server to a log-aggregation service. Following the separation of concerns principle, we implement the Sidecar pattern by deploying a second container that ships the error and access logs from nginx. Nginx does one thing, and it does it well - serving web pages. The second container also specializes in its task - shipping logs. Since containers are running on the same Pod, we can use a shared emptyDir volume to read and write logs. Create a pod named webserver . Create an emptyDir volume named shared-logs . Create a regular container in the webserver pod from the nginx:latest image named nginx-container , and an init container from the ubuntu:latest image named sidecar-container . Add the following command to the sidecar-container "sh","-c","while true; do cat /var/log/nginx/access.log /var/log/nginx/error.log; sleep 30; done" Mount the shared-logs volume in both containers at /var/log/nginx . Ensure all containers are in a running state. What is a Sidecar Container? Think of a sidecar like a motorcycle sidecar – it's attached to the main vehicle and extends its capabilities without changing the main vehicle itself. ┌─────────────────────────────────────────────────────────────────────────────┐ │ The Sidecar Analogy │ │ │ │ ┌────────────────────────────────────────────────────────────────────────┐ │ │ │ Motorcycle: The Main Vehicle │ │ │ │ - Does its primary job (serving web pages) │ │ │ │ - Doesn't worry about extra tasks │ │ │ └────────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────────────────────────────────────────────────────┐ │ │ │ Sidecar: Adds Extra Functional

The Dutch Data Protection Authority is fining Uber €825 million in the second largest penalty issued under Europe’s GDPR.

Welcome back to TechCrunch Mobility — your central hub for news and insights on the future of transportation.

Control Plane (Master) & Worker Nodes Control Plane components: API Server Scheduler Control Manager etcd Worker Node components: Container Runtime Kubelet Kube-proxy Node Processes Each node has multiple Pods on it. 3 processes must be installed on every node — used to schedule and manage those Pods. Nodes are cluster services that actually do the work. Container Runtime Examples: Docker, containerd, CRI-O. containerd is used in worker nodes — it's lightweight in nature. This should be installed on every node because application Pods need to run containers inside the node. Kubelet The process which schedules the Pods and containers underneath is Kubelet. Kubelet interacts with both the container and the node. Kubelet starts the Pod with the container inside. Communication between two nodes is because of Services. Creation of Pod: Kubelet insures the Pod is always running — if not, it will inform etcd. Kube-proxy Kube-proxy forwards the request from Pod to Service. Makes use of the communication, with load balancing. Provides networking (container ID, IP address). Load balancing — basically using IP tables. It makes sure to send the request to the same machine instead of sending it to others (from same node communications). So, how do you interact with this cluster? Schedule the Pod Monitor Re-schedule/restart the Pod Join a new node Managing processes are done by master nodes (the control plane). API Server When you, as a user, want to deploy a new application in a Kubernetes cluster, you interact with the API server using some client — could be UI or CLI. It's a cluster gateway — it gets the initial request of any update into the cluster, even the queries from the cluster. It also acts as gatekeeper for authentication. It means when you want to schedule new Pods, deploy new applications, create new services, or any other components — you have to talk to it first. Flow: Some request → API server → Validates request → Other processes → Pods Only one entry point to t

At LinkedIn's scale, relying solely on human reviewers or simply putting an off-the-shelf AI reviewer in front of GitHub is not an effective way to manage PRs. To address this, LinkedIn engineers built a multi-agent AI code review platform that understands the organization’s coding context, treats code review as production infrastructure, and minimizes hallucinations and low-signal feedback. By Sergio De Simone

Introduction: Kubernetes can feel overwhelming when you first hear terms like Pods, Services, and Deployments thrown around. In this first post of my Kubernetes series, I’ll break down the fundamentals — what Kubernetes actually solves, and the core building blocks you need to understand before going further. What is Kubernetes? Kubernetes is an open-source container orchestration tool , originally developed by Google. It helps manage containerized applications across different environments — physical machines, virtual machines, and cloud environments — which makes it a great fit for hybrid deployment setups. Why Kubernetes? The Problem It Solves To understand why Kubernetes exists, look at the trend that led to it: Applications moved from monolith to microservices. That shift drastically increased the number of containers teams had to manage. Managing hundreds of containers by hand became unsustainable — teams needed a proper way to orchestrate them. Key Features High Availability — no downtime Scalability — scale up or down based on load and performance needs Disaster Recovery — backup and restore built into the ecosystem Main Kubernetes Components Pods: Abstraction over containers Services: Stable networking & communication Ingress: Routes external traffic into the cluster ConfigMaps & Secrets: External configuration Volumes : Data persistence Deployments & StatefulSets: Replication (stateless vs. stateful) DaemonSets: One Pod per node, auto-scaled with the cluster

Alphabet-owned company is seeking to persuade US regulators to clear a path for fully autonomous taxi services.

Iran Doesn't Need to Mine Hormuz — Your requirements.txt Is Already Rigged Every headline you've read this week is a diversion. The Strait of Hormuz is not the target. You are. And you have been for months, possibly years, while you retweeted tanker tracking maps and debated whether Brent crude would touch $150. Iranian state-sponsored groups — OilRig, APT33, MuddyWater, Agrius — did not spend the last decade pivoting to cloud infrastructure so they could watch you panic about a waterway. They did it so they could own your build pipeline while you were distracted. And they have. This is not speculation. CISA Advisory AA24-038A explicitly maps Iranian APT campaigns against U.S. and allied critical infrastructure to cloud identity, Kubernetes targets, and software supply chains. Not SCADA. Not PLCs. Your kubectl binary. Your Helm charts. That FastAPI microservice running payment webhooks that you deployed on a Friday and haven't touched since March. The Revolutionary Guard does not need a mine. They need a maintainer who hasn't updated python-jose in fourteen months. The Theater and the Operation You watched the Strait. They watched your CI/CD. Geopolitical analysis is a spectator sport for infrastructure engineers, and Iranian cyber command is the bookie. While your LinkedIn feed filled with satellite imagery and retired admirals explained chokepoint logistics, the actual operation ran silently against: Public Helm charts with hardcoded cluster-admin ServiceAccounts FastAPI services with python-multipart handling unbounded file uploads on single-threaded Uvicorn workers .kube/config files exfiltrated from developer laptops in a dev-legacy namespace that predates your current CTO Terraform state stored in a single S3 bucket with versioning disabled and a policy written by someone who left in 2021 The Hormuz closure narrative is Information Operations . The closure of your API gateway due to an unpatched ASGI memory exhaustion vulnerability is the kinetic effect. You a

Dan Finneran discusses the risks of unowned AI-generated code in production and demonstrates how eBPF can intercept and control AI API traffic in Kubernetes. He explains how kernel-level socket hooks enable transparent prompt filtering, model swapping, token limits, and syscall restrictions to secure AI agents without modifying application source code or restarting containers. By Dan Finneran

Together, these permits would allow up to 8,000 robotaxis to be deployed over the next 12 months.

Flux has introduced Flux Mirror, a CLI plugin that mirrors container images, Helm charts and OCI artifacts between registries from a declarative configuration. The plugin is part of the Flux v2.9 CLI plugin system and is presented as a way to keep Kubernetes clusters reconciling only from registries that teams operate themselves. By Matt Saunders

I Deliberately Destroyed My Kubernetes Cluster at 2 AM. Here's What Died First. Chaos engineering is not about breaking things. It's about discovering that your "production-grade" homelab is held together by hope and a single etcd snapshot before someone else finds out for you. The Setup I was lying in bed at 1:47 AM, staring at the ceiling, unable to sleep. Not because of caffeine. Because of a thought that had been gnawing at me for weeks: If one of my nodes died right now, would my cluster actually survive? I run a 4-node bare-metal Kubernetes cluster on Talos Linux. Dell OptiPlex control plane. Three Raspberry Pi workers. Cilium eBPF. ArgoCD. Longhorn distributed storage. Prometheus. Grafana. The whole cloud-native stack, shoehorned into $220 of scrap hardware and stubbornness. From the outside, it looks solid. ArgoCD syncs green. Cilium status shows healthy. Longhorn volumes are replicated across three nodes. I have etcd snapshots every 6 hours to S3. On paper, I'm resilient. But I had never actually tested it. Not a controlled test. Not a graceful node drain. I mean chaos . Sudden death. The kind of failure that happens at 3 AM when a power supply dies, or a kernel panics, or a neighbor's construction crew hits the wrong breaker. So I got out of bed, walked to my desk, and installed Chaos Mesh. Why Chaos Engineering on a Homelab? Professionally, I design AWS infrastructure with multi-AZ failover, auto-scaling groups, and managed services that abstract failure away. At Siemens, if an EKS node dies, the managed node group replaces it before I finish reading the alert. But my homelab has no managed control plane. No AWS SLA. No auto-repair. If a Pi's USB boot drive corrupts, that node is gone until I physically fix it. I needed to know: What dies first when a worker vanishes? Not "what should die" — what actually dies. Does Longhorn really failover? Three replicas sound great until you realize two of them were on the same node. Does Cilium handle network partitio