GPU Rightsizing Without Breaking Production: G5, G6, P4, P5 and the CUDA Check Nobody Mentions

CPU rightsizing is a solved, well-documented practice. GPU rightsizing is where the real money is now, and almost nobody writes about it, because GPU instances are expensive enough that people are scared to touch them and unsure how. Given how much a GPU box costs per hour, an over-provisioned one is the single most expensive rightsizing mistake in your account. Here is how to rightsize AWS GPU instances without breaking the workload, including the compatibility check that quietly bites people. Know what each GPU family is for Rightsizing starts with using the right family, not just the right size. On AWS: G5 / G6 (NVIDIA A10G / L4): inference, graphics, smaller training. The workhorses for serving models and lighter ML. Cheaper per hour. P4 / P5 (A100 / H100): large-scale training and heavy inference. The expensive tier, built for jobs that genuinely need the horsepower and interconnect. The most common GPU waste is running a training-class P-family instance for an inference workload that a G-family instance would serve fine at a fraction of the cost. Wrong family is a bigger error than wrong size. Rightsize on the binding resource, and it is usually not CPU GPU workloads have several resources that can be the bottleneck, and CPU utilization, the thing you would check for a normal instance, is often the least relevant: GPU utilization: is the GPU actually busy, or idle between requests? (CloudWatch does not report this by default; you need the CloudWatch agent with GPU metrics or nvidia-smi telemetry.) GPU memory: many inference workloads are GPU-memory-bound, not compute-bound. A model that fits in less VRAM can move to a smaller GPU. Host CPU and RAM: sometimes the GPU is fine but the instance is over-sized on host resources. The rightsizing signal is a GPU sitting at low utilization or using a fraction of its VRAM over a sustained window (a 90-day-style baseline, same idea as CPU rightsizing). That is your candidate to move down a size or across to a cheaper fam