Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess

Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess OK, you're a senior SRE, you've been hearing incessantly about AI models, but aren't quite sure how to determine the correct node size to host your model. - If so ... you're in the right place. Part of a series on running vLLM on AKS. Companion piece: How to avoid flapping . GPU infrastructure setup — coming soon. This piece walks through estimating GPU memory requirements from both a model's parameter count or a concurrent requests requirement. After reading this article you will have enough knowledge to pick a GPU family with confidence. Disclaimer: this process is a rule-of-thumb filter, not a precise calculation — the last step covers how to get exact numbers once the model is actually running. Background: What actually consumes GPU memory AI models live in GPU memory — VRAM — and engines such as vLLM provide novel techniques for managing that memory efficiently [ paper ], but the model isn't the only thing consuming it. Below is a short list of things that consume our precious VRAM: Model weights — the parameters themselves. The big fixed cost: loaded once, never shrinks. KV cache — working memory for in-flight requests. Every token of every active request holds its attention keys/values here. This is the one that determines throughput : more KV cache = more concurrent requests. Everything else — activations (the temporary tensors of a forward pass) plus CUDA/framework overhead. You don't calculate these by hand; vLLM measures activations with a profiling pass at start-up and prints it for our consumption. The sizing question is really: after weights and overhead, how much is left for the KV cache — and is that enough for your traffic? OK, lets get started Step 1 — Choose a model Guidance on which model to choose is outside the bounds of this article. What matters here: once you have a candidate, everything below can be read off its spec sheet — you can then run this method on every model on y