Why a 24 GB GPU Does Not Give Your Local LLM 24 GB

I keep seeing the same local LLM sizing mistake: "The model file is smaller than my GPU, so it should fit." That is only the first check. A 24 GB GPU does not give your model a clean 24 GB memory budget. The display stack, runtime, temporary buffers, model weights, and KV cache all compete for the same space. Here is the worksheet I use before I download a model or rent a GPU. 1. Start with the weight floor The simplest weight estimate is: weight_memory_gib = parameters * bits_per_parameter / 8 / 1024^3 For a simple 4-bit estimate: Model size Weight floor 7B 3.3 GiB 13B 6.1 GiB 70B 32.6 GiB These are floors, not promises. Real quantized files can also contain scales, metadata, and layers stored at higher precision. If you know the exact checkpoint size, use that instead of the simple bits-per-parameter estimate. Also use total parameters for a sparse mixture-of-experts model unless your runtime really offloads inactive experts. Active parameters describe compute per token. They do not automatically describe how many weights must be stored. 2. Reduce the physical capacity to a usable budget I normally start with 90 percent usable VRAM for planning: usable_vram = physical_vram * usable_fraction For a 24 GB card: 24 * 0.90 = 21.6 GiB usable The exact reserve depends on the OS, display use, driver, runtime, graph capture, allocator behavior, and other processes. The important part is to stop treating the number on the box as fully available. 3. Add the KV cache The KV cache is where context length and concurrency become expensive. A useful planning formula is: kv_cache_bytes = 2 * layers * kv_heads * head_dimension * context_tokens * concurrent_sequences * bytes_per_kv_value The factor of two stores keys and values. Take a model with: 32 layers 8 KV heads 128 dimensions per head 8,192 cached tokens 1 concurrent sequence 16-bit KV values, which use 2 bytes The KV cache is about 1 GiB. Raise the context to 32,768 tokens and it becomes about 4 GiB. Keep that context and ru