How much VRAM do open LLMs really need? Real GGUF sizes for 35 models

“Parameters × bits ÷ 8” is the rule everyone quotes for how much GPU memory a model needs. It gets you close and then lets you down exactly where it matters: on the card that is 2 GB too small, or on the 32k-token chat that suddenly no longer fits.

So instead of estimating, the numbers below come from the source for 35 current open models:

• Weights: the real size of the published file — GGUF builds by unsloth, ggml-org, bartowski and the model authors (Q4_K_M unless the model ships quantised natively, like GPT-OSS in MXFP4), read from the Hugging Face API.
• Context (KV) cache: computed from each model's own config.json — layers, KV heads and head size — including the architectures where the naive formula is wrong.
• Runtime: CUDA context and buffers, about 0.75 GiB plus the logits buffer of a 512-token batch (it adds up for 256k-token vocabularies).

All figures are GiB (1024³ bytes), the unit GPU memory is sold in, so they compare directly with “24 GB”. Hugging Face lists decimal gigabytes, which is why its file sizes look about 7% larger.

The table

One GPU, one request, FP16 cache. “Smallest single card” is the smallest common card where the 8k total fits in 97.5% of the memory (the driver and CUDA take the rest). Each model links to its page with every quantisation.

What stands out

Hybrid and sliding-window attention change the picture. Qwen3 32B and Qwen3.8 27B are about the same size on disk, but going from an 8k to a 32k context costs Qwen3 32B 6.0 GiB more and Qwen3.8 27B only 1.5 GiB. Qwen3.8 keeps a KV cache on only 16 of its 64 layers; the other 48 are linear-attention layers with a fixed state per sequence. Gemma 4 does something similar with a 1024-token sliding window on 5 of every 6 layers. For long contexts the architecture matters as much as the parameter count.

MoE models need all of their weights in memory. Qwen3 30B-A3B uses 3.3B parameters per token, but all 30.5B have to be loaded: it needs the same 24 GB card as a dense 27B model, it just runs much faster on it. (llama.cpp can keep experts in system RAM with --n-cpu-moe, at a speed cost.)

GPT-OSS is the best fit for its size class. GPT-OSS 20B ships in MXFP4 and needs 12.6 GiB at 8k, so it fits a 16 GB card; GPT-OSS 120B needs 60.5 GiB and fits a single 80 GB card such as an H100.

70B still means 48 GB. Llama 3.3 70B in Q4_K_M is 43.1 GiB at 8k context: a 48 GB card, or two 24 GB cards with tensor parallelism. At 32k it grows to 50.6 GiB and no longer fits in 48 GB without a quantised KV cache.

DeepSeek's MLA cache is tiny. DeepSeek V3.2 and R1 cache one 576-wide latent per token and layer instead of full keys and values: about 70 KB per token for the whole model. The weights are the problem there, not the context.

The formula, if you want to check a model yourself

weights   = file size of the quantisation you run (or params × bits ÷ 8)
kv_cache  = tokens × full_attention_layers × 2 × kv_heads × head_dim × 2 bytes
          + min(tokens, window) × sliding_layers × 2 × kv_heads × head_dim × 2 bytes
runtime   ≈ 0.75 GiB + 512 × vocab × 4 bytes
total     = weights + kv_cache × parallel_requests + runtime

A q8_0 KV cache (-ctk q8_0 -ctv q8_0 in llama.cpp, --kv-cache-dtype fp8 in vLLM) roughly halves the second line. vLLM preallocates 90% of the card for its cache by default, so nvidia-smi always looks full there; the numbers above are what the model needs, not what vLLM grabs.

Try other settings

The VRAM calculator takes any of these models plus FLUX, Qwen-Image, Wan and other image and video models, a quantisation, the context length and the number of parallel requests, and shows the memory breakdown and the GPUs that hold it. Every model also has a page with the full quantisation table in model GPU requirements.