Qwen3.5 4B in Q4_K_M with an 8k context takes 4.1 GB of GPU memory. The cheapest option right now is RTX 3060 12 GB, from $0.084/h.
The weight file and the total GPU memory with the context cache. One GPU, one request.
| Quantisation | File | 8K context | 32K context | 128K context | Fits on |
|---|---|---|---|---|---|
| BF16 | 7.8 | 9.4 | 10.1 | 13.1 | RTX 3080 10 GB |
| Q8_0 | 4.2 | 5.7 | 6.4 | 9.4 | RTX A2000 6 GB |
| Q6_K | 3.3 | 4.8 | 5.6 | 8.6 | RTX A2000 6 GB |
| Q5_K_M | 2.9 | 4.4 | 5.2 | 8.2 | RTX A2000 6 GB |
| Q4_K_M | 2.6 | 4.1 | 4.8 | 7.8 | RTX A2000 6 GB |
| Q3_K_M | 2.1 | 3.7 | 4.4 | 7.4 | RTX A2000 6 GB |
| UD-Q2_K_XL | 1.8 | 3.3 | 4.1 | 7.1 | RTX A2000 6 GB |
How much memory the cache takes per request in FP16. A Q8 cache is about half. 32 KB per token.
GPUs from the catalog the model fits on, by price per hour.
The price is the cheapest GPU of the model × the GPU count. A 2–8 GPU server can cost a little more per GPU.
For an API under load there is the vLLM template with an OpenAI-compatible API.
File sizes checked against Hugging Face on 2026-10-19. 1 GB here is 1024³ bytes, like GPU memory. Hugging Face lists decimal gigabytes, so its numbers are 7% larger.