Hermes 4.3 36B in Q4_K_M with an 8k context takes 23.3 GB of GPU memory. The cheapest option right now is RTX 3090 24 GB, from $0.302/h.
The weight file and the total GPU memory with the context cache. One GPU, one request.
| Quantisation | File | 8K context | 32K context | 128K context | Fits on |
|---|---|---|---|---|---|
| BF16 | ≈67.3 | 70.4 | 76.4 | 100.4 | A800 PCIe 80 GB |
| Q8_0 | 35.8 | 38.8 | 44.8 | 68.8 | A100 PCIe 40 GB |
| Q6_K | 27.6 | 30.7 | 36.7 | 60.7 | Tesla V100 Super 32 GB |
| Q5_K_M | 23.8 | 26.9 | 32.9 | 56.9 | Tesla V100 Super 32 GB |
| Q4_K_M | 20.3 | 23.3 | 29.3 | 53.3 | RTX 3090 24 GB |
| Q3_K_M | 16.4 | 19.5 | 25.5 | 49.5 | RTX 4000 Ada 20 GB |
How much memory the cache takes per request in FP16. A Q8 cache is about half. 256 KB per token.
GPUs from the catalog the model fits on, by price per hour.
The price is the cheapest GPU of the model × the GPU count. A 2–8 GPU server can cost a little more per GPU.
The “Hermes 4.3 36B” template starts the model in a ready environment: open the link once the server has loaded.
For an API under load there is the vLLM template with an OpenAI-compatible API.
File sizes checked against Hugging Face on 2026-10-19. 1 GB here is 1024³ bytes, like GPU memory. Hugging Face lists decimal gigabytes, so its numbers are 7% larger.