RTX PRO 6000 Blackwell 96 GB for AI: which models fit on one card

One RTX PRO 6000 Blackwell with 96 GB of video memory holds 29 of the 35 open LLMs in our calculator at an 8K context, and 23 of them uncompressed. Llama 3.3 70B runs at Q8_0 with a 32K context (80.8 GB), and GPT-OSS 120B takes 60.5 GB. Below are a table for every model, a comparison with 32, 48 and 80 GB cards, images and video, and how to run it on a rented card.

What is the card: 96 GB of GDDR7 in three editions

The RTX PRO 6000 Blackwell is an NVIDIA professional GPU with 96 GB of GDDR7 memory with error correction (ECC). It comes in three editions: the Workstation Edition for workstations, the 300 W Max-Q Workstation Edition and the Server Edition for servers. The memory is the same on all of them; power, cooling and memory bandwidth differ.

In the gpu.nz catalog they appear as RTX PRO 6000, RTX PRO 6000 Max-Q and RTX PRO 6000 Server. The main card, with its live price, is on the RTX PRO 6000 96 GB page; this article covers what runs on it.

Which LLMs fit in 96 GB?

29 of 35: that many models fit in the 93.6 GB a program can use on a 96 GB card at an 8K context. The driver and CUDA keep the other 2.5% or so. For each model we looked for the highest precision among BF16, FP8, Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M and Q2_K at which it still fits.

Precision is how the weights are stored: the higher it is, the closer the model stays to the original. Q8_0 and FP8 usually differ little from the original, Q4_K_M is the usual compromise, and answers at Q2_K are noticeably weaker.

The gap between 48 and 80 GB is large; between 80 and 96 GB it is zero by this measure: the same 23 models fit uncompressed. The extra 16 GB serve other purposes, covered below.

The full table, from the largest models to the smallest. Each cell gives the precision and the memory of the whole configuration: weights, context cache and runtime buffers. “Tight” means more than 92% of the memory is in use.

What do 96 GB add over 32, 48 and 80 GB?

That is three times the memory of a 32 GB card, twice that of a 48 GB card and 16 GB more than an A100 or H100. The table shows what that means for large models: the highest precision at an 8K context.

• Against 32 GB (RTX 5090). Llama 3.3 70B fits 32 GB only at Q2_K, but 96 GB at Q8_0. GPT-OSS 120B does not fit 32 GB at all. The RTX 5090 and the RTX PRO 6000 Workstation have the same memory bandwidth, 1792 GB/s (NVIDIA figures), so do not expect extra speed from the extra gigabytes: they change what you can run at all.
• Against 48 GB (RTX A6000, L40S). For Llama 3.3 70B the ceiling on 48 GB is Q4_K_M, or Q3_K_M with a 32K context. GPT-OSS 120B and GLM-4.5 Air do not fit 48 GB at a good precision.
• Against 80 GB (A100, H100). The gain is smaller: whatever fits 80 GB uncompressed also fits 96 GB. But GLM-4.5 Air fits at Q5_K_M instead of Q4_K_M, Llama 3.3 70B with a 32K context at Q8_0 instead of Q6_K, and Qwen3 235B-A22B runs on one card, though at Q2_K.

A long context is another reason. The cache of Llama 3.3 70B takes 10.0 GB for every 32K tokens, so at Q4_K_M with a 128K context the model needs 80.6 GB. That does not fit an 80 GB card but fits 96 GB.

Speed is another matter. The memory bandwidth is 2039 GB/s on the A100 SXM, 3.35 TB/s on the H100 SXM (NVIDIA figures) and 1792 GB/s on the RTX PRO 6000. For a single request, generation speed usually depends on bandwidth rather than capacity, so a model that fits 80 GB may answer faster on an H100. We have not measured these cards.

How many requests can the card serve at once?

As many as there are context caches that fit next to the weights. The calculation for requests of 32K tokens at 90% of the card's memory (vLLM's default share) shows a wide spread:

• GPT-OSS 120B — up to 23: its cache grows slowly.
• Qwen3 Next 80B-A3B at Q4_K_M — up to 48: some of its layers keep a fixed-size state instead of a growing cache.
• Llama 3.3 70B at Q4_K_M — up to 4: a dense model, each request holds 10.0 GB.

This is an upper bound: it counts weights, cache and runtime, but not load peaks. Leave headroom for a service.

Which image and video models run on 96 GB?

All 14 image and video models in our calculator. 13 of them stay fully in memory at BF16, uncompressed. On a 32 GB card, 6 do.

The first number in the table is the minimum: ComfyUI unloads the text encoder before generating, which is what it does when memory is short. The second is with everything kept in memory, so repeat runs do not wait for a reload. For video we took the largest resolution with data, usually 720p.

There is one exception: FLUX.2 [dev] at BF16 does not fit 96 GB with everything resident (96.3 GB), but runs in 63.1 GB with the encoder unloaded. Wan 2.2 A14B at BF16 with both experts in memory takes 71.5 GB, which needs an 80 GB card. More on the models in the guides to FLUX and video on your own GPU.

What does not fit even in 96 GB?

Large MoE models: Qwen3 235B-A22B and DeepSeek V4 Flash fit only at Q2_K (V4 Flash takes 98% of the memory), and the rest do not fit at all. The table shows how many 96 GB cards the default file needs when the model is split across cards (tensor parallel).

Servers with 2, 4 or 8 of these cards are not always available: check the catalog. DeepSeek V4 Flash has its own guide.

Check your own model

Pick a model, a quantisation and a context length: the calculator shows the memory and the catalog GPUs that run it for the least money.

Buy an RTX PRO 6000 or rent one?

If your model fits 32 or 48 GB, there is no reason to pay for 96 GB. Buying makes sense when the card works almost every day, and the formula in Buy an H100 or rent one works for this card too.

Add everything around the card to its price. The Workstation Edition is rated at 600 W, which is up to 14.4 kWh a day at full load, plus a power supply, cooling and space. Before you buy, rent the card for an hour and check on your own model that it fits and the speed suits you.

How to run a model on a rented RTX PRO 6000

We will take Llama 3.3 70B at Q8_0 through the Ollama template. GPT-OSS 120B is simpler: it has a ready template.

1. 1

   Pick the RTX PRO 6000 in the catalog

   Open the catalog, find the RTX PRO 6000 with 96 GB (any edition will do) and press Rent.

2. 2

   Choose a template

   For GPT-OSS 120B take the GPT-OSS 120B template: it downloads the model and opens a chat and an API. For the other models in the table, use Open WebUI + Ollama.

3. 3

   Set the model and the disk

   In the Model field enter the tag from the Ollama library (the file is about 75 GB), or run the command over SSH. Book a disk of at least 100 GB.

   ollama pull llama3.3:70b-instruct-q8_0

4. 4

   Wait for the model to download

   The server starts in 2–5 minutes, then the file downloads: for 65–75 GB that is usually 10–30 minutes, sometimes longer. When the model is ready, open the chat with the Open WebUI button.

5. 5

   Connect the API if you need it

   The address is on the «Ollama API» button, the token on the server page, the model is the same tag.

   curl "https://<id>-11434.run.gpu.nz/v1/chat/completions" \
     -H "Authorization: Bearer <token>" \
     -H "Content-Type: application/json" \
     -d '{"model": "llama3.3:70b-instruct-q8_0", "messages": [{"role": "user", "content": "Hello"}]}'

6. 6

   Stop the server when you are done

   Billing is per minute: a stopped server costs nothing for the GPU, only its disk.

For an API with several users, take the vLLM template and restart the server with the FP8 build of Qwen3 Next 80B-A3B: the file is about 82 GB, and with a 32K context the model takes 78.3 GB. Book a disk of at least 150 GB. The context is capped with --max-model-len, as the model card on Hugging Face advises; pass the token to --api-key explicitly, otherwise the API stays open.

pkill -f api_server
nohup vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 --host 0.0.0.0 --port 8000 \
  --api-key <rental token> --max-model-len 32768 > /var/log/vllm.log 2>&1 &
tail -f /var/log/vllm.log

What can go wrong

• “Fits” does not mean “fast”. We counted memory, not speed. The Max-Q edition is rated at 300 W and the Workstation edition at 600 W: the memory size is the same, but we have not compared how that affects speed.
• A tight fit. Above 92% of the card's memory (marked in the table) the model starts, but a long conversation or a second request may not fit. DeepSeek V4 Flash at Q2_K takes 98% of the memory, and answers from a two-bit quantisation are noticeably weaker.
• Parallel requests multiply the cache. The figures in the tables are for one request. Every concurrent request keeps its own context cache, so a service for several people needs headroom.
• Blackwell needs recent software. NVIDIA names CUDA 12.8 as the first version to support this architecture. If you install PyTorch or your own image, use builds for CUDA 12.8 or newer: older ones can fail with errors such as “no kernel image is available for execution on the device”.

How we counted

Weights are the real file sizes: GGUF from unsloth's Hugging Face repositories, FP8 and BF16 from the model authors' repositories. The context cache is computed from each model's config.json. The runtime is about 0.75 GB plus the logits buffer. 1 GB here is 1024³ bytes, as on GPUs. A model “fits” when the whole configuration takes at most 97.5% of the card's memory. “Original precision” means BF16 or the format the model was released in (MXFP4, FP8). Data as of 2026-10-19.

The cards' memory, bandwidth and power come from NVIDIA's product pages on nvidia.com (RTX PRO 6000 Workstation, Max-Q and Server Edition, RTX 5090, A100, H100), read on 20 October 2026. Everything here is a memory calculation: we have not run the configurations from this article on an RTX PRO 6000 or measured speed. The method is in the calculator, and one table for 35 models is in How much VRAM do open LLMs really need?.