vLLM is a server for language models with an API that follows the OpenAI format. The template runs it on a rented GPU with 24 GB of memory: Qwen2.5 7B starts by default, and your rental token is the API key. The first start is slower than with Ollama, because the model downloads and compiles, which takes 10–30 minutes or more.
In short
vLLM runs on cards with 24 GB or more. The most affordable suitable card right now is the Tesla V100 32 GB from 23 ₽ per hour, billed by the minute.
Updated 13 October 2026
vLLM reserves up to 90% of the card's video memory for weights and attention cache, so other GPU jobs usually cannot run on the same card. The default model, Qwen2.5 7B, weighs about 15 GB at 16 bits, so take a 24 GB card such as an RTX 4090 24 GB or an RTX 3090. A 32B model at 16 bits needs 80 GB, such as an A100 80 GB, while a 4-bit AWQ build of 32B already fits an RTX 5090 32 GB.
Prices and availability come from the live catalog and update automatically.
For vLLM the precision of the weights matters: a model at 16 bits takes twice the memory of an 8-bit build and four times the memory of a 4-bit one. Typical sizes:
| Model and precision | Weights, approx. | Video memory | Example cards |
|---|---|---|---|
| 7B at 16 bits (default) | about 15 GB | 24 GB | RTX 3090, RTX 4090 24 GB |
| 14B at 16 bits | about 30 GB | 48 GB | RTX A6000, L40S 48 GB |
| 32B at 16 bits | about 65 GB | 80 GB | A100 80 GB, H100 80 GB |
| 32B at 4 bits (AWQ) | about 20 GB | 32 GB | RTX 5090 32 GB |
The template runs Qwen/Qwen2.5-7B-Instruct. The weights download from Hugging Face, then the model compiles for your card. The log is written to /var/log/vllm.log: follow it over SSH with tail -f.
To change the model, connect over SSH, stop the running process and start the server again with the right --model. Pass the token to --api-key explicitly rather than through an environment variable: if the variable is not set in your session, the server starts without a key and the API is open. For example, Qwen/Qwen2.5-7B-Instruct-AWQ takes about 5 GB and fits comfortably on a 24 GB card.
pkill -f api_server
nohup python3 -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 8000 \
--model <Hugging Face repository> --max-model-len 8192 \
--api-key <rental token> > /var/log/vllm.log 2>&1 &
tail -f /var/log/vllm.logFor gated models such as Llama or Gemma, set a Hugging Face token in the HF_TOKEN variable before starting. If you need another vLLM version or extra flags, run your own image with the Your own Docker image template.
The API address is the «API (port 8000)» button on the server page, with the path /v1. The model name in the request must match the --model parameter.
curl https://<id>-8000.run.gpu.nz/v1/chat/completions \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen2.5-7B-Instruct", "messages": [{"role": "user", "content": "What is vLLM?"}]}'# pip install openai
from openai import OpenAI
client = OpenAI(
base_url="https://<id>-8000.run.gpu.nz/v1",
api_key="<token>",
)
reply = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[{"role": "user", "content": "What is vLLM?"}],
)
print(reply.choices[0].message.content)If you want a chat, start with the Open WebUI + Ollama template. vLLM suits an API that many people use at the same time, and models in the Hugging Face format.
| Parameter | Ollama | vLLM |
|---|---|---|
| Purpose | a chat and an API for you or a small team | an API for a service under parallel load |
| Interface | the Open WebUI chat in the browser | API only, no chat |
| Models | the Ollama library and GGUF from Hugging Face | Hugging Face weights: 16-bit, AWQ, GPTQ |
| First start | a few minutes | 10–30 minutes or more |
| Switching models | the Model field or a download in the chat | a restart over SSH with a new --model |
A step-by-step comparison with commands is in the guide Ollama and vLLM on a rented GPU.