Qwen API with no limits: your own OpenAI-compatible server

The official Qwen API is issued against a model provider's key and runs under that provider's rates and quotas. Your own API on a gpu.nz server answers in the same OpenAI-compatible format, and the key is the token of your rental. There is no request quota here: the limits are the video memory, the speed of the card and your balance. OpenAI SDK clients work unchanged; you only change the address and the key.

How your own API differs from the official one

• The official API uses a key issued by the model provider, and its rates, quotas and free-tier terms change. Check them in the provider's documentation before you build a product on it.
• Your own API uses your rental token, and you pay only for GPU minutes. There is no request quota, because the card itself sets the ceiling.
• Requests go to your own server, not to the cloud of the model's provider. That is not the same as running at home, and the security section below covers the difference.
• You choose the model: Qwen 3.8, Qwen3, Qwen2.5 or any other model that Ollama or vLLM supports.

Two ways to run the API

If you need a chat and an API for yourself or a small team, take the ready chat. If your application sends many parallel requests, take vLLM.

Ollama and vLLM differ in more than speed. Ollama is easier to install, loads models on demand and works well with one or two clients. vLLM batches requests and keeps high throughput under parallel load, but you choose its model at start-up and it needs memory for the KV cache. If you are unsure, start with the ready chat and move to vLLM when the number of simultaneous requests becomes the limit.

Option 1: the ready chat and the Ollama API

The Qwen 3.8 template starts Ollama with the qwen3.8:27b model already loaded. The API address is the «Ollama API» button on the server page. The model needs a 24 GB card: in the 4-bit format the file takes about 18 GB.

curl https://<id>-11434.run.gpu.nz/v1/chat/completions -H "Authorization: Bearer <token>" -H "Content-Type: application/json" -d '{"model": "qwen3.8:27b", "messages": [{"role": "user", "content": "Hello"}]}'

The answer uses the OpenAI format: the text is in choices[0].message.content. The /v1/models route lists the models on the server.

Which Qwen tags to pull in Ollama

• qwen3:8b is about 5 GB, for a card with 8–12 GB.
• qwen3:14b is about 9 GB, for a 16 GB card.
• qwen3:32b is about 20 GB, for a 24 GB card.
• qwen2.5-coder:14b and qwen2.5-coder:32b are for code, at about 9 GB and 20 GB.

If you need answers in Russian, state it in the system prompt and ask for Russian replies. An explicit instruction lowers the risk that the model drifts into English in a long conversation.

Option 2: vLLM for load

The vLLM template starts an OpenAI-compatible server on port 8000 and requires a key, which is the rental token. By default it runs Qwen2.5-7B-Instruct. The first start downloads the weights and compiles the model, which can take 30 minutes or more. To run a different model, stop the template's process and start the server by hand over SSH:

pkill -f api_server
vllm serve Qwen/Qwen2.5-14B-Instruct --host 0.0.0.0 --port 8000 --api-key <token> --max-model-len 8192

The model name in a request must match the one you passed to vllm serve. Server readiness is checked on the /health route, which is open without a key, as in the template.

vLLM has one limit tied to the card type. By default the weights load in bfloat16, which needs an Ampere (RTX 30, A100) or newer card. On a T4, start with the --dtype half option. Quantised variants such as AWQ, GPTQ or FP8 take much less memory and fit 24 GB cards.

Which card to pick

For vLLM the card architecture matters: bfloat16 works on Ampere and newer, and FP8 needs Ada (RTX 40, L40S) or Hopper (H100, H200) and newer. Parallel requests need memory for the KV cache, so the more conversations run at once, the more memory you need. Current card rates are in the catalogue.

Code: curl, Python and JavaScript

In every example, replace <id> with your server's identifier, and keep the token in the GPUNZ_TOKEN environment variable. Port 11434 is Ollama, port 8000 is vLLM.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://<id>-11434.run.gpu.nz/v1",
    api_key=os.environ["GPUNZ_TOKEN"],
)
reply = client.chat.completions.create(
    model="qwen3.8:27b",
    messages=[{"role": "user", "content": "Write a sorting function in Python"}],
)
print(reply.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://<id>-11434.run.gpu.nz/v1",
  apiKey: process.env.GPUNZ_TOKEN,
});
const reply = await client.chat.completions.create({
  model: "qwen3.8:27b",
  messages: [{ role: "user", content: "Write a sorting function in JavaScript" }],
});
console.log(reply.choices[0].message.content);

For vLLM, change the port to 8000 and the model name to the one you passed to vllm serve. A streamed answer is switched on with stream set to true, and the SDKs return it in parts.

Chat history and parameters

The API does not keep a chat history: each request has to include all the earlier messages, or the model does not remember the context. A long history grows the KV cache and slows the reply, so trim old messages once they are no longer needed.

• temperature sets how varied the answers are. The lower the value, the more predictable the answer: low for code, higher for brainstorming.
• max_tokens is the maximum length of the answer in tokens. A limit that is too small cuts the answer off in the middle.
• top_p limits the choice of the next word to the most probable options. Usually you change either temperature or top_p, not both at once.

Checking that the API works

Before you write any code, check the server with one command: the /v1/models route should return the list of models. If it answers but the chat request does not, check three things. A 401 response usually means the token was rejected. A 404 usually means the server has no model with that name; the names are listed at /v1/models. And while the model is still loading into video memory, the first request can wait longer than usual.

curl https://<id>-11434.run.gpu.nz/v1/models -H "Authorization: Bearer <token>"

Streaming the answer

In a chat interface it helps to receive the answer in parts. Pass stream set to true and read the fragments in a loop:

stream = client.chat.completions.create(
    model="qwen3.8:27b",
    messages=[{"role": "user", "content": "Explain quantisation"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Timeouts and retries

For large models and long answers, set the timeout explicitly: the timeout parameter limits how many seconds the client waits for a reply. The max_retries parameter repeats the request after temporary network failures. Both are passed to the client constructor, next to the address and the key.

When your own API is not the right fit

A server of your own suits you when the model is for you or your product. If you need features the chosen engine lacks, such as tool calling or structured output, check the documentation of Ollama or vLLM for your version, because the feature set depends on the engine and its version.

Keeping the key safe

The rental token is the key to your server, so do not paste it into code that goes to a public repository. Keep it in an environment variable, as in the examples above. Call the API from your application's server rather than from a page in the browser: anyone who opens such a page can see the token.

From outside, the server port is open only through the protected link with the token. The server itself is rented and sits in a data centre, so for truly confidential data keep the model at home, as the guide local AI without a powerful computer explains. The walkthrough for calling vLLM and Ollama is in Ollama and vLLM on a rented GPU.

Price and limits

Your own server has no request quota, but it has a ceiling: one card processes a limited number of tokens per second. So count the cost in card minutes for your load, not as a price per token. A card with more memory can hold more parallel requests, so on a large volume of answers the cost per answer can be lower.