Local AI models without a powerful computer

A local AI model is one that runs on your own hardware. You download the weights once, and your graphics card handles the requests instead of a cloud service. At home, such a model runs on a video card with 12–24 GB of memory. If you do not have that card, or yours is weaker, the same open models run on a rented GPU server at gpu.nz: you start the server for the time you need, use the chat or the API, and stop it when you are done.

What a local model is

Most modern language models are released with open weights. You can download the model file, run it without a subscription, and manage it with Ollama, llama.cpp or a web interface such as Open WebUI. A cloud service receives your requests on its own servers, while a local model answers on your hardware, so your data goes nowhere unless you send it yourself.

Images and video follow the same pattern. Stable Diffusion and FLUX run in Forge and ComfyUI, Wan video runs in ComfyUI or Wan2GP, and Whisper turns speech into text. Each task needs its own amount of memory, but the rule is the same: the model has to fit entirely in video memory. If it does not, part of the work moves to system RAM and everything slows down many times over.

Cloud or your own hardware

A cloud API has clear advantages: you do not have to think about hardware, and the largest models are available at once. It also has a downside. Each request goes to someone else's server, the cost grows with the amount of work, and the terms can change without your involvement. A local model needs an investment in hardware or rental, but it gives you full control: you choose the version, the settings and the moment to update.

The practical rule is this. If the task involves personal or commercial data that must not go to third-party services, run the model yourself. If you need the strongest model for a one-off task, a cloud API is often simpler. Many teams combine the two: a local model for drafts and sensitive data, and a cloud model for hard cases.

How much video memory you need

The model weights have to sit entirely in video memory. For a language model in 4-bit quantisation (Q4), the weights take roughly the number of billion parameters multiplied by 0.5 GB. Add about 20% for context and buffers. Q4 is the most common choice in Ollama, and the table below assumes it.

These figures assume a short context. A long chat or a large document adds memory for the KV cache. The full calculation, with formulas, is in our guide how much video memory an LLM needs.

What to run: text, code, images, video and speech

Each task has its own set of models and a ready template. The template is already configured, so you do not need to install drivers or build an environment.

For multilingual text, start with Qwen or Gemma. Models of the same size differ noticeably in language quality, so test the answers on your own tasks.

Quantisation: what it is and what it affects

Quantisation lowers the precision of the weights so the model takes less memory. Q4 is the most common trade-off: quality is usually close to the original, and the file is three to four times smaller than fp16. Q8 loses almost nothing but takes one and a half to two times the space of Q4. Variants below 4 bits save even more memory, but the answers get noticeably worse, so avoid them for serious work.

A PC or a rented server

Your own graphics card pays off if you use models every day and are ready to cover the hardware and the electricity. Renting makes sense when you need a bigger card than the one you have, or when the work comes in bursts. The gpu.nz server bills per minute from your balance. While the server is stopped, the GPU is not charged, but the disk is billed as storage until you delete the server.

For a first try, a 24 GB card is enough: it fits 27–34B models in Q4. Current prices are in the catalogue; the cost of a minute depends on the card model. For ways to cut the bill, see how to save money on GPU rental.

When your own card is no longer enough

• The model does not fit even in Q4: you need a 48–80 GB card, for example an RTX A6000, an L40S or an A100.
• You need a long context: long documents grow the KV cache, so leave headroom from the start.
• Several people use the model at once: for parallel requests, a server with an API such as vLLM is more convenient.
• Video models use a lot of memory: Wan 2.2 runs on 16 GB, but with limits on clip length and resolution.

What to check before you buy or rent

• The video memory your model and quantisation need, with room for the context.
• System RAM: if the model does not fit in video memory, some layers move to RAM. For experiments with large models, take 32 GB of system RAM or more.
• Disk space: a 70B model alone is about 43 GB, and models pile up quickly. A reserve of around 100 GB is a sensible minimum.
• An NVIDIA driver: Ollama on a PC needs a recent driver, and nvidia-smi shows whether it works.
• Power supply and cooling: powerful cards draw a lot of power and run hot, so check the case and the power supply against the card's specification.

How to start on gpu.nz

1. Choose a card from the memory table. For 27–34B models you need a 24 GB card.
2. Create a server with the Open WebUI + Ollama template or a ready chat. If you enter an Ollama model name in the «Model» field, it is downloaded on the first start.
3. Wait 2–5 minutes for the server to start. Large models take longer: a 60 GB file needs 10–30 minutes.
4. Open the interface with the «Open» button on the server page, or connect over SSH and pull the model from the terminal.

ollama pull qwen3:14b
ollama run qwen3:14b "Explain quantisation in three sentences"
ollama ps

The ollama ps command shows how much memory the model uses and whether it runs on the GPU. If the processor column shows a share of CPU, the model did not fit into video memory: pick a more compact quantisation or a bigger card. Leave room on the disk as well: the weights of a 70B model take about 43 GB, and Ollama keeps every model on the server disk.

Common mistakes beginners make

• They count only the weights and forget the context. The model fits on the first message, but on a long document it slows down or crashes.
• They take full precision for quality and cannot run the model on their card. For chat, Q4 or Q8 is usually enough.
• They expect a 70B model to run on a 24 GB card. Without offloading to system RAM it runs slowly or does not start at all.
• They never check whether the model runs on the GPU. The ollama ps command shows at once where the work is done.

Privacy

A model on your PC sends your requests nowhere. Renting works differently: the server sits in a data centre rather than in your home, and the hardware belongs to someone else. Access to the interfaces is protected by a token, and SSH opens with your key. For confidential data, it is still better to keep the model at home. If you work on a server, download your results and delete the server when you no longer need it.

In short: if the video card at home has enough memory for the model, run it locally. If not, renting gives you the same model on the card you need, with no hardware purchase, and you pay only for the minutes you use.