Before you start: hardware, honestly
The question everyone asks first: what do I need? Here is what we measured on our own machine, two 16 GB NVIDIA cards (an RTX 5080 and an RTX 4070 Ti SUPER).
The rule: video memory decides
A model runs fast when it fits entirely in the card's video memory (VRAM). The file size of the model is a good first estimate of what it needs, plus room for the context (the "KV cache") and a margin.
| Model | File / VRAM | Fits on |
|---|---|---|
| Qwen3 8B, Q4_K_M | 5.0 GB file | an 8 GB card |
| A 9B model, Q8, long context | 9.4 GB used, measured | a 12 GB card |
| A 27B model, IQ3_XXS, 64k context | 14.2 GB used, measured | a 16 GB card, just |
| Qwen3.6 35B-A3B (mixture of experts), Q4_K_M | 22.1 GB file | 16 GB card + the experts in system RAM |
System RAM matters too
"Mixture of experts" models only use a small part of their weights for each word, so their experts can live in system RAM while the rest stays on the card. That trades VRAM for RAM: count 20 GB of free RAM for the 35B above.
What to buy first
- An NVIDIA card with as much VRAM as you can afford. VRAM matters more than raw speed. 12 GB is a comfortable start, 16 GB opens the 27B class.
- 32 GB of RAM or more if you want mixture-of-experts models.
- A fast SSD with room: a few models take 50 to 100 GB.
- Linux. Everything in this series runs on Linux; it is where the tools are first.
Prices change every month, so we do not quote them. Compare VRAM per euro, not benchmark scores.