Your first local model with llama.cpp

llama.cpp is the open-source engine that runs most local models. Its server speaks the same API as the large cloud providers, so most tools can use it without changes.

1. Download a model

Qwen3 8B, compressed to 4 bits (Q4_K_M), 5.0 GB: it fits on an 8 GB card.

mkdir -p ~/models
curl -L -o ~/models/Qwen3-8B-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen3-8B-GGUF/resolve/main/Qwen3-8B-Q4_K_M.gguf

2. Serve it

docker run --rm --gpus all \
  -p 127.0.0.1:8080:8080 \
  -v ~/models:/models \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Qwen3-8B-Q4_K_M.gguf \
  -ngl 99 -c 8192 -fa on \
  --host 0.0.0.0 --port 8080

• -ngl 99: put every layer on the GPU.
• -c 8192: the context, how much text the model can keep in mind at once. More context, more VRAM.
• -fa on: flash attention, faster and lighter on memory.
• -p 127.0.0.1:8080:8080: reachable from this machine only. Do not open it to your network before you have put something in front of it.

3. Talk to it

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Say hello in one sentence."}]}'

Or open http://127.0.0.1:8080 in a browser: the server ships a small chat page.

A trap we measured

To save memory you can compress the context cache with -ctk q8_0 -ctv q8_0. Do not go further on the V side: with -ctv q4_0, flash attention silently switches off and reading a long prompt became 21 times slower on our card (62.7 instead of 1,330 tokens per second), while generation speed did not change. A test that only measures generation would never see it.