Running Ollama on 8GB VRAM — the honest setup guide

Marketing pages claim you can run LLMs locally on any laptop. The reality on 8GB VRAM cards (RTX 3060 8GB, RTX 4060, laptop GPUs) is more nuanced. Here is what actually works, at what speed, and how to avoid the most common failures.

What Ollama does

Ollama wraps llama.cpp with a simple CLI + REST API. Pull a model, run it, chat with it. Handles quantization, GPU offloading, and model swapping automatically.

Install (any platform):

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
winget install Ollama.Ollama

Verify:

ollama --version

The 8GB VRAM ceiling

VRAM constraints determine which models load without spilling to system RAM (which slows inference 5-10x).

Practical limits on 8GB VRAM:

Reality: 7B models are the sweet spot for 8GB. 13B/14B run but slower. 30B+ is not practical.

Recommended models for 8GB VRAM

Best all-around:

ollama pull qwen3:7b
# Fallback if qwen3 isn't in your Ollama registry yet:
ollama pull qwen2.5:7b

Qwen 3 7B (or its 2.5 predecessor) is competitive with much larger models on reasoning and coding.

Best for coding:

ollama pull qwen3-coder:7b
# Fallback:
ollama pull qwen2.5-coder:7b

Trained specifically for code. Outperforms 13B general models on programming tasks.

Best for speed:

ollama pull llama3.2:3b

3B model runs at 60-80 tokens/sec on 8GB. Great for autocomplete/fast tasks. Also try Gemma 4 (small variant) — very fast, tuned for on-device use.

Best for instruction-following:

ollama pull mistral:7b

Strong at following complex prompts. Good default. Mistral Small variants are also available if you want the latest generation.

Understanding quantization

Quantization = compressing model weights to reduce memory. Trade-offs:

For 8GB VRAM, Q4_K_M is the standard trade-off. Use Q5_K_M if you have headroom.

Specify quantization explicitly:

ollama pull qwen3:7b-instruct-q5_K_M

Common OOM errors and fixes

Error: CUDA out of memory

Model + context window exceeds VRAM. Fix by reducing context length:

# Create custom Modelfile with smaller context
cat > Modelfile <<EOF
FROM qwen3:7b
PARAMETER num_ctx 2048
EOF

ollama create qwen-small -f Modelfile
ollama run qwen-small

Default context is 4096. Reducing to 2048 frees ~500MB VRAM.

Error: Model loads but responses are slow (< 5 tokens/sec)

Model spilled to CPU. Check GPU utilization:

# NVIDIA
nvidia-smi

# See if Ollama process is using GPU

If GPU is idle during inference, model didn’t fit. Try smaller model or heavier quantization.

Speed optimizations

1. Flash Attention (2-3x speedup on supported GPUs)

Set environment variable before starting Ollama:

$env:OLLAMA_FLASH_ATTENTION=1   # Windows PowerShell
export OLLAMA_FLASH_ATTENTION=1  # Mac/Linux

Then restart the Ollama service. Supported on RTX 3000+ / Ampere and newer.

2. Keep model loaded between requests

Ollama unloads models after 5 min of inactivity by default. Extend:

$env:OLLAMA_KEEP_ALIVE="30m"

3. Batch prompts

For batch processing, use the API directly instead of ollama run:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:7b",
  "prompt": "Explain Kubernetes RBAC in 2 sentences",
  "stream": false
}'

Measuring actual performance

Time a response:

time echo "Explain Kubernetes in one paragraph" | ollama run qwen3:7b

Check tokens/second in Ollama logs:

# Windows
Get-Content "$env:LOCALAPPDATA\Ollama\server.log" -Tail 20

# Mac/Linux
tail -f ~/.ollama/logs/server.log

Look for eval rate: X tokens/s in the output.

Reproduce this yourself (no download required)

Try Ollama in browser without local install:

Google Colab (free GPU): https://colab.google

Create new notebook, run:

!curl -fsSL https://ollama.com/install.sh | sh
!ollama serve &
!ollama pull qwen3:7b
!echo "Explain Docker" | ollama run qwen3:7b

Free Colab GPU access is subject to Colab’s current usage limits (which change) — quotas may not last long enough for extensive testing. Good for a first look at which model quality matches your needs before committing to a local install.

What NOT to expect

• Frontier-model quality (Claude Opus 5, GPT-5.6-sol) on any local 7B model. Local 7B LLMs cover most day-to-day tasks well but aren’t a full replacement for hosted frontier models.
• Long context — Local models handle 4K-32K tokens well. 128K+ context is memory-prohibitive on 8GB VRAM.
• Multi-modal — most 7B models are text-only. Vision models exist (LLaVA, Qwen-VL small variants) but require more VRAM.

When 8GB isn’t enough

If frequent OOM or slow speeds:

• Upgrade to 12GB (RTX 3060 12GB, RTX 4070) — comfortably runs 13B models
• Upgrade to 16GB+ (RTX 4080 / 4090 / RTX 5090) — runs 30B models
• Use API instead (Claude, OpenAI, or Groq for cloud-hosted open-source at API convenience)

The honest bottom line

8GB VRAM runs 7B models well. That is enough for:

• Local code completion
• Draft article writing (with human editing)
• Simple summarization
• Development experimentation
• Learning how LLMs work

It is not enough for production applications requiring frontier-model quality. For those, use hosted APIs (Claude, OpenAI, Anthropic) and treat local models as a backup / dev tool.

Ollama + Qwen 3 7B (or Qwen 2.5 7B as fallback) on 8GB VRAM = the current best free local AI setup for developers.