Local LLM Gpu Selection

Local LLM GPU Selection & Performance Guide

Problem

Choosing the right hardware for local LLM inference requires understanding the relationship between VRAM capacity, memory bandwidth, GPU compatibility with server chassis, and realistic model quality expectations.

Context / Trigger Conditions

• User asks about running quantized models locally (Ollama, llama.cpp)
• User wants to know which GPU fits their server (Dell R730 or similar 2U)
• User asks about Apple Silicon (Mac Mini/Studio) vs datacenter GPUs for LLMs
• User wants to compare local model quality to Claude (Opus/Sonnet/Haiku) for coding

Key Principle: Memory Bandwidth Is Everything

LLM token generation is memory-bandwidth bound, not compute bound. The formula:

approx tokens/sec = memory_bandwidth_GB_s / model_size_GB

This is why Apple Silicon (high bandwidth unified memory) competes with datacenter GPUs despite having less raw compute.

VRAM Requirements by Model Size

Add ~1-2 GB overhead for KV cache and context. Longer conversations use more.

Dell R730 GPU Compatibility

Constraints

• 2U chassis: Full-height cards fit, but limited to dual-slot width
• PCIe 3.0 x16 slots: 2-3 usable slots depending on riser configuration
• Power: Needs Dell GPU power cable (P/N 0D4J0T) for GPUs >75W TDP
• PSU: Check wattage headroom (dual 750W or 1100W typical)

Compatible GPUs

No external power needed (<=75W):

• Tesla T4: 16 GB, 320 GB/s, 70W — best drop-in option
• Tesla P4: 8 GB, 192 GB/s, 75W — too little VRAM for modern LLMs
• NVIDIA L4: 24 GB, 300 GB/s, 72W — T4 successor, Ada Lovelace, expensive
• NVIDIA A2: 16 GB, 200 GB/s, 60W — worse than T4 in every way, avoid

Requires power cable (>75W):

• Tesla P40: 24 GB, 346 GB/s, 250W — best value per GB
• Tesla V100 PCIe: 32 GB, 900 GB/s, 250W — excellent bandwidth
• Tesla P100 PCIe: 16 GB, 732 GB/s, 250W — same VRAM as T4, not worth it

Won't fit:

• RTX 3090/4090: Too thick (3-slot), too long
• A100: Fits physically but very expensive
• Any consumer RTX: Generally too large for 2U

Multi-GPU Considerations

• Ollama splits model layers across GPUs automatically
• PCIe 3.0 cross-GPU transfer adds ~30-40% latency penalty
• Mismatched GPUs (e.g., T4 + P40) work but the slower card bottlenecks
• R730 PCIe 3.0 limits newer GPU bandwidth (L4 runs at half its rated speed)

Apple Silicon Comparison

Apple Silicon unified memory means ALL system RAM = VRAM with no bus penalty.

A Mac Mini M4 Pro 48GB often matches or beats a T4+L4 multi-GPU setup for LLM inference due to zero cross-GPU overhead and high unified bandwidth.

Best Coding Models (for Ollama)

For coding tasks specifically, prefer dedicated coding models:

1. Qwen 2.5 Coder 32B — best open-source coding model in this size class
2. Codestral 22B — Mistral's dedicated coding model
3. DeepSeek Coder V2 — good quality, efficient
4. Llama 3.1 70B — strong general purpose but needs ~40 GB

Realistic Quality Comparison to Claude

For Claude Code-style agentic coding workflows:

Local models work for simple completions and code questions. They struggle badly with Claude Code's complex multi-step tool-use workflows, long context windows, and self-correction capabilities.

Quantization Quality Guide

From best to worst quality (and largest to smallest):

• FP16: Full precision, baseline quality
• Q8_0: Near-lossless, ~50% size reduction
• Q6_K: Minimal quality loss
• Q5_K_M: Good balance
• Q4_K_M: Recommended default — best quality/size tradeoff
• Q3_K_M: Noticeable degradation on complex reasoning
• Q2_K: Significant quality loss, emergency only

Verification

• Check GPU compatibility: lspci | grep -i nvidia on the host
• Check available VRAM: nvidia-smi inside the GPU VM
• Check model fit: Ollama shows VRAM usage during ollama run
• Check inference speed: Count tokens/sec in Ollama output

Notes

• GPU prices fluctuate significantly in the used market; check current prices
• The T4 is PCIe 3.0 only; newer GPUs in PCIe 3.0 slots run at reduced bandwidth
• Power consumption matters for 24/7 homelab use (electricity cost)
• For Claude Code specifically, API-based Claude models remain significantly superior to any local model for agentic coding workflows