Ollama Local Setup

ollama-local-setup

Production guide for standing up and fixing a local Ollama server on Windows, where models live on a non-default disk and must be visible to clients connecting via http://localhost:11434 (both the native /api/tags and the OpenAI-compatible /v1/models). Also covers performance tuning — diagnosing and fixing GPU starvation caused by oversized context windows (KV cache spilling to CPU RAM).

When to Use

Trigger this skill when any of the following are true:

• ollama list is empty, /api/tags returns {}, or /v1/models returns {"object":"list","data":null}.
• User wants local-only inference (Hermes provider=ollama, base_url=http://localhost:11434/v1, api_key=ollama).
• Models were moved to another drive and the server no longer sees them.
• After a reboot or relaunch the local models "vanished."
• You are about to start an ollama serve / ollama app and port 11434 may already be in use.
• Local inference is very slow (single-digit tok/s) or GPU utilization sits near ~5% even on a powerful GPU (e.g. RTX 3090 Ti). Classic KV-cache spill from an oversized context_length.

Prerequisites

• Windows host (PowerShell primary). Linux/macOS notes provided where relevant.
• Ollama installed (app or CLI). Default install path: $env:LOCALAPPDATA\Programs\Ollama\ollama app.exe.
• A target models directory on a non-default drive, e.g. YOUR_MODELS_DIR, containing blobs/ (multi-GB files) and manifests/ (model tags). Set $env:OLLAMA_MODELS to that path.
• For performance tuning: an NVIDIA GPU with nvidia-smi available.
• For Hermes integration: Hermes config must set provider=ollama, base_url=http://localhost:11434/v1, api_key=ollama, and context_length>=64000 (Hermes refuses to init below 64K).

Procedure

Phase 1 — Fix "empty models" on Windows (three gotchas)

There are three silent failure modes on Windows that cause ollama list to return empty:

1. OLLAMA_MODELS must use BACKSLASHES on Windows (e.g. YOUR_MODELS_DIR via $env:OLLAMA_MODELS). Forward slashes are silently rejected — Ollama falls back to the empty default C:\Users\<user>\.ollama\models, and /api/tags returns {}. This is silent: no error, just empty.
2. The Windows Ollama app (ollama app.exe, the tray app) IGNORES OLLAMA_MODELS entirely. It always serves from its default path. Fix by redirecting the default path itself with a directory junction (see below), not by setting the env var alone.
3. Only ONE server per port. Two ollama serve/app processes on 11434 → /v1/models returns null and clients get 404s. Kill all Ollama processes, then start exactly one.

Step 1 — Set the env var correctly (backslashes)

# Point at YOUR_MODELS_DIR (backslashes on Windows)
$env:OLLAMA_MODELS = "YOUR_MODELS_DIR"
# User scope (re-logon to apply; new shells only)
setx OLLAMA_MODELS $env:OLLAMA_MODELS

# Machine scope (requires admin)
setx OLLAMA_MODELS $env:OLLAMA_MODELS /M

HARD RULE: Always use backslashes on Windows. Forward slashes are the #1 silent failure.

Step 2 — Redirect the default path with a directory junction

The app ignores the env var, so redirect the default path C:\Users\<user>\.ollama\models to your custom directory:

# Only if the default dir is empty or you've moved its contents
cmd /c rmdir "C:\Users\<user>\.ollama\models"

# Create the junction (reboot-safe, filesystem-level)
cmd /c mklink /J "C:\Users\<user>\.ollama\models" $env:OLLAMA_MODELS

A junction is filesystem-level and reboot-safe — it survives restarts and does not depend on an env var being set.

HARD RULE: Do not trust the app to honor OLLAMA_MODELS. Use the junction. Setting the env var alone will not move the app's models.

Step 3 — Kill stale processes, then start exactly ONE server

# Kill all Ollama processes
taskkill /F /IM "ollama app.exe" /T
taskkill /F /IM ollama.exe /T

# Launch the normal app (tray + supervised server)
start "" "$env:LOCALAPPDATA\Programs\Ollama\ollama app.exe"

HARD RULE: Do not run both ollama app.exe and a manual ollama serve. Pick one. Two listeners on 11434 break /v1/models.

Linux/macOS note: OLLAMA_MODELS works normally there and forward slashes are fine. The junction trick is Windows-specific. On Linux a symlink (ln -s) achieves the same redirect.

Step 4 — Verify

See Verification below. If still empty, re-read OLLAMA_MODELS with backslashes and confirm the junction target exists and contains blobs/ + manifests/.

Load references/windows-troubleshooting.md when you need copy-paste command recipes (env set, junction, kill+relaunch, verify) with placeholders for a specific user/machine.

Phase 2 — Performance tuning (GPU starvation from oversized context)

Symptom: a large model (e.g. qwen3.6:27b, 27B) generates ~3 tok/s and nvidia-smi shows ~5% GPU compute util during generation, even on a strong GPU (RTX 3090 Ti, 24 GB). VRAM shows the model loaded but the KV cache is elsewhere.

Root cause: Ollama sizes the KV cache from the effective context length at load = min(client context_length request, OLLAMA_CONTEXT_LENGTH env var). A 27B model at 256K context needs a ~40 GB KV cache. When that exceeds free VRAM, Ollama silently allocates it in CPU RAM and shuffles KV data per token → GPU idle, throughput collapses. The binding cap is the server env var OLLAMA_CONTEXT_LENGTH; lowering the Hermes/client context_length alone does NOT free VRAM unless the env var is also reduced.

Step 1 — Persist tuning in Windows User env

setx OLLAMA_CONTEXT_LENGTH 65536
setx OLLAMA_KV_CACHE_TYPE q4_0
setx OLLAMA_FLASH_ATTENTION true

• OLLAMA_CONTEXT_LENGTH=65536 — 64K satisfies Hermes's context_length>=64000 requirement and is small enough for the KV cache to fit on GPU for most chat.
• OLLAMA_KV_CACHE_TYPE=q4_0 — 4-bit KV cache. qwen3.6:27b ≈ 17.8 GB + q4 KV ≈ 2.6 GB ≈ 20.4 GB of 24 GB, leaving headroom.
• OLLAMA_FLASH_ATTENTION=true — faster attention kernels.

Step 2 — Restart the server

taskkill /F /IM "ollama app.exe" /T
taskkill /F /IM ollama.exe /T
# Then launch one server (app or ollama serve, not both)
start "" "$env:LOCALAPPDATA\Programs\Ollama\ollama app.exe"

Step 3 — Verify performance improvement

In one shell, monitor GPU:

nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 1

In another shell, run a 200-token generation. Use think:false for Qwen3 (it defaults to thinking mode, which inflates latency).

Load references/performance-tuning.md when diagnosing GPU starvation or when you need the exact generation-speed probe and before/after numbers (3→33 tok/s, 5%→90% GPU util on a 3090 Ti).

Pitfalls

• Forward-slash env var is the #1 silent failure. Always set backslashes on Windows. A forward-slash YOUR_MODELS_DIR is silently rejected; Ollama falls back to the empty default with no error.
• Don't trust the app to honor OLLAMA_MODELS. Use the junction. Setting the env var alone will not move the app's models.
• Don't run both ollama app.exe and a manual ollama serve. Pick one. Two listeners on 11434 break /v1/models (returns null, clients get 404s).
• Oversized context_length starves the GPU (performance, not visibility). A 27B model at 256K context allocates a ~40 GB KV cache that won't fit VRAM, so Ollama spills it to CPU RAM → ~5% GPU util, ~3 tok/s. Fix: cap OLLAMA_CONTEXT_LENGTH (e.g. 65536) and set OLLAMA_KV_CACHE_TYPE=q4_0. Reducing the client context_length alone does not help — the server env var is the binding cap.
• Verify with ollama list and raw curl, not fragile JSON-parsing one-liners. Prefer the official CLI and a direct curl http://localhost:11434/api/tags. A mis-parsed probe can report "empty" when the server is actually healthy (false alarm). See scripts/verify-ollama.ps1.
• Confirm models are actually on disk at the junction target before blaming the server: blobs/ should contain multi-GB files and manifests/ should contain model tags.
• Qwen3 defaults to thinking mode, which inflates latency during speed probes. Always pass think:false when measuring generation throughput.

Verification

Run scripts/verify-ollama.ps1 (PowerShell, no external deps) for a deterministic probe that checks /api/tags, /v1/models, OLLAMA_MODELS, and the default-vs-custom path; reports a clear PASS/FAIL.

Or manually:

# CLI check
ollama list

# Native API
curl -s http://localhost:11434/api/tags

# OpenAI-compatible API
curl -s http://localhost:11434/v1/models

Expected: all configured models appear under both endpoints, including the one the client needs (e.g. qwen3.6:27b for Hermes).

For Hermes integration, ensure config sets:

• provider=ollama
• base_url=http://localhost:11434/v1
• api_key=ollama
• context_length>=64000 (Hermes refuses to init below 64K)

For performance verification, watch nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 1 in one shell while running a 200-token generation in another. Expect GPU util to jump from ~5% to ~90% and throughput from ~3 tok/s to ~33 tok/s on a 3090 Ti with the tuning recipe applied.

Support Files

• references/windows-troubleshooting.md — Load when you need copy-paste command recipes (env set, junction, kill+relaunch, verify) with placeholders for a specific user/machine.
• references/performance-tuning.md — Load when diagnosing GPU starvation or when you need the exact generation-speed probe and before/after numbers.
• scripts/verify-ollama.ps1 — Load and run for a deterministic end-to-end probe (checks /api/tags, /v1/models, OLLAMA_MODELS, and default-vs-custom path; reports PASS/FAIL).