Local LLM

Local LLM Management

This skill provides comprehensive guidance for managing local LLM models via Ollama for development, testing, and production use.

Overview

Ollama provides a simple way to run large language models locally with:

• Easy model management (pull, run, stop)
• OpenAI-compatible API at http://localhost:11434/v1
• GPU acceleration with automatic VRAM management
• Custom model creation via Modelfiles

Two-Tier Model Strategy

For efficient development workflows, use a two-tier approach:

Recommended Model Combinations

Create custom models using the templates in templates/ directory.

Quick Reference Commands

Running Models

# Interactive chat
ollama run <model-name>

# Single prompt (non-interactive)
echo "Your prompt" | ollama run <model-name>

# With verbose output (shows speed metrics)
echo "Your prompt" | ollama run <model-name> --verbose

Model Management

# List all models
ollama list

# Check currently loaded models (VRAM usage)
ollama ps

# Unload model to free VRAM
ollama stop <model-name>

# View model details
ollama show <model-name>

# View Modelfile configuration
ollama show <model-name> --modelfile

# Delete a model
ollama rm <model-name>

# Pull/update a model
ollama pull <model-name>

Service Management

# Check Ollama service status
systemctl status ollama

# Restart Ollama service
sudo systemctl restart ollama

# View Ollama logs
journalctl -u ollama -f

# Manual start (if service not running)
ollama serve

OpenAI-Compatible API

Ollama provides an OpenAI-compatible API at http://localhost:11434/v1.

Python Integration

import openai

# Configure client for local Ollama
client = openai.OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # Required but not validated locally
)

# Chat completion
response = client.chat.completions.create(
    model="your-model-name",
    messages=[{"role": "user", "content": "Your prompt"}],
    temperature=0.6
)
print(response.choices[0].message.content)

Environment Variables

# Standard configuration
export OLLAMA_HOST=http://localhost:11434
export OPENAI_API_BASE=http://localhost:11434/v1
export OPENAI_API_KEY=ollama
export OPENAI_MODEL=your-model-name

cURL Examples

# Test API availability
curl http://localhost:11434/v1/models

# Chat completion
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-name",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Creating Custom Models

Custom models are created using Modelfiles. See templates/ for ready-to-use templates.

Modelfile Structure

# Base model
FROM model-name:tag

# Parameters
PARAMETER temperature 0.6
PARAMETER top_p 0.95
PARAMETER num_ctx 8192
PARAMETER repeat_penalty 1.1

# System prompt
SYSTEM """Your system prompt here"""

# Stop sequences
PARAMETER stop "<|end|>"

Create Model from Modelfile

ollama create my-model -f /path/to/Modelfile

Key Parameters

Performance Optimization

VRAM Guidelines

Speed vs Quality Trade-offs

Troubleshooting

Common Issues

Model won't load / CUDA out of memory

ollama stop --all
sudo systemctl restart ollama

Slow generation speed

• Check nvidia-smi for GPU utilization
• Ensure model fits in VRAM (ollama ps)
• Use smaller model or higher quantization

API connection refused

curl http://localhost:11434/api/tags
ollama serve  # Start if needed

Model not found

ollama list
ollama pull model-name

Performance Diagnostics

# Test with verbose output
echo "test" | ollama run model-name --verbose 2>&1 | grep -E "(eval rate|total duration)"

# Monitor GPU during inference
watch -n 1 nvidia-smi

Recommended Workflow

1. Development: Use fast model (4B) for rapid iteration
2. Pre-commit: Run sanity tests with fast model
3. CI/CD: Use fast model for pipeline tests
4. Validation: Run full tests with quality model (overnight if needed)
5. Release: Final validation with reasoning model

File Locations

Available Templates

See the templates/ directory for ready-to-use Modelfiles:

• fast-model.Modelfile - Quick iteration (4B base)
• reasoning-model.Modelfile - Quality validation (8B reasoning)
• code-generation.Modelfile - Code-focused tasks
• json-output.Modelfile - Structured data generation
• analysis.Modelfile - Scientific analysis and reasoning