Dspy Ollama

Ollama — Run DSPy with Local Models

Guide the user through running DSPy with local models via Ollama. No API keys, no cloud costs, full privacy.

Step 1 — Gather context

Before diving in, confirm:

1. Model: Do you have a specific model in mind (e.g., llama3.1:8b, qwen2.5:7b), or do you need a recommendation based on your hardware?
2. Location: Is Ollama running locally, or on a remote GPU server you want to point DSPy at?
3. Task: Classification, RAG, optimization, or general pipelines? (This affects context window and model recommendations.)

What is Ollama

Ollama is a local LLM runner that wraps llama.cpp. It downloads, manages, and serves models locally with a simple CLI. DSPy connects to it through LiteLLM's ollama_chat/ provider.

Setup

Install Ollama

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows: download from ollama.com

Start the server and pull a model

# Start the Ollama server (runs in background)
ollama serve

# Pull a model (one-time download)
ollama pull llama3.1

# Quick test
ollama run llama3.1 "What is DSPy?"

Connect DSPy to Ollama

import dspy

lm = dspy.LM(
    "ollama_chat/llama3.1",
    api_base="http://localhost:11434",
    api_key="",  # required but ignored
    temperature=0.7,
    num_ctx=8192,  # IMPORTANT: set context window explicitly
)
dspy.configure(lm=lm)

# Test it
classify = dspy.Predict("text -> sentiment")
result = classify(text="DSPy makes AI development easier")
print(result.sentiment)

Note: dspy.OllamaLocal is deprecated. Use dspy.LM("ollama_chat/...") instead.

Model selection guide

Quick recommendations

Prototyping (fast iteration, good quality):
  → llama3.1:8b or qwen2.5:7b

Best quality on consumer hardware (16GB+ RAM):
  → llama3.1:8b or phi4:14b

Complex reasoning:
  → deepseek-r1:14b or qwen3:14b

Coding tasks:
  → qwen2.5-coder:7b or codellama:13b

Minimal resources (8GB RAM):
  → llama3.2:3b or gemma2:2b or qwen2.5:3b

Context window gotcha (critical)

Ollama defaults to 4096 tokens regardless of the model's actual capacity. This is the #1 source of issues when running DSPy with Ollama. DSPy prompts with few-shot demos can easily exceed 4096 tokens.

Always set num_ctx explicitly:

# BAD — defaults to 4096 tokens, will silently truncate
lm = dspy.LM("ollama_chat/llama3.1", api_base="http://localhost:11434", api_key="")

# GOOD — set context window to match model capability
lm = dspy.LM(
    "ollama_chat/llama3.1",
    api_base="http://localhost:11434",
    api_key="",
    num_ctx=8192,  # 8K is a safe default for most tasks
)

Larger context = more VRAM. If you get OOM errors, reduce num_ctx:

Environment variables

Setting env vars per platform:

• macOS (app): launchctl setenv OLLAMA_HOST "0.0.0.0:11434" then restart Ollama
• Linux (systemd): sudo systemctl edit ollama.service, add Environment= lines under [Service], then systemctl daemon-reload && systemctl restart ollama
• Everywhere else: export OLLAMA_HOST="0.0.0.0:11434" before ollama serve

Remote Ollama (GPU server)

If Ollama runs on a different machine (e.g., a GPU server), point DSPy to it:

lm = dspy.LM(
    "ollama_chat/llama3.1:8b",
    api_base="http://gpu-server:11434",  # remote Ollama host
    api_key="",
    num_ctx=8192,
)

Make sure the server has OLLAMA_HOST=0.0.0.0:11434 set.

Performance tuning

GPU acceleration

Ollama automatically uses GPU if available. Check with:

ollama ps  # shows which models are loaded and GPU/CPU split

Apple Silicon optimization

Ollama runs natively on Apple Silicon using Metal. Performance tips:

• M1/M2 (8GB): 8B models work well with num_ctx=4096
• M1/M2 Pro (16GB): 8B models with num_ctx=8192, or 14B with num_ctx=4096
• M1/M2 Max (32GB+): 70B quantized models with num_ctx=4096
• M3/M4 Max (64GB+): 70B models with num_ctx=8192

Embedding models (for retrieval pipelines)

Ollama can serve embedding models for dspy.Embedder:

embedder = dspy.Embedder(
    "ollama/nomic-embed-text",
    api_base="http://localhost:11434",
    api_key="",
    batch_size=200,  # default; reduce if Ollama OOMs on large batches
)

Pull embedding models the same way: ollama pull nomic-embed-text

Model management

ollama list                    # show installed models and sizes
ollama pull llama3.1:8b        # download or update a model
ollama rm codellama:7b         # remove a model to free disk space
ollama ps                      # show running models and GPU/CPU split
ollama show llama3.1:8b        # show model details (parameters, template, license)
curl http://localhost:11434/api/tags  # verify Ollama is running (useful in scripts)

Run Ollama as a background service instead of ollama serve &:

• macOS: The Ollama app runs as a menu bar service automatically
• Linux: sudo systemctl enable --now ollama (installed by the install script)

Per-module model assignment

Use a big model for hard tasks and a small model for simple ones:

import dspy

big = dspy.LM("ollama_chat/llama3.1:8b", api_base="http://localhost:11434",
              api_key="", num_ctx=8192)
small = dspy.LM("ollama_chat/llama3.2:3b", api_base="http://localhost:11434",
                api_key="", num_ctx=4096)

dspy.configure(lm=small)  # default: cheap model

class Pipeline(dspy.Module):
    def __init__(self):
        self.classify = dspy.Predict("text -> category")
        self.analyze = dspy.ChainOfThought("text, category -> analysis")

    def forward(self, text):
        cat = self.classify(text=text)
        return self.analyze(text=text, category=cat.category)

pipeline = Pipeline()
pipeline.classify.set_lm(small)   # simple task → small model
pipeline.analyze.set_lm(big)      # complex task → big model

Running DSPy optimization with Ollama

Optimization works with local models but is significantly slower than cloud APIs. Tips:

import dspy

lm = dspy.LM("ollama_chat/llama3.1:8b", api_base="http://localhost:11434",
             api_key="", num_ctx=8192)
dspy.configure(lm=lm)

# Tip 1: Start with BootstrapFewShot (fastest optimizer)
optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
optimized = optimizer.compile(program, trainset=trainset)

# Tip 2: For MIPROv2, use auto="light" (fewest trials)
optimizer = dspy.MIPROv2(metric=metric, auto="light")
optimized = optimizer.compile(program, trainset=trainset)

# Tip 3: Use a bigger model as teacher, smaller as student
teacher_lm = dspy.LM("ollama_chat/llama3.1:70b", api_base="http://localhost:11434",
                     api_key="", num_ctx=8192)
optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
with dspy.context(lm=teacher_lm):
    optimized = optimizer.compile(program, trainset=trainset)
# Deploy optimized program with the smaller model

Expect hours, not minutes for optimization with local models. A MIPROv2 auto="medium" run that takes 5 minutes with GPT-4o-mini might take 2-4 hours with a local 8B model.

Ollama vs vLLM

Recommended workflow: Develop with Ollama locally, deploy with vLLM in production. The DSPy code is identical — only the LM config line changes:

# Development (Ollama)
lm = dspy.LM("ollama_chat/llama3.1:8b", api_base="http://localhost:11434", api_key="")

# Production (vLLM)
lm = dspy.LM("openai/meta-llama/Llama-3.1-8B-Instruct", api_base="http://gpu-server:8000/v1", api_key="none")

Gotchas

1. Context window defaults to 4096 — always set num_ctx explicitly. DSPy optimized prompts with few-shot demos easily exceed 4096 tokens.
2. api_key="" is required — even though Ollama doesn't use it, LiteLLM requires the parameter.
3. First request is slow — Ollama loads the model into memory on the first call. Subsequent calls are fast.
4. OOM errors — reduce num_ctx or switch to a smaller model. Check VRAM with ollama ps.
5. dspy.OllamaLocal is deprecated — use dspy.LM("ollama_chat/...") instead.

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• LM configuration basics (providers, parameters, caching) — /dspy-lm
• Production serving with vLLM — /dspy-vllm
• Reducing costs (model routing, caching) — /ai-cutting-costs
• Switching models without breaking things — /ai-switching-models
• Install /ai-do if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

Additional resources

• dspy.LM API docs
• dspy.Embedder API docs
• Ollama model library
• For API details, see reference.md
• For worked examples, see examples.md