# Integration Llama Cpp

> llama.cpp integration for DSPy

- Skill: `j33bs/integration-llama-cpp` (Agent Skill)
- Install (CLI): `npx skillmds@latest add j33bs/integration-llama-cpp`
- Raw SKILL.md: https://api.skillmd.com/api/skills/j33bs/integration-llama-cpp/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: j33bs (https://skillmd.com/u/j33bs)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/j33bs/integration-llama-cpp

---


# llama.cpp Integration

## 🎯 Trigger Conditions
Use when asked about running DSPy programs with llama.cpp, CPU inference, or GGUF model support.

## 📚 Prerequisites
- `llama-cpp-python` package installed
- GGUF model available
- Python 3.8+

## 🛠️ llama.cpp Integration Patterns

### 1. Basic llama.cpp Setup
```python
from llama_cpp import Llama

# Load GGUF model
llm = Llama(
    model_path="./model.gguf",
    n_ctx=2048,
    n_threads=8
)

# Generate
output = llm(
    "Your prompt",
    max_tokens=1024,
    temperature=0.7
)
```

### 2. llama.cpp with DSPy
```python
import dspy
from llama_cpp import Llama

# Create llama.cpp engine
llama = Llama(model_path="./model.gguf")

# Create DSPy LM wrapper
class LlamaCPP_LM(dspy.LM):
    def __init__(self, llama):
        super().__init__()
        self.llama = llama
    
    def __call__(self, prompt, **kwargs):
        # Convert to llama.cpp format
        output = self.llama(prompt, **kwargs)
        return [output["choices"][0]["text"]]

# Use with DSPy
llama_lm = LlamaCPP_LM(llama)
dspy.settings.configure(lm=llama_lm)
```

### 3. Quantization Options
```python
# Different quantization levels
models = {
    "q4_k_m": "./model-q4_k_m.gguf",  # Good balance
    "q8_0": "./model-q8_0.gguf",  # Higher quality
    "q2_k": "./model-q2_k.gguf"  # Lower quality, smaller size
}

llm = Llama(model_path=models["q4_k_m"])
```

### 4. Performance Tuning
```python
llm = Llama(
    model_path="./model.gguf",
    # CPU settings
    n_threads=8,
    n_batch=512,
    # GPU offload
    n_gpu_layers=35,
    # Context
    n_ctx=2048
)
```

## ⚠️ Pitfalls
- **Speed**: llama.cpp is slower than GPU inference
- **Quality**: Quantization affects output quality
- **Memory**: Large models require significant RAM
- **Compatibility**: Not all models support GGUF

## 📖 References
- [llama.cpp Documentation](https://github.com/ggerganov/llama.cpp)
- [DSPy llama.cpp](https://dspy-docs.vercel.app/docs/integration/llama-cpp)