# Local AI Stack > Set up and optimize a complete local AI infrastructure — Ollama, Open WebUI, local RAG pipelines, and private model serving - Skill: `mouadja02/local-ai-stack` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add mouadja02/local-ai-stack` - Raw SKILL.md: https://api.skillmd.com/api/skills/mouadja02/local-ai-stack/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: mouadja02 (https://skillmd.com/u/mouadja02) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/mouadja02/local-ai-stack --- # Local AI Stack **Tier:** POWERFUL **Category:** AI Agents **Domain:** Local AI Infrastructure / Self-Hosted LLM / Privacy-First AI ## Overview Build a complete, private AI stack on your own hardware — from local model serving to RAG pipelines to chat interfaces. This eliminates cloud API dependencies, keeps sensitive data local, and reduces per-token costs to zero after hardware investment. Covers Ollama for model management, Open WebUI for chat interface, and local embedding/RAG pipelines. ## When to Use - Running LLMs locally for privacy, compliance, or cost savings - Setting up Ollama with optimized model configurations - Deploying Open WebUI as a self-hosted ChatGPT alternative - Building local RAG pipelines with local embeddings - Configuring GPU acceleration (CUDA/ROCm/Metal) for inference - Creating an air-gapped AI environment (no internet required) - Benchmarking local models for specific use cases - Setting up a team-shared local AI server ## Quick Start: Full Local Stack in 10 Minutes ### Step 1: Install Ollama ```bash # macOS / Linux curl -fsSL https://ollama.ai/install.sh | sh # Windows (PowerShell) winget install Ollama.Ollama # Verify ollama --version ``` ### Step 2: Pull Models ```bash # Recommended starter models (by use case) ollama pull llama3.1:8b # General purpose (fast, 4.7GB) ollama pull qwen2.5:14b # Best quality/speed balance (8.9GB) ollama pull codellama:13b # Code generation ollama pull nomic-embed-text # Embeddings for RAG (274MB) ollama pull llama3.1:70b # High quality (requires 48GB+ VRAM) # Check downloaded models ollama list ``` ### Step 3: Deploy Open WebUI ```bash # Docker (recommended) docker run -d -p 3000:8080 \ --add-host=host.docker.internal:host-gateway \ -v open-webui:/app/backend/data \ --name open-webui \ --restart always \ ghcr.io/open-webui/open-webui:main # Access at http://localhost:3000 ``` ## GPU Configuration ### NVIDIA (CUDA) ```bash # Check GPU nvidia-smi # Ollama auto-detects CUDA. Verify: ollama run llama3.1:8b "Hello" --verbose # Look for "gpu" in output # For Docker with GPU passthrough: docker run -d --gpus all -p 3000:8080 \ --add-host=host.docker.internal:host-gateway \ -v open-webui:/app/backend/data \ ghcr.io/open-webui/open-webui:main ``` ### Apple Silicon (Metal) ```bash # Ollama uses Metal automatically on macOS # Check memory: models need ~60% of unified memory # 8B model → 8GB Mac minimum # 14B model → 16GB Mac minimum # 70B model → 64GB Mac minimum (M2 Ultra/M3 Max) ``` ### AMD (ROCm) ```bash # Install ROCm toolkit first, then: HSA_OVERRIDE_GFX_VERSION=11.0.0 ollama serve ``` ## Model Selection Guide | Use Case | Model | Size | Speed | Quality | |---|---|---|---|---| | Chat/General | Llama 3.1 8B | 4.7GB | ⚡⚡⚡ | ⭐⭐⭐ | | Best balance | Qwen 2.5 14B | 8.9GB | ⚡⚡ | ⭐⭐⭐⭐ | | Coding | CodeLlama 13B | 7.4GB | ⚡⚡ | ⭐⭐⭐⭐ | | Coding (best) | DeepSeek Coder V2 | 8.9GB | ⚡⚡ | ⭐⭐⭐⭐⭐ | | Reasoning | Qwen 2.5 32B | 19GB | ⚡ | ⭐⭐⭐⭐⭐ | | Max quality | Llama 3.1 70B | 40GB | 🐌 | ⭐⭐⭐⭐⭐ | | Embeddings | nomic-embed-text | 274MB | ⚡⚡⚡ | ⭐⭐⭐⭐ | | Vision | LLaVA 1.6 13B | 8GB | ⚡⚡ | ⭐⭐⭐⭐ | ## Local RAG Pipeline ### Architecture ``` Documents → Chunking → Local Embeddings → Vector Store → Query ↓ ↓ ↓ ↓ LangChain nomic-embed-text ChromaDB Ollama LLM ``` ### Implementation ```python from langchain_ollama import OllamaEmbeddings, ChatOllama from langchain_chroma import Chroma from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain_community.document_loaders import DirectoryLoader # 1. Load documents loader = DirectoryLoader('./docs', glob='**/*.md') docs = loader.load() # 2. Chunk splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200) chunks = splitter.split_documents(docs) # 3. Embed locally (no API calls!) embeddings = OllamaEmbeddings(model='nomic-embed-text') vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory='./chroma_db') # 4. Query with local LLM llm = ChatOllama(model='qwen2.5:14b', temperature=0) retriever = vectorstore.as_retriever(search_kwargs={'k': 5}) def ask(question: str) -> str: docs = retriever.invoke(question) context = '\n'.join([d.page_content for d in docs]) response = llm.invoke(f"Context:\n{context}\n\nQuestion: {question}") return response.content ``` ## Ollama API for Custom Integrations ```typescript // Ollama exposes an OpenAI-compatible API at localhost:11434 const response = await fetch('http://localhost:11434/api/chat', { method: 'POST', body: JSON.stringify({ model: 'qwen2.5:14b', messages: [{ role: 'user', content: 'Hello!' }], stream: false, }), }); // Or use OpenAI SDK with Ollama backend: import OpenAI from 'openai'; const client = new OpenAI({ baseURL: 'http://localhost:11434/v1', apiKey: 'ollama', // required but unused }); ``` ## Performance Optimization ```bash # Ollama environment variables OLLAMA_NUM_PARALLEL=4 # Concurrent requests OLLAMA_MAX_LOADED_MODELS=2 # Models in VRAM simultaneously OLLAMA_GPU_OVERHEAD=0 # Reduce if OOM OLLAMA_FLASH_ATTENTION=1 # Enable flash attention (faster) # Custom Modelfile for fine-tuned inference cat > Modelfile <