Ollama RAG

Ollama RAG Guide

Build RAG systems with Ollama - run locally or use cloud for massive models.

Ollama Cloud Models (Dec 2025)

Access via ollama signin (v0.12+). No local storage needed, privacy preserved.

# Sign in to access cloud
ollama signin

# Run cloud models
ollama run deepseek-v3.2:cloud
ollama run qwen3-coder:480b-cloud
ollama run minimax-m2:cloud

Local Models (Dec 2025)

Reasoning Models

Fast/Efficient Models

Embedding Models

Recommendation: snowflake-arctic-embed2 for accuracy, nomic-embed-text for speed.

Quick Start

Cloud (No Local Resources)

ollama signin
ollama run deepseek-v3.2:cloud  # GPT-5 level
ollama run qwen3-coder:480b-cloud  # 1M context for huge repos

Local

ollama pull nemotron-3-nano  # 1M context, 24GB VRAM
ollama pull snowflake-arctic-embed2

# Or for lower RAM (8GB)
ollama pull llama3.2:3b
ollama pull nomic-embed-text

Stack Options

Option A: LangChain + ChromaDB (Most Common)

from langchain_ollama import OllamaLLM, OllamaEmbeddings
from langchain_chroma import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyPDFLoader

# Load and split
loader = PyPDFLoader("document.pdf")
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
docs = splitter.split_documents(loader.load())

# Embed and store
embeddings = OllamaEmbeddings(model="snowflake-arctic-embed2")
vectorstore = Chroma.from_documents(docs, embeddings, persist_directory="./db")

# Query - LOCAL
llm = OllamaLLM(model="nemotron-3-nano")

# Or CLOUD (GPT-5 level, no local resources)
llm = OllamaLLM(model="deepseek-v3.2:cloud")

retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

from langchain.chains import RetrievalQA
qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)
answer = qa.invoke("What is the main topic?")

Option B: LlamaIndex (Better Accuracy)

from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings

# Configure
Settings.llm = Ollama(model="nemotron-3-nano", request_timeout=300.0)
Settings.embed_model = OllamaEmbedding(model_name="snowflake-arctic-embed2")

# Load and index
documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)

# Query
query_engine = index.as_query_engine()
response = query_engine.query("Summarize the key findings")

Option C: Direct Ollama API (Minimal Dependencies)

import ollama
import chromadb

# Embed
def embed(text):
    return ollama.embed(model="nomic-embed-text", input=text)["embeddings"][0]

# Store in ChromaDB
client = chromadb.PersistentClient(path="./db")
collection = client.get_or_create_collection("docs")
collection.add(ids=["1"], documents=["text"], embeddings=[embed("text")])

# Retrieve and generate
results = collection.query(query_embeddings=[embed("query")], n_results=3)
context = "\n".join(results["documents"][0])

response = ollama.chat(
    model="nemotron-3-nano",
    messages=[{"role": "user", "content": f"Context:\n{context}\n\nQuestion: ..."}]
)

Vector Database Options

Nemotron 3 Nano Deep Dive

Why Nemotron for RAG:

• 1M token context = entire codebases, long documents
• Hybrid Mamba-Transformer = 4x faster inference
• MoE (3.6B active params) = runs on 24GB VRAM
• Apache 2.0 license = commercial use OK

# For very long documents
llm = OllamaLLM(
    model="nemotron-3-nano",
    num_ctx=131072,  # 128K context, increase as needed
    temperature=0.1,  # Lower for factual RAG
)

Hardware Requirements

References

• Model selection guide
• Ollama Library
• Nemotron on Ollama