Research Library & RAG Guide

Research Library & RAG Guide

This guide covers the Research Library for document management and the RAG (Retrieval-Augmented Generation) system for semantic search.

Table of Contents

• Overview
• Managing Documents
• Collections
• RAG Indexing
• Semantic Search
• Embedding Models
• Configuration

Overview

The Research Library allows you to:

• Upload documents (PDFs, text files, markdown)
• Organize into collections for different projects or topics
• Index for semantic search using RAG (vector embeddings)
• Search your documents using natural language queries

Access the library at: http://localhost:5000/library

Managing Documents

Supported File Types

Uploading Documents

1. Navigate to Library in the sidebar
2. Click Upload or drag files into the upload area
3. Select a collection (or use the default "Library")
4. Documents are processed and text is extracted

Storage Modes

Document Actions

• View - Open document details and extracted text
• Download PDF - Get original file (if stored)
• Download Text - Export extracted text
• Delete - Remove from library

Collections

Collections organize your documents into groups.

Creating a Collection

1. Go to Library → Collections
2. Click Create Collection
3. Enter a name and optional description
4. Click Create

Managing Collections

• Add documents - Upload directly to collection or move existing docs
• Remove documents - Documents can exist in multiple collections
• Delete collection - Choose to keep or delete orphaned documents
• Index collection - Build RAG index for semantic search

Default Collection

The "Library" collection is created automatically and serves as the default destination for uploads.

RAG Indexing

RAG (Retrieval-Augmented Generation) enables semantic search over your documents.

How It Works

Document → Split into Chunks → Generate Embeddings → Store in Vector Index

1. Chunking - Documents split into overlapping segments
2. Embedding - Each chunk converted to a vector using AI model
3. Indexing - Vectors stored in FAISS for fast similarity search

Indexing a Collection

1. Go to Library → Collections
2. Select a collection
3. Click Index for Search (or Rebuild Index)
4. Wait for indexing to complete (progress shown)

Index Status

Semantic Search

Once indexed, search your documents using natural language.

Using Collection Search

1. Select a collection with indexed documents
2. Enter a natural language query
3. Results ranked by semantic similarity

Using in Research

When conducting research, you can:

1. Set search tool to your collection name
2. LDR will search your documents instead of the web
3. Combine with web search using "auto" mode

Example with Python API:

from local_deep_research.api import quick_summary

result = quick_summary(
    query="What does the documentation say about authentication?",
    search_tool="my_collection",  # Use your collection name
    programmatic_mode=True
)

Embedding Models

Choose the embedding model based on your needs.

Available Providers

Sentence Transformers (Local - Default)

Runs locally, no API key required.

Ollama (Local)

Uses your local Ollama installation.

• Default model: nomic-embed-text
• Requires Ollama running locally
• Configure URL in Settings → LLM → Ollama

OpenAI (Cloud)

Uses OpenAI's embedding API.

• Default model: text-embedding-3-small
• Requires OpenAI API key
• Higher quality, requires internet

Changing Embedding Model

1. Go to Library → Embedding Settings
2. Select provider and model
3. Click Save

Note: Changing models requires reindexing existing collections.

Configuration

Chunking Settings

Splitter Types:

• recursive - Split by paragraphs, then sentences (recommended)
• token - Split by token count
• sentence - Split by sentences
• semantic - Split by semantic similarity

Index Settings

Distance Metrics:

• cosine - Angle-based similarity (recommended)
• l2 - Euclidean distance
• dot_product - Dot product similarity

File Locations

API Reference

Collection Endpoints

Document Endpoints

RAG Endpoints

Troubleshooting

Documents Not Appearing

• Check file format is supported
• Verify upload completed successfully
• Refresh the library page

Search Not Working

• Ensure collection is indexed (check status)
• Try rebuilding the index
• Check embedding model is configured

Slow Indexing

• Large documents take longer
• Consider using smaller chunk sizes
• Local embedding models are slower than cloud

Memory Issues

• Reduce chunk size
• Index fewer documents at once
• Use a lighter embedding model

See Also

• Architecture Overview - System architecture
• Extension Guide - Adding custom retrievers
• API Quickstart - Using the API