Developer Guide

Developer Guide

Architecture Documentation

• Architecture Overview - System components, research flow, and module responsibilities
• Database Schema - Database models and relationships
• Extension Guide - How to add custom search engines, strategies, and LLM providers
• Testing and CI - GitHub Actions workflows, pre-commit hooks, and security scanning

Development Setup

Prerequisites

• Python: 3.11+
• Node.js: 24.0.0+
• Docker: Latest version (for production builds)
• PDM: Python package manager
• SQLCipher: Required for encrypted database support. See SQLCIPHER_INSTALL.md for platform-specific instructions.

Initial Setup

1. Clone and Prepare Environment

   git clone git@github.com:LearningCircuit/local-deep-research.git
   # Or via HTTPS: https://github.com/LearningCircuit/local-deep-research.git
   cd local-deep-research
   

2. Backend Setup

   # Create and activate virtual environment
   python -m venv .venv
   source .venv/bin/activate
   
   # Install dependencies
   pip install pdm
   pdm install --no-self
   

3. Frontend Setup

   # Install frontend dependencies
   npm install
   

4. Enable Git Hooks

   # Install pre-commit hooks
   pre-commit install
   pre-commit install-hooks
   

   We use the pre-commit framework to manage git hooks. This repository includes both standard checks (ruff, eslint, gitleaks) and custom local checks located in .pre-commit-hooks/.

Running the Application

Development Mode

1. Start the Backend

   source .venv/bin/activate
   
   # Option A: Using the installed entry point
   ldr-web
   
   # Option B: Using Python module directly
   python -m local_deep_research.web.app
   

2. Start the Frontend (in a new terminal)

   npm run dev
   

   Access the app at http://localhost:5173.

Development Environment Variables

For local development and testing, you may want to configure these environment variables:

Warning: Never set LDR_BOOTSTRAP_ALLOW_UNENCRYPTED=true in production. User data will be stored unencrypted.

Docker (Production-like)

To build and run the entire stack in Docker:

docker build -t localdeepresearch/local-deep-research:dev .
docker run -p 5000:5000 -e LDR_DATA_DIR=/data -v ldr_data:/data localdeepresearch/local-deep-research:dev

Building

Building a Package

To build a wheel and source distribution, simply run pdm build.

Building Frontend Assets

If you're developing from source and want to use the Web UI, you need to build the frontend assets:

npm install
npm run build

This builds the Vite frontend into src/local_deep_research/web/static/dist/.

Note: pip users don't need this step - pre-built assets are included in the PyPI package.

Dependency Lockfile Management

This project uses PDM with pdm.lock to pin exact dependency versions. If you see this warning during Docker builds:

WARNING: Lockfile hash doesn't match pyproject.toml, packages may be outdated

It means pyproject.toml has changed but pdm.lock hasn't been regenerated.

How to fix:

pdm lock

Always commit pdm.lock alongside pyproject.toml changes to ensure reproducible builds.

Testing

Backend Tests (Pytest)

We support two modes of backend testing:

1. Isolated Testing (No Server Required)

This is the default and recommended way to run unit and integration tests. It uses Flask's test_client to mock the server.

How to run:

source .venv/bin/activate
# Run all isolated tests
pytest tests/

# Run specific API or Auth tests
pytest tests/api_tests/
pytest tests/auth_tests/

2. Live System Testing (Requires Running Server)

These tests make real HTTP requests to a running application instance.

Prerequisites:

1. Start the backend server in one terminal: ldr-web (or via Docker).
2. Run the live test scripts in another terminal.

How to run:

python tests/ui_tests/test_simple_research_api.py

Frontend & E2E Tests (Puppeteer)

We use Puppeteer for UI and End-to-End testing.

Prerequisites:

1. Navigate to the tests directory: cd tests
2. Install test dependencies: npm install

How to run:

1. Ensure the application is running (locally on port 5000 or via Docker).
2. Run the test suite:

   # Run all UI tests
   node tests/ui_tests/run_all_ui_tests.js
   
   # Run specific test
   node tests/ui_tests/test_simple_auth.js
   

Database Management

Backup & Restore

Before testing changes, you may wish to backup your Local Deep Research data volume.

Create a Backup

docker run --rm \
  -v ldr_data:/from \
  -v ldr_data-backup:/to \
  debian:latest \
  bash -c "cd /from ; tar -cf - . | (cd /to ; tar -xpf -)"

Restore from Backup

Warning: This overwrites existing data

docker run --rm \
  -v ldr_data:/target \
  -v ldr_data-backup:/source \
  debian:latest \
  bash -c "rm -rf /target/* /target/.[!.]* ; \
           cd /source ; tar -cf - . | (cd /target ; tar -xpf -)"

Troubleshooting

SQLCipher Issues

See SQLCIPHER_INSTALL.md for SQLCipher-related errors.

Permission Denied on Docker Volume

Error: PermissionError: [Errno 13] Permission denied: '/app/.config/...'

Solution: The volume may have been created with different ownership. Reset it:

docker volume rm ldr_data
# Re-run the container to create a fresh volume

Session Lost After Server Restart

Cause: The application's secret key is used to sign session cookies.

Solution: The secret key is automatically generated on first run and persisted to a .secret_key file in your data directory (controlled by LDR_DATA_DIR). Sessions are only lost if this file is deleted. If using Docker, ensure your data volume (ldr_data:/data) is persistent.

NoSettingsContextError in Background Threads

Error: NoSettingsContextError: No settings context available

Cause: Background threads don't have access to the Flask request context.

Solution: This is handled automatically by the application. If you encounter this during development, ensure your LLM settings are configured via the web UI settings page. For CI/testing environments, the LDR_TESTING_USE_FALLBACK_LLM=true environment variable enables a mock LLM fallback.

PDM Lockfile Out of Sync

Error: Lockfile hash doesn't match pyproject.toml

Solution:

pdm lock
git add pdm.lock