1532 CI CD Infrastructure C2acbe56

CI/CD and Infrastructure Documentation

This document describes the continuous integration, security scanning, and development infrastructure used by the Local Deep Research project.

Overview

The project uses many GitHub Actions workflows and 20+ pre-commit hooks to ensure code quality, security, and reliability.

┌─────────────────────────────────────────────────────────────────┐
│                        Developer Workflow                        │
├─────────────────────────────────────────────────────────────────┤
│  Local Development          │  Pull Request        │  Main/Dev  │
│  ─────────────────          │  ────────────        │  ────────  │
│  • Pre-commit hooks         │  • All tests         │  • Deploy  │
│  • Unit tests               │  • Security scans    │  • Publish │
│  • Linting                  │  • Code review       │  • Release │
└─────────────────────────────────────────────────────────────────┘

Pre-Commit Hooks

Pre-commit hooks run locally before each commit. Install with:

pre-commit install
pre-commit install-hooks

Standard Hooks

Security Hooks

Code Quality Hooks

Project-Specific Hooks

GitHub Actions Workflows

Test Workflows

Security Scanning

Dependency Management

UI/Accessibility

Build & Deploy

Code Quality

Repository Management

Dependabot Configuration

Dependabot automatically creates PRs for dependency updates:

Coverage Reporting

Coverage reports are generated by the docker-tests.yml workflow (pytest-tests job):

• HTML Report: Deployed to GitHub Pages at https://learningcircuit.github.io/local-deep-research/coverage/
• PR Comments: Each PR receives a comment with coverage percentage
• Badge: Coverage badge updated via GitHub Gist

Configuration in pyproject.toml:

[tool.coverage.run]
source = ["src"]
omit = ["*/tests/*", "*/migrations/*"]

[tool.coverage.report]
exclude_lines = ["pragma: no cover", "if TYPE_CHECKING:"]

Security Architecture

Supply Chain Security

1. Dependency Pinning: All GitHub Actions use SHA256 digests
2. Docker Image Pinning: All base images use SHA256 digests
3. Lock Files: pdm.lock and package-lock.json committed
4. Vulnerability Scanning: OSV-Scanner, npm audit, RetireJS

Runtime Security

1. SSRF Protection: safe_get(), safe_post(), SafeSession wrappers
2. XSS Prevention: DOMPurify for HTML sanitization
3. SQL Injection: SQLAlchemy ORM (no raw SQL)
4. Secret Management: Environment variables via SettingsManager

Container Security

1. Non-root User: Containers run as ldruser:1000
2. Minimal Base Image: Python slim images
3. Health Checks: Docker health check endpoints
4. Read-only Where Possible: Minimal write permissions

Running Tests Locally

Quick Test (Unit Tests Only)

pdm run pytest tests/test_settings_manager.py tests/test_utils.py -v

Full Test Suite

pdm run pytest tests/ --ignore=tests/ui_tests --ignore=tests/fuzz -v

With Coverage

pdm run pytest tests/ --cov=src --cov-report=html -v
open coverage/htmlcov/index.html

UI Tests (Requires Server)

# Terminal 1: Start server
pdm run ldr-web

# Terminal 2: Run UI tests
cd tests/ui_tests && npm test

Docker Testing

Build and run tests in Docker:

# Build test image
docker build --target ldr-test -t ldr-test .

# Run tests
docker run --rm -v "$PWD":/app -w /app ldr-test \
  pytest tests/ --ignore=tests/ui_tests -v

Environment Variables for CI

Adding New Workflows

When adding a new workflow:

1. Use pinned action versions with SHA256 digests
2. Add permissions: {} at top level (minimal permissions)
3. Add job-level permissions as needed
4. Include step-security/harden-runner step
5. Add workflow to this documentation

Example template:

name: New Workflow

on:
  pull_request:
    branches: [main]

permissions: {}

jobs:
  example:
    runs-on: ubuntu-latest
    permissions:
      contents: read

    steps:
      - name: Harden the runner
        uses: step-security/harden-runner@... # pinned
        with:
          egress-policy: audit

      - uses: actions/checkout@... # pinned
        with:
          persist-credentials: false