Windmill Workflows

Windmill Workflow Engine

Windmill is an open-source workflow engine for building internal tools with TypeScript, Python, Go, Bash, and more. This skill focuses on self-hosting and custom worker images for PDF/image processing.

Quick Reference

Architecture

Caddy → Server (port 8000) → PostgreSQL
              ↓
         Workers (N replicas) → Job Containers
              ↓
            LSP (port 3001)

• Server: API + frontend
• Workers: Execute jobs, spawn Docker containers
• LSP: Editor intellisense
• PostgreSQL: State and job queue

Custom Worker with PDF/Image Tools

Build a worker with pdfcpu, Poppler, and libvips (~150MB overhead):

FROM ghcr.io/windmill-labs/windmill:main

USER root

RUN apt-get update && apt-get install -y --no-install-recommends \
    poppler-utils \
    libvips-tools \
    && rm -rf /var/lib/apt/lists/*

# Add pdfcpu
RUN curl -fsSL "https://github.com/pdfcpu/pdfcpu/releases/download/v0.11.1/pdfcpu_0.11.1_Linux_x86_64.tar.xz" \
    | tar -xJf - -C /usr/local/bin --strip-components=1

Included Tools

Common Scripts

Extract text from PDF

import { $ } from "bun";

export async function main(pdfPath: string): Promise<string> {
  return await $`pdftotext -layout ${pdfPath} -`.text();
}

Convert PDF to images

import { $ } from "bun";

export async function main(pdfPath: string): Promise<string[]> {
  const outDir = `/tmp/pdf_${Date.now()}`;
  await $`mkdir -p ${outDir}`;
  await $`pdftoppm -png -r 300 ${pdfPath} ${outDir}/page`;
  return (await $`ls ${outDir}/*.png`.text()).trim().split('\n');
}

Resize image (fast, memory-efficient)

import { $ } from "bun";

export async function main(imagePath: string, width: number): Promise<string> {
  const output = `/tmp/resized_${Date.now()}.jpg`;
  await $`vipsthumbnail ${imagePath} -s ${width} -o ${output}[Q=85,strip]`;
  return output;
}

Merge PDFs

import { $ } from "bun";

export async function main(pdfs: string[]): Promise<string> {
  const output = `/tmp/merged_${Date.now()}.pdf`;
  await $`pdfunite ${pdfs.join(' ')} ${output}`;
  return output;
}

libvips vs ImageMagick

libvips is chosen for memory-constrained environments:

• 4-8x faster than ImageMagick
• 90% less memory (~200MB vs ~3GB for large images)
• Streaming architecture (processes in chunks)

Docker Compose Example

windmill-worker:
  build:
    context: ./windmill-worker
    dockerfile: Dockerfile
  image: windmill-worker-custom:latest
  environment:
    - DATABASE_URL=${DATABASE_URL}
    - MODE=worker
    - WORKER_GROUP=default
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock:ro
  mem_limit: 1.5g
  cpus: 1.0
  deploy:
    replicas: 2

Resource Guidelines

Scaling rule: 1 worker per 1 vCPU and 1-2 GB RAM

Reference Files

Official Documentation

Troubleshooting Quick Guide