Databricks Unstructured PDF Generation

name: databricks-unstructured-pdf-generation tags: [databricks, document-processing]

Unstructured PDF Generation

Generate realistic synthetic PDF documents using LLM for RAG (Retrieval-Augmented Generation) and unstructured data use cases.

Overview

This skill uses the generate_pdf_documents MCP tool to create professional PDF documents with:

• LLM-generated content based on your description
• Accompanying JSON files with questions and evaluation guidelines (for RAG testing)
• Automatic upload to Unity Catalog Volumes

Quick Start

Use the generate_pdf_documents MCP tool:

• catalog: "my_catalog"
• schema: "my_schema"
• description: "Technical documentation for a cloud infrastructure platform including setup guides, troubleshooting procedures, and API references."
• count: 10

This generates 10 PDF documents and saves them to /Volumes/my_catalog/my_schema/raw_data/pdf_documents/ (using default volume and folder).

With Custom Location

Use the generate_pdf_documents MCP tool:

• catalog: "my_catalog"
• schema: "my_schema"
• description: "HR policy documents..."
• count: 10
• volume: "custom_volume"
• folder: "hr_policies"
• overwrite_folder: true

Parameters

Document Size Guide

• SMALL: ~1 page, concise content. Best for quick demos or testing.
• MEDIUM: ~4-6 pages, comprehensive coverage. Good balance for most use cases.
• LARGE: ~10+ pages, exhaustive documentation. Use for thorough RAG evaluation.

Output Files

For each document, the tool creates two files:

1. PDF file (<model_id>.pdf): The generated document
2. JSON file (<model_id>.json): Metadata for RAG evaluation

JSON Structure

{
  "title": "API Authentication Guide",
  "category": "Technical",
  "pdf_path": "/Volumes/catalog/schema/volume/folder/doc_001.pdf",
  "question": "What authentication methods are supported by the API?",
  "guideline": "Answer should mention OAuth 2.0, API keys, and JWT tokens with their use cases."
}

Common Patterns

Pattern 1: HR Policy Documents

Use the generate_pdf_documents MCP tool:

• catalog: "ai_dev_kit"
• schema: "hr_demo"
• description: "HR policy documents for a technology company including employee handbook, leave policies, performance review procedures, benefits guide, and workplace conduct guidelines."
• count: 15
• folder: "hr_policies"
• overwrite_folder: true

Pattern 2: Technical Documentation

Use the generate_pdf_documents MCP tool:

• catalog: "ai_dev_kit"
• schema: "tech_docs"
• description: "Technical documentation for a SaaS analytics platform including installation guides, API references, troubleshooting procedures, security best practices, and integration tutorials."
• count: 20
• folder: "product_docs"
• overwrite_folder: true

Pattern 3: Financial Reports

Use the generate_pdf_documents MCP tool:

• catalog: "ai_dev_kit"
• schema: "finance_demo"
• description: "Financial documents for a retail company including quarterly reports, expense policies, budget guidelines, and audit procedures."
• count: 12
• folder: "reports"
• overwrite_folder: true

Pattern 4: Training Materials

Use the generate_pdf_documents MCP tool:

• catalog: "ai_dev_kit"
• schema: "training"
• description: "Training materials for new software developers including onboarding guides, coding standards, code review procedures, and deployment workflows."
• count: 8
• folder: "courses"
• overwrite_folder: true

Workflow

1. Ask for destination: Default to ai_dev_kit catalog, ask user for schema name
2. Get description: Ask what kind of documents they need
3. Generate PDFs: Call generate_pdf_documents MCP tool with appropriate parameters
4. Verify output: Check the volume path for generated files

Best Practices

1. Detailed descriptions: The more specific your description, the better the generated content

   • BAD: "Generate some HR documents"
   • GOOD: "HR policy documents for a technology company including employee handbook covering remote work policies, leave policies with PTO and sick leave details, performance review procedures with quarterly and annual cycles, and workplace conduct guidelines"

2. Appropriate count:

   • For demos: 5-10 documents
   • For RAG testing: 15-30 documents
   • For comprehensive evaluation: 50+ documents

3. Folder organization: Use descriptive folder names that indicate content type

   • hr_policies/
   • technical_docs/
   • training_materials/

4. Use overwrite_folder: Set to true when regenerating to ensure clean state

Integration with RAG Pipelines

The generated JSON files are designed for RAG evaluation:

1. Ingest PDFs: Use the PDF files as source documents for your vector database
2. Test retrieval: Use the question field to query your RAG system
3. Evaluate answers: Use the guideline field to assess if the RAG response is correct

Example evaluation workflow:

# Load questions from JSON files
questions = load_json_files(f"/Volumes/{catalog}/{schema}/{volume}/{folder}/*.json")

for q in questions:
    # Query RAG system
    response = rag_system.query(q["question"])

    # Evaluate using guideline
    is_correct = evaluate_response(response, q["guideline"])

Environment Configuration

The tool requires LLM configuration via environment variables:

# Databricks Foundation Models (default)
LLM_PROVIDER=DATABRICKS
DATABRICKS_MODEL=databricks-meta-llama-3-3-70b-instruct

# Or Azure OpenAI
LLM_PROVIDER=AZURE
AZURE_OPENAI_ENDPOINT=https://your-resource.cognitiveservices.azure.com/
AZURE_OPENAI_API_KEY=your-api-key
AZURE_OPENAI_DEPLOYMENT=gpt-4o

Common Issues