Zo Dataset Creator

Zo Dataset Creator

Create Zo Datasets with correct formatting for the Zo Datasets UI.

Quick Start

Creating a New Dataset

Use scripts/create_dataset.py to scaffold a new dataset:

python3 scripts/create_dataset.py <dataset-name>

This creates:

• datapackage.json with required metadata
• generate_schema.py for schema generation
• schema.yaml (auto-generated after database creation)
• ingest/, source/, assets/ directories
• README.md and PROCESS.md templates

Fixing an Existing Dataset

If tables don't show in the Zo UI:

1. Verify datapackage.json exists
2. Ensure data.duckdb is a valid database
3. Re-generate schema: python3 generate_schema.py

See TROUBLESHOOTING.md for common issues.

Dataset Structure

A valid Zo Dataset requires:

dataset-name/
├── datapackage.json          # Required: Dataset metadata
├── schema.yaml                # Required: Auto-generated from database
├── data.duckdb                # Required: DuckDB database
├── generate_schema.py         # Required: Script to generate schema
├── README.md                  # Recommended: Dataset documentation
├── PROCESS.md                 # Recommended: Ingestion instructions
├── ingest/                    # Optional: Ingestion scripts
├── source/                    # Optional: Raw source files
└── assets/                    # Optional: Generated files

Critical: Schema Format

ALWAYS auto-generate schema.yaml from the database. Never write it manually.

The Zo UI expects this format (list of tables with name: keys):

tables:
- name: my_table
  row_count: 10
  columns:
  - name: id
    type: VARCHAR
  - name: title
    type: VARCHAR
  - name: created_at
    type: TIMESTAMP

This format is generated by generate_schema.py.

Do NOT use manual YAML format (nested dictionaries):

# WRONG - Will not display in Zo UI
tables:
  my_table:
    columns:
      id:
        type: VARCHAR

Using the Scripts

create_dataset.py

Create a new dataset scaffold:

python3 scripts/create_dataset.py my-dataset

Creates the dataset in Datasets/my-dataset/ with all required files.

generate_schema.py

Generate schema from an existing database:

cd /home/workspace/Datasets/my-dataset
python3 generate_schema.py

When to run:

• After creating data.duckdb
• After modifying table structure (add/remove columns, create tables)
• After changing COMMENT annotations
• After any database schema changes

validate_dataset.py

Validate a dataset structure:

python3 scripts/validate_dataset.py /home/workspace/Datasets/my-dataset

Checks:

• datapackage.json exists and is valid
• data.duckdb exists and is readable
• schema.yaml exists and is in correct format
• Tables in schema match tables in database

Workflow Examples

Example 1: New Dataset from CSV

# 1. Create scaffold
python3 scripts/create_dataset.py sales-data

# 2. Copy CSV to source/
cp sales.csv /home/workspace/Datasets/sales-data/source/

# 3. Create database and import data
cd /home/workspace/Datasets/sales-data
python3 -c "
import duckdb
con = duckdb.connect('data.duckdb')
con.execute(\"CREATE TABLE sales AS SELECT * FROM 'source/sales.csv'\")
con.execute(\"COMMENT ON TABLE sales IS 'Monthly sales data'\")
con.close()
"

# 4. Generate schema
python3 generate_schema.py

# 5. View in Zo UI at /?t=datasets

Example 2: Fixing Broken Dataset

# 1. Validate to identify issues
python3 scripts/validate_dataset.py /home/workspace/Datasets/broken-dataset

# 2. If schema format is wrong, re-generate
cd /home/workspace/Datasets/broken-dataset
python3 generate_schema.py

# 3. Validate again
python3 scripts/validate_dataset.py /home/workspace/Datasets/broken-dataset

Example 3: Adding Comments to Tables

import duckdb

con = duckdb.connect('data.duckdb')

# Add table comment
con.execute("COMMENT ON TABLE videos IS 'YouTube videos from watchlist playlist'")

# Add column comments
con.execute("COMMENT ON COLUMN videos.title IS 'Video title from YouTube'")
con.execute("COMMENT ON COLUMN videos.view_count IS 'Total view count'")

con.close()

# Re-generate schema to include comments
# (run: python3 generate_schema.py)

Best Practices

Use COMMENT Annotations

Add inline comments to tables and columns — they're extracted into schema.yaml:

CREATE TABLE videos (
    id VARCHAR PRIMARY KEY COMMENT 'YouTube video ID',
    title VARCHAR COMMENT 'Video title from YouTube metadata',
    published_at TIMESTAMP COMMENT 'When the video was published'
) COMMENT 'Collection of videos from the watchlist playlist'

Keep Schema in Sync

Always re-run generate_schema.py after database changes:

• Adding/removing columns
• Changing column types
• Creating new tables
• Updating COMMENT annotations

Use Descriptive Names

• Tables: playlist_videos, user_sessions, transactions
• Columns: video_id, created_at, total_amount
• Use snake_case consistently

Document in README.md

Include:

• Purpose of the dataset
• What data it contains
• How to query it
• Example queries
• Business rules and caveats

Common Pitfalls

See TROUBLESHOOTING.md for detailed solutions to:

• Tables not showing in Zo UI
• Invalid schema.yaml format
• Missing datapackage.json
• Stale schema after database changes
• Locked database files

Reference Materials

• TROUBLESHOOTING.md - Common issues and solutions
• SCHEMA_GUIDE.md - Schema formatting details
• DATAPACKAGE_SPEC.md - datapackage.json reference
• DUCKDB_BASICS.md - DuckDB usage examples