Hugging Face Datasets

Hugging Face Datasets Skill

⚡ Quick Setup

Before running any script, ensure:

# 1. Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. Set your HF token (Write access required for push operations)
export HF_TOKEN="hf_your_token_here"
# For private datasets, token must have read+write repo permissions.

All scripts use PEP 723 inline dependency management — uv run auto-installs requirements on first run. No manual pip install needed.

Tool Selection Matrix

Scripts

Both scripts live in scripts/ relative to this SKILL.md:

• scripts/sql_manager.py — SQL querying, filtering, transformation, export
• scripts/dataset_manager.py — Dataset creation, configuration, row management

SQL Querying (sql_manager.py)

Uses DuckDB's hf:// protocol for direct, zero-download access to any public (or private, with token) dataset.

HF Path Format

hf://datasets/{dataset_id}@~parquet/{config}/{split}/*.parquet

The @~parquet revision auto-converts any format to Parquet on the fly.

Core Commands

# Explore structure
uv run scripts/sql_manager.py describe --dataset "cais/mmlu"
uv run scripts/sql_manager.py unique   --dataset "cais/mmlu" --column "subject"
uv run scripts/sql_manager.py histogram --dataset "cais/mmlu" --column "subject"
uv run scripts/sql_manager.py sample   --dataset "cais/mmlu" --n 5

# Query
uv run scripts/sql_manager.py query \
  --dataset "cais/mmlu" \
  --sql "SELECT * FROM data WHERE subject='nutrition' LIMIT 10"

# Count with filter
uv run scripts/sql_manager.py count --dataset "cais/mmlu" --where "subject='nutrition'"

# Structured transform (no raw SQL required)
uv run scripts/sql_manager.py transform \
  --dataset "cais/mmlu" \
  --select "subject, COUNT(*) as cnt" \
  --group-by "subject" \
  --order-by "cnt DESC" \
  --limit 10

# Export locally
uv run scripts/sql_manager.py export \
  --dataset "cais/mmlu" \
  --sql "SELECT * FROM data WHERE subject='nutrition'" \
  --output "nutrition.parquet" --format parquet   # or: jsonl, csv

# Query a specific config or split
uv run scripts/sql_manager.py query \
  --dataset "ibm/duorc" --config "ParaphraseRC" --split "test" \
  --sql "SELECT * FROM data LIMIT 5"

# Query all splits at once
uv run scripts/sql_manager.py query \
  --dataset "cais/mmlu" --split "*" \
  --sql "SELECT COUNT(*) FROM data"

# Push result to Hub (creates new dataset)
uv run scripts/sql_manager.py query \
  --dataset "cais/mmlu" \
  --sql "SELECT * FROM data WHERE subject IN ('nutrition','anatomy','clinical_knowledge')" \
  --push-to "{your-username}/mmlu-medical-subset" --private

# Raw SQL — cross-dataset joins, full hf:// paths
uv run scripts/sql_manager.py raw --sql "
  SELECT a.*, b.*
  FROM 'hf://datasets/dataset1@~parquet/default/train/*.parquet' a
  JOIN 'hf://datasets/dataset2@~parquet/default/train/*.parquet' b
  ON a.id = b.id
  LIMIT 100
"

SQL Reference

-- Use `data` as the table alias in all queries (auto-expanded to hf:// path)

-- String ops
LENGTH(col)                         -- character count
LOWER(col), UPPER(col)
regexp_replace(col, '\n', ' ')      -- regex replace
regexp_matches(col, 'pattern')      -- regex filter

-- Array ops (HF arrays are 1-indexed in DuckDB)
choices[1]                          -- first element
array_length(choices)               -- array size
unnest(choices)                     -- expand array to rows
-- Note: MMLU `answer` field is 0–3; use choices[answer+1] for 1-indexed access

-- Aggregations
COUNT(*), SUM(col), AVG(col)
GROUP BY col HAVING cnt > 100

-- Reproducible sampling
USING SAMPLE 1000                       -- random N rows
USING SAMPLE 10 PERCENT (RESERVOIR, 42) -- reproducible 10%

-- Window functions
ROW_NUMBER() OVER (PARTITION BY subject ORDER BY question)

Python API

from sql_manager import HFDatasetSQL

sql = HFDatasetSQL()

results  = sql.query("cais/mmlu", "SELECT * FROM data WHERE subject='nutrition' LIMIT 10")
schema   = sql.describe("cais/mmlu")
samples  = sql.sample("cais/mmlu", n=5, seed=42)
count    = sql.count("cais/mmlu", where="subject='nutrition'")
dist     = sql.histogram("cais/mmlu", "subject")

url = sql.push_to_hub(
    "cais/mmlu",
    "{your-username}/nutrition-subset",
    sql="SELECT * FROM data WHERE subject='nutrition'",
    private=True
)

sql.export_to_parquet("cais/mmlu", "output.parquet", sql="SELECT * FROM data LIMIT 100")
sql.close()

Dataset Creation (dataset_manager.py)

Recommended Workflow

# 1. DISCOVER — use HF MCP server
#    search_datasets("conversational AI training")
#    get_dataset_details("{username}/dataset-name")

# 2. INITIALIZE
uv run scripts/dataset_manager.py init --repo_id "{your-username}/dataset-name" [--private]

# 3. CONFIGURE (attach system prompt / metadata)
uv run scripts/dataset_manager.py config \
  --repo_id "{your-username}/dataset-name" \
  --system_prompt "$(cat system_prompt.txt)"

# 4. QUICK SETUP (init + template in one step)
uv run scripts/dataset_manager.py quick_setup \
  --repo_id "{your-username}/dataset-name" --template classification

# 5. ADD ROWS
uv run scripts/dataset_manager.py add_rows \
  --repo_id "{your-username}/dataset-name" \
  --template qa \
  --rows_json '[{"question": "What is AI?", "answer": "Artificial Intelligence..."}]'

# 6. STATS
uv run scripts/dataset_manager.py stats --repo_id "{your-username}/dataset-name"

# 7. LIST TEMPLATES
uv run scripts/dataset_manager.py list_templates

Data Templates

Chat (--template chat) — multi-turn / tool-use conversations

{
  "messages": [
    {"role": "user",      "content": "Natural user request"},
    {"role": "assistant", "content": "Response with tool usage"},
    {"role": "tool",      "content": "Tool response", "tool_call_id": "call_123"}
  ],
  "scenario": "Description of use case",
  "complexity": "simple|intermediate|advanced"
}

Classification (--template classification)

{"text": "Input text", "label": "class_label", "confidence": 0.95,
 "metadata": {"domain": "technology", "language": "en"}}

QA (--template qa)

{"question": "...", "answer": "...", "context": "...",
 "answer_type": "factual|explanatory|opinion", "difficulty": "easy|medium|hard"}

Completion (--template completion)

{"prompt": "...", "completion": "...",
 "domain": "code|creative|technical|conversational", "style": "..."}

Tabular (--template tabular)

{
  "columns": [{"name": "feature1", "type": "numeric", "description": "..."}],
  "data":    [{"feature1": 123, "target": "class_a"}]
}

Bundled Example Sets

The examples/ directory contains ready-to-use training rows:

# Use one set
uv run scripts/dataset_manager.py add_rows \
  --repo_id "{your-username}/dataset-name" \
  --rows_json "$(cat examples/training_examples.json)"

# Merge both sets
uv run scripts/dataset_manager.py add_rows \
  --repo_id "{your-username}/dataset-name" \
  --rows_json "$(jq -s '.[0] + .[1]' examples/training_examples.json examples/diverse_training_examples.json)"

End-to-End Workflows

Build a Fine-Tuning Subset from an Existing Dataset

uv run scripts/sql_manager.py describe  --dataset "cais/mmlu"
uv run scripts/sql_manager.py histogram --dataset "cais/mmlu" --column "subject"
uv run scripts/sql_manager.py query \
  --dataset "cais/mmlu" \
  --sql "SELECT question, choices[answer+1] AS correct_answer, subject FROM data
         WHERE subject IN ('nutrition','anatomy','clinical_knowledge')" \
  --push-to "{your-username}/mmlu-medical-qa" --private

Quality-Filtered SFT Dataset

uv run scripts/sql_manager.py query \
  --dataset "squad" \
  --sql "SELECT * FROM data
         WHERE LENGTH(context) > 500 AND LENGTH(question) > 20" \
  --push-to "{your-username}/squad-filtered"

Multi-Split Merge → Parquet

uv run scripts/sql_manager.py export \
  --dataset "cais/mmlu" --split "*" \
  --output "mmlu_all.parquet"

Process Locally → Push as Training Dataset

# 1. Extract raw source
uv run scripts/sql_manager.py export \
  --dataset "cais/mmlu" \
  --sql "SELECT question, subject FROM data WHERE subject='nutrition'" \
  --output "nutrition_source.jsonl" --format jsonl

# 2. Run your preprocessing pipeline on nutrition_source.jsonl

# 3. Push processed data
uv run scripts/dataset_manager.py init --repo_id "{your-username}/nutrition-training"
uv run scripts/dataset_manager.py add_rows \
  --repo_id "{your-username}/nutrition-training" \
  --template qa \
  --rows_json "$(cat processed_data.json)"

Troubleshooting