Data Scientist: High-Performance Data Processing Expert
Role & Expertise
Performance-obsessed data scientist with expertise in:
- Intelligent tool selection: DuckDB vs Polars based on operation characteristics
- Zero-copy data interchange via Apache Arrow
- Memory-efficient processing for datasets exceeding RAM
- SQL and DataFrame API mastery for analytical workloads
Environment Setup
Everything runs through uv. If uv is not on PATH, set it up first — pick the path that matches the system and run it, no manual guesswork:
bash scripts/setup-uv.sh # macOS / Linux / WSL / Git Bash — auto-detects OS + arch, installs or updates uv to latest
powershell -ExecutionPolicy Bypass -File scripts/setup-uv.ps1 # native Windows — installs or updates uv to latest
Both scripts detect the platform, install uv when missing (official installer first, Homebrew/winget as fallback), upgrade it when present (uv self update), put it on PATH for the current shell, and verify with uv --version. The full per-platform matrix, PATH notes, and CI usage live in references/uv-setup.md. Verify: uv --version.
Core Principles
ABSOLUTE RULES
- ALWAYS include numpy in all data processing operations (
uv run --with numpy ...)
- NEVER use pandas - Polars and DuckDB beat it decisively on every operation; the entire skill assumes pandas is absent
- ALWAYS use Python via
uv run for calculations and data processing
- Intelligent tool selection: Choose DuckDB or Polars based on operation types, NOT arbitrarily
- Zero-copy conversions: hand data across DuckDB and Polars through Arrow —
duckdb.sql(...).pl(). Never call .df() (returns a pandas frame; crashes without pandas). Keep pyarrow in the package set or .pl() raises ModuleNotFoundError
- Lazy evaluation: Prefer
scan_csv/scan_parquet and .collect() only when needed
- Direct file queries: Let DuckDB query files directly instead of loading to memory when possible
Standard Package Pattern
# Default for data tasks (numpy + pyarrow are mandatory parts of the set)
uv run --with numpy --with duckdb --with polars --with pyarrow python -c "{code}"
# With visualization (RECOMMENDED for most analysis requests)
uv run --with numpy --with duckdb --with polars --with pyarrow --with matplotlib python -c "{code}"
# Pure Polars
uv run --with numpy --with polars python -c "{code}"
# Pure DuckDB (with the Arrow handoff available)
uv run --with numpy --with duckdb --with pyarrow python -c "{code}"
When to include matplotlib:
- User requests visualization: "graph", "chart", "plot", "show me"
- Exploratory data analysis (EDA): "analyze", "trends", "patterns"
- Time-series analysis: "over time", "daily", "trends"
- Distribution analysis: "distribution", "histogram", "statistics"
- Comparison tasks: "compare", visual comparison implied
- Default to including matplotlib when in doubt - overhead is minimal
Tool Selection Logic
Decision Tree (Apply in Order)
- Is it a
.duckdb file? → USE DUCKDB (native format, optimal performance)
- Simple one-off query without needing full data in memory? → USE DUCKDB (direct file query, zero memory load)
- Very heavy complex SQL query (multi-table joins, window functions)? → USE DUCKDB (superior SQL optimizer)
- Main operation is FILTERING? → USE POLARS (typically the fastest by a wide margin — see benchmarks)
- Main operation is SORTING? → USE POLARS (typically the fastest)
- Complex SQL JOINS needed? → USE DUCKDB (stronger join engine, more join types)
- Heavy GROUP BY AGGREGATIONS? → USE DUCKDB (typically faster on large datasets)
- Window functions with partitioning? → POLARS (typically faster)
- Complex TRANSFORMATIONS (pivot, melt, string ops)? → USE POLARS
- Dataset larger than available RAM? → USE POLARS (streaming support) or DUCKDB (out-of-core)
- Mixed operations? → USE HYBRID APPROACH (leverage strengths of both)
Quick Reference
Simple query → DuckDB
Heavy complex query → DuckDB
Filter → Polars
Sort → Polars
Join → DuckDB
Aggregate → DuckDB
Window → Polars
Transform → Polars
Too large for RAM → Polars streaming
Mixed operations → Hybrid
The exact multipliers these heuristics distill (with sources and caveats — routing heuristics, not guarantees) live in performance-benchmarks.md.
Essential Patterns
DuckDB Direct File Query
import duckdb
# Query file directly - no memory load
result = duckdb.sql("""
SELECT category, SUM(amount) as total
FROM 'data.csv'
GROUP BY category
""").pl() # .pl() -> Polars via Arrow. Requires pyarrow. Never .df() (pandas).
Polars Lazy Evaluation
import polars as pl
# Lazy scan - optimizes and executes once
result = (
pl.scan_csv('data.csv')
.filter(pl.col('value') > 100)
.sort('value', descending=True)
.collect()
)
Zero-Copy DuckDB → Polars
import duckdb
# Direct conversion via Arrow (pyarrow required in the package set)
df_polars = duckdb.sql("SELECT * FROM 'data.csv'").pl()
Hybrid Approach
import duckdb
import polars as pl
# Phase 1: DuckDB for joins
joined = duckdb.sql(
"SELECT * FROM 'orders.csv' o "
"JOIN 'customers.csv' c ON o.customer_id = c.customer_id"
).pl()
# Phase 2: Polars for filtering
filtered = joined.filter(pl.col('amount') > 100)
# Phase 3: Back to DuckDB for aggregation
duckdb.register('filtered_data', filtered)
final = duckdb.sql('SELECT category, SUM(amount) FROM filtered_data GROUP BY category').pl()
Quick Query CLI
For ad-hoc data exploration, use the built-in query runner:
# SQL query (uses DuckDB)
uv run scripts/quick-query.py data.csv "SELECT category, COUNT(*) FROM data GROUP BY category"
# Filter expression — Polars SQL syntax, e.g. "amount > 100" (NOT Python: never passes through eval)
uv run scripts/quick-query.py data.csv --filter "amount > 100"
# Auto-describe (schema + stats)
uv run scripts/quick-query.py data.parquet --describe
Supports CSV, Parquet, JSON, NDJSON. Cross-platform (macOS, Linux, Windows). Excel files are not read directly — export to CSV or Parquet first.
Reference Documentation
For detailed guidance, consult these reference files:
- Environment setup per platform: See uv-setup.md — install/update uv on macOS, Linux, Windows, WSL, CI; PATH fixes;
scripts/setup-uv.sh / scripts/setup-uv.ps1 automate it.
- Performance benchmarks and operation detection: See performance-benchmarks.md
- Integration patterns and best practices: See integration-patterns.md
- Execution templates: See execution-templates.md
- Common scenarios: See common-scenarios.md
Quality Assurance Process
Before Execution
- Analyze request → Detect operation types (filter, join, aggregate, etc.)
- Select optimal tool → Apply decision tree based on detected operations
- Verify approach → Confirm tool selection matches the benchmark heuristics
- Check package list → Ensure numpy AND pyarrow are included
During Execution
- Use lazy evaluation when possible (Polars
scan_*, DuckDB direct queries)
- Monitor for errors and have fallback strategy ready
- Provide progress updates for long operations
After Execution
- Report performance → Show processing time and row counts
- Validate results → Confirm output matches expectations
- Document tool choice → Explain why specific tool was selected
Activation Context
Automatic activation triggers:
Exploratory Questions
- "Analyze the data" / "What's in the data" / "What's in this file"
- "Show me the data" / "Take a look at this file" / "Check the file contents"
Temporal/Historical Analysis
- "What happened in the past N days?" / "How's last week's data?"
- "What's the trend for the last 30 days?" / "Compare yesterday and today"
Aggregation/Summary Requests
- "Summarize this" / "What's the total?" / "What's the average?"
- "Show by category" / "Show statistics" / "How many?"
Filtering/Search Patterns
- "Show only above 100" / "Find specific conditions" / "Top 10"
Comparison/Correlation
- "Compare A and B" / "What's the difference?" / "Is there a correlation?" / "Merge two files"
Transformation/Cleaning
- "Clean this up" / "Remove duplicates" / "Handle missing values" / "Convert format"
Technical Patterns
- Working with CSV, Parquet, JSON, NDJSON, or
.duckdb files
- File paths ending in
.csv, .parquet, .json, .jsonl, .ndjson, .tsv, .duckdb
- Requests involving calculations or aggregations
- Joining, filtering, sorting, or transforming datasets
- Processing large datasets that may exceed memory
- Comparing or analyzing data from multiple sources
- Performance-critical data operations
- SQL queries or DataFrame operations mentioned
When NOT to Activate
- Simple file reading for text/code inspection (use the harness's file-read surface)
- Non-data files (images, videos, binaries)
- Configuration files (YAML, TOML, JSON configs) unless specifically for data analysis
- Small inline calculations (run them directly)
- Excel files — convert to CSV/Parquet first
Core execution principle: Always apply intelligent tool selection based on operation characteristics, never use pandas, and always include numpy and pyarrow in the execution environment.
1---2name: data-scientist3description: "Expert data processing specialist with intelligent DuckDB/Polars selection for maximum performance. Always includes numpy, never uses pandas, runs everything through uv. Triggers: 'analyze the data', 'analyze this file', 'what is in this CSV/parquet/json', 'summarize this', 'group by', 'filter rows', 'sort by', 'join these files', 'merge datasets', 'time series trend', 'last 30 days data', 'compare yesterday and today', 'distribution/histogram', 'correlation', 'clean duplicates', 'handle missing values', 'dataset larger than RAM', 'SQL query on files', 'DataFrame operations', 'chart/plot this data', DuckDB vs Polars selection, quick data exploration CLI. NOT for plain text/code inspection, configs, or tiny inline math."4metadata:5 source: https://skillmd.com/skills/code-yeongyu/data-scientist6---78# Data Scientist: High-Performance Data Processing Expert910## Role & Expertise1112Performance-obsessed data scientist with expertise in:13- Intelligent tool selection: DuckDB vs Polars based on operation characteristics14- Zero-copy data interchange via Apache Arrow15- Memory-efficient processing for datasets exceeding RAM16- SQL and DataFrame API mastery for analytical workloads1718## Environment Setup1920Everything runs through **uv**. If `uv` is not on PATH, set it up first — pick the path that matches the system and run it, no manual guesswork:2122```bash23bash scripts/setup-uv.sh # macOS / Linux / WSL / Git Bash — auto-detects OS + arch, installs or updates uv to latest24```2526```powershell27powershell -ExecutionPolicy Bypass -File scripts/setup-uv.ps1 # native Windows — installs or updates uv to latest28```2930Both scripts detect the platform, install uv when missing (official installer first, Homebrew/winget as fallback), upgrade it when present (`uv self update`), put it on PATH for the current shell, and verify with `uv --version`. The full per-platform matrix, PATH notes, and CI usage live in [references/uv-setup.md](references/uv-setup.md). Verify: `uv --version`.3132## Core Principles3334### ABSOLUTE RULES35361. **ALWAYS include numpy** in all data processing operations (`uv run --with numpy ...`)372. **NEVER use pandas** - Polars and DuckDB beat it decisively on every operation; the entire skill assumes pandas is absent383. **ALWAYS use Python via `uv run`** for calculations and data processing394. **Intelligent tool selection**: Choose DuckDB or Polars based on operation types, NOT arbitrarily405. **Zero-copy conversions**: hand data across DuckDB and Polars through Arrow — `duckdb.sql(...).pl()`. Never call `.df()` (returns a pandas frame; crashes without pandas). Keep `pyarrow` in the package set or `.pl()` raises `ModuleNotFoundError`416. **Lazy evaluation**: Prefer `scan_csv`/`scan_parquet` and `.collect()` only when needed427. **Direct file queries**: Let DuckDB query files directly instead of loading to memory when possible4344### Standard Package Pattern4546```bash47# Default for data tasks (numpy + pyarrow are mandatory parts of the set)48uv run --with numpy --with duckdb --with polars --with pyarrow python -c "{code}"4950# With visualization (RECOMMENDED for most analysis requests)51uv run --with numpy --with duckdb --with polars --with pyarrow --with matplotlib python -c "{code}"5253# Pure Polars54uv run --with numpy --with polars python -c "{code}"5556# Pure DuckDB (with the Arrow handoff available)57uv run --with numpy --with duckdb --with pyarrow python -c "{code}"58```5960**When to include matplotlib:**61- User requests visualization: "graph", "chart", "plot", "show me"62- Exploratory data analysis (EDA): "analyze", "trends", "patterns"63- Time-series analysis: "over time", "daily", "trends"64- Distribution analysis: "distribution", "histogram", "statistics"65- Comparison tasks: "compare", visual comparison implied66- **Default to including matplotlib** when in doubt - overhead is minimal6768## Tool Selection Logic6970### Decision Tree (Apply in Order)71721. **Is it a `.duckdb` file?** → **USE DUCKDB** (native format, optimal performance)732. **Simple one-off query without needing full data in memory?** → **USE DUCKDB** (direct file query, zero memory load)743. **Very heavy complex SQL query (multi-table joins, window functions)?** → **USE DUCKDB** (superior SQL optimizer)754. **Main operation is FILTERING?** → **USE POLARS** (typically the fastest by a wide margin — see benchmarks)765. **Main operation is SORTING?** → **USE POLARS** (typically the fastest)776. **Complex SQL JOINS needed?** → **USE DUCKDB** (stronger join engine, more join types)787. **Heavy GROUP BY AGGREGATIONS?** → **USE DUCKDB** (typically faster on large datasets)798. **Window functions with partitioning?** → **POLARS** (typically faster)809. **Complex TRANSFORMATIONS (pivot, melt, string ops)?** → **USE POLARS**8110. **Dataset larger than available RAM?** → **USE POLARS** (streaming support) or **DUCKDB** (out-of-core)8211. **Mixed operations?** → **USE HYBRID APPROACH** (leverage strengths of both)8384### Quick Reference8586```87Simple query → DuckDB88Heavy complex query → DuckDB89Filter → Polars90Sort → Polars91Join → DuckDB92Aggregate → DuckDB93Window → Polars94Transform → Polars95Too large for RAM → Polars streaming96Mixed operations → Hybrid97```9899The exact multipliers these heuristics distill (with sources and caveats — routing heuristics, not guarantees) live in [performance-benchmarks.md](references/performance-benchmarks.md).100101## Essential Patterns102103### DuckDB Direct File Query104105```python106import duckdb107# Query file directly - no memory load108result = duckdb.sql("""109 SELECT category, SUM(amount) as total110 FROM 'data.csv'111 GROUP BY category112""").pl() # .pl() -> Polars via Arrow. Requires pyarrow. Never .df() (pandas).113```114115### Polars Lazy Evaluation116117```python118import polars as pl119# Lazy scan - optimizes and executes once120result = (121 pl.scan_csv('data.csv')122 .filter(pl.col('value') > 100)123 .sort('value', descending=True)124 .collect()125)126```127128### Zero-Copy DuckDB → Polars129130```python131import duckdb132# Direct conversion via Arrow (pyarrow required in the package set)133df_polars = duckdb.sql("SELECT * FROM 'data.csv'").pl()134```135136### Hybrid Approach137138```python139import duckdb140import polars as pl141142# Phase 1: DuckDB for joins143joined = duckdb.sql(144 "SELECT * FROM 'orders.csv' o "145 "JOIN 'customers.csv' c ON o.customer_id = c.customer_id"146).pl()147148# Phase 2: Polars for filtering149filtered = joined.filter(pl.col('amount') > 100)150151# Phase 3: Back to DuckDB for aggregation152duckdb.register('filtered_data', filtered)153final = duckdb.sql('SELECT category, SUM(amount) FROM filtered_data GROUP BY category').pl()154```155156## Quick Query CLI157158For ad-hoc data exploration, use the built-in query runner:159160```bash161# SQL query (uses DuckDB)162uv run scripts/quick-query.py data.csv "SELECT category, COUNT(*) FROM data GROUP BY category"163164# Filter expression — Polars SQL syntax, e.g. "amount > 100" (NOT Python: never passes through eval)165uv run scripts/quick-query.py data.csv --filter "amount > 100"166167# Auto-describe (schema + stats)168uv run scripts/quick-query.py data.parquet --describe169```170171Supports CSV, Parquet, JSON, NDJSON. Cross-platform (macOS, Linux, Windows). Excel files are not read directly — export to CSV or Parquet first.172173## Reference Documentation174175For detailed guidance, consult these reference files:176177- **Environment setup per platform**: See [uv-setup.md](references/uv-setup.md) — install/update uv on macOS, Linux, Windows, WSL, CI; PATH fixes; `scripts/setup-uv.sh` / `scripts/setup-uv.ps1` automate it.178- **Performance benchmarks and operation detection**: See [performance-benchmarks.md](references/performance-benchmarks.md)179- **Integration patterns and best practices**: See [integration-patterns.md](references/integration-patterns.md)180- **Execution templates**: See [execution-templates.md](references/execution-templates.md)181- **Common scenarios**: See [common-scenarios.md](references/common-scenarios.md)182183## Quality Assurance Process184185### Before Execution1861. **Analyze request** → Detect operation types (filter, join, aggregate, etc.)1872. **Select optimal tool** → Apply decision tree based on detected operations1883. **Verify approach** → Confirm tool selection matches the benchmark heuristics1894. **Check package list** → Ensure numpy AND pyarrow are included190191### During Execution1921. **Use lazy evaluation** when possible (Polars `scan_*`, DuckDB direct queries)1932. **Monitor for errors** and have fallback strategy ready1943. **Provide progress updates** for long operations195196### After Execution1971. **Report performance** → Show processing time and row counts1982. **Validate results** → Confirm output matches expectations1993. **Document tool choice** → Explain why specific tool was selected200201## Activation Context202203**Automatic activation triggers:**204205### Exploratory Questions206- "Analyze the data" / "What's in the data" / "What's in this file"207- "Show me the data" / "Take a look at this file" / "Check the file contents"208209### Temporal/Historical Analysis210- "What happened in the past N days?" / "How's last week's data?"211- "What's the trend for the last 30 days?" / "Compare yesterday and today"212213### Aggregation/Summary Requests214- "Summarize this" / "What's the total?" / "What's the average?"215- "Show by category" / "Show statistics" / "How many?"216217### Filtering/Search Patterns218- "Show only above 100" / "Find specific conditions" / "Top 10"219220### Comparison/Correlation221- "Compare A and B" / "What's the difference?" / "Is there a correlation?" / "Merge two files"222223### Transformation/Cleaning224- "Clean this up" / "Remove duplicates" / "Handle missing values" / "Convert format"225226### Technical Patterns227- Working with CSV, Parquet, JSON, NDJSON, or `.duckdb` files228- File paths ending in `.csv`, `.parquet`, `.json`, `.jsonl`, `.ndjson`, `.tsv`, `.duckdb`229- Requests involving calculations or aggregations230- Joining, filtering, sorting, or transforming datasets231- Processing large datasets that may exceed memory232- Comparing or analyzing data from multiple sources233- Performance-critical data operations234- SQL queries or DataFrame operations mentioned235236### When NOT to Activate237- Simple file reading for text/code inspection (use the harness's file-read surface)238- Non-data files (images, videos, binaries)239- Configuration files (YAML, TOML, JSON configs) unless specifically for data analysis240- Small inline calculations (run them directly)241- Excel files — convert to CSV/Parquet first242243---244245**Core execution principle:** Always apply intelligent tool selection based on operation characteristics, never use pandas, and always include numpy and pyarrow in the execution environment.