# Parquet Analysis

> Analyze parquet files using Python and Ibis. Use when the user wants to explore, transform, or analyze parquet data files, perform aggregations, joins, or export results. Works with local parquet files and provides database-agnostic data operations.

- Skill: `brojonat/parquet-analysis` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add brojonat/parquet-analysis`
- Raw SKILL.md: https://api.skillmd.com/api/skills/brojonat/parquet-analysis/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- Author: brojonat (https://skillmd.com/u/brojonat)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/brojonat/parquet-analysis

---


# Parquet Analysis with Python and Ibis

Analyze parquet files using [Ibis](https://ibis-project.org), a database-agnostic Python DataFrame API. Ibis translates Python operations into optimized queries for the underlying backend (DuckDB by default for parquet files).

## Quick Start

### Basic workflow

```python
import ibis

# Connect to DuckDB (optimized for parquet)
con = ibis.duckdb.connect()

# Read parquet file
table = con.read_parquet("data.parquet")

# Explore
print(table.schema())      # View schema
print(table.head(10))      # Preview data
print(table.describe())    # Summary statistics

# Filter and aggregate
summary = (
    table
    .filter(table.amount > 100)
    .group_by("category")
    .aggregate(
        total=table.amount.sum(),
        avg=table.amount.mean(),
        count=table.count()
    )
)
print(summary.execute())

# Export
con.to_parquet(summary, "output.parquet")
```

## Core Operations

### Select and filter

```python
# Select columns
selected = table.select("id", "amount", "date")

# Filter rows
filtered = table.filter(
    (table.amount > 100) &
    (table.date >= "2024-01-01")
)

# Sort
sorted_data = table.order_by(table.amount.desc())
```

### Transform

```python
# Add computed columns
enriched = table.mutate(
    revenue=table.quantity * table.unit_price,
    year=table.date.year(),
    size=ibis.case()
        .when(table.amount < 100, "small")
        .when(table.amount < 1000, "medium")
        .else_("large")
        .end()
)
```

### Aggregate

```python
# Group by and summarize
by_category = (
    table.group_by("category")
    .aggregate(
        total=table.amount.sum(),
        avg=table.amount.mean(),
        count=table.count()
    )
)
```

### Join

```python
# Read and join multiple files
customers = con.read_parquet("customers.parquet")
orders = con.read_parquet("orders.parquet")

joined = (
    orders
    .join(customers, orders.customer_id == customers.id, how="left")
    .select(
        orders.order_id,
        orders.amount,
        customers.name
    )
)
```

### Export

```python
# To parquet
con.to_parquet(result, "output.parquet")

# To CSV (via pandas)
df = result.execute()
df.to_csv("output.csv", index=False)
```

## Common Patterns

### Data quality checks

```python
# Row count
row_count = table.count().execute()

# Check for nulls
null_counts = table.select([
    col.isnull().sum().name(f"{col}_nulls")
    for col in table.columns
]).execute()

# Value distribution
table.group_by("category").aggregate(count=table.count()).execute()
```

### Time-based analysis

```python
# Monthly aggregation
monthly = (
    table.mutate(month=table.date.truncate("M"))
    .group_by("month")
    .aggregate(total=table.amount.sum())
    .order_by("month")
)

# Extract date components
dated = table.mutate(
    year=table.date.year(),
    month=table.date.month(),
    quarter=table.date.quarter()
)
```

### Window functions

```python
# Rank within groups
ranked = table.mutate(
    rank=table.amount.rank().over(
        ibis.window(group_by="category", order_by=table.amount.desc())
    )
)

# Running total
with_cumsum = table.mutate(
    cumulative=table.amount.sum().over(
        ibis.window(order_by="date", rows=(None, 0))
    )
)
```

## Best Practices

1. **Filter early**: Apply filters before aggregations to reduce data volume
2. **Use lazy evaluation**: Ibis operations don't execute until `.execute()` is called - chain operations before executing
3. **Handle nulls**: Check for and handle null values explicitly
4. **Leverage selectors** for column operations (see [REFERENCE.md](REFERENCE.md))

## Detailed Resources

- **[EXAMPLES.md](EXAMPLES.md)** - Complete examples for all common tasks
- **[REFERENCE.md](REFERENCE.md)** - Detailed API reference for Ibis operations
- **[scripts/analyze.py](scripts/analyze.py)** - Complete working example script
- **[scripts/template.py](scripts/template.py)** - Quick-start template for your analyses
- **[../ibis-data/SKILL.md](../ibis-data/SKILL.md)** - Additional Ibis context and patterns

## Scripts

### Complete analysis workflow

Run the full example script:

```bash
python scripts/analyze.py
```

This demonstrates:
- Data exploration and quality checks
- Transformations and enrichment
- Aggregations and filtering
- Joins (if multiple files available)
- Multiple export formats
- Optional visualization

### Quick-start template

Copy and customize the template:

```bash
cp scripts/template.py my_analysis.py
# Edit with your file paths and logic
python my_analysis.py
```

## Installation

Install required packages:

```bash
uv add "ibis-framework[duckdb]"

# Optional for visualization
uv add matplotlib
```

## Troubleshooting

**Import error**: Ensure `ibis-framework[duckdb]` is installed

**Large files**: DuckDB handles large parquet files efficiently. For very large datasets:
- Filter early and aggressively
- Use selective column reading: `con.read_parquet("file.parquet", columns=["id", "amount"])`
- Process in chunks or use more selective queries

**Schema mismatches in joins**: Ensure column types match using `.cast()`:
```python
table = table.mutate(id=table.id.cast("int64"))
```

**Performance**: For complex queries, check the generated SQL with `ibis.to_sql(table)` to understand what's being executed