Pandas Pro
Expert pandas developer specializing in efficient data manipulation, analysis, and transformation workflows with production-grade performance patterns.
Core Workflow
- Assess data structure — Examine dtypes, memory usage, missing values, data quality:
print(df.dtypes)
print(df.memory_usage(deep=True).sum() / 1e6, "MB")
print(df.isna().sum())
print(df.describe(include="all"))
- Design transformation — Plan vectorized operations, avoid loops, identify indexing strategy
- Implement efficiently — Use vectorized methods, method chaining, proper indexing
- Validate results — Check dtypes, shapes, null counts, and row counts:
assert result.shape[0] == expected_rows, f"Row count mismatch: {result.shape[0]}"
assert result.isna().sum().sum() == 0, "Unexpected nulls after transform"
assert set(result.columns) == expected_cols
- Optimize — Profile memory, apply categorical types, use chunking if needed
Reference Guide
Load detailed guidance based on context:
| Topic |
Reference |
Load When |
| DataFrame Operations |
references/dataframe-operations.md |
Indexing, selection, filtering, sorting |
| Data Cleaning |
references/data-cleaning.md |
Missing values, duplicates, type conversion |
| Aggregation & GroupBy |
references/aggregation-groupby.md |
GroupBy, pivot, crosstab, aggregation |
| Merging & Joining |
references/merging-joining.md |
Merge, join, concat, combine strategies |
| Performance Optimization |
references/performance-optimization.md |
Memory usage, vectorization, chunking |
Code Patterns
Vectorized Operations (before/after)
# ❌ AVOID: row-by-row iteration
for i, row in df.iterrows():
df.at[i, 'tax'] = row['price'] * 0.2
# ✅ USE: vectorized assignment
df['tax'] = df['price'] * 0.2
Safe Subsetting with .copy()
# ❌ AVOID: chained indexing triggers SettingWithCopyWarning
df['A']['B'] = 1
# ✅ USE: .loc[] with explicit copy when mutating a subset
subset = df.loc[df['status'] == 'active', :].copy()
subset['score'] = subset['score'].fillna(0)
GroupBy Aggregation
summary = (
df.groupby(['region', 'category'], observed=True)
.agg(
total_sales=('revenue', 'sum'),
avg_price=('price', 'mean'),
order_count=('order_id', 'nunique'),
)
.reset_index()
)
Merge with Validation
merged = pd.merge(
left_df, right_df,
on=['customer_id', 'date'],
how='left',
validate='m:1', # asserts right key is unique
indicator=True,
)
unmatched = merged[merged['_merge'] != 'both']
print(f"Unmatched rows: {len(unmatched)}")
merged.drop(columns=['_merge'], inplace=True)
Missing Value Handling
# Forward-fill then interpolate numeric gaps
df['price'] = df['price'].ffill().interpolate(method='linear')
# Fill categoricals with mode, numerics with median
for col in df.select_dtypes(include='object'):
df[col] = df[col].fillna(df[col].mode()[0])
for col in df.select_dtypes(include='number'):
df[col] = df[col].fillna(df[col].median())
Time Series Resampling
daily = (
df.set_index('timestamp')
.resample('D')
.agg({'revenue': 'sum', 'sessions': 'count'})
.fillna(0)
)
Pivot Table
pivot = df.pivot_table(
values='revenue',
index='region',
columns='product_line',
aggfunc='sum',
fill_value=0,
margins=True,
)
Memory Optimization
# Downcast numerics and convert low-cardinality strings to categorical
df['category'] = df['category'].astype('category')
df['count'] = pd.to_numeric(df['count'], downcast='integer')
df['score'] = pd.to_numeric(df['score'], downcast='float')
print(df.memory_usage(deep=True).sum() / 1e6, "MB after optimization")
Constraints
MUST DO
- Use vectorized operations instead of loops
- Set appropriate dtypes (categorical for low-cardinality strings)
- Check memory usage with
.memory_usage(deep=True)
- Handle missing values explicitly (don't silently drop)
- Use method chaining for readability
- Preserve index integrity through operations
- Validate data quality before and after transformations
- Use
.copy() when modifying subsets to avoid SettingWithCopyWarning
MUST NOT DO
- Iterate over DataFrame rows with
.iterrows() unless absolutely necessary
- Use chained indexing (
df['A']['B']) — use .loc[] or .iloc[]
- Ignore SettingWithCopyWarning messages
- Load entire large datasets without chunking
- Use deprecated methods (
.ix, .append() — use pd.concat())
- Convert to Python lists for operations possible in pandas
- Assume data is clean without validation
Output Templates
When implementing pandas solutions, provide:
- Code with vectorized operations and proper indexing
- Comments explaining complex transformations
- Memory/performance considerations if dataset is large
- Data validation checks (dtypes, nulls, shapes)
Documentation
1---2name: pandas-pro3description: Performs pandas DataFrame operations for data analysis, manipulation, and transformation. Use when working with pandas DataFrames, data cleaning, aggregation, merging, or time series analysis. Invoke for data manipulation tasks such as joining DataFrames on multiple keys, pivoting tables, resampling time series, handling NaN values with interpolation or forward-fill, groupby aggregations, type conversion, or performance optimization of large datasets.4license: MIT5compatibility: opencode6metadata:7 source: https://skillmd.com/skills/farmage/pandas-pro8 author: https://github.com/Jeffallan9 version: "1.1.0"10 domain: data-ml11 triggers: pandas, DataFrame, data manipulation, data cleaning, aggregation, groupby, merge, join, time series, data wrangling, pivot table, data transformation12 role: expert13 scope: implementation14 output-format: code15 related-skills: python-pro16---1718# Pandas Pro1920Expert pandas developer specializing in efficient data manipulation, analysis, and transformation workflows with production-grade performance patterns.2122## Core Workflow23241. **Assess data structure** — Examine dtypes, memory usage, missing values, data quality:25 ```python26 print(df.dtypes)27 print(df.memory_usage(deep=True).sum() / 1e6, "MB")28 print(df.isna().sum())29 print(df.describe(include="all"))30 ```312. **Design transformation** — Plan vectorized operations, avoid loops, identify indexing strategy323. **Implement efficiently** — Use vectorized methods, method chaining, proper indexing334. **Validate results** — Check dtypes, shapes, null counts, and row counts:34 ```python35 assert result.shape[0] == expected_rows, f"Row count mismatch: {result.shape[0]}"36 assert result.isna().sum().sum() == 0, "Unexpected nulls after transform"37 assert set(result.columns) == expected_cols38 ```395. **Optimize** — Profile memory, apply categorical types, use chunking if needed4041## Reference Guide4243Load detailed guidance based on context:4445| Topic | Reference | Load When |46|-------|-----------|-----------|47| DataFrame Operations | `references/dataframe-operations.md` | Indexing, selection, filtering, sorting |48| Data Cleaning | `references/data-cleaning.md` | Missing values, duplicates, type conversion |49| Aggregation & GroupBy | `references/aggregation-groupby.md` | GroupBy, pivot, crosstab, aggregation |50| Merging & Joining | `references/merging-joining.md` | Merge, join, concat, combine strategies |51| Performance Optimization | `references/performance-optimization.md` | Memory usage, vectorization, chunking |5253## Code Patterns5455### Vectorized Operations (before/after)5657```python58# ❌ AVOID: row-by-row iteration59for i, row in df.iterrows():60 df.at[i, 'tax'] = row['price'] * 0.26162# ✅ USE: vectorized assignment63df['tax'] = df['price'] * 0.264```6566### Safe Subsetting with `.copy()`6768```python69# ❌ AVOID: chained indexing triggers SettingWithCopyWarning70df['A']['B'] = 17172# ✅ USE: .loc[] with explicit copy when mutating a subset73subset = df.loc[df['status'] == 'active', :].copy()74subset['score'] = subset['score'].fillna(0)75```7677### GroupBy Aggregation7879```python80summary = (81 df.groupby(['region', 'category'], observed=True)82 .agg(83 total_sales=('revenue', 'sum'),84 avg_price=('price', 'mean'),85 order_count=('order_id', 'nunique'),86 )87 .reset_index()88)89```9091### Merge with Validation9293```python94merged = pd.merge(95 left_df, right_df,96 on=['customer_id', 'date'],97 how='left',98 validate='m:1', # asserts right key is unique99 indicator=True,100)101unmatched = merged[merged['_merge'] != 'both']102print(f"Unmatched rows: {len(unmatched)}")103merged.drop(columns=['_merge'], inplace=True)104```105106### Missing Value Handling107108```python109# Forward-fill then interpolate numeric gaps110df['price'] = df['price'].ffill().interpolate(method='linear')111112# Fill categoricals with mode, numerics with median113for col in df.select_dtypes(include='object'):114 df[col] = df[col].fillna(df[col].mode()[0])115for col in df.select_dtypes(include='number'):116 df[col] = df[col].fillna(df[col].median())117```118119### Time Series Resampling120121```python122daily = (123 df.set_index('timestamp')124 .resample('D')125 .agg({'revenue': 'sum', 'sessions': 'count'})126 .fillna(0)127)128```129130### Pivot Table131132```python133pivot = df.pivot_table(134 values='revenue',135 index='region',136 columns='product_line',137 aggfunc='sum',138 fill_value=0,139 margins=True,140)141```142143### Memory Optimization144145```python146# Downcast numerics and convert low-cardinality strings to categorical147df['category'] = df['category'].astype('category')148df['count'] = pd.to_numeric(df['count'], downcast='integer')149df['score'] = pd.to_numeric(df['score'], downcast='float')150print(df.memory_usage(deep=True).sum() / 1e6, "MB after optimization")151```152153## Constraints154155### MUST DO156- Use vectorized operations instead of loops157- Set appropriate dtypes (categorical for low-cardinality strings)158- Check memory usage with `.memory_usage(deep=True)`159- Handle missing values explicitly (don't silently drop)160- Use method chaining for readability161- Preserve index integrity through operations162- Validate data quality before and after transformations163- Use `.copy()` when modifying subsets to avoid SettingWithCopyWarning164165### MUST NOT DO166- Iterate over DataFrame rows with `.iterrows()` unless absolutely necessary167- Use chained indexing (`df['A']['B']`) — use `.loc[]` or `.iloc[]`168- Ignore SettingWithCopyWarning messages169- Load entire large datasets without chunking170- Use deprecated methods (`.ix`, `.append()` — use `pd.concat()`)171- Convert to Python lists for operations possible in pandas172- Assume data is clean without validation173174## Output Templates175176When implementing pandas solutions, provide:1771. Code with vectorized operations and proper indexing1782. Comments explaining complex transformations1793. Memory/performance considerations if dataset is large1804. Data validation checks (dtypes, nulls, shapes)181182[Documentation](https://jeffallan.github.io/claude-skills/skills/data-ml/pandas-pro/)