Data Analysis

Data Analysis Agent Skill

Overview

A production-grade agent skill for end-to-end data analysis workflows. Use this skill when you need to load datasets, inspect structure, clean messy data, perform statistical analyses, create visualizations, or generate structured reports from tabular data.

When to Trigger

Activate this skill when the user asks to:

• Analyze data: "Analyze this CSV", "What patterns do you see in sales.csv?", "Explore this dataset", "Summarize the data in users.json"
• Create charts/graphs: "Make a bar chart of…", "Plot revenue over time", "Visualize the correlation matrix", "Show me a heatmap of…"
• Find patterns: "Find trends in this data", "Is there a correlation between X and Y?", "Cluster customers from this data", "Detect anomalies in…"
• Generate reports: "Create a report from survey_results.xlsx", "Summarize quarterly metrics", "Build a dashboard from sales data"
• Clean data: "Clean this messy dataset", "Fix missing values in…", "Normalize these columns", "Deduplicate this CSV"
• Statistical testing: "Run a t-test on group A vs B", "Check if this distribution is normal", "Perform regression analysis", "Calculate confidence intervals"

Near-Miss Negatives — Do NOT Trigger

• Questions about database schema design without actual data (e.g., "What columns should my users table have?")
• Questions about spreadsheet software UI (e.g., "How do I freeze a row in Google Sheets?")
• General math / statistics theory questions without a dataset context (e.g., "Explain the central limit theorem")
• Pure SQL query writing without a data-analysis intent (e.g., "Write a query to join three tables" — use a SQL skill instead)
• Questions about ETL pipeline architecture or data engineering (e.g., "Design a data ingestion pipeline")

Step-by-Step Workflow

Follow these phases in order. Skip phases that don't apply (e.g., if data is already clean) but always state that you're skipping and why.

Phase 1: Load

Determine the data source and load it into a DataFrame.

CSV    → pd.read_csv(filepath, ...)
JSON   → pd.read_json(filepath, ...)
Excel  → pd.read_excel(filepath, sheet_name=...)
SQL    → pd.read_sql_query(query, connection)

Checklist:

• Identify encoding (try UTF-8, Latin-1, detect automatically)
• For CSV: inspect delimiter (comma, tab, semicolon), quote character
• For Excel: list available sheets, load the right one
• For JSON: handle nested structures with pd.json_normalize() if needed
• For SQL: confirm read-only access, never run destructive queries
• Load a sample first if the dataset is large (>100k rows)

Phase 2: Inspect

Understand what you're working with before touching anything.

df.shape          # rows × columns
df.info()         # dtypes, non-null counts, memory
df.head(10)       # first rows
df.tail(5)        # last rows
df.describe()     # numeric summary stats
df.describe(include='object')  # categorical summary
df.dtypes         # column types
df.columns.tolist()  # column names

Checklist:

• Report shape: rows × columns
• List all columns with their dtypes
• Show summary statistics for numeric columns
• Show value counts for low-cardinality categorical columns
• Flag potential issues: wrong dtypes, placeholder values, suspicious zeros

Phase 3: Clean

Address data quality issues. Never modify the source file without prompting the user first. Work on a copy.

Common operations:

Rules:

1. Always work on df_clean = df.copy(), never mutate the original in-place without explicit user consent.
2. Report every change: "Dropped 47 duplicate rows (2.3% of data)", "Imputed missing age values with median (142 cells)".
3. Flag suspicious patterns even if you don't fix them: "Column 'salary' has 340 zero values — verify if these are legitimate."
4. If a cleaning decision is irreversible, ask first.

Phase 4: Analyze

Apply appropriate analytical methods based on the question.

Exploratory Data Analysis (EDA):

• Univariate: histograms, box plots, value counts per column
• Bivariate: scatter plots, correlation coefficients, grouped means
• Multivariate: pair plots, correlation matrix heatmap, PCA

Statistical Methods (see references/statistical-methods.md):

Time Series specifics:

• Set datetime index: df.set_index('date', inplace=True)
• Resample: df.resample('M').mean()
• Rolling windows: df['value'].rolling(7).mean()
• Decomposition: trend, seasonal, residual

Phase 5: Visualize

Choose the right chart for the data and question. See references/visualization-patterns.md for the full guide.

Library selection:

• Static, publication-quality: matplotlib + seaborn
• Interactive, exploratory: plotly
• Statistical plots: seaborn (box, violin, pair, joint, heatmap)

Quick reference:

*Pie charts: use only when ≤5 categories and values sum to a meaningful whole. Prefer bar charts otherwise.

Best practices:

• Always label axes and add a title
• Use accessible color palettes (avoid red-green for colorblind users)
• Sort bar charts by value unless categories have a natural order
• Add data source and date to chart footnotes
• For interactive charts, include hover tooltips

Phase 6: Report

Synthesize findings into a structured report.

Report structure:

1. Executive Summary — 2–3 sentences with the key finding
2. Data Overview — source, shape, date range, columns
3. Data Quality — issues found, actions taken
4. Key Findings — bullet points with numbers, ranked by importance
5. Visualizations — inline charts with captions
6. Statistical Results — test statistics, p-values, effect sizes
7. Limitations & Caveats — data gaps, assumptions, edge cases
8. Recommendations — actionable next steps or further analysis

Output formats:

• Quick answer: plain text summary in chat with key numbers
• Detailed report: Markdown document with embedded charts
• Dashboard: interactive HTML with plotly (offer if >5 charts)
• Export: offer to save cleaned data and charts as files

Tool-Aware Implementation

Python Libraries

This skill assumes Python 3.9+ with the following libraries available. Check availability before use; install missing packages as needed.

# Core
import pandas as pd
import numpy as np

# Visualization
import matplotlib.pyplot as plt
import seaborn as sns

# Interactive
import plotly.express as px
import plotly.graph_objects as go

# Statistics
from scipy import stats
from scipy.stats import norm, ttest_ind, f_oneway, pearsonr, spearmanr

# Optional: machine learning
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.linear_model import LinearRegression, LogisticRegression

Matplotlib setup for non-interactive environments:

import matplotlib
matplotlib.use('Agg')  # headless rendering

Plotly in notebooks vs scripts:

# In Jupyter/notebook environments:
import plotly.io as pio
pio.renderers.default = 'notebook'

# For saving to files:
fig.write_html('chart.html')
fig.write_image('chart.png')

Platform-Specific Notes

Safety & Guardrails

1. Never modify source data in-place — always create a copy or backup before transformations. Offer to save cleaned data as a new file.
2. Flag data quality issues — don't silently fix problems. Report missing values, outliers, and type inconsistencies before and after cleaning.
3. Statistical honesty — report p-values and effect sizes, not just "significant" or "not significant". Don't p-hack by running multiple tests without correction. Mention when sample sizes are too small for reliable inference.
4. Privacy awareness — if a dataset appears to contain PII (emails, phone numbers, names), warn the user and suggest anonymization before analysis.
5. Large file handling — for files >100MB, use chunked reading (chunksize parameter) or sample before full analysis. Warn about memory constraints.
6. SQL safety — use read-only connections. Never run INSERT, UPDATE, DELETE, DROP, or ALTER. Use transactions or connection strings that enforce read-only mode.
7. Deterministic results — set random seeds for reproducible analysis: np.random.seed(42).

Scripts

scripts/validate_dataset.py

PEP 723 compliant data quality validation script. Run with:

python scripts/validate_dataset.py path/to/dataset.csv
# or
python scripts/validate_dataset.py path/to/dataset.json
# or
python scripts/validate_dataset.py path/to/dataset.xlsx --sheet "Sheet1"

Produces a structured quality report covering missing values, outliers, type consistency, duplicates, and basic statistics. See script docstring for details.

References

• data-cleaning-guide.md — Handling nulls, outliers, type coercion, deduplication, and normalization patterns.
• visualization-patterns.md — Chart type selection guide, color best practices, accessibility considerations.
• statistical-methods.md — Descriptive statistics, hypothesis testing, regression, correlation, confidence intervals, and when to use each method.