Data Analyst

CrushData AI - Data Analyst Skill

A data analyst intelligence skill that guides you through structured, professional data analysis workflows.

How to Use This Skill

When user requests data analysis work (analyze, query, dashboard, metrics, EDA, cohort, funnel, A/B test), follow this workflow:

Step 1: Discovery Protocol (MANDATORY)

Before writing any code, ask the user:

## Discovery Questions

1. **Business Context**
   - What business question should this analysis answer?
   - Who is the audience? (Executive, Analyst, Engineer)
   - What action will this analysis inform?

2. **Data Context**
   - Which tables/databases contain the relevant data?
   - What time range should I analyze?
   - Any known data quality issues?

3. **Metric Definitions**
   - How does YOUR company define the key metrics?
   - Any filters to apply? (exclude test users, internal accounts?)
   - What timezone should I use for dates?

4. **Script Organization**
    - Save all analysis scripts in an `analysis/` folder.
    - Create this folder if it does not exist.
   
5. **Python Environment**
   - Check for `venv` or `.venv`. If missing, `python3 -m venv venv`.
   - Install dependencies in venv (`venv/bin/pip install`).
   - Run scripts using venv python (`venv/bin/python`).

6. **Reports**
   - Save all profiling, validation, and findings to `reports/` folder.
   - Create this folder if it does not exist.

1b. Secure Data Access

Credentials are stored in the project's .env file - never hardcoded.

• Check Connections: Run npx crushdataai connections first.
• Missing Data?: If the data source is not listed, INSTRUCT the user to run: npx crushdataai connect
• Discover Schema: Use npx crushdataai schema <connection> [table] to see available tables and columns.
• Get Code: ALWAYS use npx crushdataai snippet <name> to get loading code (uses env vars).
• Load .env: Scripts use os.environ["VAR"]. Ensure python-dotenv is installed and .env is loaded.
• Security: DO NOT ask user to copy/move files to data/. Treat connected data as read-only.

Step 2: Search Relevant Domains

Use search.py to gather comprehensive information:

python3 .claude/skills/data-analyst/scripts/search.py "<query>" --domain <domain> [-n 3]

Available domains:

Industry-specific search:

python3 .claude/skills/data-analyst/scripts/search.py "<query>" --industry saas|ecommerce|finance|marketing

Recommended search order:

1. workflow - Get the step-by-step process for this analysis type
2. metric or --industry - Get relevant metric definitions
3. sql or python - Get code patterns for implementation
4. chart - Get visualization recommendations
5. validation - Check for common mistakes to avoid

Step 3: Data Profiling (MANDATORY Before Analysis)

Before any analysis, run profiling:

Python:

print(f"Shape: {df.shape}")
print(f"Date range: {df['date'].min()} to {df['date'].max()}")
print(f"Missing values:\n{df.isnull().sum()}")
print(f"Sample:\n{df.sample(5)}")

SQL:

SELECT 
    COUNT(*) as total_rows,
    COUNT(DISTINCT user_id) as unique_users,
    MIN(date) as min_date,
    MAX(date) as max_date
FROM table;

Report findings to user before proceeding:

"I found X rows, Y unique users, date range from A to B. Does this match your expectation?"

Step 3b: Data Cleaning & Transformation (ETL)

Address data quality issues found in Step 3:

1. Cleaning: Handle missing values, remove invalid duplicates, fix types.
2. Transformation: Standardize categories, parse dates, normalize text.
3. Feature Engineering: Create calculated columns needed for metrics.

Tip: Save cleaning scripts to etl/ folder. Save processed data to data/processed/.

Step 4: Execute Analysis with Validation

Before JOINs:

• Run on 100 rows first
• Check: Did row count change unexpectedly?
• Ask: "The join produced X rows from Y. Expected?"

Before Aggregations:

• Check for duplicates that could inflate sums
• Verify granularity: "Is this one row per user per day?"
• Ask: "Total = $X. Does this seem reasonable?"

Before Delivery:

• Sanity check order of magnitude
• Compare to benchmark or prior period
• Present for user validation before finalizing

Workflow Reference

Common Rules

1. Always ask before assuming - Metric definitions vary by company
2. Profile data first - Never aggregate without understanding the data
3. Validate results - Check totals, compare to benchmarks
4. Document assumptions - State what filters and definitions you used
5. Show your work - Explain the logic behind complex queries

Step 5: Generate Dashboard Output

After analysis, save results for dashboard visualization:

1. Search for best chart type:

   python3 .claude/skills/data-analyst/scripts/search.py "<metric description>" --domain chart
   

2. Create dashboard JSON:

   from pathlib import Path
   import json
   from datetime import datetime
   
   Path("reports/dashboards").mkdir(parents=True, exist_ok=True)
   
   dashboard = {
       "metadata": {
           "title": "Analysis Dashboard",
           "generatedAt": datetime.now().isoformat(),
           "dataRange": f"{start_date} to {end_date}"
       },
       "kpis": [
           {"id": "kpi-1", "label": "Total Revenue", "value": "$50,000", "trend": "+12%", "trendDirection": "up"}
       ],
       "charts": [
           {
               "id": "chart-1",
               "type": "line",  # line, bar, pie, area, scatter, donut, table
               "title": "Monthly Trend",
               "data": {
                   "labels": ["Jan", "Feb", "Mar"],
                   "datasets": [{"label": "Revenue", "values": [10000, 15000, 25000]}]
               }
           }
       ]
   }
   
   with open("reports/dashboards/dashboard.json", "w") as f:
       json.dump(dashboard, f, indent=2)
   

5b. Making Charts Refreshable (Recommended)

To allow the user to refresh data directly from the dashboard:

1. Include a query object in the chart definition.
2. Run npx crushdataai connections to list available connection names (this is secure - no passwords shown).
3. Set connection to one of the listed names.
4. Set sql to the query used to generate the data.

SECURITY: Never read .env directly to find connection names. Always use npx crushdataai connections.

"query": {
    "connection": "my_postgres_db",
    "sql": "SELECT date, revenue FROM sales WHERE..."
}

Database Specifics:

• SQL/Databases: Provide the full SQL query.
• Shopify: Provide the resource name (e.g. orders).
• CSV: Provide the connection name. sql is ignored but required (set to "default").
• MongoDB: Provide the collection name in the sql field.

Script-Based Refresh (for Python-aggregated charts):

If your chart requires Python aggregation (e.g., grouping, custom calculations), use script instead of connection:

"query": {
    "script": "analysis/my_dashboard_script.py"
}

When the user clicks Refresh, the CLI will re-run your Python script. The script should update the dashboard JSON file with fresh aggregated data.

TIP: Use script for Shopify/API charts that need aggregation. Use connection + sql only for SQL databases where the query returns pre-formatted chart data.

3. Tell user:

   "Dashboard ready! Run npx crushdataai dashboard to view."

Pre-Delivery Checklist

Before presenting final results:

• Confirmed business question is answered
• Data was profiled and validated
• Metric definitions match user's expectations
• Sanity checks pass (order of magnitude, trends, etc.)
• Visualizations follow best practices (search --domain chart)
• Assumptions and filters are documented

Converted and distributed by TomeVault — claim your Tome and manage your conversions.