Data Explorer

Data explorer skill

Profile any tabular dataset (CSV, JSON, Parquet) and produce a structured summary the other skills can consume.

Workflow

1. Scan workspace: list all data files in the workspace directory.
2. Load and profile each file:
   • Row count, column count
   • Column names, data types, null counts, unique counts
   • Basic statistics (min, max, mean, median, std for numerics)
   • Value counts for categorical columns (top 10)
   • Correlation matrix for numeric columns
3. Assess data quality:
   • Missing value percentage per column
   • Potential data type issues (e.g., numbers stored as strings)
   • Duplicate row detection
   • Outlier detection (IQR method)
4. Output a structured profile as JSON for downstream skills.
5. Recommend analysis directions based on what you found.

Output format

{
  "files": [
    {
      "filename": "customers.csv",
      "rows": 91,
      "columns": 7,
      "schema": [
        {"name": "CustomerID", "dtype": "object", "nulls": 0, "unique": 91},
        {"name": "CompanyName", "dtype": "object", "nulls": 0, "unique": 91}
      ],
      "quality": {
        "missing_pct": {"Region": 0.60},
        "duplicates": 0
      },
      "recommendations": [
        "CustomerID is a unique string identifier",
        "Region column has a high missing percentage (60%)",
        "Can be joined with orders.csv on CustomerID to analyze customer behavior"
      ]
    }
  ]
}

Key rules

• Never assume a specific dataset. Profile whatever is present.
• If no data files are found, inform the user and ask them to upload.
• Use pandas for profiling. It is pre-installed in the sandbox.
• Use select_dtypes(include=["object", "str"]) for categorical columns.
• For large files (>100K rows), profile a sample first and note the sampling.