When profiling a dataset, generate a comprehensive data profile:
- Basic Info: Number of rows, columns, memory usage
- Column Types: Data type for each column, count of numeric vs categorical
- Summary Statistics: Mean, median, std, min, max, quartiles for numeric columns
- Missing Values: Count and percentage of nulls per column
- Unique Values: Count of unique values per column
- Distribution Shape: Skewness and kurtosis for numeric columns
- Outlier Detection: Count of outliers per column using the IQR method
- Correlation: Top 5 most correlated feature pairs
Save the profile as a markdown report to output/data_profile.md.
Use polars for data manipulation. Follow the project's coding standards.