Data Analysis
Run an end-to-end data analysis following sewage project conventions.
Input: $ARGUMENTS — a dataset path, analysis goal description, or specification to estimate.
Project-Specific Context
Analysis Organisation
Scripts in scripts/R/09_analysis/ by approach:
01_descriptive/ — Maps, scatter plots, Google Trends
02_hedonic/ — Cross-sectional hedonic regressions
03_repeat_sales/ — Repeat-transaction regressions
04_long_difference/ — 250m grid-level long differences
05_news/ — DiD and event studies with media coverage
06_upstream_downstream/ — Directional spillover
07_dry_spills/ — Dry spill analysis
Datasets
data/final/ — Analysis-ready datasets
data/processed/ — Intermediate pipeline outputs (parquet)
- All data loaded via
arrow::read_parquet() or arrow::open_dataset()
Output Destinations
- Tables:
output/tables/*.tex (modelsummary → LaTeX with tabularray)
- Figures:
output/figures/*.pdf or *.png
- Regression objects:
output/regs/*.rds
- HTML interactive:
output/html_plots/
Required R Conventions
here::here() for all paths
- Native pipe
|>
fixest::feols() for regressions with vcov = "hetero"
modelsummary for table output (tabularray format, [H] placement)
arrow for parquet I/O
snake_case naming
forcats::as_factor() for factors
Workflow
Step 1: Context Gathering
- Understand the analysis goal from
$ARGUMENTS
- Read existing analysis scripts in the relevant subdirectory for patterns
- Read
scripts/R/utils/spill_aggregation_utils.R if spill metrics are involved
- Check
data/final/ for available datasets
- Read the relevant manuscript section in
docs/overleaf/ if the analysis feeds into the paper
Step 2: Write Analysis Script
Follow the analysis script structure:
# ================================================================
# [Descriptive Title]
# Purpose: [What this script does]
# Inputs: [Data files]
# Outputs: [Figures, tables, RDS files]
# ================================================================
# === 1. Setup ============================================
library(tidyverse)
library(fixest)
library(modelsummary)
library(arrow)
library(here)
# === 2. Data Loading =====================================
df <- read_parquet(here("data", "final", "dataset.parquet"))
# === 3. Main Analysis ====================================
model <- feols(
log_price ~ spill_count | lsoa + year_quarter,
data = df,
vcov = "hetero"
)
# === 4. Tables and Figures ================================
modelsummary(
list("Main" = model),
output = here("output", "tables", "table_name.tex"),
fmt = 3
)
# === 5. Export ============================================
saveRDS(model, here("output", "regs", "model_name.rds"))
Step 3: Code Review
After writing the script, review it against the 9 categories from /review-r:
- Script structure, console hygiene, reproducibility
- Function design, figure quality, data persistence
- Comments, error handling, polish
Fix any Critical or Major issues before presenting.
Step 4: Run the Script
If the user wants execution:
cd /Users/jacopoolivieri/Library/CloudStorage/Dropbox/01_projects/sewage
Rscript scripts/R/09_analysis/[subdir]/[script_name].R
Step 5: Present Results
- Results summary — Key estimates with SEs and economic interpretation
- Script created — Path and description
- Output files — Tables and figures generated
- Code review notes — Any conventions to flag
- TODO items — Missing data, additional specifications needed
Principles
- Reproduce, don't guess. If a specific regression is requested, implement exactly that.
- Strategy alignment. If an analysis feeds into a manuscript section, the code must implement what the paper claims.
- Publication-ready output. Tables and figures should be directly includable in the paper.
- Follow existing patterns. Read neighbouring scripts in the same subdirectory for style consistency.
- Save everything. Every regression object saved as RDS, every table as LaTeX, every figure as PDF.
1---2name: data-analysis3description: End-to-end R data analysis for the sewage project. Writes analysis scripts following project conventions (here::here, arrow/parquet, fixest, modelsummary, native pipe), runs code review, and produces publication-ready tables and figures. This skill should be used when asked to "run an analysis", "estimate the model", "add a specification", or "write an R script".4argument-hint: "[dataset path, analysis goal, or specification description]"5allowed-tools: ["Read", "Grep", "Glob", "Write", "Edit", "Bash", "Agent"]6metadata:7 source: https://skillmd.com/skills/brycewang-stanford/data-analysis8---910# Data Analysis1112Run an end-to-end data analysis following sewage project conventions.1314**Input:** `$ARGUMENTS` — a dataset path, analysis goal description, or specification to estimate.1516---1718## Project-Specific Context1920### Analysis Organisation2122Scripts in `scripts/R/09_analysis/` by approach:23- `01_descriptive/` — Maps, scatter plots, Google Trends24- `02_hedonic/` — Cross-sectional hedonic regressions25- `03_repeat_sales/` — Repeat-transaction regressions26- `04_long_difference/` — 250m grid-level long differences27- `05_news/` — DiD and event studies with media coverage28- `06_upstream_downstream/` — Directional spillover29- `07_dry_spills/` — Dry spill analysis3031### Datasets3233- `data/final/` — Analysis-ready datasets34- `data/processed/` — Intermediate pipeline outputs (parquet)35- All data loaded via `arrow::read_parquet()` or `arrow::open_dataset()`3637### Output Destinations3839- Tables: `output/tables/*.tex` (modelsummary → LaTeX with tabularray)40- Figures: `output/figures/*.pdf` or `*.png`41- Regression objects: `output/regs/*.rds`42- HTML interactive: `output/html_plots/`4344### Required R Conventions4546- `here::here()` for all paths47- Native pipe `|>`48- `fixest::feols()` for regressions with `vcov = "hetero"`49- `modelsummary` for table output (tabularray format, `[H]` placement)50- `arrow` for parquet I/O51- `snake_case` naming52- `forcats::as_factor()` for factors5354---5556## Workflow5758### Step 1: Context Gathering59601. Understand the analysis goal from `$ARGUMENTS`612. Read existing analysis scripts in the relevant subdirectory for patterns623. Read `scripts/R/utils/spill_aggregation_utils.R` if spill metrics are involved634. Check `data/final/` for available datasets645. Read the relevant manuscript section in `docs/overleaf/` if the analysis feeds into the paper6566### Step 2: Write Analysis Script6768Follow the analysis script structure:6970```r71# ================================================================72# [Descriptive Title]73# Purpose: [What this script does]74# Inputs: [Data files]75# Outputs: [Figures, tables, RDS files]76# ================================================================7778# === 1. Setup ============================================7980library(tidyverse)81library(fixest)82library(modelsummary)83library(arrow)84library(here)8586# === 2. Data Loading =====================================8788df <- read_parquet(here("data", "final", "dataset.parquet"))8990# === 3. Main Analysis ====================================9192model <- feols(93 log_price ~ spill_count | lsoa + year_quarter,94 data = df,95 vcov = "hetero"96)9798# === 4. Tables and Figures ================================99100modelsummary(101 list("Main" = model),102 output = here("output", "tables", "table_name.tex"),103 fmt = 3104)105106# === 5. Export ============================================107108saveRDS(model, here("output", "regs", "model_name.rds"))109```110111### Step 3: Code Review112113After writing the script, review it against the 9 categories from `/review-r`:114- Script structure, console hygiene, reproducibility115- Function design, figure quality, data persistence116- Comments, error handling, polish117118Fix any Critical or Major issues before presenting.119120### Step 4: Run the Script121122If the user wants execution:123```bash124cd /Users/jacopoolivieri/Library/CloudStorage/Dropbox/01_projects/sewage125Rscript scripts/R/09_analysis/[subdir]/[script_name].R126```127128### Step 5: Present Results1291301. **Results summary** — Key estimates with SEs and economic interpretation1312. **Script created** — Path and description1323. **Output files** — Tables and figures generated1334. **Code review notes** — Any conventions to flag1345. **TODO items** — Missing data, additional specifications needed135136---137138## Principles139140- **Reproduce, don't guess.** If a specific regression is requested, implement exactly that.141- **Strategy alignment.** If an analysis feeds into a manuscript section, the code must implement what the paper claims.142- **Publication-ready output.** Tables and figures should be directly includable in the paper.143- **Follow existing patterns.** Read neighbouring scripts in the same subdirectory for style consistency.144- **Save everything.** Every regression object saved as RDS, every table as LaTeX, every figure as PDF.