Data Analysis

Data Analysis

Run an end-to-end data analysis following sewage project conventions.

Input: $ARGUMENTS — a dataset path, analysis goal description, or specification to estimate.

Project-Specific Context

Analysis Organisation

Scripts in scripts/R/09_analysis/ by approach:

• 01_descriptive/ — Maps, scatter plots, Google Trends
• 02_hedonic/ — Cross-sectional hedonic regressions
• 03_repeat_sales/ — Repeat-transaction regressions
• 04_long_difference/ — 250m grid-level long differences
• 05_news/ — DiD and event studies with media coverage
• 06_upstream_downstream/ — Directional spillover
• 07_dry_spills/ — Dry spill analysis

Datasets

• data/final/ — Analysis-ready datasets
• data/processed/ — Intermediate pipeline outputs (parquet)
• All data loaded via arrow::read_parquet() or arrow::open_dataset()

Output Destinations

• Tables: output/tables/*.tex (modelsummary → LaTeX with tabularray)
• Figures: output/figures/*.pdf or *.png
• Regression objects: output/regs/*.rds
• HTML interactive: output/html_plots/

Required R Conventions

• here::here() for all paths
• Native pipe |>
• fixest::feols() for regressions with vcov = "hetero"
• modelsummary for table output (tabularray format, [H] placement)
• arrow for parquet I/O
• snake_case naming
• forcats::as_factor() for factors

Workflow

Step 1: Context Gathering

1. Understand the analysis goal from $ARGUMENTS
2. Read existing analysis scripts in the relevant subdirectory for patterns
3. Read scripts/R/utils/spill_aggregation_utils.R if spill metrics are involved
4. Check data/final/ for available datasets
5. Read the relevant manuscript section in docs/overleaf/ if the analysis feeds into the paper

Step 2: Write Analysis Script

Follow the analysis script structure:

# ================================================================
# [Descriptive Title]
# Purpose: [What this script does]
# Inputs: [Data files]
# Outputs: [Figures, tables, RDS files]
# ================================================================

# === 1. Setup ============================================

library(tidyverse)
library(fixest)
library(modelsummary)
library(arrow)
library(here)

# === 2. Data Loading =====================================

df <- read_parquet(here("data", "final", "dataset.parquet"))

# === 3. Main Analysis ====================================

model <- feols(
  log_price ~ spill_count | lsoa + year_quarter,
  data = df,
  vcov = "hetero"
)

# === 4. Tables and Figures ================================

modelsummary(
  list("Main" = model),
  output = here("output", "tables", "table_name.tex"),
  fmt = 3
)

# === 5. Export ============================================

saveRDS(model, here("output", "regs", "model_name.rds"))

Step 3: Code Review

After writing the script, review it against the 9 categories from /review-r:

• Script structure, console hygiene, reproducibility
• Function design, figure quality, data persistence
• Comments, error handling, polish

Fix any Critical or Major issues before presenting.

Step 4: Run the Script

If the user wants execution:

cd /Users/jacopoolivieri/Library/CloudStorage/Dropbox/01_projects/sewage
Rscript scripts/R/09_analysis/[subdir]/[script_name].R

Step 5: Present Results

1. Results summary — Key estimates with SEs and economic interpretation
2. Script created — Path and description
3. Output files — Tables and figures generated
4. Code review notes — Any conventions to flag
5. TODO items — Missing data, additional specifications needed

Principles

• Reproduce, don't guess. If a specific regression is requested, implement exactly that.
• Strategy alignment. If an analysis feeds into a manuscript section, the code must implement what the paper claims.
• Publication-ready output. Tables and figures should be directly includable in the paper.
• Follow existing patterns. Read neighbouring scripts in the same subdirectory for style consistency.
• Save everything. Every regression object saved as RDS, every table as LaTeX, every figure as PDF.