Pipeline Audit

Pipeline Audit — Adversarial Data Review

v1.0 — Adversarial reviewer that challenges data pipeline outputs, assumptions, and integrity. Complements /code-review (which reviews code quality) by testing what the code actually produced.

Review the outputs of a data cleaning or analysis pipeline by reading the data, checking for anomalies, and challenging the assumptions embedded in the pipeline. This is the "red team" for your data work.

Argument: $ARGUMENTS

• Path to a directory containing .dta output files (e.g., data/for_analysis/)
• Or a project name (will look in ~/Dropbox/Github/[project]/)
• Or a specific .dta file to audit

Modes (append to argument):

• quick (default) — Scan outputs for red flags: duplicate IDs, unexpected missingness, implausible values
• deep — Full adversarial review: all quick checks + cross-file consistency, merge integrity, distribution analysis
• pre-submission — Publication-readiness check: all deep checks + AEA-style output verification, table-to-code tracing

Flags:

• scope:cleaning — Focus on cleaning pipeline outputs (raw -> for_analysis)
• scope:analysis — Focus on analysis outputs (regressions, tables)
• vars:consumption,assets — Focus on specific variable families
• baseline:path/to/old.dta — Compare against a previous version

Example: /pipeline-audit ~/Dropbox/Github/graduation-coaching/data/for_analysis/ deep Example: /pipeline-audit data/for_analysis/endline_analysis.dta quick vars:consumption Example: /pipeline-audit bd-social scope:cleaning

Instructions

Step 0: Locate Data and Context

1. Resolve $ARGUMENTS to find the target .dta file(s):
   • If a directory: glob for *.dta files, report count
   • If a project name: check ~/Dropbox/Github/[project]/data/for_analysis/, data/, output/
   • If a single file: use directly
2. Read the project's AGENTS.md, CLAUDE.md, or README.md if available — look for:
   • Expected unique identifiers (hhid, id, respondent_id)
   • Expected sample sizes
   • Variable naming conventions
   • Known data quality issues
3. If a docs/ or codebook/ directory exists, scan for data documentation

Parse mode and flags from $ARGUMENTS. Default to quick.

Step 1: Structural Integrity

Write a temporary Python script for all checks (use uv run python or python3). Never use python -c.

For each .dta file:

1.1 ID Integrity

• Detect likely ID columns (hhid, id, respondent_id, or columns ending in _id)
• Test uniqueness: df[id_col].is_unique
• Report any duplicates with sample rows
• If multiple .dta files share an ID column, verify consistent ID sets across files

1.2 Shape and Completeness

• Report: rows, columns, memory usage
• Flag files with 0 rows (empty output = pipeline failure)
• Flag files where >50% of columns are entirely missing
• Flag any column that is 100% missing

1.3 Variable Type Audit

• Count: numeric, string, categorical columns
• Flag columns where the name suggests numeric but the type is string (e.g., income_total as string)
• Flag columns where >90% of values are the same (constant or near-constant)

Step 2: Value Plausibility (all modes)

2.1 Distributional Red Flags

For each numeric column:

• Flag if min < 0 for variables that should be non-negative (expenditure, count, age, income)
• Flag extreme outliers: values > 10x the 99th percentile or < -10x the 1st percentile
• Flag suspiciously round distributions (all values multiples of 100, 1000, etc.)
• Report top 5 columns by coefficient of variation (most dispersed)

2.2 Missing Value Patterns

• Report overall missingness rate per column
• Flag columns where missingness is suspiciously correlated with treatment/control (if treatment variable detected)
• Flag columns where missingness jumps sharply compared to nearby columns in the dataset
• Check for sentinel codes that survived cleaning (-99, -88, -77, -66 in numeric columns)

2.3 Categorical/Indicator Checks

• For binary (0/1) columns: report balance. Flag if <1% or >99% are 1 (degenerate)
• For categorical columns: report number of unique values. Flag if >100 unique values (may be continuous stored as categorical)
• Check value labels exist for categorical variables (via pyreadstat metadata)

Step 3: Cross-File Consistency (deep and pre-submission modes)

Skip for quick mode.

3.1 ID Coverage

• If multiple .dta files share an ID column:
  • Are the ID sets identical? Report any IDs present in one file but not another
  • Do key structural variables (treatment arm, strata, demographics) match across files?

3.2 Variable Consistency

• If the same variable name appears in multiple files:
  • Do the distributions match? (Compare means, SDs, ranges)
  • Flag any where the mean differs by >10%

3.3 Merge Residue

• Flag any _merge variables that survived into the output (should be dropped)
• Flag any variables named _* that look like temporary leftovers

Step 4: Assumption Challenges (deep and pre-submission modes)

Skip for quick mode.

This is the adversarial core. Challenge the pipeline's implicit assumptions:

4.1 Sample Selection Bias

• If treatment variable exists: compare sample sizes by arm. Flag imbalance >10%
• Check if missingness patterns differ by treatment arm for key outcomes
• If attrition variable exists: profile attriters vs non-attriters on baseline characteristics

4.2 Construction Validity

• For index/composite variables: check correlation with components. Flag if a component has near-zero correlation with the composite (may not belong)
• For per-capita variables: verify denominator is reasonable (household size > 0, not missing)
• For currency-converted variables: verify conversion factor is plausible (not 0, not extreme)

4.3 Winsorization / Trimming

• Detect winsorized variables (names containing wins, _w, _tr)
• Verify winsorization was applied (compare min/max to percentile boundaries)
• Flag if winsorization thresholds seem too aggressive (>5% each tail)

4.4 Time / Date Variables

• Check date variables for future dates (data entry errors)
• Check for dates outside the study period
• Flag if interview date range is implausibly short or long

Step 5: Analysis Output Checks (pre-submission mode only)

Skip for quick and deep modes.

5.1 Regression Output Tables

• If results .dta or .csv files exist, check:
  • Do coefficient signs match economic intuition? (Treatment effects positive for graduation programs)
  • Are standard errors plausible relative to coefficient size?
  • Are sample sizes consistent across specifications?
  • Do R-squared values seem reasonable for the model type?

5.2 Table-to-Data Traceability

• For each regression output: can we identify which .dta was the input?
• Are all outcome variables in the output traceable to a column in the analysis dataset?

Step 6: Generate Report

Save report to:

• Same directory as reviewed data: pipeline_audit_[YYYY-MM-DD].md
• Or project root if reviewing a project

Classify each finding:

• RED FLAG — Data integrity violation. Likely pipeline bug or data corruption. Investigate immediately.
• ORANGE FLAG — Suspicious pattern that may indicate a problem. Verify before proceeding.
• YELLOW FLAG — Unusual but potentially intentional. Document the decision if keeping.
• CLEAN — Passed the check.

Output Format

# Pipeline Audit: [project/directory]

**Date:** [YYYY-MM-DD]
**Mode:** [quick / deep / pre-submission]
**Files reviewed:** [N] .dta files ([total rows] observations, [total cols] variables)
**Reviewer:** /pipeline-audit v1.0

---

## Executive Summary

**Overall assessment:** [Clean / Minor Issues / Investigate / Significant Problems]

[2-3 sentences: what was checked, what was found, what needs attention]

**Red flags:** [N]  |  **Orange flags:** [N]  |  **Yellow flags:** [N]  |  **Clean checks:** [N]

---

## Red Flags

### RF1: [Title]
- **File:** [filename]
- **Issue:** [description]
- **Evidence:** [specific values, counts, or statistics]
- **Action:** [what to investigate or fix]

---

## Orange Flags

### OF1: [Title]
- **File:** [filename]
- **Issue:** [description]
- **Evidence:** [evidence]
- **Question:** [what to verify — phrased as a question for the PI]

---

## Yellow Flags

[Brief list format]
- **YF1:** [file] — [issue] — [why it might be OK]

---

## Clean Checks

| Check | Status | Notes |
|-------|--------|-------|
| ID uniqueness | PASS | [N] files, all unique on [hhid] |
| No empty files | PASS | All [N] files have data |
| No sentinel codes | PASS | No -99/-88/-77 in numeric columns |
| ... | ... | ... |

---

## File Inventory

| File | Rows | Cols | ID Column | Missing Rate | Notes |
|------|------|------|-----------|-------------|-------|
| [name] | [N] | [N] | [col] | [%] | [any flags] |

---

## Next Steps

1. [Highest-priority investigation]
2. [Second priority]
3. [Third priority]

Principles

• Adversarial, not adversary. The goal is to find problems before reviewers, referees, or replicators do. Every flag is an opportunity to strengthen the work.
• Evidence-based. Every finding includes specific numbers, rows, or statistics. "Something looks wrong" is not a finding.
• Calibrated severity. A duplicate ID is a red flag. An unusual distribution is a yellow flag. Don't cry wolf.
• Ask, don't assume. When a pattern is ambiguous, phrase the finding as a question for the PI: "Is it expected that 23% of treated households have missing consumption data?"
• Respect the pipeline. The cleaning code may have already handled an issue. Check the code/logs before flagging something the pipeline explicitly addresses.
• No false comfort. A clean audit doesn't mean the data is perfect — it means the checks you ran didn't find problems. State what you checked and what you didn't.