# Pipeline Audit

> Adversarial data pipeline review — audits variable construction, sample restrictions, and analytical decisions against pre-analysis plans

- Skill: `thinkingwithagents/pipeline-audit` (Agent Skill)
- Install (CLI): `npx skillmds@latest add thinkingwithagents/pipeline-audit`
- Raw SKILL.md: https://api.skillmd.com/api/skills/thinkingwithagents/pipeline-audit/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: thinkingwithagents (https://skillmd.com/u/thinkingwithagents)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/thinkingwithagents/pipeline-audit

---


# Pipeline Audit — Adversarial Data Review

*v1.0 — Adversarial reviewer that challenges data pipeline outputs, assumptions, and integrity. Complements /code-review (which reviews code quality) by testing what the code actually produced.*

Review the outputs of a data cleaning or analysis pipeline by reading the data, checking for anomalies, and challenging the assumptions embedded in the pipeline. This is the "red team" for your data work.

**Argument:** `$ARGUMENTS`
- Path to a directory containing .dta output files (e.g., `data/for_analysis/`)
- Or a project name (will look in `~/Dropbox/Github/[project]/`)
- Or a specific .dta file to audit

**Modes** (append to argument):
- `quick` (default) — Scan outputs for red flags: duplicate IDs, unexpected missingness, implausible values
- `deep` — Full adversarial review: all quick checks + cross-file consistency, merge integrity, distribution analysis
- `pre-submission` — Publication-readiness check: all deep checks + AEA-style output verification, table-to-code tracing

**Flags:**
- `scope:cleaning` — Focus on cleaning pipeline outputs (raw -> for_analysis)
- `scope:analysis` — Focus on analysis outputs (regressions, tables)
- `vars:consumption,assets` — Focus on specific variable families
- `baseline:path/to/old.dta` — Compare against a previous version

Example: `/pipeline-audit ~/Dropbox/Github/graduation-coaching/data/for_analysis/ deep`
Example: `/pipeline-audit data/for_analysis/endline_analysis.dta quick vars:consumption`
Example: `/pipeline-audit bd-social scope:cleaning`

---

## Instructions

### Step 0: Locate Data and Context

1. Resolve `$ARGUMENTS` to find the target .dta file(s):
   - If a directory: glob for `*.dta` files, report count
   - If a project name: check `~/Dropbox/Github/[project]/data/for_analysis/`, `data/`, `output/`
   - If a single file: use directly
2. Read the project's `AGENTS.md`, `CLAUDE.md`, or `README.md` if available — look for:
   - Expected unique identifiers (hhid, id, respondent_id)
   - Expected sample sizes
   - Variable naming conventions
   - Known data quality issues
3. If a `docs/` or `codebook/` directory exists, scan for data documentation

Parse mode and flags from `$ARGUMENTS`. Default to `quick`.

---

### Step 1: Structural Integrity

Write a temporary Python script for all checks (use `uv run python` or `python3`). Never use `python -c`.

For each .dta file:

#### 1.1 ID Integrity
- Detect likely ID columns (hhid, id, respondent_id, or columns ending in `_id`)
- Test uniqueness: `df[id_col].is_unique`
- Report any duplicates with sample rows
- If multiple .dta files share an ID column, verify consistent ID sets across files

#### 1.2 Shape and Completeness
- Report: rows, columns, memory usage
- Flag files with 0 rows (empty output = pipeline failure)
- Flag files where >50% of columns are entirely missing
- Flag any column that is 100% missing

#### 1.3 Variable Type Audit
- Count: numeric, string, categorical columns
- Flag columns where the name suggests numeric but the type is string (e.g., `income_total` as string)
- Flag columns where >90% of values are the same (constant or near-constant)

---

### Step 2: Value Plausibility (all modes)

#### 2.1 Distributional Red Flags
For each numeric column:
- Flag if min < 0 for variables that should be non-negative (expenditure, count, age, income)
- Flag extreme outliers: values > 10x the 99th percentile or < -10x the 1st percentile
- Flag suspiciously round distributions (all values multiples of 100, 1000, etc.)
- Report top 5 columns by coefficient of variation (most dispersed)

#### 2.2 Missing Value Patterns
- Report overall missingness rate per column
- Flag columns where missingness is suspiciously correlated with treatment/control (if treatment variable detected)
- Flag columns where missingness jumps sharply compared to nearby columns in the dataset
- Check for sentinel codes that survived cleaning (-99, -88, -77, -66 in numeric columns)

#### 2.3 Categorical/Indicator Checks
- For binary (0/1) columns: report balance. Flag if <1% or >99% are 1 (degenerate)
- For categorical columns: report number of unique values. Flag if >100 unique values (may be continuous stored as categorical)
- Check value labels exist for categorical variables (via pyreadstat metadata)

---

### Step 3: Cross-File Consistency (deep and pre-submission modes)

*Skip for `quick` mode.*

#### 3.1 ID Coverage
- If multiple .dta files share an ID column:
  - Are the ID sets identical? Report any IDs present in one file but not another
  - Do key structural variables (treatment arm, strata, demographics) match across files?

#### 3.2 Variable Consistency
- If the same variable name appears in multiple files:
  - Do the distributions match? (Compare means, SDs, ranges)
  - Flag any where the mean differs by >10%

#### 3.3 Merge Residue
- Flag any `_merge` variables that survived into the output (should be dropped)
- Flag any variables named `_*` that look like temporary leftovers

---

### Step 4: Assumption Challenges (deep and pre-submission modes)

*Skip for `quick` mode.*

This is the adversarial core. Challenge the pipeline's implicit assumptions:

#### 4.1 Sample Selection Bias
- If treatment variable exists: compare sample sizes by arm. Flag imbalance >10%
- Check if missingness patterns differ by treatment arm for key outcomes
- If attrition variable exists: profile attriters vs non-attriters on baseline characteristics

#### 4.2 Construction Validity
- For index/composite variables: check correlation with components. Flag if a component has near-zero correlation with the composite (may not belong)
- For per-capita variables: verify denominator is reasonable (household size > 0, not missing)
- For currency-converted variables: verify conversion factor is plausible (not 0, not extreme)

#### 4.3 Winsorization / Trimming
- Detect winsorized variables (names containing `wins`, `_w`, `_tr`)
- Verify winsorization was applied (compare min/max to percentile boundaries)
- Flag if winsorization thresholds seem too aggressive (>5% each tail)

#### 4.4 Time / Date Variables
- Check date variables for future dates (data entry errors)
- Check for dates outside the study period
- Flag if interview date range is implausibly short or long

---

### Step 5: Analysis Output Checks (pre-submission mode only)

*Skip for `quick` and `deep` modes.*

#### 5.1 Regression Output Tables
- If results .dta or .csv files exist, check:
  - Do coefficient signs match economic intuition? (Treatment effects positive for graduation programs)
  - Are standard errors plausible relative to coefficient size?
  - Are sample sizes consistent across specifications?
  - Do R-squared values seem reasonable for the model type?

#### 5.2 Table-to-Data Traceability
- For each regression output: can we identify which .dta was the input?
- Are all outcome variables in the output traceable to a column in the analysis dataset?

---

### Step 6: Generate Report

**Save report to:**
- Same directory as reviewed data: `pipeline_audit_[YYYY-MM-DD].md`
- Or project root if reviewing a project

**Classify each finding:**
- **RED FLAG** — Data integrity violation. Likely pipeline bug or data corruption. Investigate immediately.
- **ORANGE FLAG** — Suspicious pattern that may indicate a problem. Verify before proceeding.
- **YELLOW FLAG** — Unusual but potentially intentional. Document the decision if keeping.
- **CLEAN** — Passed the check.

---

## Output Format

```markdown
# Pipeline Audit: [project/directory]

**Date:** [YYYY-MM-DD]
**Mode:** [quick / deep / pre-submission]
**Files reviewed:** [N] .dta files ([total rows] observations, [total cols] variables)
**Reviewer:** /pipeline-audit v1.0

---

## Executive Summary

**Overall assessment:** [Clean / Minor Issues / Investigate / Significant Problems]

[2-3 sentences: what was checked, what was found, what needs attention]

**Red flags:** [N]  |  **Orange flags:** [N]  |  **Yellow flags:** [N]  |  **Clean checks:** [N]

---

## Red Flags

### RF1: [Title]
- **File:** [filename]
- **Issue:** [description]
- **Evidence:** [specific values, counts, or statistics]
- **Action:** [what to investigate or fix]

---

## Orange Flags

### OF1: [Title]
- **File:** [filename]
- **Issue:** [description]
- **Evidence:** [evidence]
- **Question:** [what to verify — phrased as a question for the PI]

---

## Yellow Flags

[Brief list format]
- **YF1:** [file] — [issue] — [why it might be OK]

---

## Clean Checks

| Check | Status | Notes |
|-------|--------|-------|
| ID uniqueness | PASS | [N] files, all unique on [hhid] |
| No empty files | PASS | All [N] files have data |
| No sentinel codes | PASS | No -99/-88/-77 in numeric columns |
| ... | ... | ... |

---

## File Inventory

| File | Rows | Cols | ID Column | Missing Rate | Notes |
|------|------|------|-----------|-------------|-------|
| [name] | [N] | [N] | [col] | [%] | [any flags] |

---

## Next Steps

1. [Highest-priority investigation]
2. [Second priority]
3. [Third priority]
```

---

## Principles

- **Adversarial, not adversary.** The goal is to find problems before reviewers, referees, or replicators do. Every flag is an opportunity to strengthen the work.
- **Evidence-based.** Every finding includes specific numbers, rows, or statistics. "Something looks wrong" is not a finding.
- **Calibrated severity.** A duplicate ID is a red flag. An unusual distribution is a yellow flag. Don't cry wolf.
- **Ask, don't assume.** When a pattern is ambiguous, phrase the finding as a question for the PI: "Is it expected that 23% of treated households have missing consumption data?"
- **Respect the pipeline.** The cleaning code may have already handled an issue. Check the code/logs before flagging something the pipeline explicitly addresses.
- **No false comfort.** A clean audit doesn't mean the data is perfect — it means the checks you ran didn't find problems. State what you checked and what you didn't.