# Psy Ana Reviewer

> Audit a behavioral-data analysis plan or R/Python script without modifying it. Use for statistical-method review, reproducibility review, publication readiness, seeds, exclusion logging, effect sizes, multiple-comparison correction, assumptions, sensitivity analyses, figures, session information, and “检查分析代码/统计方法审查/分析脚本有没有问题”. Select the review mode from the available input and report graded findings plus a readiness label. Do not generate or fix analysis code.

- Skill: `soupandpsy/psy-ana-reviewer` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add soupandpsy/psy-ana-reviewer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/soupandpsy/psy-ana-reviewer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Security
- Author: soupandpsy (https://skillmd.com/u/soupandpsy)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/soupandpsy/psy-ana-reviewer

---


# Analysis Code Reviewer

## Version

v1.4.0 — unified evidence-gated contract, 2026-07-23. Sub-skill of [amazing-psycoder](../SKILL.md).

## Purpose

Audit analysis plans, scripts, execution evidence, and generated results for statistical correctness, reproducibility, and reporting completeness. Static review can approve execution testing; publication readiness additionally requires a successful clean run and review of the actual tables/figures/logs supporting the claims.

This is the **analysis audit layer**. It evaluates code generated by psy-ana-coder, identifies issues, and works with the coder to fix them. The reviewer enters a **check → fix → re-check loop** — each audit round identifies remaining issues, the coder applies fixes, and the reviewer re-audits. Zero Critical/Major permits the maximum label supported by that mode's evidence; it never upgrades a static audit to publication readiness.

## Review Modes

| Mode | Input | Maximum Label |
|------|-------|--------------|
| `analysis-audit` | Complete analysis script + config/data schema | `ready_for_execution` |
| `result-audit` | Script + config + execution log + generated results | `ready_for_publication` |
| `plan-review` | Analysis config YAML | `analysis_plan_ready` |
| `triage-only` | Research question | None (missing-info list only) |
| `blocked` | Insufficient input | None |

## Readiness Labels

| Label | Meaning |
|-------|---------|
| `ready_for_publication` | Zero Critical/Major + successful clean execution + reviewed outputs/environment evidence |
| `ready_for_execution` | Static audit passed; successful clean execution and result review remain |
| `not_ready` | Critical or Major issues exist |
| `analysis_plan_ready` | Analysis design complete, ready for code generation |
| `blocked` | Input insufficient for any review |

## Severity Classification

| Severity | Definition |
|----------|-----------|
| **Critical** | The selected method/implementation cannot estimate the claimed quantity, reverses/mislabels results, or makes the reported results unrecoverable |
| **Major** | Materially biases estimates/uncertainty, ignores dependence/missingness central to the design, or blocks independent execution/result verification |
| **Minor** | Does not affect correctness; fix when convenient |

## Intake Protocol

Before any review, confirm the input:

| Mode | Intake Action |
|------|--------------|
| `analysis-audit` | Request the script, confirmed config, declared dependency artifact, and data/schema. Verify artifacts are readable; private data may be replaced by a schema plus user-executed logs for static review. |
| `result-audit` | Require script, config, declared dependency artifact, `analysis-run.json`/equivalent clean-run log, generated tables/figures, and environment snapshot. |
| `plan-review` | > "Please provide the analysis config YAML (paste content or provide file path)." Verify the YAML structure is complete. |
| `triage-only` | > "Please describe your research question, experimental design, and data type." |
| `blocked` | > "The information provided is currently insufficient for any review. Please provide at least a research question description." |

Mode auto-detection: complete execution evidence + outputs → `result-audit`; script + config/data schema → `analysis-audit`; config only → `plan-review`; question only → `triage-only`; none → `blocked`.

## Review Checklist — analysis-audit

### Gate 0: Quality Gate (minimum bar)

For `analysis-audit`/`result-audit`: Re-run the Coder Quality Gate, record every failure, and continue the remaining safe checks so the user receives a complete evidence-backed audit. A gate failure affects the verdict but must not truncate diagnosis.

When config and code are available, select an interpreter that passes `import yaml`, then run `scripts/validate_analysis.py <analysis_config.yaml> --code <script> --language r|python`. When a run log is provided, add `--execution-log <configured-log>` and reject any hash/environment mismatch. Treat a static pass as deterministic static evidence and a matching run manifest as execution evidence; neither proves the reported claims are correct without result review.

For `plan-review` mode: Skip Gate 0 (no script to check). Proceed directly to design-level review.

### 1. Statistical Validity

| # | Check | Verify by |
|---|-------|-----------|
| 1 | Estimand and model align | Verify outcome family, observation level, contrast, and interpretation answer the declared estimand; design labels alone do not select a model. |
| 2 | Dependence represented | Verify subject, item, session, site, and other clustering/repeated units declared by the sampling design are handled or explicitly justified. |
| 3 | Contrasts specified correctly | Are contrast weights orthogonal? Are planned comparisons justified? |
| 4 | Multiplicity strategy appropriate | Define the family of claims first, then verify the selected control (planned contrasts, Tukey, Holm/Bonferroni, FDR, hierarchical testing, or justified no adjustment) matches that family and the inferential goal. |
| 5 | Effect estimate correct | Verify every inferential claim has a compatible estimate and uncertainty (e.g. mean contrast, standardized contrast, OR/probability difference); model R² does not replace the focal effect. |

### 2. Reproducibility

| # | Check | Verify by |
|---|-------|-----------|
| 1 | Stochastic control recorded | Require seed/backend controls only for stochastic steps; confirm config and code agree |
| 2 | Session info output | `grep "sessionInfo\|session_info\|version"` |
| 3 | Runtime/dependency evidence | Exact language version is checked; the declared pin/lock artifact exists, covers packages actually imported, and agrees with the clean-run environment snapshot |
| 4 | Data path configurable | No hardcoded paths. Acceptable: relative paths from project root (`data/subject.csv`), `here::here()`, or path from config. Unacceptable: absolute paths (`/Users/...`, `C:\...`), `setwd()` |
| 5 | Exclusion log complete | Every excluded trial/subject documented with reason |
| 6 | Parameter provenance | Are analysis parameters (cutoffs, thresholds) referenced to config or literature? |

### 3. Assumption Checking

| # | Check | Verify by |
|---|-------|-----------|
| 1 | Distribution/model diagnostics | Use diagnostics appropriate to the estimator and target (e.g. paired differences, residuals, dispersion, convergence, posterior predictive checks), not universal Shapiro tests per condition |
| 2 | Sphericity | Only for an ANOVA whose within-subject factor/contrast structure makes sphericity relevant; use a justified diagnostic/correction rather than a ceremonial test |
| 3 | Variance structure | Check homoscedasticity/variance modeling when required by the chosen Gaussian test/model; Levene is not a universal gate |
| 4 | Diagnostic response | Is the prespecified remedy appropriate to this estimator and estimand (e.g. covariance correction, alternative likelihood, robust uncertainty, sensitivity model)? |

**Check adaptation rules**: Adjust assumption checks based on model type:
- `lmer`/`glmer` → skip Mauchly's sphericity (not applicable); check convergence warnings + singular fit + random effects variance
- paired t-test → inspect the distribution/robustness of paired differences (not raw condition scores)
- `aov` within-subjects → check Mauchly's sphericity + Greenhouse-Geisser correction
- `glmer(binomial)` → check overdispersion

### 4. Reporting Completeness

| # | Check | Verify by |
|---|-------|-----------|
| 1 | All conditions reported | Every condition from design has descriptive stats |
| 2 | Effect estimates for inferential claims | Each substantive claim has an estimate and uncertainty; diagnostic-test p-values do not require ceremonial effect sizes |
| 3 | Confidence intervals | Effect sizes reported with CI, not just point estimates |
| 4 | n reported per analysis | After cleaning, how many subjects/trials per condition? |
| 5 | Exclusion documented | Are excluded subjects/trials listed with reasons? Counts and percentages reported? |

### 5. Figure Quality

| # | Check | Verify by |
|---|-------|-----------|
| 1 | Error bars defined | SE or CI stated in caption or code |
| 2 | Individual data shown | For within-subjects designs, individual data points visible |
| 3 | Axes labeled | Clear axis titles with units |
| 4 | Color-safe | Colorblind-friendly palette? |

## R Anti-Patterns

- `attach()` — don't; use `with()` or `dplyr::` verbs
- `setwd()` — don't; use relative paths or `here::here()`
- `save.image()` — don't; save specific objects with `saveRDS()`
- `options(stringsAsFactors = TRUE)` — don't; modern R defaults to FALSE
- `summary(model)$r.squared` for mixed models — wrong R²; use `performance::r2()`
- Automatic Type III ANOVA without matching contrasts/hypotheses — choose sums of squares from the estimand/design, not imbalance alone

## R Anti-Pattern Grep Patterns

When auditing R scripts, scan for these patterns:

| # | Anti-Pattern | Grep | Why |
|---|-------------|------|-----|
| 1 | `attach(` | `grep -q "attach(" script.R` | Namespace pollution |
| 2 | `setwd(` | `grep -q "setwd(" script.R` | Non-reproducible |
| 3 | `save.image()` | `grep -q "save\.image" script.R` | Non-reproducible |
| 4 | `summary(lmer.*r.squared` | `grep -q "summary.*lmer.*r\.sq" script.R` | Doesn't exist |
| 5 | `aov()` repeated measures | `grep -q "aov(" script.R` + inspect formula/Error structure | Requires design-aware review; not automatically wrong |
| 6 | Absolute paths | `grep -qE "/Users/|/home/|C:\\\\" script.R` | Non-portable |

## Python Anti-Patterns

- `import *` — don't; use `import pandas as pd` or explicit imports
- Hardcoded paths — don't; use `pathlib.Path` or config-driven paths
- Missing `random_state`/seed on an actually stochastic function — deterministic `scipy.stats` tests do not require one
- `print(df)` without `.head()` — floods output; use `df.info()` or `print(df.head())`
- No column existence check — use `assert set(expected).issubset(df.columns)`
- `pd.set_option('mode.chained_assignment', None)` — hides warnings; use `.loc[]` instead
- No claim-compatible effect estimate — each inferential claim needs an estimate on an interpretable scale plus uncertainty; `pingouin` is optional and is not suitable for every model
- Figure not saved — `plt.savefig()` required, not just `plt.show()`
- `scipy.stats.f_oneway()` for within-subjects designs — use `pingouin.rm_anova()` instead
- Plain `statsmodels.Logit()` for within-subjects binary data — use a design-appropriate, verified GLMM/GEE/Bayesian implementation; `pymer4` and Bambi are examples, not universal defaults
- `scipy.stats.ttest_ind()` for within-subjects — use `scipy.stats.ttest_rel()` or `pingouin.ttest()`
- Aggregation that discards trial/item structure needed by the confirmed model — inspect intent rather than banning `groupby().mean()`
- `smf.ols()` for repeated measures — use `smf.mixedlm()` with groups='subject_id'
- `pingouin.compute_effsize(eftype='cohen')` for between-subjects when design is within — use `paired=True`
- `np.corrcoef()` for within-subjects repeated measures — use `pingouin.rm_corr()`

## Python Anti-Pattern Grep Patterns

| # | Anti-Pattern | Grep | Why |
|---|-------------|------|-----|
| 1 | `import *` | `grep -q "import \*" script.py` | Namespace pollution |
| 2 | Absolute paths | `grep -qE "/Users/|/home/|C:\\\\" script.py` | Non-portable |
| 3 | No saved figure despite declared figures | inspect output code | Declared artifacts not persisted |
| 4 | `chained_assignment` | `grep -q "chained_assignment" script.py` | Hides warnings |
| 5 | `f_oneway()` within-subjects | `grep -q "f_oneway" script.py` + inspect design | Ignores dependence |
| 6 | Plain `Logit()` for repeated binary trials | `grep -q "Logit(" script.py` + inspect hierarchy | No random effects; use a validated GLMM/GEE implementation |
| 7 | `ttest_ind()` within-subjects | `grep -q "ttest_ind" script.py` + inspect design | Ignores pairing |
| 8 | `smf.ols()` repeated measures | `grep -q "smf.ols" script.py` + inspect design | Ignores dependence unless justified with another covariance estimator |
| 9 | `compute_effsize.*cohen` no paired | inspect call and design | Pairing must match the estimand |
| 10 | `np.corrcoef()` within-subjects | `grep -q "np.corrcoef" script.py` + inspect design | Ignores repeated measures |

## Scope Limitation Rule

At the start of every review output, state what was and was not reviewed:

> **Scope**: Review covers statistical correctness, reproducibility, assumption checking, and reporting completeness of the provided analysis script. If only a config YAML was provided (plan-review mode), this review cannot verify implementation details such as correct API usage, data import robustness, or figure rendering. If only a research question was provided (triage-only mode), this review can only identify missing design information.

**Note**: The output format adapts to the review mode. In `analysis-audit` mode, include the full output below. In `plan-review` mode, skip Anti-Pattern Scan Results (no script to scan). In `triage-only` mode, output only the missing-information list.

## Output Format

```
## Review Mode
{result-audit | analysis-audit | plan-review | triage-only | blocked}

## Scope
{What was and was not reviewed}

## Readiness Label
{ready_for_publication | ready_for_execution | not_ready | analysis_plan_ready | blocked}

## Critical Issues
- {issue} — {how to fix}

## Major Issues
- {issue} — {how to fix}

## Minor Issues
- {issue} — {how to fix}

## Anti-Pattern Scan Results
- Context-relevant patterns checked: {list}
- Findings requiring semantic inspection: {list}

## Assumption Check Results
- Distribution/model diagnostics: {results and applicability}
- Dependence/convergence/dispersion: {results and applicability}
- Prespecified fallback/sensitivity status: {results}

## Reproducibility Evidence
Exact runtime + dependency artifact: {verified/mismatch/missing}
Stochastic controls: {verified/not applicable/missing}
Session/environment info: {present/absent}
Exclusion log: {present/absent}
Relative paths: {yes/no}

## Execution and Result Evidence (result-audit only)
Command/exit/timestamps: {verified/mismatch/missing}
Config + code + dependency + input hashes: {verified/mismatch/missing}
Output inventory + hashes: {verified/mismatch/missing}
Warnings and diagnostics reviewed: {yes/no + summary}
Claim-to-table/figure trace: {verified/mismatch/missing}

## Overall Verdict
{1-2 sentence summary}
```

## Verdict Rules

- `plan-review`: complete design contract → `analysis_plan_ready`; otherwise `not_ready`/missing-information report.
- `analysis-audit`: zero unresolved Critical/Major → `ready_for_execution`; static inspection alone cannot yield publication readiness.
- `result-audit`: require successful clean execution; matching hashes for the reviewed config/code/dependency/input; reviewed output inventory, tables/figures, warnings/diagnostics and environment evidence; and zero unresolved Critical/Major. Then use `ready_for_publication`; report any non-claim-affecting Minor findings separately.
- Any failed execution, result/code mismatch, or unresolved Critical/Major → `not_ready`.

## Recovery Loop

The audit is not a one-shot report — after issues are found, enter a **check → fix → re-check** loop until passing.

### Loop Flow

```
psy-ana-coder generates code
       │
       ▼
psy-ana-reviewer audits
       │
       ├── Critical/Major issues → fix (see fix paths below) → re-audit
       │                              ↑                    │
       │                              └────────────────────┘
       │                                   loop until 0 Critical + 0 Major
       │
       └── 0 Critical + 0 Major → static pass (`ready_for_execution`); only result-audit evidence can advance further
```

### Fix Paths

| Issue Type | Who Fixes | How to Fix |
|---------|--------|---------|
| Incorrect statistical method choice | psy-ana-designer | Revise `questions[].selected_method`/estimand rationale → psy-ana-coder regenerates |
| Code implementation error (API misuse) | psy-ana-coder | Based on specific line numbers and correct API pointed out by reviewer, directly fix the code |
| Inappropriate scientific parameter/threshold | psy-ana-designer | Confirm the revised config decision → regenerate |
| Missing effect size / figure / environment info | psy-ana-coder | Add missing code block → reviewer re-audits |
| Hardcoded data path | psy-ana-coder | Replace with config-driven path → reviewer re-audits |

> "Round N audit: {X} issues found. After fixes, proceed to round N+1."

A re-audit is mandatory after each fix. The audit round count and issues per round are recorded in the final report.