Cross Language Check

Cross-Language Replication Check

Level 1 of the verification hierarchy: same specification → same estimate across languages. If two independent implementations disagree, at least one has a bug.

Output Path

Per rules/review-artefact-routing.md (auto-loads in research projects (path-scoped to paper-*/ and paper/)):

• Source slug: cross-language-check
• Write reports to: reviews/<scope>/cross-language-check/<YYYY-MM-DD-HHMM>.md inside the project, where <scope> is the paper slug (e.g., paper-jtp) for paper-level checks or _project for project-level checks. Path is relative to the research project root, not the Task-Management repo.
• Never at project root (./CRITIC-REPORT.md-style filenames are forbidden — pre-rule layout).
• Idempotency: if today's file exists, append a same-day descriptor ({date}-revision.md, {date}-r2.md, {date}-pre-submission.md) — never overwrite.
• Index update: if reviews/INDEX.md exists, write a one-line entry under "Latest per source" pointing at the new file. Otherwise review-recap will rebuild the index next time it runs.
• Infrastructure repos (Task-Management, atlas-workspace, etc.): this section does not apply — the path-scoped rule won't load there.

When to Use

• Before submitting a paper with quantitative results
• When you suspect a subtle bug in estimation code
• After refactoring analysis scripts
• As a robustness check that reviewers increasingly expect
• When switching languages for a collaborator

When NOT to Use

• Pure simulation code with no statistical estimation → computational-experiments
• The analysis is trivial (descriptive stats only) — not worth the overhead
• The source script uses language-specific packages with no equivalent (e.g., bespoke Bayesian MCMC)

Workflow

Phase 1: Parse Source Script

1. Read the source script — identify language, packages, estimation calls
2. Extract the specification:
   • Data loading and cleaning steps
   • Variable construction and transformations
   • Estimation command(s) with exact formula/model specification
   • Standard error clustering, weights, fixed effects
   • Sample restrictions and filters
3. Identify key outputs — point estimates, standard errors, p-values, confidence intervals, N
4. Flag untranslatable elements — language-specific features that may need adaptation (e.g., R formula syntax, Stata factor variables, Python sklearn pipelines)

Phase 2: Choose Target Language

If --target is specified, use that. Otherwise:

Ask the user to confirm if the default seems wrong for the specific analysis.

Phase 3: Translate

Write the replication script to code/replication/ (or src/replication/):

code/replication/{original_name}_{target_lang}.{ext}

Translation rules:

1. Mirror the specification exactly — same formula, same controls, same sample restrictions
2. Use equivalent packages (see shared/multi-language-conventions.md for mappings)
3. Match output format — both scripts should produce a CSV with columns: estimate, se, pvalue, ci_lower, ci_upper, n, model_label
4. Document every adaptation — comment blocks explaining where the translation required judgment calls
5. Use the same data file — both scripts read from the same cleaned dataset

Phase 4: Run Both & Compare

1. Run the source script, capture output CSV
2. Run the replication script, capture output CSV
3. Comparison thresholds:

4. Generate comparison table saved to code/replication/comparison.md:

| Model | Estimate (source) | Estimate (replica) | Diff (%) | SE (source) | SE (replica) | Diff (%) | N match | Verdict |

Phase 5: Diagnose Discrepancies

If any FAIL or WARN:

1. Check N first — if sample sizes differ, the data pipeline diverged (most common source of bugs)
2. Check SE type — HC1 vs HC3 vs clustered vs bootstrap defaults differ across languages
3. Check optimizer — MLE/GLM may converge to different optima with different starting values
4. Check missing value handling — NA dropping rules differ (R drops per-variable, Stata drops listwise, Python varies)
5. Check factor variable encoding — reference category defaults differ across languages

Report the root cause, not just the symptom.

Phase 6: Report

Save to code/replication/cross-language-report.md:

# Cross-Language Replication Report

**Source:** {source_path} ({source_language})
**Replica:** {replica_path} ({target_language})
**Date:** {date}

## Summary
- Models checked: N
- PASS: N | WARN: N | FAIL: N

## Comparison Table
[from Phase 4]

## Discrepancies
[from Phase 5, if any]

## Verdict
[REPLICATED | REPLICATED WITH NOTES | FAILED — action required]

Common Package Mappings

Log to REVIEW-STATE.md (final step)

Write the comparison report to reviews/<scope>/cross-language-check/<YYYY-MM-DD-HHMM>.md (where <scope> is the paper slug for paper-level checks or _project for project-level checks; mkdir -p reviews/<scope>/cross-language-check/ first). Then append a row to the project's REVIEW-STATE.md:

bash <skills-root>/_shared/review-state-log.sh \
  --check cross-language-check \
  --paper "<paper-{venue} dir, or — for project-level cross-language checks>" \
  --verdict "<MATCH|DIVERGENCE>" \
  --score "<pass-count>/<total-comparisons>" \
  --open-issues "<fail-count>/<total-comparisons>" \
  --report "reviews/<scope>/cross-language-check/<YYYY-MM-DD-HHMM>.md" \
  --notes "<one-line: e.g. 'all match within tol'; or 'IV SE differs in §4'>" \
  [--trigger "pre-submission-report|review-cluster"]

• Verdict: MATCH if every comparison passes (within tolerance); DIVERGENCE if any FAIL.
• Score: PASS count / total comparisons.
• Open issues: FAIL count / total at run time.
• Trigger: pass orchestrator name only if invoked as a sub-agent. Otherwise omit.

Schema: the installed shared resource shared/review-state-schema.md.

Cross-References