Pipeline Auditor

Pipeline Auditor (draft audit + regression)

Purpose: a deterministic “regression test” for the writing stage.

It answers:

• did we leak placeholders or planner talk?
• did citation scope drift?
• did the draft fall back to generator voice (navigation/narration templates)?
• is citation density/health sufficient for a survey-like draft?

This skill is analysis-only. It does not edit content. For survey/deep, style/citation-shape violations are blocking by default.

Inputs

• output/DRAFT.md
• outline/outline.yml
• Optional (recommended):
  • outline/evidence_bindings.jsonl
  • citations/ref.bib

Outputs

• output/AUDIT_REPORT.md

What it checks (deterministic)

A150++ citation targets (used by the auditor):

• Per-H3: >=12 unique citations (deep: >=14).

• Global: >=150 unique citations across the full draft (recommended target: 165; deep floor: 165).

• Placeholder leakage: ellipsis (..., …), TODO markers, scaffold tags.

• Outline alignment: section/subsection order vs outline/outline.yml.

• Survey tables (survey deliverable): require >=2 Markdown tables in the merged draft (index tables live in outline/tables_index.md) (inserted by section-merger from outline/tables_appendix.md).

• Paper voice anti-patterns:

  • narration templates (This subsection ..., In this subsection ...)
  • slide navigation (Next, we move ..., We now turn to ...)
  • pipeline voice (this run, “pipeline/stage/workspace” in prose)

• Evidence-policy disclaimer spam: repeated “abstract-only/title-only/provisional” boilerplate inside H3 bodies.

• Meta survey-guidance phrasing: survey synthesis/comparisons should ....

• Synthesis stem repetition: repeated Taken together, ... and similar high-signal generator stems.

• Numeric claim context: numbers without minimal evaluation context tokens (benchmark/dataset/metric/budget/cost).

• Citation health (if citations/ref.bib exists): undefined keys, duplicates, basic formatting red flags.

• Citation-shape hard gate (survey/deep): no adjacent citation blocks ([@a] [@b]), no duplicate keys inside one block ([@a; @a]), and per-H3 mid-sentence citation ratio >=30%.

• Citation scope (if outline/evidence_bindings.jsonl exists): citations used per H3 should stay within the bound evidence set.

How to use the report (routing table)

Treat output/AUDIT_REPORT.md as a “what to fix next” router.

Common FAIL families -> responsible stage/skill:

• Placeholders / leaked scaffolds

  • Fix: C2–C4 artifacts are not clean. Route to subsection-briefs / evidence-draft / writer-context-pack, then rewrite affected sections.

• Missing overview tables (draft has <2 tables)

  • Fix: ensure table-schema + appendix-table-writer produced outline/tables_appendix.md (>=2 tables, citation-backed, no placeholders), then rerun section-merger (tables insert as an Appendix block by default).

• Planner talk in transitions / narrator bridges

  • Fix: rerun transition-weaver (and ensure briefs include bridge_terms / contrast_hook), then re-merge.

• Narration templates / slide navigation inside H3

  • Fix: rewrite the failing sections/S*.md via writer-selfloop (local, section-level) or subsection-polisher.

• Evidence-policy disclaimer spam

  • Fix: keep evidence policy once in Intro/Related Work (front matter), delete repeats in H3 (use draft-polisher or local section rewrites).

• Citation scope drift (out-of-scope bibkeys)

  • Fix: either (a) rewrite the subsection to stay in-scope, or (b) fix mapping/bindings (section-mapper → evidence-binder) and regenerate packs.

• Global unique citations too low

  • Fix: citation-diversifier → citation-injector (NO NEW FACTS), then draft-polisher.

• Intro/Related Work too thin / too few cites

  • Fix: rewrite the corresponding sections/S<sec_id>.md front-matter file via writer-selfloop (front-matter path) using dense positioning + method paragraph.

Prevention guidance (what upstream writers should do)

If you want the auditor to PASS without a heavy polish loop:

• Start each H3 with a content claim + thesis (avoid narration templates).
• Use explicit contrasts and at least one evaluation anchor paragraph.
• Embed citations per claim (avoid trailing cite dumps).
• Put evidence-policy limitations once in the front matter, not in every H3.

Script

Quick Start

• python .codex/skills/pipeline-auditor/scripts/run.py --help
• python .codex/skills/pipeline-auditor/scripts/run.py --workspace workspaces/<ws>

All Options

• --workspace <dir>
• --unit-id <U###> (optional; for logs)
• --inputs <semicolon-separated> (rare override; prefer defaults)
• --outputs <semicolon-separated> (rare override; default writes output/AUDIT_REPORT.md)
• --checkpoint <C#> (optional)

Examples

• Run audit after global-reviewer and before LaTeX/PDF:
  • python .codex/skills/pipeline-auditor/scripts/run.py --workspace workspaces/<ws>

Troubleshooting

Issue: audit fails due to undefined citations

Fix:

• Regenerate citations with citation-verifier and ensure citations/ref.bib contains every cited key.

Issue: audit fails due to narration-style navigation phrases

Fix:

• Rewrite as argument bridges (content-bearing handoffs, no navigation commentary) in the failing sections/* files, then re-merge.

Issue: audit fails due to "unique citations too low"

Fix:

• Run citation-diversifier to produce output/CITATION_BUDGET_REPORT.md.
• Apply it via citation-injector (edits output/DRAFT.md, writes output/CITATION_INJECTION_REPORT.md).
• Then run draft-polisher → global-reviewer → auditor.