Adversarial Doc Review

Adversarial Doc Review

Quick start

1. Get the document (path/URL/pasted text) plus the intended audience and stakes.
2. Build a claim ledger: every definition, assumption, factual claim, normative claim, control claim, and route-authorization claim must be traceable to a location in the document.
3. Stress-test internally for consistency, logic, edge cases, evidence, citations, non-vacuity, and specification gaming.
4. Verify externally by default: use a web browsing/search tool to fact-check time-sensitive and externally-checkable claims.
5. When the document authorizes research evidence or stage advancement, apply references/epistemic-assurance-contract.md and test whether labels correspond to actual properties.
6. Produce a structured report in chat by default using references/report-template.md and references/review-checklist.md. If a file is requested, write <docname>.review.md next to the document.

Workflow decision points

• If web browsing is unavailable or disallowed, ask for permission or user-provided sources; otherwise label items as unverified and state what would verify them.
• If the document is long, ask which sections matter; still scan the rest for contradictions, definitional drift, unsupported strong claims, and stage-advancing statements.
• If the document claims validation, verification, independence, immutability, isolation, replay, auditability, or route readiness, inspect the mechanism and evidence rather than accepting the label.
• Default deliverable: structured report in chat. If a file is requested, write <docname>.review.md in the same folder as the input document.

Inputs supported

• Pasted text, Markdown (.md file paths), LaTeX, PDF, DOCX, web pages.
• Prefer reviewing the canonical version (repo path, published PDF). Record the version/date you reviewed.
• For .md inputs, cite path:line plus the nearest heading, for example docs/P2.md:143 (## Assumptions).

Adversarial review workflow

1) Intake and scope

• Ask for purpose, target audience, required rigor, whether you may browse the web, and whether the document is intended to authorize evidence promotion or a consequential decision.
• Record title, authors, version/commit, and review date.
• Identify the key conclusions the document wants the reader to accept.
• Identify any claimed evidence class: exploratory, confirmatory, independently verified, or operational/high-stakes.

2) Build a claim ledger exhaustively

• Extract and list:
  • definitions: terms, symbols, acronyms, assurance labels, and where they first appear
  • assumptions and scope limits, explicit and implicit
  • factual claims, including numbers, dates, rankings, “state of the art,” “most,” and “first”
  • normative claims such as “should” and “must,” plus policy requirements
  • causal claims, correlations, and generalizations
  • control claims such as immutable, isolated, replayed, blinded, independent, verified, validated, or audited
  • route-authorization claims such as proceed, gate passed, ready, safe, or sufficient
• For each entry, tag:
  • type: definitional / factual / quantitative / normative / causal / speculative / control / route authorization
  • verifiability: internal-only vs external-checkable
  • time sensitivity: stable vs time-sensitive
  • evidence class and provenance when relevant
• Keep every entry traceable with short quotes and location pointers: Markdown path:line plus nearest heading; PDF page; LaTeX section; web URL plus heading when possible.

3) Internal stress-test

• Consistency:
  • Find contradictions, changing terminology, inconsistent notation, unit mismatches, and evidence-class drift.
  • Verify every variable, symbol, assurance term, and gate condition is defined and used consistently.
• Logic and argumentation:
  • Find missing steps, unsupported leaps, hidden quantifiers, category errors, and conclusions stronger than the premises.
  • Construct counterexamples and edge cases; check whether the document already rules them out.
• Evidence and methodology:
  • Check that strong claims have commensurate evidence.
  • Flag selection bias, cherry-picked examples, unclear baselines, missing ablations, missing failure states, and missing sanity checks.
  • Check whether cases, metrics, or conclusions were selected after inspecting relevant outcomes.
• Citations:
  • Flag missing citations and citation laundering: a citation is present but does not support the attached claim.

4) Run the assurance and specification-gaming pass

For documents that claim controls, verification, or research-route readiness:

• Distinguish the property from its label or proxy.
• Test common substitutions explicitly:
  • self-hash versus external immutability
  • environment flag versus actual isolation
  • copied digest versus regenerated replay
  • role name versus material independence
  • file or field presence versus semantic completeness
  • shared implementations agreeing versus genuinely diverse evidence
  • feature-absence comparator versus substantive distinctness
• Check non-vacuity:
  • Can competing systems, policies, or actions differ on any plausible case?
  • Can the comparator win under plausible conditions?
  • Does the loss or decision contract penalize every decision-relevant error, including failure to act?
• Check complete outcome accounting:
  • successes, wrong actions, missed actions, skips, nulls, retries, timeouts, initial failures, and resource failures
• Check hidden-information boundaries and whether the evaluated process could read oracle truth or outcome labels.
• State actual independence across context, data, implementation, evaluation, and advancement authority. The same agent or team performing multiple roles is self-review unless materially separated.
• Carry material predecessor failures forward. Reclassification, replacement, renaming, or omission is not resolution.

5) External verification

• For each externally-checkable or time-sensitive claim:
  • Search the web and verify against reputable, current sources, preferring primary sources, standards bodies, official documentation, peer-reviewed venues, or widely trusted data providers.
  • Record source, publication/update date, and what exactly it confirms or refutes.
  • If sources disagree, report the disagreement and what would resolve it.
• Verify references:
  • Confirm the cited work exists and plausibly supports the statement it is attached to.
• Do not guess. If you cannot verify, label the claim unverified and state what evidence is needed.

6) Produce the adversarial report

• Use references/report-template.md.
• Prioritize major issues that would change conclusions, correctness, safety, evidence class, or route authorization.
• Provide concrete fixes: rewrites, added definitions, missing citations, alternative framing, corrected evidence classification, or additional experiments/analyses.
• Separate:
  • incorrect or misleading statements
  • true-but-misleading statements
  • unsupported statements
  • ambiguous statements
  • outdated statements
  • assurance labels not established by the mechanism
  • unresolved predecessor failures
• Use bounded verdicts such as structurally valid only, internally consistent only, supports exploratory follow-up, supports the confirmatory claim, or independently verified.
• Do not infer malicious intent from a misleading artifact alone. Distinguish intentional misconduct, agent violation of explicit instructions, skill-contract defects, orchestration defects, and ordinary error according to the available provenance.

Format-specific tips

• Markdown (.md):
  • For every issue/claim, cite path:line plus the nearest heading.
  • Preserve code fences and tables when quoting; avoid paraphrases that change semantics.
  • Check that internal anchors and external links resolve where feasible.
• LaTeX:
  • Check \\cite{} keys exist in .bib and that claims align with cited works.
  • Compile if possible; otherwise review .tex plus any generated .bbl/PDF if available.
• PDF/DOCX:
  • Prefer extracting text for searchability; keep page/section references for every issue.
• Specs/policies:
  • Treat MUST, SHOULD, and MAY as normative; look for loopholes, contradictions, proxies, and missing definitions.

Resources

• references/review-checklist.md: full pedantic checklist.
• references/report-template.md: report skeleton to fill.
• references/epistemic-assurance-contract.md: proportionate evidence-promotion and independence contract.