Reviewing Paper Sources

Reviewing Paper Sources

Overview

A paper review is an evidence-gathering exercise, not a reading exercise. The findings that change a paper come from things you compute or verify, not from things you notice while reading: recomputing the headline statistic, checking every reference against its publisher record, compiling the sources yourself, rendering the figures at print size.

Core principle: never trust an artifact you did not produce. The committed PDF is stale until proven current. A reference is wrong until checked against a primary source. A reported rate means nothing until you have its sample size.

Mode: establish this first

If it is ambiguous, ask. "My student's paper" ⇒ Mode A. "I'm reviewing for X" ⇒ Mode B.

Then establish what you were given. With LaTeX sources, follow the workflow below as written. With only a PDF — the usual case for refereeing — the compile step, the .bbl cross-check and the source-based tools are all unavailable, and two failure modes appear that do not exist otherwise. Build the pseudo-source first and read reference/pdf-only-review.md:

python3 assets/run-pdfshim.py paper.pdf --out review-assets/

Also establish where in its lifecycle the paper is, because it changes what the review is for: pre-submission (triage by what fits the deadline), already submitted (target what reviewers will attack, and prepare answers), or post-rejection (framing and structural weaknesses).

An unfinished draft is a fourth case, and it needs the findings split three ways: content, not-yet-written, and container. Content is the only one that bears on the judgement — claims that are written and wrong, framing the paper's own data refutes, work missing from the argument. Container is everything that makes the artifact submittable rather than good, and it belongs in its own late section with the severity tag saying so ("Blocks submission; not a judgement on the work") rather than sitting at the top of the concerns list outranking the science. The temptation runs the other way, because container defects are the easiest to find and the easiest to state with certainty — which is exactly why they crowd out the findings that took real work to establish.

For the not-yet-written third: empty sections, note-form prose and TODO stubs are normal at this stage and the authors already know about them; reporting them as defects wastes the review. What they cannot see is which of the text they have written is wrong. So give this third an inventory table — location, current state, and a yes/no column for whether it blocks submission — rather than prose. The yes/no column is the deliverable: on one draft it distinguished an empty Conclusions section and a section heading with nothing under it, both of which had to be written, from an appendix the author was mid-way through rewriting, which could ship as it stood. Read the notes rather than skipping them, too: an author's own \todo{is this correct?} against a method description means the method is not settled, which is a finding about the work and not about the prose — promote that one to content.

Two boundary calls worth making the same way each time. A venue-mandated statement that is only a stub is container, not not-yet-written, however unwritten it looks: writing it is mechanical and it earns no credit. An absent Limitations section is content, because what goes in it is the paper's own account of what its evidence cannot support, and that is a judgement only the authors can make.

One trap specific to this case: an unfinished Discussion often lists contributions in note form that the Results section does not contain, because the experiment was cut or commented out. Cross-check every claimed contribution in the notes against the built paper before assuming it is merely unwritten.

Identify the real artifact before reviewing

The paper is what was submitted, and the repository may not contain it. Before writing anything, reconcile:

• A commented-out \input does not mean the content is absent. Sections get consolidated: check where each heading actually lives (grep -n '^\\section' sections/*.tex) before concluding anything is missing. One file routinely holds two top-level sections. A file named eval_*.tex that nothing includes is usually a superseded draft, not a lost result.
• Map the built structure against the submitted PDF heading by heading, then compare word multisets per page. Differences that are only ligatures (di+erent for different) or missing figure-embedded numerals mean the submitted PDF was re-saved through a viewer (check pdfinfo Producer for Quartz, etc.) and its text layer, not its content, differs.
• Diff the reference lists. A name present in the submitted bibliography but absent from yours is either a citation removed post-submission or a phantom entry from a stale .bbl. Both are findings.
• If the repository still cannot produce the submitted paper, ask for the submitted PDF rather than reviewing a reconstruction. Say what you would otherwise be guessing at.

Workflow

Create a todo per phase. Do not skip phases 1–4 to get to the writing.

0. Scope the review. Before any work, present the optional modules with their costs and ask which to run. Record the answer in the review header so the artifact states its own coverage. Ask once; do not re-ask mid-review.

Before running a gap sweep, run it with --seeds-only first: it makes no requests and shows exactly which queries will be used. Wrong angles mean wrong results, and OpenAlex bills each search against a small daily budget.

Offer bibliography verification pre-selected: measured against two bibliographies with known ground truth, it has the highest finding-rate per minute of any phase, and it found a defect a careful manual pass had missed.

1. Ground truth. Compile the sources yourself into a scratch dir: latexmk -pdf -interaction=nonstopmode -outdir=/tmp/pbuild main.tex Then compare against any committed PDF. A stale committed PDF is a finding in itself — it can carry the wrong title, or real author names in a double-blind submission. Record page count, LaTeX errors, undefined citations/references, and overfull boxes. Also list the directory: unused files whose names identify authors (figure_JH.pdf) are an anonymity risk worth reporting.

If it does not compile, fix it before reviewing — a review of a document you could not build is worth little. Then ask which artifact is authoritative before writing anything (see Identify the real artifact).

A build that stops before BibTeX ships a paper with every citation unresolved. latexmk abandons the run on the first fatal error, so BibTeX never executes and no .blg appears — yet a PDF from an earlier pass may still be sitting there, with all (?). If you see errors, re-run with -f and compare: the difference between the two builds is what the authors will ship if they do not look. On one draft that difference was 47 citations against 14 genuinely missing entries.

Grep the rendered text for the authors' own editorial macros, not the source. In-progress comment macros (\todo, \fixme, first-initial macros like \ab{...}) are usually defined to print, in colour, and survive into a double-blind submission carrying their author's name. pdftotext main.pdf - | grep -c 'Name:' settles it in one command; the source cannot, because a macro that looks inert may be \newcommand-ed to render. Count them and say where the reader-visible ones fall — abstract and first body page are the ones that matter. This is a desk-reject risk at every double-blind venue and it is the most common anonymity leak by a wide margin, ahead of identifying filenames.

Delete stale .bbl before testing a bibliography. \bibliography{...} inputs ./main.bbl, so a leftover .bbl in the source directory silently shadows your test even with -outdir. If citations resolve suspiciously well, or no .blg appears, that is what happened. This also explains a class of real defect: a submission built past a fatal error against an old .bbl ships a bibliography that no longer matches its own citations.

2. Venue rules. Find the CFP (WebSearch/WebFetch) and check: page limit and whether references count, anonymity requirements, format/template, deadline. The deadline determines how you triage. Report compliance explicitly.

Read the template's own comments — they list the commands that block publication. grep '^%' <venue>.sty main.tex typically yields lines like "\nocopyright -- Your paper will not be published if you use this command". Collect them, then grep the comment-stripped source for live usage. Check \vspace too: negative vertical space to win room is the most common silent violation. Verify the page split by extracting text per page rather than trusting the page count — "7 pages of content, 9 total" means the References heading must fall at the top of page 8, and that is a one-command check.

A venue-mandated statement can be present in the source and absent from the PDF. Templates gate required blocks on class conditionals — a track option, a camera-ready switch — and an unset option commonly falls through to a branch that sets none of them. The statement then sits in main.tex looking done while the PDF has no trace of it, which is worse than a visible placeholder because nothing prompts anyone to fix it. Never confirm a mandatory statement from the source: grep the rendered text, then read the class to find which flag gates it (grep -n 'newif\|needs' <venue>.cls). An unset track option usually also prints its own header banner — "Unspecified Track" or similar — which is the visible symptom of the invisible defect.

3. Read the rendered pages, not just the source. pdftoppm -r 110 -png main.pdf /tmp/pg, then read the images. This is the only way to catch illegible figures, encoding inconsistencies, and space allocation. Estimate what fraction of the page budget the figures consume.

Extract the text layer of every figure — pdftotext Figures/*.pdf -. Plots are generated by scripts, so their legends contain series the prose may never mention. On one submission this surfaced a fourth series in the headline scalability figure — a CPU build of the authors' own engine — whose name appeared zero times in the .tex. It was the ablation separating the paper's GPU claim from the effect of reimplementation alone, and no reader could have known it existed. Compare the series names against the text and ask about any that are unexplained.

4. Audit the claims. For every headline number: find its sample size, recompute the statistic, and check whether the comparison is licensed. Read the primary sources for numbers the paper quotes from others — protocols rarely match. See reference/claim-audit.md.

4a. Check baseline provenance. Before comparing protocols, establish whether the baseline rows were re-run or quoted. Take the reported values, pull that paper's own source (curl -sL arxiv.org/e-print/<id> | tar -xz, or CVF/ACL Anthology for camera-readies), and diff. An exact match across every column to two decimals settles it. Then read the baseline's methods section and compare protocol and search budget against the submission's. On one review this took fifteen minutes and produced the finding that set the rating. While you are there: does every baseline named in a results table have a bibliography entry? On one paper four of nine did not.

For the baseline-scale and seed-count checks specifically, invoke comparing-papers: python3 ~/.claude/skills/comparing-papers/assets/run-compare.py . --against "<baseline title>" --out review-assets/ It fetches the baseline's LaTeX source, extracts updates x batch against the baseline's own published scale, and computes the ratio with both quotes. On one real paper this reproduced a critical finding (6.4M vs 60M examples, ~11%) that had taken a manual pass through two appendices.

4b. Check the mathematics. For any paper with theorem or proof environments, invoke verifying-proofs: python3 ~/.claude/skills/verifying-proofs/assets/run-proofcheck.py main.tex --out review-assets/ Its default mode needs no external tooling and reports what the structure of the argument gives up: an induction with no base case, a claim dependency cycle, a restatement that drops a hypothesis, a division by something nobody proved non-zero. Read the coverage table before the findings — "54 of 138 inference steps were mechanically checkable" is usually the more important number, and a dense cluster of UNVERIFIED inside one proof is a finding in itself.

Then do what the tool cannot: reference/structural-audit.md in that skill is the non-mechanical half — whether the hypothesis is used, whether quantifier order survives, whether the induction covers its claim.

When a specific derivation is load-bearing and you cannot follow it, invoke explaining-derivations. A step nobody can make explicit is evidence against the derivation, and its gap ledger converts that into review findings with severities.

5. Audit the bibliography — every entry, no exceptions. Invoke verifying-bibliography: python3 ~/.claude/skills/verifying-bibliography/assets/run-bibcheck.py refs.bib --bbl main.bbl --out review-assets/ Consume bibcheck-report.md, then do what the tool cannot: read each load-bearing citation's abstract and check the submission's characterisation against it (selective citation), and write refs-corrected.bib. See reference/bibliography-audit.md for the audit-table format and the corrected-bib rules.

WEAK and UNVERIFIED rows are not passes — they are the entries a human must check, and a fabricated reference is exactly the one no database can find. Run the checker over the whole file, not only the cited subset: uncited entries are also unaudited.

This is the phase most likely to be skipped and most likely to produce a finding.

6. Write the review into review-<reviewer>-<YYYYMMDD>.md using reference/review-template.md, then render it and every audit report:

MD2PDF=~/.claude/skills/_shared/md2pdf/md2pdf
"$MD2PDF" --review review-<reviewer>-<date>.md
"$MD2PDF" --review review-assets/*.md        # bibcheck, proofcheck, gaps, comparison

--review breaks the metadata block per label, turns the repository-state blockquote into a callout box, and tightens the audit tables. It is one self-contained script, so copying just that file into review-assets/ keeps the build reproducible after the skill is gone.

Read what it prints. A character no font can render is dropped silently by the engine — the sentence then reads as though you never wrote it. md2pdf reports every dropped character; treat that warning as a defect in the review, not noise. It also says when it had to degrade something (a quoted macro typeset literally, math typeset as source text).

Do not hand-tune the LaTeX for wide tables. Wide tables already step down a font size and long DOIs already get break opportunities inserted; a table still running off the page means the content needs splitting, not the preamble.

7. Annotate the sources (Mode A only). Produce main-annotated.tex with changes.sty markup and a corrected .bib. See reference/annotating-with-changes.md — check the document class for a forbidden-package list first, and expect the [final] accept-all build to be less reliable than the markup build. Render the marked-up pages and look at them; markup that compiles can still be garbage.

8. Verify, then offer both formats. Every artifact compiles; every claim in the review traces to something you ran or read.

Refute your own findings before committing the rating, not after. Across four PDF-only reviews the pass changed the rating on three of them, because the findings it overturned were the ones carrying the recommendation. Dispatch one subagent with the finished review and the sources, instructed to refute each critical and major finding and to default to "refuted" under uncertainty. Every surviving objection must be visibly resolved — the finding is amended, or it is kept with the reason the objection fails. An objection neither answered nor recorded is the one that comes back in the authors' response.

Two things this needs to be told, both learned from running it:

• Adjudicate against the artifact the review pins, not the working tree. On the one review this was tried against, the tree had moved one commit past the reviewed artifact and already conceded five of seven findings. Judging a review by revisions made in response to it is circular, and the refuter has to be told so or it will report those concessions as refutations.
• Have it check the review's own arithmetic and labels, not only its conclusions. The value showed up inside sound findings rather than against them: 0 of 7 findings were overturned, and 3 carried a wrong sub-claim — a confidence interval attributed to the wrong method, a "the metric cannot fail by construction" assertion contradicted by the paper's own figure, and a parenthetical whose numbers could not average to the figure they were said to average to. Each would have been corrected by an author, and a review's authority rests on getting exactly that kind of thing right.

Nothing else in the workflow re-checks the review's own statistics — phase 4 audits the paper's arithmetic, not yours.

Close by listing what exists, .md and .pdf side by side, so the user can read it either way without asking — the .md is what they will edit and diff, the .pdf is what they will circulate or annotate:

review-claude-20260815.md   review-claude-20260815.pdf   (8 pp)
review-assets/bibcheck-report.md   .../bibcheck-report.pdf   (2 pp)

Offer to open the PDF. Report page counts: a review that ran to 30 pages is a finding about the review, not about the paper.

Hard rules

• Never modify the originals. main.tex and refs.bib are inputs. Write main-annotated.tex, refs-corrected.bib. If you must test a corrected bib against the real build, back up, test, restore, and say so.
• Never install tooling without asking. Missing pandoc/LaTeX is a question for the user, not a brew install.
• Never commit anything. Leave the artifacts untracked; integration is the author's call.
• Every number in the review must be one you computed or read from a source. No "approximately" derived from memory.
• Report what you could not verify. An unverifiable reference or an anonymous artifact is a stated limitation, not a silent pass.
• A disqualifying finding sets a floor the scores cannot lift. Never sum or average the §2 criteria. If soundness of evaluation fails, the recommendation follows soundness, whatever novelty scored — and the §2 disqualifying line must name every finding that gates the outcome. Across five reviews on file the verdict line once named M1–M3 while its own prose named a fourth item carrying desk-reject risk; a reader acting on the bold line would have submitted with it outstanding.
• On an early draft, separate the two gates and let only one of them score. The rule above was written for a finished paper, where a desk-reject risk and a soundness failure both stop the same submission. On a draft still being written they are different questions, and merging them produces a verdict about the container instead of the work. Anonymisation leaks, an unset template switch, page count, missing bibliography entries and LaTeX errors are all real and none can be skipped — and all of them are a morning's work that the authors were always going to do. State them, in their own section, under their own heading, listed after the content findings; give the §2 line as two lines — what gates the science, and what gates the submission — and score only the first. A review that opens with four mechanical items and one substantive one has told the author their paper is bad when what it means is that their build is unfinished. This was learned the hard way: a review on file did exactly that and had to be restructured after the author pointed it out.

Findings that recur

Run this list against every quantitative paper; each is expanded in reference/claim-audit.md: sample size behind a 0%/100% rate (rule of three) · protocol match to each baseline · baseline training scale vs its published scale · seed count · self-built baselines · ablations for the claimed mechanism · metric saturation by construction · selection on the reported construct · per-item denominators hidden inside an aggregate · unevaluated contributions · cost of the method · selective citation · does the hyperparameter grid contain the value that turns the contribution off · are the constants in a stated guarantee ever measured on real data · does the appendix concede something the main text presents favourably · do the ranking gaps survive the variance the paper's own appendix reports · does every derived annotation on a figure match the table it came from · does the paper's own comparison table support the superiority its caption claims.

Read the commented-out text (grep -n '^\s*%' sections/*.tex). Authors delete their own caveats under page pressure, and a caveat they wrote and cut is the strongest recommendation you can make — you are asking them to restore their own sentence, not accept yours.

Common mistakes

• Reviewing the committed PDF. It may not correspond to the sources. Compile first.
• Spot-checking the bibliography. In practice errors cluster in the references the authors were least likely to re-read — the classic ones. Sampling misses them.
• Clearing an entry after checking authors only. Check authors, title, venue, volume/issue, pages, and DOI. A correct DOI beside a wrong title is how fabrication becomes detectable.
• Softening the central objection. If the headline claim is unsupported, that belongs in the summary, not in concern #7.
• Writing "should be improved" without the fix. In Mode A every concern needs a concrete, costed action.
• Skipping the figure render. Illegible figures are invisible in the .tex.
• Writing "the paper does not report X" after one failed search. The largest source of wrong findings in PDF-only reviewing, six times across four reviews. Read the caption and heading inventory end to end, search the concept in several wordings, and read the whole subsection rather than the paragraph you landed on. Never write "I would accept this if the paper did Y" without checking that it has not already done Y.
• Shipping a typographic finding straight from the text layer. Extraction artifacts masquerade as author errors: a caption clipped out of the printed page, a hyphen removed at a line break, axis labels producing doubled words. Render the page and look before reporting.
• Treating WEAK/UNVERIFIED as a pass. They mean no database confirmed the entry — which is what a fabricated reference looks like. Check them by hand.
• Reporting a rate-limited run as a clean bill. When the checker's circuit breaker drops a source, most entries fall through to title-search only. Say "nothing wrong was found in a degraded run", not "the bibliography is correct", and re-run when quotas reset.
• Letting the review PDF drop characters. md2pdf warns, but you have to read the warning: a missing glyph is deleted silently by the engine, so the sentence reads as though you never wrote it.
• Delivering only one format. The .md is for editing and diffing, the .pdf for circulating. Produce both and say where they are; do not make the user ask.

Quick reference

Every .md in that table gets a .pdf from md2pdf --review; deliver both.

assets/: changes-preamble.tex, run-pdfshim.py + pdfshim/ (PDF -> pseudo-source) reference/: bibliography-audit.md, claim-audit.md, annotating-with-changes.md, review-template.md, pdf-only-review.md (no LaTeX sources) PDF rendering: _shared/md2pdf/md2pdf --review (see its README.md) Sibling skills: verifying-bibliography (phase 5), surveying-literature (phase 0), comparing-papers (phases 0 and 4), verifying-proofs and explaining-derivations (phases 0 and 4b). Shared layers: _shared/scholarly (retrieval), _shared/latexmath (proof parsing).