# Paper Review Evaluator

> Evaluate, compare, calibrate, and rank research papers or LLM/foundation-model technical reports from PDFs, extracted text, or paper titles/topics. Use when Codex is asked to score papers on innovation/value/rigor, find related comparison papers, compare model technical reports, use public reviews for calibration, analyze historical score drift, create tier lists, or write image-generation prompts summarizing rankings.

- Skill: `mathieu0905/paper-review-evaluator` (Agent Skill, multi-file: 11 files)
- Install (CLI): `npx skillmds@latest add mathieu0905/paper-review-evaluator`
- Raw SKILL.md: https://api.skillmd.com/api/skills/mathieu0905/paper-review-evaluator/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: mathieu0905 (https://skillmd.com/u/mathieu0905)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/mathieu0905/paper-review-evaluator

---


# Paper Review Evaluator

## Core Contract

Treat each invocation as one self-contained evaluation run. For a single target paper, Codex finds or accepts a comparison pool, scores that target with the pool, calibrates the judgment, and ranks only that run's papers.

For multiple target papers that are candidate submissions, manuscripts to review, or otherwise separate evaluation objects, use **multi-target local-review mode** by default: keep one coordinator run, but split the work into one independent sub-run per target paper. Each target gets its own closest-topic comparison pool and its own target-focused judgment. Do not rank the target papers against each other unless the user explicitly asks for a head-to-head ranking.

When the user asks for submission outcome, "中稿几率", PC-style review, or venue decision risk for SE venues, use the 10-point Innovation / Paper Value / Rigor rubric as the default scoring output. Do not output four-point reviewer recommendation scores unless the user explicitly asks for four-point PC simulation. Instead, provide several independent reviewer-style textual concerns and an area-chair style synthesis. The reviewers are independent complete reviewers, not preassigned personas: each reviewer considers novelty, value, rigor, presentation, fit, and risk. Do not force one reviewer to be "strict rigor", one to be "innovation", and one to be "balanced"; any emphasis differences should emerge naturally from independent readings. If four-point recommendations are explicitly requested, derive them under the same standard as the local comparison pool and warn that the scale is coarse and can hide meaningful 10-point quality differences.

Three rules make repeated use reliable:

1. Keep every run isolated. Store target papers, downloaded comparison PDFs, extracted text, review payloads, pairwise judgments, and the final report inside that run's own directory. Inventory, extraction, and scoring must touch only that directory.
2. Never carry over rankings. Do not pull in, re-score, re-rank, or update papers from earlier runs unless the user explicitly asks to reuse or merge a previous run. If reusing material, confirm the exact directory and state what is being reused.
3. Use history as calibration evidence only. Public reviews, historical scores, anchors, and prior pairwise judgments can reveal bias or drift, but every run still needs fresh reading, fresh comparison, and fresh scoring.
4. In multi-target local-review mode, isolate target judgments. Other target papers from the same user request are not comparison baselines unless they are genuinely same-topic anchors and the user asks to compare them.
5. Treat recurring comparison papers as calibration rulers, not disposable competitors. Their relative order and score bands should remain stable across runs unless fresh paper evidence, public reviews, or a documented anchor audit justifies movement.
6. Separate **topic-nearest comparison papers** from **scale rulers**. A paper belongs in the primary comparison pool only when it is close on task, input/output contract, method family, evaluation object, or benchmark semantics. Broad famous papers such as general LLM-for-SE agent systems may be used as venue-bar scale rulers, but they must be labeled separately and must not dominate the target-local ranking or replace true same-topic neighbors.

## Multi-Target Local-Review Mode

Use this mode when the user supplies several papers as "targets", "submissions", "待审文章", or similar, and asks for review-quality evaluation.

- Start one coordinator run directory, then create one subdirectory per target such as `per_target/<target_id>/{target,pool,reviews,analysis,judgments}`.
- Spawn one independent subagent per target when subagent tooling is available. Each subagent evaluates exactly one target paper, builds or verifies that target's local pool, and writes a target-local review. If subagents are unavailable, simulate one isolated pass per target and disclose that limitation.
- Give each subagent only its target paper, the rubric, and its own local pool instructions. Do not give it the other target papers' scores or conclusions.
- For each target, build a two-layer local pool:
  - **Primary nearest-neighbor pool**: normally 4-6 papers that are genuinely same-task, same-mechanism, same benchmark family, or same evaluation object. Include accepted or published top-venue papers as positive anchors, plus rejected, withdrawn, arXiv-only, secondary-track, workshop, or otherwise unaccepted/unverified papers as boundary controls when they are topically close.
  - **Scale-ruler set**: at most 2-3 broader accepted papers used only to calibrate the venue bar or stabilize historical score bands. Reused broad LLM-SE anchors across multiple targets are scale rulers by default unless the target's topic signature makes them truly same-topic.
- Label each pool paper's status clearly: accepted main-track, accepted secondary-track, rejected, withdrawn, arXiv-only, unverified, or metadata-only. Use non-accepted papers for calibration, not as evidence that the target is weak by association.
- Label each pool paper's **pool role** clearly: `target`, `primary-nearest`, `accepted-nearest-anchor`, `boundary-control`, or `scale-ruler`. Do not hide broad scale rulers inside the primary nearest-neighbor ranking.
- Score the target, primary nearest-neighbor papers, boundary controls, and any scale rulers with the same rubric, but report scale rulers in a separate section or clearly separated rows. The target-local placement must be argued mainly against the primary nearest-neighbor pool. For SE submission judgments, include independent reviewer-style textual concerns and an AC synthesis, but do not include four-point reviewer scores by default. Mark pool-paper scores as target-local calibration scores, not global or historical truth.
- The coordinator reconciles scale after subagents finish: check review evidence, historical drift, pairwise consistency within each local pool, and **run a pool-overlap audit** before creating the cross-target summary matrix. If the same broad paper appears in more than two target pools, demote it to scale-ruler unless a concrete same-topic reason is written. The final report is organized by target paper, not by a global rank of all target papers.

## Workflow

1. Set up the isolated run directory before any search or extraction.
   - Identify the target paper paths, extracted-text files, titles, or topic supplied by the user.
   - Build a run slug from the primary target's filename stem, title, or topic, for example `Run_Less_ISSTA.pdf` to `run_less`.
   - Create a fresh run directory at the invocation cwd root, not inside the user's source-paper folder:
     ```bash
     RUN_DIR="$PWD/eval_runs/<slug>__$(date +%Y%m%d_%H%M)"
     mkdir -p "$RUN_DIR/targets" "$RUN_DIR/pool" "$RUN_DIR/reviews" "$RUN_DIR/analysis" "$RUN_DIR/judgments"
     ```
   - For multi-target local-review mode, also create one isolated workspace per target:
     ```bash
     mkdir -p "$RUN_DIR/per_target/<target_id>"/{target,pool,reviews,analysis,judgments}
     ```
   - Link local target PDFs into `targets/` when possible so source files remain untouched:
     ```bash
     ln -s "/abs/path/to/Target.pdf" "$RUN_DIR/targets/Target.pdf"
     ```
   - If the user supplies extracted text instead of PDFs, copy or link it into `analysis/text/` and record the source in `run.md`.
   - Write `run.md` with date, target source paths or topic, and an initially empty pool list. This manifest is the source of truth for what is in scope.
   - From this point on, read PDFs and write artifacts only under `$RUN_DIR`.

2. Assemble the comparison pool, unless the user supplied a complete pool.
   - Check the local evidence library before web search:
     ```bash
     python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py search "<target topic keywords>"
     ```
   - For same-topic, accepted, cached-PDF hits, reuse them by linking:
     ```bash
     python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py link <id> "$RUN_DIR/pool"
     ```
   - Use web/search/browser tools when the user has not supplied a complete pool or asks to find/gather related papers. Derive a topic signature from the target: title, problem statement, method family, task, input/output contract, evaluation object, benchmark semantics, and 3-8 keywords.
   - Prefer same-topic comparability over broad fame. Before keeping a paper in the primary pool, run this nearest-neighbor test: it should match the target on at least two of task, input/output contract, method family, evaluation object, benchmark/dataset semantics, or failure mode. If it only shares "LLM agent", "software engineering", "benchmark", or venue prestige, label it as a scale ruler instead of a primary comparison.
   - Aim for about 4-6 primary nearest-neighbor papers plus at most 2-3 scale rulers unless the user asks for a different pool size. If the primary pool is thin, say so and lower confidence rather than padding the primary pool with generic anchors.
   - Search across relevant top venues, not only the target paper's home field. Typical venues:
     - SE: ICSE, FSE/ESEC-FSE, ASE, ISSTA, TSE, TOSEM
     - ML/AI: NeurIPS, ICML, ICLR, AAAI, JMLR
     - NLP: ACL, EMNLP, NAACL, TACL
     - CV: CVPR, ICCV, ECCV, TPAMI
     - Security, systems, databases, and other fields: use the corresponding top venues
   - Apply a hard default acceptance gate for ordinary primary pools: keep accepted or published top-venue papers when enough same-topic papers exist. Exclude arXiv-only papers without confirmed venue unless the user explicitly wants them or they are close boundary controls needed for local calibration.
   - In multi-target local-review mode, or whenever the user asks for rejected/unaccepted calibration, keep labeled subsets instead of one blended pool: primary accepted nearest-neighbor anchors, primary boundary controls, and scale rulers.
   - Prefer main/research-track papers. Label or down-rank secondary tracks such as workshops, industry/in-practice tracks, Findings, short papers, demos, and posters.
   - Prefer recent papers from the last roughly 18 months. Include older baselines only when they are essential or requested, and label them as older baselines.
   - For OpenReview-hosted venues, fetch public review evidence with the bundled harvester:
     ```bash
     python3 ~/.codex/skills/paper-review-evaluator/scripts/openreview_reviews.py \
       --paper-id <paper_id> \
       --forum <openreview_forum_id> \
       --output "$RUN_DIR/reviews/<paper_id>.openreview.json"
     ```
     Record average rating, rating scale, and decision when available. Use scores only as within-venue quality signals, not cross-scale absolutes.
   - Download every kept PDF into `$RUN_DIR/pool/`; never download into the repo root or the user's source folder. For arXiv PDFs:
     ```bash
     curl -L -o "$RUN_DIR/pool/<slug>.pdf" "https://arxiv.org/pdf/<arxiv-id>"
     ```
   - If no open PDF is available, record title, venue, year, abstract, and source URL. Either score from metadata with a lower-confidence note or ask the user for the PDF.
   - Append each kept paper to `run.md` with title, venue, year, type, pool role, PDF or metadata-only status, and why it is comparable. Show this pool to the user before scoring when the pool was auto-discovered. If the pool contains recurring broad anchors, explicitly state that they are scale rulers, not nearest-neighbor evidence.
   - Add newly confirmed pool papers back to the evidence library:
     ```bash
     python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py add --id <id> --title "<title>" --pdf "$RUN_DIR/pool/<file>.pdf" --venue "<venue>" --year <year> --ptype "<type>" --accepted yes --keywords "<keywords>" --source-url "<url>"
     ```
     The library is a material and calibration evidence store, not a ranking store. Scores are recomputed fresh for each run.

3. Inventory and extract the run, scoped to `$RUN_DIR` only.
   - List PDFs with:
     ```bash
     rg --files -g '*.pdf' "$RUN_DIR"
     ```
   - Extract text with:
     ```bash
     python3 ~/.codex/skills/paper-review-evaluator/scripts/extract_pdf_reports.py "$RUN_DIR"
     ```
   - Outputs go under `$RUN_DIR/analysis/`:
     - `analysis/text/<pdf-stem>.txt`
     - `analysis/pdf_summaries.json`
     - optional page PNGs when using `--render-pages`
   - Prefer local extracted text for scoring. Browse only for pool discovery, acceptance verification, missing paper metadata, or when the user asks for external verification.

4. Read each report with a paper-value lens.
   - Prioritize method sections, training or infrastructure details, ablations, evaluation protocol, limitations, and appendices.
   - Do not over-weight raw benchmark wins, business value, deployment popularity, or marketing claims.
   - Treat internal benchmarks as useful but weaker than transparent ablations, reproducible protocols, and failure analysis.

5. Use anonymous multi-reviewer scoring by default.
   - For multi-target local-review mode, first parallelize by target: spawn one independent subagent per target paper. Within each target subagent, use the reviewer styles below when tooling and budget allow.
   - When subagent tooling is available, run multiple blind reviewers concurrently and point each reviewer only at `$RUN_DIR/analysis/text/` plus the rubric. Use available multi-agent tooling rather than leaking one reviewer's conclusions into another.
   - Recommended reviewer styles: `strict-rigor`, `innovation`, `transferable-value`, and `skeptical-meta`.
   - Give each reviewer the same fixed rubric. Let style affect scrutiny and evidence selection, not the scoring formula.
   - Ask reviewers to provide both dimension scores and pairwise comparisons for close papers. Store useful pairwise judgments as JSONL in `$RUN_DIR/judgments/pairwise_<dimension>.jsonl`.
   - For SE submission judgment, ask each reviewer pass for independent textual concerns and a confidence note. Do not ask for four-point recommendation scores unless explicitly requested. Reviewers should be independent complete reviewers rather than fixed personas. Each reviewer should evaluate the full paper, including novelty, value, rigor, presentation, fit, and risk; do not preassign one reviewer to be strict, positive, or skeptical.
   - The main Codex instance acts as area chair: collect reviews, reconcile large disagreements, and produce the final ranking with a short consensus/uncertainty note.
   - If subagents are unavailable, simulate separate blind review passes locally and explicitly disclose that real anonymous subagents were not used.

6. Score, compare, and calibrate with the fixed rubric.
   - Use `references/rubric.md` for dimension definitions, score anchors, tier labels, output format, calibration rules, and image-prompt templates.
   - Use `references/calibration.md` when public reviews, historical scores, anchors, or pairwise judgments are present.
   - Use `references/stability.md` whenever a pool contains papers that appeared in earlier runs, the user complains about ranking drift, or the task is a repeated venue/submission judgment. In that mode, lock recurring pool papers to stable score bands and place the target by pairwise insertion instead of freely re-ranking the whole pool.
   - `Total = 0.40 * Innovation + 0.40 * Intrinsic Paper Value + 0.20 * Rigor`
   - Keep `Aesthetics` as a separate reference column only. Do not include it in total unless the user changes the rule.
   - Use one decimal place for dimension scores and two decimals for totals.
   - For recurring anchors, report canonical or banded scores separately from current-run target scores. Do not let a new target's topic framing move unrelated anchors by more than the stability thresholds without an explicit drift note. Recurring broad anchors should usually appear as scale rulers, not as primary nearest-neighbor competitors.
   - Aggregate pairwise JSONL when available:
     ```bash
     python3 ~/.codex/skills/paper-review-evaluator/scripts/pairwise_rank.py "$RUN_DIR/judgments/pairwise_innovation.jsonl" --markdown
     ```
   - Before finalizing each recurring paper, run `paper_db.py reviews <id>`, `paper_db.py history <id>`, `paper_db.py drift-check <id> ...`, and when checking a group, `paper_db.py stability-report --ids <comma-separated-ids>`. Record material score changes or justified disagreements with `record-calibration`.

7. Rank and explain.
   - Provide a primary nearest-neighbor table with rank, report/model, pool role, type, Innovation, Paper Value, Rigor, Aesthetics, Total, confidence, and calibration note. If scale rulers are used, provide a separate scale-ruler table or clearly separated rows. If the user's question is about submission outcome, keep the main table 10-point based and add likely AC read plus decisive risks. Include four-point recommendation scores only when explicitly requested.
   - Mark the user's target paper rows so they stand out from the comparison pool.
   - In multi-target local-review mode, organize the report by target paper. For each target, include its topic signature, primary nearest-neighbor anchors, non-accepted/boundary controls, scale rulers if used, relative position, calibrated score, confidence, and likely review-style verdict. Include target-local scoring tables that separate primary nearest-neighbor rows from scale rulers. Include a final cross-target summary matrix only as a coordinator calibration view, not as the main ranking, and include a short pool-overlap audit.
   - Add concise per-paper rationales grounded in concrete contributions, close comparisons, public review evidence when available, and any score-drift explanation.
   - Rank only this run's target and pool papers. If the user adds another paper mid-run, add it to this run's pool and re-rank within this run.
   - Save the final table and rationales to `$RUN_DIR/REPORT.md`.
   - After finalization, record run membership and final scores in the evidence library with `record-run` and `record-score` so future runs can audit drift.

8. Optional tiering and image prompts.
   - Use five tiers from strongest to weakest: `S 夯爆`, `A 很夯`, `B 够硬`, `C 能打`, `D 偏拉`.
   - For image prompts, use the A4 Chinese infographic templates in `references/rubric.md`.

## Bundled Resources

### Rubric

Read `references/rubric.md` when scoring, ranking, tiering, or writing image-generation prompts. It contains the fixed scoring formula, score anchors, reviewer protocol, output table format, calibration rules, and prompt templates.

### Calibration

Read `references/calibration.md` when public reviews, historical scores, anchors, pairwise comparisons, or calibration notes are involved. It explains how to use review evidence and history without converting the library into a stale leaderboard.

### Stability

Read `references/stability.md` when repeated runs reuse comparison papers or when anchor ranking drift is a concern. It defines stable anchors, score bands, target insertion, drift thresholds, and the final report shape that keeps target judgment local without re-sorting the whole historical universe.

### PDF Extraction

Run:

```bash
python3 ~/.codex/skills/paper-review-evaluator/scripts/extract_pdf_reports.py "$RUN_DIR"
```

The extractor recurses through `targets/` and `pool/` while avoiding the output directory. It prefers PyMuPDF (`fitz`) and falls back to `pypdf`. If neither exists, install one with `pip install pymupdf` or `pip install pypdf`.

### Evidence Library

`scripts/paper_db.py` manages a local SQLite evidence library plus cached PDFs under this skill directory:

- `cache/papers.db`
- `cache/pdfs/<id>.pdf`

Use it to avoid re-downloading already-vetted comparison PDFs and to calibrate future runs with public reviews, historical score drift, anchors, pairwise judgments, and calibration events. Do not treat cached scores or past rankings as authoritative for a new run.

Common commands:

```bash
python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py init
python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py search "<keywords>"
python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py link <id> "$RUN_DIR/pool"
python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py add --id <id> --title "<title>" --pdf <path>
python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py get <id> --with-evidence
python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py reviews <id>
python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py history <id>
python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py drift-check <id> --innovation <n> --value <n> --rigor <n> --aesthetics <n> --total <n>
python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py stability-report --ids <id1,id2,...>
python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py record-run ...
python3 ~/.codex/skills/paper-review-evaluator/scripts/paper_db.py record-score ...
```

Use `--cache-dir` for isolated testing.

### OpenReview Reviews

Use `scripts/openreview_reviews.py` for OpenReview-hosted venues:

```bash
python3 ~/.codex/skills/paper-review-evaluator/scripts/openreview_reviews.py --paper-id <id> --forum <forum_id>
```

It stores public review and decision evidence into the evidence library.

### Pairwise Ranking

Use `scripts/pairwise_rank.py` to aggregate JSONL pairwise judgments into a transparent ranking cross-check. Pairwise ranking informs calibration; it does not replace the final area-chair synthesis.