Paper-Audit Skill Design Document

Paper-Audit Skill Design Document

Date: 2026-02-23 Status: Approved Author: Feasibility analysis & design by collaborative brainstorming session

1. Overview

1.1 Problem Statement

Academic researchers need a unified, local tool that audits papers across multiple formats (LaTeX, Typst, PDF) and languages (Chinese, English) before submission. Existing solutions are either:

• Commercial SaaS (Paperpal, Trinka, Writefull) — cloud-only, no offline, no source-file awareness
• Anti-plagiarism focused (PaperPass, PaperFace) — Chinese ecosystem focuses on plagiarism/AI-trace reduction, not quality audit
• Single-format (existing Claude Code skills) — English-only or single-format
• AI/ML-domain only (PaperReview.ai) — no LaTeX/Typst awareness, no Chinese support

1.2 Solution

Create a paper-audit skill for Claude Code that provides:

• 3 audit modes: pre-submission self-check, peer review simulation, quality gate
• 3 input formats: LaTeX (.tex), Typst (.typ), PDF (.pdf with dual extraction modes)
• 2 languages: English and Chinese (auto-detected)
• Structured Markdown report with 4-dimensional scoring (Quality, Clarity, Significance, Originality)

1.3 Design Principles

• 70-80% code reuse from existing sibling skills (latex-paper-en, latex-thesis-zh, typst-paper)
• KISS: Monolithic orchestrator with mode-to-checks dict (no plugin system, no YAML config)
• YAGNI: Only build what's needed now; extensibility via simple dict modification
• DRY: Import existing check scripts directly, zero duplication

2. Industry Landscape Research

2.1 Commercial Tools

2.2 Academic/Research Tools

2.3 Chinese Ecosystem

2.4 Existing Claude Code Skills

2.5 Competitive Advantage

No existing tool provides: PDF + LaTeX + Typst + Chinese + English + 3-mode audit + local/offline + structured scoring in a single skill.

3. Architecture

3.1 File Structure

academic-writing-skills/paper-audit/
├── SKILL.md                              # Skill definition (3 modes, triggers, modules)
├── scripts/
│   ├── audit.py                          # Main orchestrator (entry point)
│   ├── pdf_parser.py                     # PdfParser class (basic + enhanced modes)
│   ├── detect_language.py                # Auto language detection (CJK ratio)
│   ├── report_generator.py              # Markdown report builder with scoring
│   └── parsers.py                        # Re-exports DocumentParser, LatexParser, TypstParser
└── resources/
    └── references/
        ├── REVIEW_CRITERIA.md            # Unified 4-dimensional review criteria
        ├── CHECKLIST.md                  # Consolidated pre-submission checklists
        └── AUDIT_GUIDE.md               # Audit modes & interpretation guide

3.2 Component Responsibilities

3.3 Approach Rationale

Three architectures were evaluated:

1. Monolithic Orchestrator (CHOSEN) — Single audit.py entry point with mode-to-checks dict
2. Plugin Registry — Auto-discovered check plugins with CheckPlugin interface
3. Config-Driven Pipeline — YAML-defined check pipeline

Why Monolithic: With ~10 stable check modules, a plugin system is over-engineering (YAGNI). The monolithic orchestrator matches existing codebase patterns, is easy to test, and can be refactored to plugins later if needed.

4. Three Audit Modes

4.1 Mode: self-check (Pre-submission Self-Check)

Target user: Researcher checking their own paper.

Checks:

SELF_CHECK = ["format", "grammar", "logic", "sentences", "deai", "bib", "figures"]
# + "consistency", "gbt7714" if language == "zh"

Output: Structured Markdown report with executive summary, per-dimension scores (1-6 scale), issue list sorted by severity, improvement suggestions, and pre-submission checklist results.

4.2 Mode: review (Peer Review Simulation)

Target user: Researcher wanting simulated conference-style peer review.

Checks: Same as self-check + LLM-driven analysis:

REVIEW = SELF_CHECK + ["reviewer_strengths", "reviewer_weaknesses", "reviewer_questions"]

Output: Conference-style review with summary, strengths, weaknesses, questions for authors, missing references, overall score with confidence, and accept/reject recommendation.

4.3 Mode: gate (Quality Gate)

Target user: Advisor/editor enforcing minimum quality.

Checks (fast, mandatory only):

GATE = ["format", "bib", "figures", "checklist"]

Output: Pass/fail gate report with per-item checkmarks, blocking issues, and binary PASS/FAIL verdict.

4.4 CLI Interface

# Self-check (default mode)
python audit.py paper.tex
python audit.py paper.typ
python audit.py paper.pdf --pdf-mode enhanced

# Specific modes
python audit.py paper.tex --mode review
python audit.py paper.pdf --mode gate --pdf-mode basic

# Venue-specific checklist
python audit.py paper.tex --mode self-check --venue neurips

# Force language
python audit.py paper.tex --lang zh

5. PDF Parser Design

5.1 Class Design

class PdfParser(DocumentParser):
    """PDF document parser implementing DocumentParser interface."""

    def __init__(self, mode: str = "basic"):
        """mode: 'basic' (pymupdf) or 'enhanced' (pymupdf4llm)"""
        self.mode = mode

    def split_sections(self, content: str) -> dict[str, tuple[int, int]]:
        """Basic: font-size heuristics. Enhanced: Markdown header detection."""

    def extract_visible_text(self, line: str) -> str:
        """Return text as-is (no markup in PDF-extracted text)."""

    def clean_text(self, content: str, keep_structure: bool = False) -> str:
        """Remove page numbers, headers/footers, ligatures."""

    def get_comment_prefix(self) -> str:
        return ">"  # Markdown blockquote for PDF audit comments

5.2 Extraction Modes

5.3 Factory Function Update

def get_parser(file_path: str, pdf_mode: str = "basic") -> DocumentParser:
    ext = Path(file_path).suffix.lower()
    if ext == ".tex":
        return LatexParser()
    elif ext == ".typ":
        return TypstParser()
    elif ext == ".pdf":
        return PdfParser(mode=pdf_mode)
    else:
        raise ValueError(f"Unsupported format: {ext}")

5.4 Known Limitations

• Math formulas: Lost in PDF extraction; audit will flag "unable to verify math" and recommend source file
• Multi-column layouts: Basic mode may interleave columns; enhanced mode handles most cases
• Scanned PDFs: Not supported (would require OCR); clear error message provided

6. Scoring & Report Engine

6.1 Dimension Mapping

Each automated check maps to one or more review dimensions:

DIMENSION_MAP = {
    "format":      ["clarity"],
    "grammar":     ["clarity"],
    "logic":       ["quality", "significance"],
    "sentences":   ["clarity"],
    "deai":        ["clarity", "originality"],
    "bib":         ["quality"],
    "figures":     ["clarity"],
    "consistency": ["clarity"],
    "gbt7714":     ["quality"],
    "checklist":   ["quality", "clarity", "significance", "originality"],
}

6.2 Scoring Algorithm

For each dimension D in {Quality, Clarity, Significance, Originality}:
    1. Collect all issues mapped to D
    2. Base score = 6.0
    3. Deductions: Critical = -1.5, Major = -0.75, Minor = -0.25
    4. Floor at 1.0
    5. Overall = weighted average:
       Quality: 30%, Clarity: 30%, Significance: 20%, Originality: 20%
    6. Map to NeurIPS scale:
       5.5-6.0 → Strong Accept, 4.5-5.4 → Accept,
       3.5-4.4 → Borderline Accept, 2.5-3.4 → Borderline Reject,
       1.5-2.4 → Reject, 1.0-1.4 → Strong Reject

6.3 Report Template (Markdown)

# Paper Audit Report

**File**: paper.tex | **Language**: English | **Mode**: self-check
**Generated**: 2026-02-23 | **Venue**: NeurIPS

## Executive Summary
[1-2 sentence summary of findings]

## Scores
| Dimension     | Score | Issues (C/M/m) | Key Finding              |
|---------------|-------|-----------------|--------------------------|
| Quality       | 4.5   | 0/2/3           | Missing ablation study   |
| Clarity       | 3.75  | 1/3/5           | Long sentences in Methods|
| Significance  | 5.0   | 0/0/1           | Strong contribution      |
| Originality   | 4.25  | 0/1/2           | Some AI-trace detected   |
| **Overall**   | **4.3**|                 | **Borderline Accept**    |

## Issues (sorted by severity)
### Critical
- [FORMAT] (Line 42) [Severity: Critical] [Priority: P0]: Missing figure reference...

### Major
- [GRAMMAR] (Line 87) [Severity: Major] [Priority: P1]: Subject-verb disagreement...

### Minor
- [SENTENCES] (Line 123) [Severity: Minor] [Priority: P2]: Sentence exceeds 60 words...

## Improvement Suggestions
### Abstract
- [Specific actionable suggestion]

### Introduction
- [Specific actionable suggestion]

## Pre-Submission Checklist
- [x] Paper compiles without errors
- [x] All figures referenced in text
- [ ] TODO placeholder found on line 256
- [ ] Anonymous submission: author name detected

7. Execution Flow

Input: paper.{tex,typ,pdf} + --mode + --pdf-mode + --venue + --lang
                |
                v
    +--- Format Detection ---------------+
    |  .tex -> LatexParser               |
    |  .typ -> TypstParser               |
    |  .pdf -> PdfParser(mode)           |
    +----------------+-------------------+
                     v
    +--- Language Detection -------------+
    |  --lang override OR                |
    |  CJK ratio > 0.3 -> "zh"          |
    |  else -> "en"                      |
    +----------------+-------------------+
                     v
    +--- Mode -> Check Selection --------+
    |  MODE_CHECKS[mode]                 |
    |  + language-specific extras        |
    +----------------+-------------------+
                     v
    +--- Sequential Check Execution -----+
    |  For each check in selected:       |
    |    import from sibling skill       |
    |    run(parsed_content, lang)       |
    |    collect CheckResult             |
    +----------------+-------------------+
                     v
    +--- Report Generation --------------+
    |  aggregate issues by dimension     |
    |  calculate scores                  |
    |  render Markdown report            |
    |  output to stdout / save file      |
    +------------------------------------+

8. Cross-Skill Import Strategy

import sys
from pathlib import Path

SKILLS_ROOT = Path(__file__).parent.parent.parent
sys.path.insert(0, str(SKILLS_ROOT / "latex-paper-en" / "scripts"))
sys.path.insert(0, str(SKILLS_ROOT / "latex-thesis-zh" / "scripts"))
sys.path.insert(0, str(SKILLS_ROOT / "typst-paper" / "scripts"))

# Import existing check modules
from check_format import check_format
from analyze_grammar import analyze_grammar
from analyze_logic import analyze_logic
from analyze_sentences import analyze_sentences
from deai_check import check_deai
from verify_bib import verify_bib
from check_figures import check_figures
# Chinese-specific
from check_consistency import check_consistency

9. Dependencies

New Dependencies

pymupdf >= 1.24.0        # PDF basic mode extraction
pymupdf4llm >= 0.0.17    # PDF enhanced mode (Markdown output)

Existing Dependencies (no changes)

Pillow                    # Image analysis (already in project)
Python >= 3.9             # Already required

10. Testing Strategy

11. Feasibility Assessment

Risks & Mitigations

12. Implementation Order

1. parsers.py — Thin re-export layer (verify imports work)
2. detect_language.py — Simple CJK ratio detection (~40 lines)
3. pdf_parser.py — PdfParser implementing DocumentParser interface (~150 lines)
4. audit.py — Main orchestrator with 3 modes (~300 lines)
5. report_generator.py — Scoring engine + Markdown rendering (~200 lines)
6. SKILL.md — Skill definition with triggers, modules, output protocol (~200 lines)
7. Reference docs — REVIEW_CRITERIA.md, CHECKLIST.md, AUDIT_GUIDE.md
8. Tests — Unit + integration tests for all new components
9. Documentation — Update VitePress docs with paper-audit skill page

13. Conclusion

The paper-audit skill is highly feasible with a clear competitive advantage:

• No existing tool provides PDF + LaTeX + Typst + Chinese + English + 3-mode audit in one skill
• 70-80% of core logic already exists in well-tested sibling skills
• The DocumentParser abstraction makes adding PDF support clean
• The REVIEWER_PERSPECTIVE.md already defines the scoring framework
• Main new work: orchestration layer (300 lines) + PDF parser (150 lines) + report generator (~200 lines)

The biggest challenge is PDF extraction quality for academic documents, mitigated by dual-mode extraction and recommending source files for full accuracy.