Read Code

Verify Code Claims

Check each claim tagged evidence_type: "code" from read-txt against the actual codebase.

Workflow

Verify Code:
- [ ] Step 1: Load claims
- [ ] Step 2: Read code index
- [ ] Step 3: Verify each code claim
- [ ] Step 4: Save report

Step 1: Load Claims

Read {paper_dir}/reports/check_claim.json

Filter to claims where evidence_type includes "code". These are the claims to verify.

Each claim's id, source, quote, claim, evidence_type fields should be copied as-is into the output — do not modify them.

Step 2: Read Code Index

Read {paper_dir}/metadata/code/index.json

This is the tree-sitter index from prepare-paper. Use it to locate relevant files, symbols, and config entries.

Step 3: Verify Each Code Claim

For each code claim, find the relevant code and judge whether it matches.

How to find code: Use Grep, Glob, and the code index. For hyperparameters, check config files first. For architecture claims, check class definitions and __init__ / forward methods.

For each claim, produce:

{
  "id": 7,
  "source": "experiments",
  "quote": "model/encoder.py:42 — num_layers=12, hidden_size=768\n\nWe use a 12-layer transformer with embedding dimension 768",
  "claim": "Model architecture is a 12-layer, 768-dim transformer",
  "evidence_type": ["code"],
  "status": "pass",
  "reason": ""
}

Field definitions (paper triple → fields):

Statuses:

When status is warning or error, reason should explain the discrepancy and suggest should propose a concrete fix (e.g., "Update the paper text to match the code, or change the code to AdamW to match the paper claim").

Step 4: Save Report

Output: {paper_dir}/reports/check_code.json

{
  "summary": {
    "total": 8,
    "pass": 5,
    "warning": 2,
    "error": 1
  },
  "results": [
    {
      "id": 9,
      "source": "experiments",
      "quote": "configs/train.yaml:14 — optimizer: Adam\n\nWe train all models using the AdamW optimizer",
      "claim": "Training optimizer is AdamW",
      "evidence_type": ["code"],
      "status": "warning",
      "reason": "Paper says AdamW but configs/train.yaml line 14 sets optimizer: Adam",
      "suggest": "Update the paper text to AdamW or change the config to match the paper claim."
    },
    {
      "id": 10,
      "source": "method",
      "quote": "src/trainer/fgt_prediction_trainer.py:0 — no margin-loss entry point present\n\nWe use a custom margin-loss training objective",
      "claim": "Margin loss is the training objective",
      "evidence_type": ["code"],
      "status": "error",
      "reason": "No margin-loss entry point in src/trainer/; the released default uses binary cross-entropy.",
      "suggest": "Add the margin-loss entry point to the released code, or remove the claim from the paper."
    }
  ]
}

Coverage Rule

Every claim with code in evidence_type MUST appear in results. No claim may be silently skipped. Valid statuses are pass / warning / error. If a claim cannot be verified at all from the codebase (no relevant file found, repo too incomplete), use status: "warning" with quote like path/to/expected/file.ext:0 — file not in repo so the reviewer can still see what was being checked.

Tips

• Check config files (.json, .yaml) first — most hyperparameters live there, not in Python code
• Map paper terms to code terms (e.g., "learning rate" → lr, "layers" → depth)
• If the index has config_entries, search those before grepping full files
• A paper saying "Adam" when code uses "AdamW" is a warning, not an error