QinghongLin Verify Theory
Check each claim tagged evidence_type: "theoretical" by reading the paper's arguments and verifying them step by step.
Workflow
- [ ] Step 1: Load claims
- [ ] Step 2: Verify each argument
- [ ] Step 3: Save report
Step 1: Load Claims
Read {paper_dir}/reports/check_claim.json. Filter to claims where evidence_type includes "theoretical".
Each claim's id, source, quote, claim, evidence_type fields should be copied as-is into the output — do not modify them.
Read the full paper text from {paper_dir}/metadata/{arxiv_id}/mathpix/{arxiv_id}.md.
Step 2: Verify Each Argument
For each theoretical claim, locate the relevant reasoning in the paper. Check:
- Correctness — does each step follow from the previous one? (algebraic, logical, or conceptual)
- Assumptions — are any implicit assumptions missing or unjustified?
- Edge cases — does the result hold at boundary conditions?
- Notation consistency — are symbols used consistently throughout?
For each claim, produce:
{
"id": 3,
"source": "method",
"quote": "Eqn. 4 → 6, Section 3.1\n\nBy substituting Eqn. 4 into Eqn. 5, we obtain the closed-form loss in Eqn. 6",
"claim": "The soft-label loss simplifies to a closed-form KL divergence",
"evidence_type": ["theoretical"],
"status": "pass",
"reason": ""
}
Field definitions (paper triple → fields):
| Field | When | Purpose |
|---|---|---|
quote |
always | The Where evidence. Start with where in the paper the argument lives (equation range, section, theorem), then a blank line, then the original sentence under scrutiny. |
reason |
warning/error only | The Why. What issue was found (e.g. "implicit assumption that distributions share support"). For pass you may include a one-line note about the verification work performed (e.g. "re-derived substitution from Eqn. 4 to 6"). |
suggest |
warning/error only | The How. One-sentence concrete fix the author can apply. |
Statuses:
| Status | Meaning |
|---|---|
pass |
Argument is correct |
warning |
Correct but has implicit assumptions, ambiguity, or missing edge-case discussion |
error |
Error found — wrong sign, dropped term, invalid logical step |
When status is warning or error, reason should explain the problem precisely and suggest should propose a concrete fix.
Step 3: Save Report
Output: {paper_dir}/reports/check_theory.json
{
"summary": {
"total": 3,
"pass": 2,
"warning": 1,
"error": 0
},
"results": [
{
"id": 3,
"source": "method",
"quote": "Eqn. 4 → 6, Section 3.1\n\nBy substituting Eqn. 4 into Eqn. 5, we obtain the closed-form loss in Eqn. 6",
"claim": "The soft-label loss simplifies to a closed-form KL divergence",
"evidence_type": ["theoretical"],
"status": "warning",
"reason": "Re-derived substitution from Eqn. 4 to 6 — derivation assumes teacher and student distributions share the same support, but this is not stated.",
"suggest": "State the shared-support assumption explicitly before Eqn. 4."
}
]
}
Coverage Rule
Every claim with theoretical in evidence_type MUST appear in results. No claim may be silently skipped. If a claim's theory is trivial or self-evident, still include it with status: "pass" and a brief note.
Tips
- Reproduce each derivation from scratch — do not just skim the steps
- For math: pay attention to summation indices and what gets absorbed vs cancelled
- For non-math arguments: check if the logic actually supports the conclusion
- A missing qualifier is a
warning, not anerror