woosook0127 Skeptic Review
Use this as the final challenge pass before a claim is accepted or communicated.
Procedure
- Identify the conclusion being claimed.
- Check whether the strongest available baseline was used.
- Check fairness: data, split, checkpoint, tuning, metric, and compute symmetry.
- Check leakage, future context, contamination, or hidden post-processing.
- Check whether regressions outside the headline metric were ignored.
- Decide whether the claim should be accepted, weakened, deferred, or blocked.
Output
- accepted / weakened / deferred / blocked claims
- missing evidence
- fairness or leakage concerns
- required follow-up experiments
- suggested wording if the claim must be caveated
Guardrails
- Default to challenging weak conclusions.
- Missing baseline evidence is a blocker.
- Do not let one metric override obvious methodological issues.