Exp Eval

/exp-eval

Convert completed experiment results into wiki knowledge updates. Review LLM acts as an impartial judge (following cross-model-review), independently evaluating how experimental results affect the target claim. Four verdict paths: supported → claim↑ + idea validated / partially_supported → supplementary experiments / not_supported → claim↓ + idea failed / inconclusive → debug. Auto-updates claims confidence and evidence, ideas status, and graph edges.

Inputs

• experiment: slug from wiki/experiments/ (status must be completed)
• --auto (optional): automatic mode — do not pause for user confirmation before wiki updates (used when called by /research)

Outputs

• wiki/claims/{slug}.md — updated confidence, status, evidence list
• wiki/ideas/{slug}.md — updated status (validated/failed), pilot_result, failure_reason
• wiki/experiments/{slug}.md — ## Claim updates section filled in
• wiki/graph/edges.jsonl — new supports/invalidates edges added
• wiki/graph/context_brief.md — rebuilt
• wiki/graph/open_questions.md — rebuilt
• wiki/log.md — appended log entry
• VERDICT_REPORT (printed to terminal) — verdict result, wiki change summary, next step suggestions

Wiki Interaction

Reads

• wiki/experiments/{slug}.md — experiment results: outcome, key_result, metrics, full Results section
• wiki/claims/{target-claim}.md — target claim current state: status, confidence, evidence list
• wiki/ideas/{linked-idea}.md — linked idea current state
• wiki/experiments/*.md — other experiments on the same claim (aggregate assessment)
• wiki/graph/context_brief.md — global context
• .claude/skills/shared-references/cross-model-review.md — reviewer independence principle

Writes

• wiki/claims/{target-claim}.md — update status, confidence, evidence, date_updated
• wiki/ideas/{linked-idea}.md — update status, pilot_result, failure_reason, date_resolved
• wiki/experiments/{slug}.md — fill in ## Claim updates section
• wiki/graph/edges.jsonl — add supports or invalidates edges
• wiki/graph/context_brief.md — rebuild
• wiki/graph/open_questions.md — rebuild
• wiki/log.md — append operation log

Graph edges created

• supports: experiment → claim (experiment supports the claim) — verdict = supported or partially_supported
• invalidates: experiment → claim (experiment refutes the claim) — verdict = not_supported

Workflow

Precondition:

1. Confirm working directory is the wiki project root (directory containing wiki/, raw/, tools/)
2. Confirm experiment status == completed (incomplete experiments cannot be evaluated)

Step 1: Load Context

1. Read experiment page wiki/experiments/{slug}.md:

   • outcome (succeeded/failed/inconclusive)
   • key_result
   • target_claim slug
   • linked_idea slug
   • metrics and full Results section
   • hypothesis

2. Read target claim wiki/claims/{target-claim}.md:

   • Current status and confidence
   • Existing evidence list
   • Conditions and scope

3. Read linked idea wiki/ideas/{linked-idea}.md (if it exists):

   • Current status
   • Hypothesis

4. Load other experiments on the same claim:

   • Glob: wiki/experiments/*.md, filter target_claim == same claim
   • Summarize existing experiment results (for aggregate claim confidence assessment)

5. Read global context:

   • wiki/graph/context_brief.md

6. Read cross-model-review.md: confirm Review LLM independence principle

Step 2: Review LLM Verdict (Cross-Model Verdict)

Follow cross-model-review.md: do not send Claude's pre-judgment to Review LLM.

mcp__llm-review__chat:
  system: "You are an impartial scientific judge evaluating whether experimental
           results support or refute a research claim. Be rigorous and objective.
           Consider: statistical significance, effect size, experimental validity,
           potential confounds, and whether the results generalize beyond the
           specific setup tested."
  message: |
    ## Claim Under Test
    Title: {claim title}
    Statement: {claim statement from ## Statement section}
    Current status: {status}
    Current confidence: {confidence}
    Conditions: {conditions and scope}

    ## Experiment
    Title: {experiment title}
    Hypothesis: {hypothesis}
    Setup: {model, dataset, hardware, framework}
    Metrics: {metrics list}

    ## Results
    {full Results section from experiment page}

    ## Key Finding
    {key_result}

    ## Other Experiments on This Claim
    {summary of other experiments' outcomes on the same claim, if any}

    ## Your Task
    Provide your verdict:
    1. **Verdict**: One of: supported / partially_supported / not_supported / inconclusive
    2. **Confidence adjustment**: Suggest new confidence value (0.0-1.0) with reasoning
    3. **Evidence strength**: weak / moderate / strong
    4. **Key reasoning**: 2-3 sentences explaining your verdict
    5. **Concerns**: Any methodological concerns or limitations
    6. **Suggested next steps**: What would strengthen or clarify this result?

Record Review LLM's verdict.

Step 3: Claude Synthesis

1. Form Claude's independent verdict (after reading Review LLM's verdict, Claude also analyzes independently):

   • Based on experimental results, claim context, and aggregate evidence from other experiments
   • Form Claude's own verdict and confidence suggestion

2. Synthesize both verdicts (follow cross-model-review.md composing rules):

   • Both agree (same verdict): use that verdict, average the confidence, high certainty
   • Both disagree:
     • Explicitly flag the disagreement
     • Take the more conservative verdict (supported > partially_supported > not_supported)
     • Use the lower confidence value
     • Detail the disagreement reason in the report
   • Fatal findings take priority: if either party finds a methodological issue (data leakage, unfair comparison), that finding takes precedence

3. Determine final verdict: verdict + new_confidence + evidence_strength

Step 4: Update Wiki Based on Verdict

If --auto is not set: display verdict and planned changes first, wait for user confirmation.

Path A: SUPPORTED (experiment supports claim)

1. Update claim:

   • confidence: ↑ adjust to new value (typically +0.1~0.3)
   • status: adjust based on new confidence
     • confidence >= 0.7 → supported
     • confidence 0.4–0.7 → weakly_supported
   • evidence: append new entry {source: experiment-slug, type: supports, strength: strong/moderate, detail: key_result}
   • date_updated: today's date

2. Update idea (if it exists and status is in_progress/tested):

   • If all linked claims are supported/weakly_supported:
     • status: validated
     • pilot_result: key_result summary
     • date_resolved: today's date

3. Add graph edge:

   python3 tools/research_wiki.py add-edge wiki/ \
     --from "experiments/{slug}" --to "claims/{target-claim}" \
     --type supports --evidence "{key_result}"
   

4. Suggest next steps: /paper-plan or continue ablation/robustness experiments

Path B: PARTIALLY_SUPPORTED (partial support)

1. Update claim:

   • confidence: minor adjustment (+0.05~0.15)
   • evidence: append {type: supports, strength: weak, detail: ...}
   • date_updated: today's date

2. Add graph edge:

   python3 tools/research_wiki.py add-edge wiki/ \
     --from "experiments/{slug}" --to "claims/{target-claim}" \
     --type supports --evidence "Partially supported: {limitation}"
   

3. Suggest supplementary experiments:

   • Specify what evidence is missing
   • Suggest using /exp-design to design supplementary experiments
   • If Review LLM-flagged concerns are addressable by experiment, suggest concrete experiment direction

4. Idea status unchanged: keep in_progress, wait for more evidence

Path C: NOT_SUPPORTED (experiment does not support claim)

1. Update claim:

   • confidence: ↓ significantly lower (typically -0.2~0.4)
   • status: if confidence < 0.3 → challenged
   • evidence: append {type: invalidates, strength: strong/moderate, detail: ...}
   • date_updated: today's date

2. Update idea (if it exists):

   • status: failed
   • failure_reason: specific reason for failure (extracted from experiment results and Review LLM analysis)
   • date_resolved: today's date
   • Note: failure_reason is anti-repetition memory — must be written clearly, explaining why it failed

3. Add graph edge:

   python3 tools/research_wiki.py add-edge wiki/ \
     --from "experiments/{slug}" --to "claims/{target-claim}" \
     --type invalidates --evidence "{failure_reason}"
   

4. Suggest next steps:

   • Analyze the failure reason
   • Consider pivoting (new idea addressing the same gap while avoiding the known failure)
   • Suggest /ideate to generate alternatives

Path D: INCONCLUSIVE (results are uncertain)

1. Do not modify claim status/confidence: insufficient evidence to make a judgment

2. Update experiment page: outcome is already inconclusive (set by /exp-run)

3. Suggest debugging:

   • Data issue? Implementation bug? Wrong metric?
   • Too much variance? More seeds needed?
   • Experiment setup not aligned with claim?

4. Idea status unchanged: keep current status

All Paths (common steps)

1. Fill in ## Claim updates section of the experiment page:

   ## Claim updates
   - **Verdict**: {supported/partially_supported/not_supported/inconclusive}
   - **Claim**: [[{target-claim}]] confidence {old} → {new}
   - **Judge agreement**: {Claude and Review LLM agreed / disagreed on ...}
   - **Date**: YYYY-MM-DD
   

2. Update index.md (if claim status changed)

3. Rebuild derived data:

   python3 tools/research_wiki.py rebuild-context-brief wiki/
   python3 tools/research_wiki.py rebuild-open-questions wiki/
   

4. Append log:

   python3 tools/research_wiki.py log wiki/ \
     "exp-eval | {slug} → {target-claim} | verdict: {verdict} | confidence: {old}→{new}"
   

5. Print VERDICT_REPORT to terminal:

   # Verdict Report: {experiment title}
   
   ## Verdict: {SUPPORTED / PARTIALLY_SUPPORTED / NOT_SUPPORTED / INCONCLUSIVE}
   
   ## Judge Assessment
   | | Claude | Review LLM | Final |
   |---|-------|------|-------|
   | Verdict | {verdict} | {verdict} | {verdict} |
   | Confidence | {value} | {value} | {value} |
   | Evidence strength | {strength} | {strength} | {strength} |
   
   ## Key Reasoning
   {2-3 sentences from Review LLM + Claude synthesis}
   
   ## Wiki Changes
   | Entity | Field | Before | After |
   |--------|-------|--------|-------|
   | claims/{slug} | confidence | {old} | {new} |
   | claims/{slug} | status | {old} | {new} |
   | ideas/{slug} | status | {old} | {new} |
   
   ## Graph Edges Added
   - experiments/{slug} → claims/{target} (supports/invalidates)
   
   ## Concerns
   {methodological concerns from Review LLM}
   
   ## Next Steps
   - {path-specific suggestions}
   
   ## Wiki Growth
   | Metric | Before | After | Delta |
   |--------|--------|-------|-------|
   | Claims updated | — | — | {N} |
   | Edges | {before} | {after} | +{delta} |
   | Maturity | {level} | {level} | {unchanged/upgraded} |
   (Data from comparing `python3 tools/research_wiki.py maturity wiki/ --json` calls at the start of Step 1 and end of Step 4.)
   

Constraints

• Only process completed experiments: experiments with status != completed are refused; prompt user to use /exp-run first
• Reviewer independence: strictly follow cross-model-review.md — do not send Claude's pre-judgment to Review LLM
• Confidence range 0.0–1.0: updated confidence must not exceed this range
• failure_reason must be specific: the not_supported path's failure_reason cannot be vague (e.g. "experiment failed") — must state the concrete reason
• Do not delete claims: even when not_supported, only challenge or lower confidence; do not delete the claim page. In extreme cases (multiple consistent refutations, confidence → 0), set status to deprecated rather than deleting
• Graph edges via tools/research_wiki.py: do not manually edit edges.jsonl
• Conservative principle: when Claude and Review LLM verdicts disagree, use the more conservative verdict
• Idea status advances only forward: proposed → in_progress → tested → validated/failed, irreversible
• Assess claim using all experiments: consider not just the current experiment but also other experiments on the same claim

Error Handling

• Experiment not found: prompt user to check slug, list candidates in wiki/experiments/ with status=completed
• Experiment not completed: report status, suggest running /exp-run {slug} or /exp-run {slug} --check
• Target claim does not exist: create new claim page (status: proposed, confidence: 0.3), note "auto-created by exp-eval"
• Linked idea does not exist: skip idea update, only update claim, note in report
• Review LLM unavailable: fall back to Claude single-model verdict, note "single-model verdict, cross-model verification unavailable" in report, suggest user confirm later
• Claim was modified by another experiment: read the latest state, make adjustments based on current confidence (do not overwrite other experiments' contributions)
• Results data missing: if the experiment page's Results section is empty, prompt user to run /exp-run {slug} --check first

Dependencies

Tools（via Bash）

• python3 tools/research_wiki.py add-edge wiki/ ... — add graph edge
• python3 tools/research_wiki.py rebuild-context-brief wiki/ — rebuild query_pack
• python3 tools/research_wiki.py rebuild-open-questions wiki/ — rebuild gap_map
• python3 tools/research_wiki.py log wiki/ "<message>" — append log

MCP Servers

• mcp__llm-review__chat — Step 2 Review LLM independent verdict

Claude Code Native

• Read — read wiki pages
• Glob — find other experiments on the same claim
• Edit — update wiki pages

Shared References

• .claude/skills/shared-references/cross-model-review.md — Review LLM independence principle (required reading)

Called by

• /research Stage 4 (verdict and iteration stage)
• User directly