Research Proposal

Research Proposal Loop

The artifact is a research proposal; the feedback signal is ScholarEval (literature-grounded Soundness + Contribution) turned into a 0-100 grade by a Judge against the fixed rubric.md. Each iteration evaluates → grades → revises until the grade clears <pass_threshold> or the budget runs out.

North star. Clearing the threshold is the stopping condition, not the goal. The goal is the strongest, most novel, genuinely publishable version the proposal can honestly become — every revision should ask "does this make the work more significant and more novel?", not just "does this patch a flaw?". This is grounded ambition: the lift comes from better-justified methods, a sharper-but-defensible novelty claim, and stronger baselines, all backed by real retrieved evidence. Overclaiming lowers the grade (evidence gate + Contribution axis); it never raises it.

The cast (all in this folder):

• roles/ScholarEval.md — the two-module literature-grounded evaluator; emits scholareval.json.
• roles/Judge.md — grades the feedback → 0-100 + ranked fixes; emits verdict.json (decides pass).
• roles/Reviser.md — rewrites the proposal to address the fixes (guards the research intent).
• rubrics/rubric.md — the fixed grading rubric (the Judge never edits it).
• schemas/scholareval.schema.json, schemas/verdict.schema.json — the two validated outputs.

Spawn-or-degrade. On Claude Code, spawn ScholarEval / Reviser as real Agent subagents; otherwise adopt each role inline. You are the orchestrator and the Judge.

When to use

Use when a written proposal exists and the user wants it pushed past a quality bar with literature-grounded critique. Default: run the full evaluate→grade→revise loop below. Escape hatch: if the user only wants the critique (no rewriting), run one ScholarEval + Judge pass and stop. Not for writing a proposal from a blank page, and not for a standalone literature survey.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm every value, <intent>, and the live/degraded literature tier before creating any other files.

Evaluation depth dial (<eval_scale> caps per iteration — methods · dimensions examined · queries each · papers read full-text):

Literature toolchain. Paper search goes through the sibling literature-search skill — resolve <lit_skill_dir> (it installs as a sibling, e.g. ~/.claude/skills/literature-search/), <lit_py> = python3, and <lit> = <lit_skill_dir>/tools/lit_search.py; append --cache-dir <sandbox_root>/literature/.cache after a subcommand to reuse the cache. Confirm <lit> --help works at setup; if the skill is absent, tell the user and either install it (copy the repo's loops/literature-search folder into ~/.claude/skills/) or degrade all retrieval to WebSearch/WebFetch (no ranked snippets or citation-graph expansion). The keyless S2 + arXiv core (search/snippet/cite/fulltext) needs no setup; a free S2_API_KEY makes snippet+cite (ScholarEval's two defining moves) reliable.

API key (optional, never block). The literature-search skill owns the key convention: run <lit> keys --init, then have the user fill the printed keys.env themselves and never paste secrets into chat. Re-run <lit> keys to confirm presence and record the tier in loop.run.yaml (literature_tiers, presence only). A missing key just degrades to the keyless pool → WebSearch.

Initialise the sandbox once bindings are confirmed:

<sandbox_root>/
├── loop.run.yaml        ← resolved bindings + <intent> + grade_weights + literature_tiers
├── ledger.tsv           ← header only (see Ledger)
├── literature/.cache/   ← lit_search on-disk cache
└── iter1/proposal.md    ← the input proposal (the baseline)

The loop

<N> starts at 1; iteration 1 evaluates the unmodified proposal (the baseline grade — no revision before it). Re-evaluate fresh every iteration: the grade comes only from a new ScholarEval pass on the revised proposal, never carried over.

Copy this checklist and tick items off:

• Evaluate — run roles/ScholarEval.md (spawn-or-degrade) on iter<N>/proposal.md with the <eval_scale> caps and <lit>; it writes iter<N>/scholareval.json (validates against schemas/scholareval.schema.json).
• Grade — as the Judge (roles/Judge.md) apply rubrics/rubric.md to scholareval.json: evidence gate → 0-5 per axis → weighted grade → hard gates → pass + ranked fixes; write iter<N>/verdict.json (validates against schemas/verdict.schema.json).
• Log — append one ledger.tsv row (see Ledger).
• Stop check — verdict.pass == true, or N == <budget>, or grade flat for <patience> iterations → stop (see Stops).
• Revise — run roles/Reviser.md (spawn-or-degrade) with verdict.json, scholareval.json, <intent>, <lit>, and a small revision search budget (≈<eval_scale> searches + a couple of reads); it applies one focused fix batch and writes iter<N+1>/proposal.md + iter<N+1>/revision_notes.md.
• N = N + 1 and repeat.

Every cited snippet must come from a real retrieval that iteration — never fabricated. When a lit tool returns {"error","fallback"}, fall back to WebSearch/WebFetch; never invent a paper.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in free text:

iter	grade	pass	soundness	contribution	evidence_quality	top_fix	revision_summary
1	58.0	no	3	2	4	add MM-GBSA re-scoring	baseline (no revision)
2	71.0	no	4	3	4	add head-to-head vs [C2] pipeline	added re-scoring + scoped affinity claim
3	82.0	yes	4	4	4	-	reframed contribution around integration + new benchmark

The matching scholareval.json and verdict.json for each iteration live in iter<N>/. Report the best-grade iteration when stopping on budget/plateau, not necessarily the last. Leave ledger.tsv, iter*/, and literature/ untracked.

Constraints

• Never fabricate citations or snippets — every evidence entry comes from a real <lit>/WebFetch result retrieved that iteration, verbatim; the evidence gate exists to catch fabrication.
• The rubric is fixed — the Judge never edits rubrics/rubric.md, so the passing threshold stays meaningful across iterations.
• Protect <intent> — the Reviser may strengthen but never replace the core research question or headline contribution, and never gut a central method just to lift the grade.
• Grade through evidence, not prose — eloquence earns nothing; grounded support and surviving novelty earn the score.
• One focused revision batch per iteration, so each grade move is attributable.
• No installs — the sibling literature-search skill is stdlib-only; never print or commit API keys (keys.env stays gitignored at the project root). The sandbox is self-contained — no ../ escapes.

Stops

The loop stops on the first of:

• Pass — verdict.pass == true. Report the final proposal (iter<N>/proposal.md), its grade, and the grade trajectory.
• Budget — N == <budget>. Report the best-grade iteration as the deliverable.
• Plateau — grade hasn't improved for <patience> consecutive iterations. Report the best iteration and the standing priority-1 fixes.

Always end with the deliverable proposal path, its grade and pass/fail, the grade trajectory from ledger.tsv, and — if it did not pass — the standing blockers between this proposal and the bar.