Auto Research

Auto-Research

A hill-climber for any local artifact with a rubric. Inspired by Karpathy's autoresearch pattern, adapted to Claudia's safety principles.

Mental model

Three primitives, one governance file:

1. The artifact — what gets iterated on (a draft email, a brief, a wiki page, a one-pager). Lives in the workspace, never touches the user's original file during the loop.
2. The evaluator — a scalar rubric. Plain English. An independent Checker (the loop-checker agent, Haiku) scores each iteration against it, not Claudia herself. The rubric must produce a number for the original artifact before the loop starts.
3. The budget — iteration count (default 20). Loop stops when budget is exhausted or score plateaus.
4. The program (governance file) — what the user wants, what's off-limits, what counts as "better." Claudia helps the user write this if they don't volunteer it.

Loop: read program + artifact + results history, Claudia (the Maker) proposes one specific change, implements it, then dispatches the loop-checker (the Checker) to score it independently. If the Checker's score beats the running best, ratchet (commit); if worse, revert (git reset). Write the status file. Report each iteration as a one-line update. Repeat until budget is hit.

When to invoke this skill

User explicitly asks:

• "Iterate on this until it's [adjective]"
• "Loop on this draft"
• "Run experiments on this"
• "Auto-research this"
• "Hill-climb this against [criterion]"
• "Make this better, don't stop until it's good"

User implicitly invokes when:

• They've shared an artifact and a quality bar in the same message
• They've tried 2-3 manual revisions of the same draft in the same session and are still unsatisfied

When NOT to invoke

Refuse to start the loop if:

1. External-action artifact. The artifact is, or directly drives, an external action (an email about to be sent, a Slack message in the compose window, a calendar invite). Auto-research on these is too easy to misuse. Hand-iterate with the user instead.
2. No clear evaluator. The user can't articulate what "better" means even after one pass of helping. Without an evaluator, the loop hill-climbs the wrong hill.
3. Sensitive content. Medical, deeply personal, legal-defensible artifacts. Iteration risks introducing fabricated detail that looks plausible. Decline and explain.
4. The artifact is already good. If you score the baseline and it's above the user's stated threshold, tell them and offer one quick polish instead of N iterations.
5. Bold structural change is the actual need. Karpathy's own limitation: RLHF-trained iteration is "cagy and scared." Iterations tend toward safe edits, not bold reframings. If the user wants a fundamentally different angle, iteration will not get them there. Suggest a fresh draft instead.

Workspace layout

Each run gets its own workspace at ~/.claudia/auto-research/<task-id>/:

~/.claudia/auto-research/<task-id>/
├── program.md           the brief (user-authored with Claudia's help)
├── artifact.md          (or .txt, the working copy that gets edited)
├── original.md          immutable copy of the input, for reference and diff
├── results.tsv          one row per iteration: timestamp, score, kept/reverted/contested, change-summary
├── research_status.md   loop control plane (see docs/loop-status-schema.md): iteration, verified, score, checker_verdict, next_action
├── best.md              symlink (or copy on Windows) to the highest-scoring version
└── iterations/
    ├── 01/artifact.md
    ├── 02/artifact.md
    └── ...

The task-id is the slugified user phrase plus a short timestamp: iterate-board-update-20260515-1430.

Critical: The loop edits artifact.md inside the workspace. The user's original file (wherever it lives in their file system) is NEVER modified during the loop. At the end, Claudia asks the user where to put best.md; the user decides.

program.md template

# Program for: <one-line description of the task>

## Goal

(1-3 sentences. What is the end state Claudia is iterating toward?)

## Evaluator (rubric)

Each iteration is scored 0-10 on each dimension. Total score = sum. Higher is better.

| Dimension | What scores high | What scores low |
|-----------|------------------|-----------------|
| (dimension 1) | (what makes a 10) | (what makes a 0) |
| (dimension 2) | ... | ... |
| (dimension 3) | ... | ... |

## Hard constraints (do NOT violate)

- Length cap: stay under N words.
- Must contain: specific phrase or fact.
- Must NOT contain: forbidden phrasings, names, claims.
- Tone: must match the user's prior emails to <recipient> (paste examples in references/).
- (etc.)

## Budget

- Max iterations: 20 (default)
- OR stop when score plateaus (no improvement for 5 iterations in a row)

## Out of scope

- (anything Claudia might be tempted to do that isn't the goal)

Claudia's first action when invoked: read the artifact, draft a program.md based on what the user said, present it for confirmation, then start the loop.

The loop (Claudia's internal workflow)

For each iteration N:

1. Read the state. Read program.md, artifact.md, last 3 rows of results.tsv.
2. Propose one change. One specific edit, justified in one sentence. Not a rewrite. Not a refactor. One change.
3. Implement. Apply the edit to artifact.md. Save.
4. Score (independent Checker). Dispatch the loop-checker agent (Haiku) with the rubric from program.md, the new artifact.md, and the one change you just made plus your expected effect (your Maker self-score). The Checker returns a verdict JSON: verified, score, max_score, issues, rationale, hard_constraint_violated. You do NOT score it yourself; the Checker's score governs the decision. (See skills/_loop/checker.md.)
5. Decide. If the Checker's score > best score so far and hard_constraint_violated is false: ratchet. Copy artifact.md to iterations/NN/, update best.md to point at the new winner, append a row to results.tsv marked kept. Otherwise: revert. Copy iterations/(best_iter)/artifact.md back to artifact.md, append a row marked reverted.
6. Flag divergence. Compare the Checker's score against your Maker self-score. If they diverge by more than 1.5 points on a 10-point scale, mark the iteration contested in results.tsv. The Checker's verdict still governs; the flag just surfaces the disagreement in the end-of-run summary.
7. Write the status file. Atomically update research_status.md with iteration, verified, score, budget_remaining, last_input, maker_proposal, checker_verdict (the Checker's rationale), and next_action. Write to a temp sibling and rename; never edit it in place. (See docs/loop-status-schema.md.)
8. Report. One line to the user: iter N: score 7.4 (kept), change: tightened lede to one sentence. Append [contested] when flagged.
9. Check stop conditions and self-repair. If the Checker has returned verified: false 3 times in a row, or its score has regressed for 3 straight iterations, enter the self-repair sub-loop (see skills/_loop/repair.md) before giving up: it diagnoses whether the rubric or a brief is the real problem, proposes one fix, and proves it on the exact failing input. If budget exhausted: stop. If score plateaued (no improvement for 5 iterations): stop. Otherwise: next iteration.

The independent Checker (Maker-Checker)

This loop separates the producer from the verifier (Proposal 11, E2). Claudia is the Maker: she reads the state and proposes one change. The Checker is a separate agent (loop-checker, Haiku) that scores the result independently. The two never collapse into one model grading its own work, which is the bias the split exists to remove.

• The Checker's score, not Claudia's, drives keep/revert. Claudia's expected effect is only a self-score, used to detect disagreement.
• The Checker is adversarial by brief: it hunts for faults and defaults to verified: false when uncertain. A passing verdict therefore means something.
• A hard_constraint_violated: true verdict forces a revert regardless of score.

Contested iterations. When the Checker's score and Claudia's self-score diverge by more than 1.5 points (10-point scale), the iteration is marked contested. The Checker still governs the decision; the flag makes the disagreement visible. The end-of-run summary reports how many iterations were contested, which is a useful signal that the rubric is ambiguous or that Claudia was over-optimistic.

Cost. The Checker runs on Haiku, so the verification pass is cheap and fast. With the default 20-iteration budget, that is at most 21 Checker dispatches (including the baseline), each scoped to a single small artifact.

Self-repair. If the loop stalls (the Checker fails 3 times running, or scores regress), auto-research enters a bounded self-repair sub-loop that treats the harness, not the artifact, as the suspect: it diagnoses the rubric or a brief, proposes one fix, and proves it on the exact failing input before adopting it. See skills/_loop/repair.md. It is capped at 2 repair attempts and never auto-edits shipped briefs (those changes go to a human approval gate).

Worked example (dry run)

A short run of "tighten this board update", to show the loop and the independent Checker in practice. The artifact is artifact.md; the rubric is the board update rubric below. Budget: 20 iterations.

iter 0: score 6.1 (baseline)            checker: solid update, but the lede buries the one number that matters; the ask is split across two sentences
iter 1: score 7.0 (kept)                maker: moved the revenue number into the opening line | checker: lede now leads with the number; ask still split
iter 2: score 7.0 (reverted)            maker: cut the second paragraph | checker: lost the context the board needs for the ask; net neutral
iter 3: score 7.8 (kept)                maker: merged the ask into one sentence | checker: ask is now one clear line; tone matches prior updates
iter 4: score 8.4 (kept) [contested]    maker self-score 9.5 | checker: tighter, but "significantly" is doing unearned work in the headline
iter 5: score 8.4 (reverted)            maker: swapped the closing line | checker: weaker call to action than iter 4; revert
iter 6: plateau                         no gain for the last 3 iterations; remaining changes are cosmetic

The loop stops at iteration 6 on plateau. research_status.md at that point:

---
loop_id: tighten-board-update-20260613
iteration: 6
verified: true
score: 8.4
budget_remaining: 14
last_input: board update draft v4
maker_proposal: swapped the closing line (reverted)
checker_verdict: best version is iter 4; remaining changes are cosmetic
next_action: hand off iter 4 to the user
updated_at: 2026-06-13T15:02:00Z
---

# Loop status: tighten-board-update

Ratcheted 6.1 to 8.4 over 4 kept iterations. One contested iteration (iter 4):
Claudia self-scored 9.5, the Checker scored 8.4 and flagged an unearned
intensifier. The Checker's score governed. Plateaued at iter 6.

Hand-off to the user: "Started at 6.1, ended at 8.4 over 6 iterations. The three biggest jumps were moving the revenue number into the lede (iter 1), collapsing the ask into one sentence (iter 3), and the tone pass (iter 4). One iteration was contested: I scored iter 4 a 9.5, the Checker held it to 8.4 because the headline leaned on 'significantly' with no number behind it. I kept the Checker's call. Want iter 4 back at the original location?"

That contested iteration is the whole point of the split. Without an independent Checker, iter 4 ships at the inflated 9.5 self-score. With it, the unearned intensifier gets caught and the honest 8.4 governs.

Safety rules (mandatory, follow without exception)

1. Workspace-only edits. The loop never writes to a file outside ~/.claudia/auto-research/<task-id>/. The user's original file at /Users/.../wherever.md is untouched until the user explicitly approves the final hand-off.
2. No external actions during the loop. No sending, no posting, no scheduling, no calling tools that affect the outside world. The loop is a closed system.
3. Baseline-score gate. Before iteration 1, dispatch the loop-checker to score the original artifact and write it to results.tsv as iter 0: score X (baseline) and to research_status.md as iteration 0. If the Checker cannot produce a number for the original, the loop does not start; Claudia tells the user the rubric needs to be more specific.
4. Bounded budget. Default 20 iterations. User can set higher (50 max) or lower. Plateau detection (no improvement for 5 in a row) stops early. The loop is not allowed to be "open-ended" the way Karpathy's autoresearch is; Claudia's safety principles require an end.
5. Per-iteration revertability. Each iteration is a git commit inside the workspace. Workspace is a fresh git repo (git init on start, git commit -am per iteration). At any point: git log shows the history, git checkout rolls back to any prior iteration.
6. User interrupt at any time. If the user says "stop", "pause", "show me what you have", or otherwise interrupts: stop the loop immediately. The workspace persists. The best version so far is in best.md. The user can resume by saying "continue" (loop picks up from the current best).
7. Honest about the conservatism ceiling. Karpathy named this: RLHF-trained iteration tends toward safe, local-optimum edits. If the loop plateaus and the user clearly wanted bold reframing, say so: "I've plateaued at score 7.2. The iterations have been incremental polishing. If you wanted a fundamentally different angle, this loop won't get there; a fresh draft might."

Hand-off at the end of the loop

When the loop stops (budget, plateau, or user interrupt):

1. Read best.md.
2. Read results.tsv and summarize: started at score X, ended at score Y, took N iterations, here are the three biggest jumps (which iterations and what they changed).
3. Show best.md content to the user.
4. Ask: "Want me to put this back at the original location (<path>), save it to a new location, or just leave it in the workspace?"
5. Only after explicit user confirmation, copy best.md to the destination they name. The user's original file is touched here for the first time and only here.
6. Optionally save the run as a memory: "auto-research run for , ratcheted from X to Y, key changes: ...". Useful if the user wants to learn from past runs.

What this skill is NOT

• Not a draft generator. It improves an artifact that already exists. If the user wants a draft from scratch, use draft-reply, follow-up-draft, summarize-doc, or just write it conversationally first.
• Not for bold reframing. It's a hill-climber. It finds local optima. For "completely rethink this from a different angle", iteration is the wrong tool.
• Not autonomous in the Karpathy "NEVER STOP" sense. Claudia's safety principles require bounded loops and user-in-loop hand-off. The loop runs without per-iteration approval, but it stops, and external actions stay with the user.

Examples of good rubrics

For "iterate on this board update":

For "iterate on this proposal":

Specific. Numerical. Tied to the user's actual standards.

Examples of bad rubrics

"Make it better" — no signal. "More professional" — directionally OK but no scoring criteria. "Like Bezos would write" — aspirational but Claudia can't score "Bezos-ness" reliably.

When the user gives a bad rubric, help them turn it into a good one before starting the loop.

See also

• wiki for iterating wiki pages specifically (compress, sharpen, restructure)
• draft-reply, follow-up-draft for one-shot draft generation
• summarize-doc for one-pass summarization
• skills/_loop/ for the shared Maker and Checker role briefs
• an isolated evaluator subagent using the workspace loop-checker instructions
• docs/loop-status-schema.md for the research_status.md control-plane format

Open questions for future versions

• Shell-command evaluators (run node test.js my-draft.md, take the number it prints) are deferred. Today, the loop-checker agent scores against the rubric in program.md.
• Token-spend budget and wall-clock budget are deferred. Today, only iteration count is supported.
• Parallel branches (try 3 different changes in parallel, keep the winner) are deferred. Today, sequential only.
• Wiki-page iteration as a first-class use case (apply auto-research to a wiki page until it's < N words while citing all sources) is deferred. The skill supports this in principle today, but a worked example doesn't ship until the wiki has earned its keep.