scientist-experiment-runner
Convert AI Scientist's "advance the experiment" into a Copilot-native workflow: Copilot reads ideas, edits code, runs experiments, and summarizes results. NEVER call any workspace-custom model pipeline.
Execution model
This is a Copilot-native model task. Copilot makes research decisions in-session; the terminal runs only non-model commands.
Workflow
- Read the ideas JSON; lock the
idea_idx or specific idea in use.
- Identify the code surface, config surface, and run commands.
- Make the minimum necessary code or config change.
- Run experiments / tests / evaluation via the terminal.
- Read result files, logs, and metrics.
- Continue analysis and the next-round decision in-session.
Long-run guidance — when an experiment exceeds the longest Bash timeout (10 min)
NEVER poll with repeated Bash(timeout=600000) calls to "just wait." Each call burns 10 minutes of context and forces a fresh decision turn on identical state.
Pick the tool by estimated duration:
| Estimated time |
Tool |
Why |
| < 10 min |
Bash synchronous |
Fits in one call |
| 10 min – 2 h, command exits when done |
Bash(run_in_background=true) |
Harness auto-notifies on exit; zero polling |
| Hours, you need progress events |
Monitor with tail -f log | grep -E --line-buffered "elapsed_steps=|Traceback|Error|FAILED|Killed|OOM" |
One notification per event; ends when the grep filter exits |
| Hours, no event stream |
ScheduleWakeup(delaySeconds=N, prompt="continue checking experiment X") |
Re-enter cold after delay; cheap |
| Recurring polls (e.g. every 30 min) |
CronCreate |
Standard cron, in-memory |
Verification before declaring completion
Before claiming the experiment finished successfully, you MUST produce one of:
- the exact metric value with the exact log line it came from,
- the file path of the produced artifact + a confirming
ls / Read output,
- or an explicit "could not verify — here is what I have so far."
A turn that ends with "the experiment finished successfully" without one of the above is a failure mode. A "background command exited 0" notification alone is not verification — the script could have crashed silently or produced empty logs.
Input
load_ideas or ideas-JSON pathidea_idx or target idea name- Relevant code directories, training scripts, configs, run commands
- Expected output directory or existing experiment directory
Output
- Code diff
- Commands actually run
- Key metrics, log summary, artifact paths
- Suggested next round
Operating principles
- Copilot owns experiment strategy and result interpretation; the terminal owns code execution.
- Advance one minimum experimental slice per round; get it running before expanding.
- If an experiment directory already exists, use
inspect_experiment (or read logs directly) before blindly editing.
- If the user only wants plotting / writeup / review, do NOT auto-expand into the full experiment chain.
Forbidden
- NEVER call any workspace-custom model pipeline.
- NEVER initiate model calls from workspace code on your own.
Risk boundaries
- If runtime conditions are insufficient, prepare the plan and code only; never pretend the experiment ran.
- If the experiment is very expensive, confirm the run budget and expected artifacts with the user first.
1---2name: scientist-experiment-runner3description: Use when the user asks to '开始实验', '推进实验', 'run experiment', 'advance the experiment', 'turn an idea into experiment', or wants an idea JSON converted into concrete code edits and runs directly in Copilot. Copilot-native — Copilot writes the changes and reads the results in-session; the terminal only executes non-model commands. Do NOT use for plotting (scientist-plotting) or writeup (scientist-writeup).4---56# scientist-experiment-runner78Convert AI Scientist's "advance the experiment" into a Copilot-native workflow: Copilot reads ideas, edits code, runs experiments, and summarizes results. NEVER call any workspace-custom model pipeline.910## Execution model1112This is a **Copilot-native model task**. Copilot makes research decisions in-session; the terminal runs only non-model commands.1314## Workflow15161. Read the ideas JSON; lock the `idea_idx` or specific idea in use.172. Identify the code surface, config surface, and run commands.183. Make the minimum necessary code or config change.194. Run experiments / tests / evaluation via the terminal.205. Read result files, logs, and metrics.216. Continue analysis and the next-round decision in-session.2223## Long-run guidance — when an experiment exceeds the longest Bash timeout (10 min)2425**NEVER** poll with repeated `Bash(timeout=600000)` calls to "just wait." Each call burns 10 minutes of context and forces a fresh decision turn on identical state.2627Pick the tool by estimated duration:2829| Estimated time | Tool | Why |30|---|---|---|31| < 10 min | `Bash` synchronous | Fits in one call |32| 10 min – 2 h, command exits when done | `Bash(run_in_background=true)` | Harness auto-notifies on exit; zero polling |33| Hours, you need progress events | `Monitor` with `tail -f log \| grep -E --line-buffered "elapsed_steps=\|Traceback\|Error\|FAILED\|Killed\|OOM"` | One notification per event; ends when the grep filter exits |34| Hours, no event stream | `ScheduleWakeup(delaySeconds=N, prompt="continue checking experiment X")` | Re-enter cold after delay; cheap |35| Recurring polls (e.g. every 30 min) | `CronCreate` | Standard cron, in-memory |3637## Verification before declaring completion3839**Before claiming the experiment finished successfully, you MUST produce one of:**40- the exact metric value with the exact log line it came from,41- the file path of the produced artifact + a confirming `ls` / `Read` output,42- or an explicit "could not verify — here is what I have so far."4344A turn that ends with "the experiment finished successfully" without one of the above is a failure mode. A "background command exited 0" notification alone is not verification — the script could have crashed silently or produced empty logs.4546## Input4748- `load_ideas` or ideas-JSON path49- `idea_idx` or target idea name50- Relevant code directories, training scripts, configs, run commands51- Expected output directory or existing experiment directory5253## Output5455- Code diff56- Commands actually run57- Key metrics, log summary, artifact paths58- Suggested next round5960## Operating principles61621. Copilot owns experiment strategy and result interpretation; the terminal owns code execution.632. Advance one minimum experimental slice per round; get it running before expanding.643. If an experiment directory already exists, use `inspect_experiment` (or read logs directly) before blindly editing.654. If the user only wants plotting / writeup / review, do NOT auto-expand into the full experiment chain.6667## Forbidden6869- NEVER call any workspace-custom model pipeline.70- NEVER initiate model calls from workspace code on your own.7172## Risk boundaries7374- If runtime conditions are insufficient, prepare the plan and code only; never pretend the experiment ran.75- If the experiment is very expensive, confirm the run budget and expected artifacts with the user first.
ldm2060