scientist-experiment-runner
Convert AI Scientist's "advance the experiment" into a Copilot-native workflow: Copilot reads ideas, edits code, runs experiments, and summarizes results. NEVER call any workspace-custom model pipeline.
Execution model
This is a Copilot-native model task. Copilot makes research decisions in-session; the terminal runs only non-model commands.
Workflow
- Read the ideas JSON; lock the
idea_idxor specific idea in use. - Identify the code surface, config surface, and run commands.
- Make the minimum necessary code or config change.
- Run experiments / tests / evaluation via the terminal.
- Read result files, logs, and metrics.
- Continue analysis and the next-round decision in-session.
Long-run guidance — when an experiment exceeds the longest Bash timeout (10 min)
NEVER poll with repeated Bash(timeout=600000) calls to "just wait." Each call burns 10 minutes of context and forces a fresh decision turn on identical state.
Pick the tool by estimated duration:
| Estimated time | Tool | Why |
|---|---|---|
| < 10 min | Bash synchronous |
Fits in one call |
| 10 min – 2 h, command exits when done | Bash(run_in_background=true) |
Harness auto-notifies on exit; zero polling |
| Hours, you need progress events | Monitor with tail -f log | grep -E --line-buffered "elapsed_steps=|Traceback|Error|FAILED|Killed|OOM" |
One notification per event; ends when the grep filter exits |
| Hours, no event stream | ScheduleWakeup(delaySeconds=N, prompt="continue checking experiment X") |
Re-enter cold after delay; cheap |
| Recurring polls (e.g. every 30 min) | CronCreate |
Standard cron, in-memory |
Verification before declaring completion
Before claiming the experiment finished successfully, you MUST produce one of:
- the exact metric value with the exact log line it came from,
- the file path of the produced artifact + a confirming
ls/Readoutput, - or an explicit "could not verify — here is what I have so far."
A turn that ends with "the experiment finished successfully" without one of the above is a failure mode. A "background command exited 0" notification alone is not verification — the script could have crashed silently or produced empty logs.
Input
load_ideasor ideas-JSON pathidea_idxor target idea name- Relevant code directories, training scripts, configs, run commands
- Expected output directory or existing experiment directory
Output
- Code diff
- Commands actually run
- Key metrics, log summary, artifact paths
- Suggested next round
Operating principles
- Copilot owns experiment strategy and result interpretation; the terminal owns code execution.
- Advance one minimum experimental slice per round; get it running before expanding.
- If an experiment directory already exists, use
inspect_experiment(or read logs directly) before blindly editing. - If the user only wants plotting / writeup / review, do NOT auto-expand into the full experiment chain.
Forbidden
- NEVER call any workspace-custom model pipeline.
- NEVER initiate model calls from workspace code on your own.
Risk boundaries
- If runtime conditions are insufficient, prepare the plan and code only; never pretend the experiment ran.
- If the experiment is very expensive, confirm the run budget and expected artifacts with the user first.