Run Notebook On Colab

Run Notebook on Colab

Executes a local Jupyter notebook on a Colab GPU runtime via SSH + papermill so the agent can see errors and fix them directly without manual copy-paste. File transfer is scp only — no git round-trip.

Inputs

If the user gives a short name, resolve it by searching the repo's notebooks/ directory.

References (load when relevant)

• kb/context/colab/colab-compatibility.md — papermill compatibility checklist (drive mount guards, magics, git-sync conflicts)
• kb/context/colab/colab-sync-protocol.md — scp-only file sync rule and "no git inside notebook" policy
• kb/context/colab/gpu-review-patterns.md — GPU anti-patterns to flag during preflight (Critical only by default)

Prerequisites (per-repo)

• cloudflared installed locally (brew install cloudflared)
• SSH key at ~/.ssh/colab_key matching the public key installed by the bootstrap notebook
• Repo contains notebooks/ad_hoc/colab_ssh_bootstrap.ipynb and scripts/connect_colab.sh (or equivalent)

Steps

Phase 1 — Prepare the notebook (delegate to subagent)

Dispatch a general-purpose subagent (or experiment-code-change when within an iterative experiment) to:

1. Read the notebook and check it against kb/context/colab/colab-compatibility.md.
2. Apply execution-safe fixes (drive mount guard, replace magics with os.makedirs / subprocess.run, gate any git reset --hard cells behind SKIP_GIT_REPO_SYNC).
3. Return a one-line summary of what changed.

Phase 2 — Connect to Colab

Resolve the repo path via fetch-repo-path if not already known.

Automated (preferred): run the repo's scripts/trigger_colab_bootstrap.py to launch Chrome with a saved session, connect a GPU runtime, and relay the hostname back via ntfy.

Manual fallback: open the bootstrap notebook in Colab, Run All, and have the user paste the *.trycloudflare.com hostname.

Verify connectivity:

ssh <SSH_OPTS> root@<HOSTNAME> \
  "echo OK && nvidia-smi --query-gpu=name --format=csv,noheader"

Phase 3 — Execute (delegate to subagent)

Dispatch a general-purpose subagent (or experiment-runtime when within an iterative experiment) to:

1. scp the notebook to <REMOTE_REPO_ROOT>/<NOTEBOOK_PATH> on Colab.

2. Launch detached — never run papermill in the foreground of an ssh session (a dropped tunnel or local tool timeout would SIGHUP a long training run):

   ssh <SSH_OPTS> root@<HOSTNAME> \
     "cd <REMOTE_REPO_ROOT> && nohup papermill <NOTEBOOK_PATH> <OUT_NB> --log-output \
      > run_r<round>_a<attempt>.log 2>&1 & echo \$! > run_r<round>_a<attempt>.pid"
   

3. Monitor with a per-job time budget (see "Headless-safe execution" below): poll the run log with short-lived ssh calls at interval max(1 min, budget/10); the run is done when the pid is gone or the papermill exit marker appears in the log.

4. On non-zero exit: read traceback, dispatch a code-change subagent to fix the failing cell, re-scp, re-run. Up to 3 retries; otherwise escalate.

5. On success: extract metrics / training curves from the papermill output log.

The lead never runs scp/ssh/papermill commands directly.

Headless-safe execution (mandatory)

Rules that keep unattended runs from stalling on permission prompts or killing long trainings:

• Job lifetime is decoupled from connection lifetime. Training always launches via nohup ... & with a pid file and log file (Phase 3 step 2). Local ssh timeouts bound only the connection, never the remote job.
• Per-job time budget, not fixed timeouts. Derive time_budget_minutes from the planner's estimated_runtime_class / critic's runtime_budget when inside an experiment loop, or from the user/notebook for standalone long trainings. Poll interval scales with budget: max(1 min, budget/10).
• Progress-based hang detection — never kill on elapsed time alone. Hung = run log size/mtime unchanged for max(10 min, budget/3). A progressing job is NEVER killed, even over budget: log an over_budget flag, continue, and mark the result's comparability. Remote pkill -f papermill is allowed ONLY on confirmed no-progress hang or explicit user abort, and ONLY as a remote command (ssh <SSH_OPTS> root@<HOSTNAME> 'pkill -f papermill') — never a local kill.
• No confirmation-gated local commands (rm/rmdir/unlink/trash, mv, kill/killall/pkill, sudo, git push/reset/clean/restore): they block forever in unattended runs. Fresh unique filenames per round+attempt (<nb>_out_r<round>_a<attempt>.*); never rm/mv prior outputs; place results with cp or by scp-ing directly to the final path.
• Bootstrap cleanup: the Chrome instance from trigger_colab_bootstrap.py is left running or must self-terminate; never pkill it from the agent.

SSH/SCP options (subagent applies)

-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
-o IdentityFile=~/.ssh/colab_key \
-o ConnectTimeout=5 -o ServerAliveInterval=5 -o ServerAliveCountMax=2 \
-o ProxyCommand="cloudflared access ssh --hostname <HOSTNAME>"

Connection-level timeouts only — they bound the ssh session, never the remote job (which is detached via nohup).

Paths

<project_name> defaults to the repo directory name; resolve with fetch-repo-path and override only if the user specifies one.

Output

• On success: notebook output path, parsed metrics, runtime in minutes.
• On failure after 3 retries: classified failure (per kb/context/experiments/experiment-contracts.md failure classes if within an experiment loop), error trace, suggested next step.