# Gcloud Vm > Google Cloud VMs for nub - Skill: `nubjs/gcloud-vm` (Agent Skill) - Install (CLI): `npx skillmds@latest add nubjs/gcloud-vm` - Raw SKILL.md: https://api.skillmd.com/api/skills/nubjs/gcloud-vm/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: nubjs (https://skillmd.com/u/nubjs) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/nubjs/gcloud-vm --- # Google Cloud VMs for nub Any time a task needs a real OS the Mac host + Docker can't give you — real Linux-kernel Landlock/seccomp/netns enforcement, real Windows AppContainer/MSVC, a clean high-RAM build box, a genuinely-clean first-run environment — spin up or start a VM. Project `pullfrog`; the existing boxes live in **`us-central1-a`**. The gcloud default zone is `us-west1-a`, so **always pass `--zone us-central1-a` explicitly**. ## ⛔⛔ THESE ARE GOOGLE CLOUD BOXES — THE AWS FREEZE DOES NOT TOUCH THEM Project `pullfrog`. **An instruction that "the AWS account is frozen" says NOTHING about these**, and reading it as covering them cost a whole session of 20-minute CI round-trips while two admin-capable Windows VMs sat idle and billing. **The VMs are almost never the blocker — check before you route around them.** ## The standing instances (`gcloud compute instances list` is the truth; this table rots) | Name | OS | Purpose | |---|---|---| | `nub-linux` | Ubuntu 24.04 LTS, e2-standard-4 | Linux-kernel enforcement (Landlock/seccomp/bwrap/netns); carries a path-bound AppArmor `bwrap-userns` profile + `apparmor_restrict_unprivileged_userns=1` reproducing a locked-down 24.04 host. **The one box worth preserving** — that profile is hand-built, so it auto-STOPs rather than auto-deletes. It carries an 8h `maxRunDuration` from before that flag was dropped for new boxes; that cap CANNOT be raised while it runs, so a job here longer than 8h gets stopped mid-flight. `STOP` keeps the disk, so restart and resume rather than losing it | **`nub-corpus-linux`, `nub-win2` and `nub-win3` were DELETED 2026-09-01** as part of the billing cleanup below, along with `nub-win`, `nub-devloop` and `nub-win4` before them. Do not go looking for them, and do not treat their absence as an outage: recreate what you need from the recipes below, at the spec the job actually needs. The Windows boxes in particular were confirmed admin (`IsInRole(Administrator)=True`) with `logman`/`wpr`/`tracerpt` present, so a fresh Server 2022 box gives you full ETW kernel tracing the same way. They are usually TERMINATED to save billing. Start what you need, and DELETE it when done — deleting is what stops the disk charge. ## ⛔ DO NOT PUT `--max-run-duration` ON A BOX YOU CREATE (reversed 2026-09-04) **A run-duration budget cannot be changed while the instance is running, and that is disqualifying.** Measured 2026-09-04, on a box holding a 1h25m acceptance sweep that needed more time than its budget allowed: ``` ERROR: Max run duration cannot be changed while the instance is running. ``` `gcloud compute instances update` does not accept the flag at all; `set-scheduling` accepts it and the API refuses. `--termination-time` is refused the same way, and *clearing* the duration counts as changing it. The only way out is to stop the instance — which kills the work the box exists for. So the flag turns a recoverable "this is taking longer than I thought" into an unrecoverable one, and it does it at the exact moment the box is most valuable. Use idle auto-stop instead (below). It targets the actual waste — a box sitting doing nothing — rather than capping useful work, and it can be tuned from inside the guest at any time. **The cost facts that motivated the budget are still true, and still worth knowing.** Measured 2026-09-01: five boxes alive, four idle between 10 hours and 26 days, carrying **$94/mo of disk that bills while TERMINATED**. `nub-win3` alone was ~$205/mo of runtime — 322 running hours in 30 days, of which **$119 was Windows licensing rather than compute** — on top of $34/mo for its 200 GB `pd-ssd`. - **`DELETE` is what stops the standing charge.** The boot disk is `autoDelete` by default, so deleting the VM takes it too; merely stopping one leaves a 200 GB `pd-ssd` billing ~$34/mo forever. Delete anything named `-tmp`, `-probe` or a builder the moment it is done. - **Windows is where the money is.** The licence is [$0.046 per vCPU-hour](https://docs.cloud.google.com/compute/docs/instances/windows/ms-licensing-faq), so an `e2-standard-8` Windows box costs **more in licence than in compute** ($0.368/h vs $0.268/h). Size Windows boxes by what the job needs, and prefer `--boot-disk-type pd-balanced` ($0.10/GB/mo) over `pd-ssd` ($0.17) unless the disk is genuinely the bottleneck. - **Verify a cleanup by listing EVERY instance and reading the list** — never by grepping for the name you expected. A cleanup check that matched `nub-win` passed while its own box, named `wingrants-…`, ran on for hours. Automated short-lived builders under `scripts/remote-build.ts` are a separate case and still carry the flag: they run ~45m unattended, nobody is there to extend anything, and the launcher dying is a real leak path. ### Idle auto-stop — the mechanism that replaces a run-duration cap This is the cost mechanism, now that the run-duration cap is gone: it targets idleness directly rather than capping useful work. Ship it in the same `create` call so no box exists without it, and the box powers itself off once nothing is happening. GCE reports a guest `poweroff` as `TERMINATED`, so the disk survives and a `start` brings it back. ```sh cat > /tmp/idle-stop.sh <<'SH' #!/usr/bin/env bash # power off once nothing has happened for $IDLE_MIN minutes set -euo pipefail IDLE_MIN=30 STAMP=/var/tmp/nub-idle-since busy() { who | grep -q . && return 0 # an SSH session is attached pgrep -x 'cargo|rustc|nub' >/dev/null && return 0 # a build is running detached awk '{exit ($1 > 0.2) ? 0 : 1}' /proc/loadavg # 1-min load says real work } if busy; then rm -f "$STAMP"; exit 0; fi now=$(date +%s); [ -f "$STAMP" ] || printf '%s' "$now" > "$STAMP" if [ $(( (now - $(cat "$STAMP")) / 60 )) -ge "$IDLE_MIN" ]; then logger -t nub-idle "idle ${IDLE_MIN}m, powering off" systemctl poweroff fi SH cat > /tmp/idle-startup.sh <<'SH' #!/bin/bash install -m 0755 /dev/stdin /usr/local/sbin/nub-idle-stop <<'INNER' __IDLE_SCRIPT__ INNER cat > /etc/systemd/system/nub-idle.service <<'UNIT' [Service] Type=oneshot ExecStart=/usr/local/sbin/nub-idle-stop UNIT cat > /etc/systemd/system/nub-idle.timer <<'UNIT' [Timer] OnBootSec=10min OnUnitActiveSec=5min [Install] WantedBy=timers.target UNIT systemctl daemon-reload && systemctl enable --now nub-idle.timer SH python3 - <<'PY' p='/tmp/idle-startup.sh'; s=open(p).read() open(p,'w').write(s.replace('__IDLE_SCRIPT__', open('/tmp/idle-stop.sh').read().rstrip())) PY # then add to the create call: # --metadata-from-file startup-script=/tmp/idle-startup.sh ``` - **`who` is the load-bearing check**, because an agent driving a long `cargo build` over SSH holds a session the whole time — the timer must not shoot the box out from under it. The `pgrep` arm covers a build detached from any session, and the load arm covers everything else. - **Tune `IDLE_MIN` up, never down, for a box you interact with by hand.** Thirty minutes is chosen so a poweroff never lands mid-thought. There is no run-duration backstop behind it any more, so a wedged timer means a box that bills until someone deletes it — check `instances list` when you finish with a box. - **Windows has no equivalent here.** Nothing reaps a Windows box for you, so DELETE it explicitly when the job is done and confirm with a full `instances list`. ## ⛔ LIFECYCLE HYGIENE — DIAGNOSE AT FIRST DETECTION, THEN DELETE **These boxes are THROWAWAY.** The maintainer's standing instruction: kill anything unreachable rather than nursing it, and create a fresh one at whatever spec the job needs. A borked box that keeps running is pure burn. **The rule that actually matters: the moment you find a box unreachable, DIAGNOSE IT THEN — not later.** Once it is deleted, or once weeks pass, every trace of what went wrong is gone and you are left guessing. Read the serial console (`gcloud compute instances get-serial-port-output --zone us-central1-a | tail -40`) BEFORE deleting, and write down what you find. ```sh gcloud compute instances delete --zone us-central1-a --quiet # a stopped box still bills its disk ``` **⛔ A FULL DISK IS INDISTINGUISHABLE FROM A BROKEN BOX, AND IT IS THE MOST COMMON CAUSE HERE.** Measured 2026-08-05 on `nub-linux`: `/dev/root 193G 193G 0 100%`, of which **166 GB was abandoned `~/.cache/nub-search-*` harness fixture roots** (8 of them) — no runaway process, just temp dirs nothing ever swept. The symptoms all look like a dead machine: `scp: write remote "x": Failure`, zero-byte outputs from commands that "succeeded", a transferred file that reads as `cannot execute binary file`. **Check `df -h /` FIRST on any box behaving strangely**; `rm -rf ~/.cache/nub-search-*` took it from 100% to 14% and fully restored the box, no recreation needed. **Re-test reachability rather than trusting a remembered "unreachable".** The external IP changes on every start, so a stale IP reads exactly like a dead box. ```sh gcloud compute instances list # names + STATUS + current external IP gcloud compute instances start nub-linux --zone us-central1-a # ~30-60s; Windows boot is slower gcloud compute instances stop nub-linux --zone us-central1-a # when you finish — they bill while RUNNING ``` ## SSH — user `nub`, key `~/.ssh/nub-vm`, and the IP is DYNAMIC - **User is `nub`** — not `ubuntu`/`nubuser`/`colinmcd94` (all fail `Permission denied (publickey)`). - **Key is `~/.ssh/nub-vm`.** - **The external IP changes on every start** (no static IP reserved). Never hardcode it or trust a memory/skill note: ```sh IP=$(gcloud compute instances describe nub-win --zone us-central1-a \ --format='value(networkInterfaces[0].accessConfigs[0].natIP)') ssh -i ~/.ssh/nub-vm -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ -o ConnectTimeout=15 nub@"$IP" "echo ok" ``` - **Re-resolve the IP on every reconnect, not once per session.** A VM can be stopped out from under you mid-session and return on a different IP, so an address resolved at the top of a long run can be dead an hour later. Treat a sudden `Connection refused`/timeout on a previously-working box as "it moved" and re-read the IP before reading the serial console. - **Always reachability-guard a VM dispatch** (`ConnectTimeout`, `timeout`), and after a fresh start retry with backoff for a few minutes — sshd (Windows especially) isn't up the instant STATUS flips RUNNING. Never let a sub-agent hang on a VM: check reachability, act, report, exit. ## Creating a NEW VM on demand For an isolated/ephemeral box (a clean first-run env, a second Linux box so you don't contend with `nub-linux`, a specific image), create one — don't wait for permission: ```sh # Linux — size ≥16 GB if it will COMPILE nub (see the OOM gotcha); e2-standard-4 is the proven size. # The last two lines are NOT optional: they are what stop a forgotten box billing for weeks. gcloud compute instances create nub-linux-tmp \ --zone us-central1-a --project pullfrog \ --machine-type e2-standard-4 \ --image-family ubuntu-2404-lts-amd64 --image-project ubuntu-os-cloud \ --boot-disk-size 30GB \ --metadata-from-file startup-script=/tmp/idle-startup.sh # Wire the `nub` SSH key so you can reach it the same way (Linux): gcloud compute instances add-metadata nub-linux-tmp --zone us-central1-a \ --metadata ssh-keys="nub:$(cat ~/.ssh/nub-vm.pub)" ``` ### Windows: `ssh-keys` metadata alone does NOT get you in — provision it yourself Measured end-to-end 2026-08-04 while building `nub-win3`, after `nub-win` and `nub-win2` both proved unreachable. **A Windows box created the "obvious" way is not SSH-able**, and each of the three failures below looks like a different problem, so use the failure MODE to tell them apart rather than guessing: | symptom | meaning | | --- | --- | | `Operation timed out` | OpenSSH Server is **not installed** — Windows Server 2022 does not ship it enabled, and `enable-windows-ssh=TRUE` does **not** install it | | `Connection refused` | sshd installed but not started yet (still booting) | | `Permission denied (publickey…)` | sshd is **up and listening**; only the key is missing | Do it in ONE creation, with a startup script that installs sshd *and* provisions the key itself. The guest agent did not provision keys here across two resets, so **do not depend on it**: ```sh cat > /tmp/win-ssh.ps1 < C:\\nub\\build.log 2>&1\r\necho EXIT=%%ERRORLEVEL%% >> C:\\nub\\build.log\r\n' > /tmp/dobuild.bat scp -i ~/.ssh/nub-vm /tmp/dobuild.bat nub@"$IP":C:/nub/dobuild.bat # /RU SYSTEM is fine HERE because a build only needs a toolchain — never reuse this line for a measurement. ssh -i ~/.ssh/nub-vm nub@"$IP" 'cmd /c "schtasks /Create /TN nubbuild /TR C:\nub\dobuild.bat /SC ONCE /ST 00:00 /RL HIGHEST /RU SYSTEM /F & schtasks /Run /TN nubbuild"' # then POLL: (Get-Process cargo,rustc).Count, plus the tail of build.log ``` ⛔⛔ **USE THIS FOR BUILDS ONLY — NEVER FOR A MEASUREMENT. A scheduled task runs as `SYSTEM`, and `SYSTEM` IS NOT A NORMAL USER.** For a build that costs a `PATH` fix (the rustup row above). For anything that MEASURES OS-enforced behaviour it silently changes the answer, because `SYSTEM` holds privileges an ordinary account does not — `SeCreateSymbolicLinkPrivilege` above all — and `os.homedir()` becomes `C:\Windows\system32\config\systemprofile`. Measured 2026-08-04, and the verdict did not give it away: a build-jail package whose real failure is a refused symlink was re-measured under a SYSTEM task, produced **exactly the expected grant** with a clean control, and was reported as a validated reproduction. The artifact refuted it — the per-cell log was 1,476 bytes containing only a catalog warning naming the `systemprofile` path, and the symlink error appeared in **3 of 54** logs where the real-user CI run shows **51 of 54**. Running as `SYSTEM` had bypassed the very mechanism under test while landing on the same answer by another route. - **`whoami` is the cheap guard.** Print it as the first line of any script whose result you will believe, and assert on it: `nub-win3\nub` good, `nt authority\system` void. - **Get the right context by running through SSH itself, not a task** — an SSH session already runs as `nub`. Wrap the call in a harness-tracked background command (`run_in_background`) rather than a scheduled task: the session stays alive for hours, so the ~10-minute foreground cap that pushed you toward `schtasks` never applies. Detaching *within* the session (`Start-Process`, `nohup`-alikes) still dies to the job object — the point is that the SSH call itself is the long-lived process. - **`schtasks /RU /RP ` also works but needs a password you probably do not have** — the `nub` account is created with a throwaway GUID password, and `net user nub ` fails `The user name or password is incorrect` from a non-elevated SSH token. Prefer the SSH route. - **Call the TOOLCHAIN binary, not the rustup shim** (`.rustup\toolchains\stable-x86_64-pc-windows-msvc\bin\cargo.exe`) so it does not matter which user rustup was configured for. - **The scheduled task is what makes failure VISIBLE** — it redirects to a log and records `EXIT=`, where the detached-process approach just disappears. - **Toolchain prerequisites, ~15 min before any build:** VS Build Tools (`--add Microsoft.VisualStudio.Workload.VCTools --add Microsoft.VisualStudio.Component.Windows11SDK.22621 --includeRecommended`) then `rustup-init.exe -y --default-toolchain stable --profile minimal`. Verify by running `cargo --version` from its absolute path, not by trusting the installer's exit. - **Node/git come from vendor installers**, silently: node `.msi` via `msiexec /qn`, Git-for-Windows `.exe` via `/VERYSILENT /NORESTART`. Neither is on `PATH` for an existing SSH session — read `[Environment]::GetEnvironmentVariable("Path","Machine")` or use absolute paths (`C:\Program Files\Git\cmd\git.exe`). **Delete an ephemeral box when done** — a created VM keeps billing its disk even when stopped: ```sh gcloud compute instances delete --zone us-central1-a --quiet ``` ## Auth — the service-account key is the durable path The USER credential (`colin@pullfrog.com`) has its refresh token revoked periodically by org session-control policy, so `gcloud auth login` is not durable. A service-account key is exempt and works non-interactively without changing gcloud's global state: ```sh CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE=~/.config/pullfrog/vertex-service-account.json \ gcloud compute instances list --project=pullfrog ``` The SA (`pullfrog-vertex-e2e@pullfrog.iam.gserviceaccount.com`, project `pullfrog`) has Owner, so `list`/`describe`/`start`/`stop`/`create` all work through the override. **This is the preferred path.** Fall back to `! gcloud auth login` only if the override itself errors `Reauthentication failed. cannot prompt during non-interactive execution` (key removed/rotated). ⛔ **That Owner grant is wider than this skill needs, and it is shared with something else — do not "fix" it by narrowing this SA.** The same key is the `VERTEX_SERVICE_ACCOUNT_JSON` GitHub Actions secret in `pullfrog/app`, where it is used *only* to authenticate Gemini inference. Narrowing it to `roles/aiplatform.user` — the obvious tightening if you look at the pullfrog side alone — silently breaks **every VM operation in this skill**, because that role carries no compute permission at all. The correct shape is two accounts: leave CI with a Vertex-only SA, and mint a separate `roles/compute.instanceAdmin.v1` key for VM ops that never enters CI. Until that split exists, treat this key as an operator credential and keep it off any surface an untrusted run can read. ## Gotchas - **A RUNNING instance can be a DEAD instance — read the serial console FIRST:** ```sh gcloud compute instances get-serial-port-output nub-linux --zone us-central1-a | tail -40 ``` A wedged box typically shows `Out of memory: Killed process (rustc)`. Serial output beats guessing "network problem." - **Size ≥16 GB for anything that compiles the nub Rust workspace.** An e2-small (2 GB) cannot build it and will OOM-wedge. e2-standard-4 (16 GB) is the proven size. - **Write every script you send to `nub-win` as ASCII + CRLF.** PowerShell 5.1 reads a BOM-less script in the ANSI codepage, so a UTF-8 character anywhere in the file — an em-dash in a *comment* is the usual culprit — fails with `The string is missing the terminator`, and the error points at the **last line of the file**, not the offending one. Keep remote PowerShell ASCII-only, or emit a UTF-8 BOM. - **`IsOutputRedirected` is always True over SSH, so the first-run TTY path is unreachable there.** Any `is_terminal()` branch silently takes the non-TTY path. Testing real console behavior on Windows needs a ConPTY harness or an interactive RDP session. (On Linux, wrap the run in `script(1)` and the TTY branch runs.) - **For a nub RUST BUILD, use the `remote-build` skill, not this one.** `scripts/remote-build.ts` provisions an ephemeral spot builder from a pre-baked image and cross-compiles `aarch64-apple-darwin` on Linux. This skill remains the entry point for Windows/MSVC and for an interactive box. - **Prefer cross-compile-on-Mac + scp the artifact over building on the VM.** Running a binary needs almost no RAM. For Windows, the VM's MSVC BuildTools is often a broken shell with no `cl.exe` — cross-compile for `x86_64-pc-windows-gnu` on the Mac (`rustup target add …`; `brew install mingw-w64` if the linker is missing), strip, `scp` the `.exe`, run it. `harness = false` test binaries are self-contained and ideal for this. The Windows home dir may be `C:/Users/nub./`, not `C:/Users/nub`. - **Judge results by behavioral/differential evidence** (EPERM vs success, byte counts, a before/after delta), not wall-clock — a shared VM may be contended. ## Related - `ci-adhoc-test` — the branch-scoped CI route for macOS/Windows probes (no PR needed). Use a VM when you need an interactive box or a create-from-scratch env; use `ci-adhoc-test` when a committed probe on a real runner is enough. - AGENTS.md's Docker section — for Linux-only checks that don't need a real cloud kernel.