Heliograph

heliograph

Someone can reach the machine. You cannot, and you are the one who knows what to ask it. This skill runs that gap as a loop instead of a relay: git carries the step out and the log back, and the operator types one command that never changes.

you        push a step ──────────────────────────────▶ transport repo
operator                                        ──────▶ git pull && ./run.sh
           the log is captured and pushed ──────▶ transport repo
you        git pull, read ops-logs/<step>-<UTC>.txt ◀──

Every captured line carries a UTC timestamp, ANSI is stripped, obvious secrets are masked, and the log is committed and pushed whether the run passed or failed.

When this applies

• The machine is in an environment you have no interactive access to.
• The only person who can reach it has other work to do and should not be your terminal.
• Several rounds of "run this and paste the output" have already gone badly.
• The repo that has to change is also on the far side (see references/remote-repo.md).

If you can SSH in yourself, do that instead and do not use this skill.

Where this sits

This skill is the far side and the method: the payload that runs on the machine, and how to debug across a gap. It needs nothing installed there, and that constraint is the whole proposition.

The near side - your machine - has two optional things that make the same loop easier to drive. Neither is required, and neither changes what happens on the far side.

command -v heliograph >/dev/null && echo "use the CLI" || echo "hand-edit, as below"

If the binary is there, prefer it. It writes the same station/request on the same branch that step 4 below writes by hand, so a station cannot tell the difference and there is nothing to migrate. What you get for it:

• heliograph send net-probe HOSTS="sql01 sql02" instead of editing a file and inventing an id. Values with spaces are re-quoted on the way out, which is the bug you would otherwise hit once and not understand
• heliograph logs --gaps does the timestamp arithmetic that the method section below tells you to do by eye, and reports an all-identical-timestamps log as an error rather than as "no gaps"
• send rebases onto the remote before it pushes, and watch pulls on every poll, so the git pull --rebase discipline in step 4 stops being something you have to remember at the moment it matters

If it is not there, everything below works unchanged. That is deliberate: the far side must never depend on the near side having anything.

Install and full reference: https://heliograph.dbhq.uk. The CLI lives in dbhq-uk/heliograph; it is Go, and it stays out of this repository so that "plain bash, you can read it before you run it" keeps being true here.

Prerequisites

Bash 4+, git, GNU coreutils. Nothing to install and no credentials of the skill's own. The control node needs git and whatever the step itself invokes.

The GNU spellings are load-bearing (sed -u keeps the capture unbuffered, so a line is stamped when it is produced). On macOS, coreutils and gnu-sed must be first on PATH.

Or run it in a container, when installing anything on the control node is its own change request:

${CLAUDE_SKILL_DIR}/toolkit/docker/heliograph.sh <transport-repo-url>

It builds the image if needed, clones the transport repo, and hands over to that repo's own start.sh - nothing below this point changes. The unprivileged user inside is not a security boundary; the full account, including the two ways the image gets built, is in references/container.md.

1. Set up the transport repo

${CLAUDE_SKILL_DIR}/scripts/bootstrap.sh <target-dir>

Then git init, add a private remote, and push.

The transport repo must be private, and must be its own repo. Captured logs are committed to it, so everything the operator's commands print lands in that history permanently. Never bootstrap into a repo that holds anything else, and never into a public one.

Re-running the bootstrap over an existing transport repo is safe: it installs what is missing and leaves what is already there alone.

Ask the operator to clone it on the control node. That is the only setup they do.

If the loop is to run unattended, decide the credential now. A forwarded ssh agent key is the nicest option for an attended run and is no use at all once the operator disconnects, which is exactly when toolkit/service.sh is keeping the loop alive. An unattended loop needs a key on disk, a deploy key, or a token in ~/.git-token. See references/transport.md.

2. Baseline before theorising

./run.sh env                      # OS, tools, auth, proxy, DNS, git, this repo's commit
HOSTS="hosta hostb" ./run.sh net  # DNS + ICMP + TCP, and the reverse direction

env is the right first step of any investigation, whatever it turns out to be about. A prior finding is a hypothesis to re-test, never a premise to build on.

3. Start a task

1. git checkout -b task/<slug>
2. Fill in TASK.md: the question, what is known, what would settle it. Do this first. It is what stops the steps becoming a fishing trip.
3. Write one step per question: cp steps/_template.sh steps/<name>.sh
4. Register it in the case table in run.sh and in the step-list comment above it, so --list stays honest.
5. Set DEFAULT_STEP to the one to run next, and push.

main is the template; task/<slug> is one investigation. Task branches are not merged back. Only genuinely generic tooling returns to main, stripped of anything task-specific, via a normal PR.

Writing the step itself: references/steps.md. Read it before the first one. Every rule in it cost a round trip.

4. Drive it

Ask the operator to run ./start.sh once, then stop relaying runs. It checks that the machine can capture properly and that git can push from it, then starts the station, which watches station/request and runs when the id: changes:

you       git pull --rebase, edit station/request (new id), push ──▶ transport repo
station   picks it up within seconds, runs ./run.sh
          pushes station/status, then the log ───────────────▶ transport repo
you       poll, read the log, decide the next step ◀────────

Always git pull --rebase before you push. You and the station push to the same branch, and it pushes far more often than you do: a status commit when a run starts and again when it ends, a progress snapshot every 60 seconds during a long step, and the log itself. So the remote moves under you while you are writing the next request, and a plain push is rejected:

! [rejected]  task/foo -> task/foo (fetch first)

That is not a fault, it is two writers on one branch working as intended. The station already does exactly this on its own side before every push. Rebase rather than merge: it keeps the history readable as a sequence of requests and answers instead of threading it with merge commits. Conflicts are rare in practice, since the station only ever writes station/status and ops-logs/ while you write station/request and steps/.

If they want to know whether the machine will work before committing to anything, ./start.sh --check answers that and changes nothing.

The trigger is the id, not a new commit: docs and step edits land constantly and would otherwise fire runs nobody asked for. stop: yes ends the station from your side, which matters because nobody is sitting at that terminal.

Every step declares itself in its own file - # heliograph-mode: read-only or action - and a step that declares neither will not run. A state-changing step needs CONFIRM=yes in the request's env: and run.sh's own gate, and the station refuses it altogether unless the operator started it with --allow-actions. The loop is read-only by default; a refusal is published to station/status within seconds, so you find out on the next poll rather than after a wasted round trip.

cancel: yes kills the step running right now, and cancel: <id> kills it only if that id is the one running. The station stays responsive while a step runs, so a long or wrong run does not have to be waited out. A new id does not cancel: it queues behind the running step, because an in-flight step may be mid-change.

A long run is not a black box: the partial log is pushed every 60 seconds with a line count and the last real line, so git pull shows where it has got to.

While a station is running, say so and wait for the log. Ask the operator only for what git cannot carry: an interactive cloud login, a decision, or a fact only they have.

Without a station, the operator's whole interface is git pull && ./run.sh. Never send them a command to paste; set DEFAULT_STEP and push.

Every runner, function and knob: references/runner.md.

When the control node cannot reach git at all

Everything above assumes the control node can reach the git host. Sometimes it cannot - a locked-down subnet whose default route goes to a firewall with no policy for it has no outbound path at all, and git stops being a transport and becomes a dependency that cannot be met.

pigeonhole.sh and drop.sh carry the same contract over Azure Blob Storage instead. You write the request to a container, the station polls it and writes the log back, and neither side ever reaches the other - a private endpoint is VNet-local, so that traffic never touches the route that is blocking everything else. The capture is untouched: it still calls run.sh, so the log is the same document.

./drop.sh send <id> <step>    # queue a step
./drop.sh watch <id>          # wait, then print the log

Measure before reaching for it. Git is better when git works, and an image pull succeeding proves nothing - a container platform pulls on its own side, so a container can start cleanly on a host with no network at all. When to use it, how the lane replaces branch binding, and the traps: references/pigeonhole.md.

When you can reach the station

The rarer case, and worth checking for before assuming the pigeonhole. An Azure Function App has a public HTTPS endpoint and sits inside the VNet, so the control node can call it directly. Then storage is indirection with no purpose: credentials to hold, a timer interval to wait, four blob operations to move text between two machines that can already talk.

intercom.sh submits the step over HTTPS and polls for its log. The capture is again untouched - it still calls run.sh.

export INTERCOM_URL=https://<app>.azurewebsites.net
export INTERCOM_KEY=<the function key>
./intercom.sh run steps/net-probe.sh HOSTS="sql.example" PORTS=1433

It runs the script you send it, which is what keeps the loop fast where there is no git on the far side - and also means heliograph-mode: becomes a claim the caller makes about its own file rather than a control. The function key and the IP allowlist are then the only real ones, and both must be deployed. Read references/intercom.md before exposing it.

5. Read the log

• Header block first: branch, commit, host, user. A divergence between the commit you pushed and the one they ran explains a surprising share of "but I fixed that".
• Scan the timestamp column for gaps before reading the content. A gap is a finding: in an untimed log a hang and slow progress are indistinguishable.
• probe_summary gives the tally, the footer gives the real exit code.
• Read the whole log, including the parts that worked. A passing probe beside a failing one is the control that tells you what the failure means.
• Record what was measured in TASK.md, separately from what you concluded. Measurements stay true; conclusions get revised.

A green exit means the probes that ran passed, not that the work happened. Verify the outcome, not the exit code.

Hard rules

These outrank convenience. Each one is here because breaking it cost a full round trip or worse.

1. Never truncate. No head, no tail -20, no 2>/dev/null on the thing being diagnosed. The line you cut is the one you needed.
2. Measure, do not infer. Say what a log showed, not what it implies.
3. Keep a control. A probe with nothing to compare against is an anecdote.
4. Change one thing between runs. Two changes and a different result tells you nothing.
5. Read-only until earned. A step changes state only when you can say precisely what it will do and why, and then it carries the CONFIRM=yes gate.
6. Never commit task work to main.
7. Never ask the operator to hand-edit anything. Deliver a change as a payload the step copies into place. An unlogged manual edit is exactly the divergence these logs exist to rule out.

The full method, and the mistakes behind each rule: references/method.md. Worth reading in full before a hard investigation.

Running it in Azure, instead of on somebody's terminal

Sometimes there is no willing human to start ./start.sh and leave it running. toolkit/azure/ runs the station as Azure infrastructure instead. Five hosts:

All bring-your-own: the estate passes in a VNet, subnet, plan or environment that already exists, and the template creates the compute and nothing else. The checkout is transient. Git is the persistence, so if the compute dies you run the step again.

The Function host is the exception to all of that, and the one to reach for when the estate will not give you anywhere to keep a process. It is not a loop: a timer answers at most one request per tick, so it needs PIGEONHOLE_RESUME=1 to know what it already answered. There is no git in its image, so it uses the blob transport - which means it needs no egress at all, and works in a subnet with no route off it. See references/azure.md.

Two things that will waste your time if you do not know them:

• The published image tag has no v. The git tag is v1.0.0-rc1, the image is ghcr.io/dbhq-uk/heliograph-toolkit:1.0.0-rc1.
• For a GitHub transport repo, set GIT_TOKEN_USER=x-access-token. Without it git says could not read Username, which reads like a missing credential rather than a wrong one.

Everything else, including what only showed up by deploying these, is in references/azure.md.

Secrets

Logs are committed and pushed, so anything a command prints is in git history permanently.

• cap_redact masks the obvious shapes (password=, Bearer, Basic, a credential carried in a URL, private keys). It is a safety net, not a guarantee. Never deliberately run something that prints a secret.
• Name secrets, never read them. Listing secret names settles "does this exist here". The value is never the question.
• The transport repo's .gitignore blocks the usual carriers. Do not git add -f around it.

When a value has to go the other way

Occasionally the far side needs a secret it cannot fetch for itself. secret.sh carries it as ciphertext, with the passphrase defined by a human on both machines and never committed:

./secret.sh key                    # on BOTH machines, then compare fingerprints
./secret.sh put registry-pass      # paste the value, Ctrl-D
git add secrets/ && git commit && git push

and in a step on the far side, captured, never echoed:

PASS="$("$HERE/../secret.sh" get registry-pass)"; export PASS

It is transport, not storage: the ciphertext is in history forever. Prefer short-lived, narrowly scoped credentials, and land the real secret in whatever store the far side has. Details, and why each guard is there: references/secrets.md.

References

Near side, on the web, because it changes with the binary rather than with this skill and one copy is better than two:

Far side, here, because it ships with the payload: