HITL Browser Automation
Use this skill when agent-browser automation needs a human in the loop for demonstration, verification, or credential handoff.
Do not use this skill for ordinary browser automation with no human checkpoint. Use the normal browser automation skill instead.
Default Workflow
Classify the handoff mode:
- human demonstrates a flow ->
references/demonstration-handoff.md
- agent runs automation and human verifies ->
references/verification-handoff.md
- human logs in, completes MFA, or handles sensitive entry ->
references/credential-handoff.md
Check prerequisites before startup:
hitl-browser-hub check
If checks fail, read references/prerequisites.md and report the missing tools.
Start the hub from the target project/context directory. If the command is not on PATH, invoke the skill-owned script by path; do not cd into the skill package for normal operation because runtime state is keyed from the current working directory.
hitl-browser-hub start
For a specific target app:
hitl-browser-hub start --url 'https://example.test/'
If the command is not installed on PATH:
/path/to/hitl-browser-automation/scripts/hitl-browser-hub start --url 'https://example.test/'
Give the human the VNC connection instructions printed by startup. For remote machines, use explicit local forwarding. Read references/vnc-client.md if connection details matter.
Let the human act in the VNC browser for the selected handoff mode.
Capture checkpoints around important states when useful:
hitl-browser-hub screenshot after-login
Resume automation or inspection with agent-browser and captured evidence. Use the user's prompt as the objective: report the click sequence, reproduce the final state, generate a replay script, continue automation, or state what evidence is missing. Read references/objective-demo.md for the smoke-app objective demo. For smoke validation:
hitl-browser-hub replay-smoke
Inspect paths/artifacts only as needed:
hitl-browser-hub paths
Read references/trace-artifacts-and-privacy.md before copying, committing, or pasting artifacts.
Stop the hub when done:
hitl-browser-hub stop
Safety Boundaries
- VNC and CDP must stay loopback-only by default.
- Use SSH local forwarding or an equivalent secure tunnel for remote access.
- Raw browser profiles, traces, screenshots, cookies, and logs can contain secrets. Do not commit them.
- For credential handoff, the human enters secrets directly in the browser. Do not ask the user to paste passwords, MFA codes, cookies, bearer tokens, or session data into chat.
- Run smoke validation before using a real app when the environment is new or uncertain.
Commands
Run normal workflow commands from the target project/context directory. Toolnix or another installer may put hitl-browser-hub on PATH. If not, invoke scripts/hitl-browser-hub by path from the skill package.
hitl-browser-hub check [--json]
hitl-browser-hub start [--url URL] [--no-smoke-app]
hitl-browser-hub status [--json]
hitl-browser-hub paths
hitl-browser-hub screenshot [label]
hitl-browser-hub replay-smoke
hitl-browser-hub shared-session-test
hitl-browser-hub stop
Run package validation from the skill root:
./scripts/validate-package.sh
Runtime state defaults outside the skill package under the user state directory. Override with BDH_STATE_ROOT for isolated test runs.
Read On Demand
references/prerequisites.md when setup fails or the environment is unknown.references/demonstration-handoff.md when the human will show the workflow first.references/verification-handoff.md when the human needs to approve or inspect an agent-driven result.references/credential-handoff.md when login, MFA, or other secrets are involved.references/vnc-client.md when the human needs connection instructions.references/trace-artifacts-and-privacy.md before sharing or preserving artifacts.references/real-app-validation.md before using the hub against a non-smoke target.references/objective-demo.md when the user wants the agent to infer a click sequence, reproduce an outcome, generate a replay script, or continue automation from smoke-app evidence.
Validation
Static package check:
./scripts/validate-package.sh
Runtime checks:
hitl-browser-hub check --json
./scripts/browser-debug-hub/tests/smoke-app.test.sh
./scripts/browser-debug-hub/tests/launcher-safety.test.sh
The launcher safety test exercises full startup only when prerequisites are present; otherwise it verifies that prerequisite detection reports missing tools cleanly.
1---2name: hitl-browser-automation3description: Use when browser automation needs a human checkpoint: the user wants to demonstrate a workflow once, watch and verify an agent-driven browser result before proceeding, log in or complete MFA/credential-sensitive steps manually, connect through VNC, or use agent-browser with human-in-the-loop browser control. This skill sets up and operates the bundled Browser Debug Hub runtime with safe loopback VNC/CDP defaults.4---56# HITL Browser Automation78Use this skill when `agent-browser` automation needs a human in the loop for demonstration, verification, or credential handoff.910Do not use this skill for ordinary browser automation with no human checkpoint. Use the normal browser automation skill instead.1112## Default Workflow13141. Classify the handoff mode:15 - human demonstrates a flow -> `references/demonstration-handoff.md`16 - agent runs automation and human verifies -> `references/verification-handoff.md`17 - human logs in, completes MFA, or handles sensitive entry -> `references/credential-handoff.md`182. Check prerequisites before startup:1920 ```bash21 hitl-browser-hub check22 ```2324 If checks fail, read `references/prerequisites.md` and report the missing tools.253. Start the hub from the target project/context directory. If the command is not on PATH, invoke the skill-owned script by path; do not `cd` into the skill package for normal operation because runtime state is keyed from the current working directory.2627 ```bash28 hitl-browser-hub start29 ```3031 For a specific target app:3233 ```bash34 hitl-browser-hub start --url 'https://example.test/'35 ```3637 If the command is not installed on PATH:3839 ```bash40 /path/to/hitl-browser-automation/scripts/hitl-browser-hub start --url 'https://example.test/'41 ```42434. Give the human the VNC connection instructions printed by startup. For remote machines, use explicit local forwarding. Read `references/vnc-client.md` if connection details matter.445. Let the human act in the VNC browser for the selected handoff mode.456. Capture checkpoints around important states when useful:4647 ```bash48 hitl-browser-hub screenshot after-login49 ```50517. Resume automation or inspection with `agent-browser` and captured evidence. Use the user's prompt as the objective: report the click sequence, reproduce the final state, generate a replay script, continue automation, or state what evidence is missing. Read `references/objective-demo.md` for the smoke-app objective demo. For smoke validation:5253 ```bash54 hitl-browser-hub replay-smoke55 ```56578. Inspect paths/artifacts only as needed:5859 ```bash60 hitl-browser-hub paths61 ```6263 Read `references/trace-artifacts-and-privacy.md` before copying, committing, or pasting artifacts.649. Stop the hub when done:6566 ```bash67 hitl-browser-hub stop68 ```6970## Safety Boundaries7172- VNC and CDP must stay loopback-only by default.73- Use SSH local forwarding or an equivalent secure tunnel for remote access.74- Raw browser profiles, traces, screenshots, cookies, and logs can contain secrets. Do not commit them.75- For credential handoff, the human enters secrets directly in the browser. Do not ask the user to paste passwords, MFA codes, cookies, bearer tokens, or session data into chat.76- Run smoke validation before using a real app when the environment is new or uncertain.7778## Commands7980Run normal workflow commands from the target project/context directory. Toolnix or another installer may put `hitl-browser-hub` on PATH. If not, invoke `scripts/hitl-browser-hub` by path from the skill package.8182```bash83hitl-browser-hub check [--json]84hitl-browser-hub start [--url URL] [--no-smoke-app]85hitl-browser-hub status [--json]86hitl-browser-hub paths87hitl-browser-hub screenshot [label]88hitl-browser-hub replay-smoke89hitl-browser-hub shared-session-test90hitl-browser-hub stop91```9293Run package validation from the skill root:9495```bash96./scripts/validate-package.sh97```9899Runtime state defaults outside the skill package under the user state directory. Override with `BDH_STATE_ROOT` for isolated test runs.100101## Read On Demand102103- `references/prerequisites.md` when setup fails or the environment is unknown.104- `references/demonstration-handoff.md` when the human will show the workflow first.105- `references/verification-handoff.md` when the human needs to approve or inspect an agent-driven result.106- `references/credential-handoff.md` when login, MFA, or other secrets are involved.107- `references/vnc-client.md` when the human needs connection instructions.108- `references/trace-artifacts-and-privacy.md` before sharing or preserving artifacts.109- `references/real-app-validation.md` before using the hub against a non-smoke target.110- `references/objective-demo.md` when the user wants the agent to infer a click sequence, reproduce an outcome, generate a replay script, or continue automation from smoke-app evidence.111112## Validation113114Static package check:115116```bash117./scripts/validate-package.sh118```119120Runtime checks:121122```bash123hitl-browser-hub check --json124./scripts/browser-debug-hub/tests/smoke-app.test.sh125./scripts/browser-debug-hub/tests/launcher-safety.test.sh126```127128The launcher safety test exercises full startup only when prerequisites are present; otherwise it verifies that prerequisite detection reports missing tools cleanly.
lefant