Workstation Aware Provider Orchestration

Workstation-Aware Provider Orchestration

Use this skill when the user wants Hermes to coordinate AI provider/model usage across multiple machines or workstations, especially when provider quota/credits are time-sensitive and work must still respect GitHub plan/approval gates.

Class of task

Design or operate a central AI workflow control plane that combines provider quota urgency, GitHub issue readiness, workstation availability, and safe dispatch prompts/ledgers.

When to use

• User asks Hermes to orchestrate Codex/Codex/Gemini or other providers across multiple machines.
• User wants one workstation to control most of the workflow while other workstations execute overflow work.
• Provider quota is expiring or underused and the user wants to burn it down safely.
• Work needs to be routed by agent:*, machine:*, status:*, priority:*, cat:*, or domain:* labels.
• Cross-workstation execution must avoid git contention and preserve plan-first governance.

Default workspace-hub assumptions

• ace-linux-1 is the primary Hermes/operator control-plane workstation for almost all AI workflow orchestration: provider usage decisions, queue review, prompt generation, dispatch ledger updates, GitHub state changes, and cross-workstation reconciliation.
• ace-linux-1 is also the continuous user-facing control surface: this is where approvals, plan decisions, work approvals, and morning reconciliation happen even when background lanes continue elsewhere.
• ace-linux-2 is the first overflow/execution worker node, not an equal peer control plane unless failover is explicitly chosen.
• Codex is best preserved for orchestration, planning, synthesis, and adversarial review.
• Codex is best for bounded implementation, tests, fixes, cleanup, and mechanical execution, especially when credits are expiring.
• Gemini is best for batched research, recon, risk enumeration, and architecture review where telemetry may be directional rather than exact.
• Continuous GTM objective: keep converting signals, approved repo work, and engineering outputs into client-ready material without crossing the engineering-evidence boundary.

Planning workflow

1. Open or update a GitHub issue first

   • Use gh-work-planning and github-issues.
   • Capture the objective, workstation priority, provider urgency, and hard gates.
   • Do not launch implementation until status:plan-approved is present unless the task is planning/review-only.

2. Refresh provider telemetry

   bash scripts/cron/provider-utilization-refresh.sh
   # or at minimum:
   bash scripts/ai/assessment/query-quota.sh --refresh --json
   

   Then inspect:

   • docs/reports/provider-utilization-weekly.md
   • docs/reports/provider-routing-scorecard.md
   • docs/reports/provider-work-queue.md

3. Reconcile telemetry with user-visible state

   • If local scripts disagree with a user-visible provider quota/expiry screen, record the conflict explicitly.
   • Do not claim exact utilization when the source is unavailable, estimated, stale, or contradictory.
   • Still use the user's expiring-credit signal to prioritize safe bounded work.

4. Verify control-plane readiness from ace-linux-1 Check or plan checks for:

   • repo sync and branch/worktree cleanliness
   • GitHub auth
   • Hermes config/profile and provider auth
   • provider telemetry artifacts
   • VPN reachability to worker machines
   • worker machine tool readiness
   • log/report directory availability

5. Rank issue candidates Prefer:

   • status:plan-approved
   • explicit agent:* labels
   • clear provider fit from the routing scorecard
   • non-overlapping file ownership
   • bounded validation commands

6. Map provider + workstation together A dispatch decision should consider both:

   • provider urgency / fit (for example expiring Codex credit)
   • machine readiness / ownership (for example ace-linux-1 control plane, ace-linux-2 overflow)

7. Emit a dispatch ledger Minimum fields:

   • issue number + URL
   • provider/model
   • workstation
   • reason for routing
   • quota/urgency basis
   • approval status
   • branch/worktree/path ownership
   • launch prompt or command
   • validation command(s)
   • expected evidence artifact/comment
   • fallback/stop condition

8. Get approval before long-running execution Present the operator with the shortlist and launch plan before starting cross-machine or high-credit-burn batches.

Immediate expiring-credit playbook

When a provider credit expires within about 24 hours:

1. Refresh telemetry and read provider work queue.
2. Reconcile telemetry with the user's visible account state.
3. Shortlist plan-approved issues matching that provider.
4. For Codex, prefer tests, implementation, repair, cleanup, and crisp execution issues.
5. Assign first to ace-linux-1; use ace-linux-2 for overflow only if readiness and zero-git-contention are verified.
6. Generate self-contained prompts/commands per issue/workstation.
7. Ask for final approval before launching a long-running batch.

GTM push loop overlay

Use this overlay when the goal is not just throughput, but steady movement toward outreach.

1. Ingest signals
   • Read fresh GTM/prospect inputs from docs/gtm/, issue comments, job-market outputs, or research notes.
   • Treat external programmes, conference pages, and social posts as topic signals, not engineering proof.
2. Map signals to engineering evidence
   • Link each signal to a real repo asset: demo report, methodology note, benchmark, capability page, or approved issue.
   • If the engineering evidence does not exist yet, create or update the plan/issue instead of marketing beyond the evidence.
3. Route the work by lane type
   • ace-linux-1 control surface: approvals, queue selection, dispatch ledger, morning synthesis, outreach packaging decisions.
   • ace-linux-1 local lanes: Codex/Gemini/Codex planning, synthesis, bounded implementation, or packaging work with repo-owned prompts.
   • ace-linux-2 overflow lanes: isolated implementation/review worktrees only after readiness checks.
4. Produce client-ready outputs with boundaries
   • Preferred outputs: docs/gtm/*.md, demo reports, website drafts, outreach templates, issue comments, and evidence summaries.
   • Every artifact must state what is demonstrated now vs what requires deeper project-specific engineering.
5. Feed the next outreach step
   • End each batch with one of: ready-to-send outreach material, a refined demo/report, a tighter capability note, or a clearly scoped blocker that needs approval/data.
   • Record the next human decision on ace-linux-1, not inside a remote worker lane.

Pre-delegation worker readiness gate

Before delegating work to any worker workstation, especially ace-linux-2, run and record a reviewable readiness probe. The worker may be repo-ready but still unsafe for AI-provider execution.

Minimum checks:

1. Host reachability

   getent hosts <host> || true
   ping -c 1 -W 2 <host> || true
   ssh -o BatchMode=yes -o ConnectTimeout=8 <host> 'hostname; uname -a; pwd'
   

2. Canonical workspace root

   • Prefer /mnt/local-analysis/workspace-hub for Linux workers unless evidence says otherwise.
   • Do not assume similarly named roots (for example /home/vamsee/workspace-hub) contain the tier-1 repo clones.

3. Tier-1 repo readiness For each target repo, capture:

   ssh <host> 'cd /mnt/local-analysis/workspace-hub/<repo> && \
     git branch --show-current && \
     git rev-parse --short HEAD && \
     git remote get-url origin && \
     git status --short && \
     git rev-list --left-right --count @{u}...HEAD 2>/dev/null || true && \
     test -f pyproject.toml && echo pyproject=yes || echo pyproject=no && \
     test -f uv.lock && echo uv_lock=yes || echo uv_lock=no && \
     test -d .venv && echo venv=yes || echo venv=no'
   

   Treat root/workspace-hub dirty state separately from child repo cleanliness; root dirt can still block workspace-hub-root work.

4. GitHub auth readiness

   ssh <host> 'gh auth status 2>&1'
   

   If invalid, the worker cannot safely mutate GitHub state, create PRs, or push via gh until re-authenticated.

5. AI provider runtime readiness Always check the worker's login shell as well as plain SSH. User-level installs may live in ~/.local/bin or ~/.npm-global/bin and be invisible in non-login SSH.

   ssh <host> 'for c in hermes Codex codex gemini; do command -v "$c" && "$c" --version 2>&1 | head -3 || echo "$c:not-found"; done'
   ssh <host> 'bash -lc '''for c in hermes Codex codex gemini; do command -v "$c" && "$c" --version 2>&1 | head -3 || echo "$c:not-found"; done; hermes config 2>/dev/null | grep -Ei "provider|model|base_url|gpt" | head -20''''
   

   Do not route expiring provider-credit work to a worker unless the relevant CLI/auth path exists in the launch environment and is known to consume the intended account/credit. If only the login shell exposes the tools, dispatch with ssh <host> 'bash -lc "<command>"' or explicitly source the user's environment.

   For Codex specifically, CLI presence and ~/.codex/ files are only a weak signal. Before assigning real Codex burn work to a remote/overflow machine, run a tiny real codex exec smoke through the exact login-shell/tmux path you will use for the lane and confirm it does not fail with 401 Unauthorized or Failed to refresh token: refresh token was already used. If that smoke fails, mark the host Codex-blocked and use it only for Codex fallback/validation until codex login is refreshed.

6. Engineering software readiness Check both package/command presence and a task-appropriate smoke test. Presence alone is not enough. Useful Linux engineering probes:

   ssh <host> 'command -v openfoam-selector && openfoam-selector --list || true'
   ssh <host> 'command -v gmsh && gmsh --version || true'
   ssh <host> 'command -v freecad || command -v FreeCAD || true'
   ssh <host> 'command -v blender && blender --background --version 2>&1 | head -5 || true'
   ssh <host> 'command -v pvbatch && pvbatch --version 2>&1 | head -5 || true'
   ssh <host> 'command -v ccx && ccx 2>&1 | head -5 || true'
   ssh <host> 'command -v qgis && qgis --version 2>&1 | head -3 || true'
   ssh <host> 'command -v gdalinfo && gdalinfo --version || true'
   

   GUI Qt tools may fail over SSH without display; prefer headless modes (--background, pvbatch) and record display/GPU caveats.

7. GPU/display caveat

   ssh <host> 'nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader 2>/dev/null || true'
   

   Do not assign GPU or GUI-dependent work unless driver/display/headless readiness is explicitly validated.

8. Dispatch ledger evidence Store the probe result in a durable report (for example docs/reports/YYYY-MM-DD-issue-NNN-<host>-readiness-probe.md) and link it from the GitHub issue before delegating.

Current learned ace-linux-2 baseline from 2026-04-27 probe

Use this as a starting hypothesis, not a substitute for a fresh probe:

• Reachable via SSH as ace-linux-2 and canonical repo root was /mnt/local-analysis/workspace-hub.
• Tier-1 repos digitalmodel, worldenergydata, assetutilities, and teamresumes existed and were clean on main; teamresumes lacked .venv.
• workspace-hub root itself was dirty, so root-level work needed a separate dirty-state decision.
• Open-source engineering tools detected included OpenFOAM ESI openfoam2312, Gmsh, FreeCAD, Blender, ParaView/pvbatch, CalculiX, QGIS, and GDAL/OGR.
• Proprietary/licensed tools were not detected in PATH: OrcaFlex/OrcaWave, ANSYS/AQWA, MATLAB, SALOME/Code_Aster.
• Plain non-login SSH did not expose hermes/codex, but a login shell did: bash -lc found /home/vamsee/.local/bin/hermes and /home/vamsee/.npm-global/bin/codex.
• Hermes on ace-linux-2 reported default provider/model openai-codex / gpt-5.5 with base URL https://chatgpt.com/backend-api/codex; Codex auth files existed under ~/.codex/.
• gh auth was invalid, so keep GitHub mutation authority on ace-linux-1 unless gh is repaired on ace-linux-2.
• Current conclusion: ace-linux-2 is repo-ready and Hermes/Codex-runtime-ready when launched through a login shell, but not ready for local GitHub mutation via gh.

Direct remote execution pattern

When the user asks to execute work on another workstation (not just prepare a prompt), use real remote process orchestration rather than delegate_task:

1. Copy the self-contained worker prompt to the remote host:

   scp local-worker-prompt.md ace-linux-2:/tmp/worker-prompt.md
   

2. Start a named tmux session over SSH from a login shell so user-level CLIs are on PATH:

   ssh ace-linux-2 "bash -lc 'mkdir -p /mnt/local-analysis/ace2-worker-logs /mnt/local-analysis/ace2-worker-reports; \
     SESSION=ace2-overflow-$(date +%Y%m%d); \
     tmux kill-session -t \$SESSION 2>/dev/null || true; \
     tmux new-session -d -s \$SESSION -c /mnt/local-analysis/workspace-hub \
       \"bash -lc \\\"Codex --print --dangerously-skip-permissions < /tmp/worker-prompt.md 2>&1 | tee /mnt/local-analysis/ace2-worker-logs/\$SESSION.log\\\"\"; \
     tmux list-sessions | grep \$SESSION'
   

3. Monitor with:

   ssh ace-linux-2 "bash -lc 'tmux capture-pane -t ace2-overflow-YYYYMMDD -p -S -80; find /mnt/local-analysis/ace2-worker-reports -maxdepth 1 -type f -printf \"%f %s bytes\\n\"'"
   

4. Require the worker to write report files under a known handoff directory for the control plane to post/reconcile later.

Do not use shell-level nohup ... & wrappers through Hermes foreground terminal; Hermes blocks that pattern. Use Hermes terminal(background=true) for local tracked processes, or remote tmux for SSH-launched workers.

Post-reboot / interrupted-run recovery playbook

Use this when a control-plane workstation reboots or a context handoff indicates in-flight Hermes/Codex/Codex/tmux work may have survived, stalled, or been partially landed. Work in this order:

1. Salvage current work first

   • Reconstruct active state from the handoff, todo, process tables, tmux sessions, logs, and GitHub issue labels/state.
   • Check Hermes background sessions directly by known session_id when available; do not assume an empty process list means the run is gone.
   • Inspect exact ps PIDs/PGIDs before killing anything. Avoid pkill -f; it can self-match and terminate the orchestrator.
   • Verify claimed completions with durable handles: issue URL/state/labels, remote commit SHA, branch, validation log, or report file.
   • Mark local todos complete only after external verification.

2. Research/restart ongoing work second

   • Poll local tmux panes/logs and remote worker panes/logs before relaunching.
   • For ace-linux-2, rerun readiness (scripts/operations/agent-execution/ace2-readiness.sh when available) and keep it report-only if gh auth status is invalid.
   • Do not duplicate Codex/Codex lanes until OS process state, worktree git state, and expected report artifacts have been checked.
   • If a worker must be restarted, use repo-owned prompt/script artifacts rather than /tmp prompts whenever they exist.

3. Set off future work last

   • Launch only plan-approved implementation lanes, or planning/review-only lanes for unapproved issues.
   • Keep ace-linux-1 as GitHub mutation/control plane and ace-linux-2 as overflow worker unless auth/readiness proves otherwise.
   • Persist reusable launch prompts and scripts inside the repo ecosystem, preferably under docs/plans/machine-prompts/<date>/... and scripts/operations/agent-execution/, then validate (bash -n, --help, dry-run) before committing.
   • Record final reconciliation with issue links, commit SHAs, validation results, remaining sessions, and blockers.

Repo-owned agent execution scripts

For workspace-hub orchestration, prefer committed scripts over ad hoc /tmp launch commands when they exist:

bash scripts/operations/agent-execution/ace2-readiness.sh
bash scripts/operations/agent-execution/launch-ace1-control-plane.sh --dry-run
bash scripts/operations/agent-execution/launch-ace2-overflow-worker.sh --dry-run
bash scripts/operations/agent-execution/launch-2518-finalizer.sh --dry-run

These scripts encode the current safety defaults: login-shell PATH on ace-linux-2, tmux-based remote execution, repo-owned prompts, explicit logs/reports, and dry-run/help validation.

Interactive orchestration readiness shadow session

Use this when the user wants to keep reviewing decisions in the current chat while a separate Hermes session performs read-only orchestration readiness inspection.

1. Write the handoff prompt to a durable repo path, for example:
   • docs/plans/machine-prompts/<date>/execution/orchestration-readiness-interactive-handoff.md
2. Start a named tmux session in the repo root:

   SESSION=orch-readiness-$(date +%Y%m%d)
   PROMPT=/mnt/local-analysis/workspace-hub/docs/plans/machine-prompts/$(date +%F)/execution/orchestration-readiness-interactive-handoff.md
   LOG=/mnt/local-analysis/workspace-hub/docs/plans/machine-prompts/$(date +%F)/execution/orchestration-readiness-interactive-session.log
   tmux new-session -d -s "$SESSION" -c /mnt/local-analysis/workspace-hub \
     "bash -lc 'hermes --pass-session-id 2>&1 | tee -a $LOG'"
   

3. Wait for the Hermes prompt before pasting the handoff; if pasted too early, it can be interpreted by the shell/tmux before Hermes is ready. Verify with tmux capture-pane.
4. Paste and submit the prompt:

   tmux load-buffer -b orch_prompt "$PROMPT"
   tmux paste-buffer -t "$SESSION" -b orch_prompt
   tmux send-keys -t "$SESSION" Enter
   

5. Report the tmux session name, prompt path, log path, and attach command to the user. Keep the current chat as the interactive decision/review surface.

Bounded lane-keeper cron until provider reset

When the user asks to keep work lanes going until a reset/expiry time, update or create a Hermes cron job rather than launching uncontrolled duplicate agents.

Recommended guardrails for the cron prompt:

• State the exact stop time with timezone and tell the job to stop launching after that timestamp.
• Monitor known lanes first: OS processes, tmux sessions, worktree status, remote branches, issue labels/comments, and logs/reports.
• Classify lanes as RUNNING, READY_FOR_REVIEW, STALLED_NO_OUTPUT, or BLOCKED.
• Do not merge, close issues, force-push, hard reset/clean primary checkout, or remove status:working autonomously.
• Only restart a lane when it has no live process, no ahead commit/evidence, and no duplicate active branch; restrict restarts to explicitly named stalled lanes unless the prompt has a safe shortlist rule.
• For ready branches, run only lightweight read-only validation and leave final merge/closure to the interactive review session.
• Keep ace-linux-1 as control plane and avoid ace-linux-2 GitHub mutations unless fresh auth/readiness proves safe.
• Include a final table per tick: issue, PID/session, worktree, branch, HEAD, classification, action taken, and next human action.

Use cronjob(action='update') to retarget an existing burn/controller job when one already exists, instead of creating overlapping cron jobs. Set enabled_toolsets narrowly (usually terminal,file) and a repeat count that covers the reset window with a small buffer.

Pitfalls

• Treating a stale quota script as more authoritative than user-visible expiring-credit evidence.
• Dispatching implementation for issues that are not status:plan-approved.
• Letting ace-linux-2 become an untracked peer control plane instead of a worker/overflow node.
• Routing by provider only and ignoring workstation readiness or git contention.
• Assuming a clean child repo means the workspace-hub root is clean enough for root-level work.
• Assuming installed engineering software is usable without a headless/tool-specific smoke test.
• Routing Codex/Codex/Gemini work to a workstation where the provider CLI is missing or unauthenticated.
• Mass-applying labels from heuristics without manual inspection.
• Losing reconciliation evidence because no dispatch ledger was written.

Related skill overlap note

This skill intentionally overlaps with agent-usage-optimizer for provider quota routing, but adds the workstation/control-plane layer. Future consolidation could merge this workstation section into agent-usage-optimizer if external skill write access is available.